Files
Netcatty/application/state/sftp/transferDirectoryFileBatch.test.ts
T
ab3580982d fix(sftp): parallel directory discovery, folder pre-scan, and skip-unchanged (#2712 / #2556) (#2772)
* fix(sftp): parallel folder discovery and unblock transfer stalls

Address #2712 / #2556 leftovers after the pipelined fail-closed work
(#2449 / #2720 / #2761): keep bulk body I/O on concurrent READ/WRITE only,
surface upload digest prep as verifying instead of a silent hang, make the
fastPut path-gate wait cancelable (#2755), parallelize multi-select enqueue,
and bound-parallelize directory listings so folder totals appear faster.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* feat(sftp): folder pre-scan and skip unchanged files

Absorb rsync strengths without an rsync backend: optionally walk the
full tree before transfer for stable file totals, and skip files whose
size and mtime already match the destination.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): drop unused DiscoveredTransferFile import

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix: address Codex review on PR #2772

* fix(sftp): address Codex review on path gate, aliases, and worker drain

- Invalidate same-transfer fallback when a force-released path gate gets a late truncating OPEN
- Use per-branch ancestors so sibling symlink aliases copy under pre-scan BFS
- Keep interleaved subdirectory walks sequential for deterministic manifest indices
- Drain transfer/bounded workers before propagating errors; ASCII punctuation

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): canonical pre-scan file order and pre-2001 mtime skip

Sort discovered files with compareDirectoryTraversalPaths before index
assignment, and treat timestamps >= 1e11 as milliseconds so year-2000
local mtimes still match SFTP second precision.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): bound multi-select downloads and preserve dest mtime

Address Codex review on PR #2772: serialize local blob reads, bound
multi-select root downloads before pre-scan, replace remaining Unicode
em dashes, and stamp destination mtimes after successful transfers so
skip-unchanged can match on a later pass.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): keep in-place OPEN poison until callback settles

Address Codex review on e2cc8241: do not force-release path gates for
in-place truncating OPENs, fail closed on pending-OPEN wait timeouts
without clearing poison, and replace remaining Unicode em dashes in
ASCII-only transfer modules.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): fail closed when in-place OPEN poison is held

When an isolated in-place truncating OPEN leaves the path gate poisoned
after channel error/cancel, mark the transfer terminal (noTransferFallback)
so fastPut/concurrent-shared cannot wait forever on the unreleased prior
gate.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): omit cyclic dirs from pre-scan plan and lower mtime cutoff

Only record discovered directories after a visit is accepted so symlink
cycles are not mkdir'd as empty targets, and treat magnitudes >= 1e10 as
milliseconds so early-1970s local Date values still match SFTP seconds.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): count max-depth symlink gaps and skip symlink quick-check

Pre-scan now surfaces max-depth symlink directory omissions as transfer
errors (matching interleaved walk), and discovered file symlinks carry
isSymlink so skip-unchanged does not trust link-node size/mtime.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): unit-aware skip mtimes and interleaved symlink bypass

Declare mtimeUnit on skip identities so early-1970 millisecond values
normalize without magnitude cutoffs, and bypass skip-unchanged for
symlinks on the interleaved (non-prescan) directory walk as well.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): bound local pre-scan and preserve SCP destination mtimes

Apply the same directory/entry traversal budget to local folder pre-scan,
and best-effort touch SCP upload targets after commit so skip-unchanged
can match on a later pass.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): address Codex review on pause, skip restat, mtime bound

- Honor transfer pause during folder pre-scan BFS waves/listings
- Re-stat source immediately before skip-unchanged on pre-scanned files
- Bound best-effort destination setStat/utimes so sendComplete cannot hang

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): harden skip restat and pre-transfer mtime capture

- Require a fresh source stat before skip-unchanged (prescan + interleaved)
- Capture source soft identity before bytes move for non-resumable/SCP
- Drop post-commit mtime re-stat fallback; keep ASCII in touched comments

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): propagate fresh re-stat size into interleaved transfers

Use the re-statted source size/mtime for child tasks on the non-prescan
path so grown files are not truncated to the stale listing size.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): omit stale listing size after re-stat failure

When source re-stat fails, clear child totalBytes/mtime so
startStreamTransfer omits an explicit snapshot and the bridge
re-stats on the dedicated session instead of truncating to the
stale listing size.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): make soft-identity capture cancelable

Race pre-transfer source soft-identity capture against cancel and
pass the transfer abort signal into requireSftpChannel so a dead
SFTP reopen cannot block cancel for the channel timeout.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
Co-authored-by: netcatty-bot <308658023+netcatty-bot@users.noreply.github.com>
2026-08-06 18:46:03 +08:00

224 lines
6.8 KiB
TypeScript

import assert from "node:assert/strict";
import test from "node:test";
import type { MutableRefObject } from "react";
import type { TransferTask } from "../../../domain/models";
import { STORAGE_KEY_SFTP_TRANSFER_CONCURRENCY } from "../../../infrastructure/config/storageKeys";
import { transferDiscoveredFiles } from "./transferDirectoryFileBatch";
const memoryStore = new Map<string, string>();
(globalThis as { localStorage?: Storage }).localStorage = {
getItem: (key: string) => memoryStore.get(key) ?? null,
setItem: (key: string, value: string) => {
memoryStore.set(key, String(value));
},
removeItem: (key: string) => {
memoryStore.delete(key);
},
clear: () => memoryStore.clear(),
key: () => null,
length: 0,
} as Storage;
memoryStore.set(STORAGE_KEY_SFTP_TRANSFER_CONCURRENCY, "1");
const rootTask = (): TransferTask => ({
id: "root",
fileName: "source",
sourcePath: "/source",
targetPath: "/target",
sourceConnectionId: "local",
targetConnectionId: "local",
direction: "upload",
status: "transferring",
totalBytes: 1,
transferredBytes: 0,
speed: 0,
startTime: 0,
isDirectory: true,
});
test("transferDiscoveredFiles re-stats source before skip-unchanged", async () => {
const transfersRef = { current: [rootTask()] } as MutableRefObject<TransferTask[]>;
const cancelledTasksRef = { current: new Set<string>() } as MutableRefObject<Set<string>>;
const activeChildIdsRef = {
current: new Map<string, Set<string>>(),
} as MutableRefObject<Map<string, Set<string>>>;
let transfers = transfersRef.current;
let transferCalls = 0;
let sourceStatCalls = 0;
const staleMtime = 1_700_000_000_000;
const freshMtime = 1_700_000_100_000;
await transferDiscoveredFiles({
rootTask: rootTask(),
files: [{
name: "a.txt",
sourcePath: "/source/a.txt",
targetPath: "/target/a.txt",
size: 10,
lastModified: staleMtime,
}],
sourceSftpId: null,
targetSftpId: null,
sourceIsLocal: true,
targetIsLocal: true,
sourceEncoding: "auto",
targetEncoding: "auto",
rootTaskId: "root",
skipUnchanged: true,
startEntryIndex: 0,
cancelledTasksRef,
activeChildIdsRef,
transfersRef,
setTransfers: (updater) => {
transfers = typeof updater === "function" ? updater(transfers) : updater;
transfersRef.current = transfers;
},
waitWhileTransferPaused: async () => {},
isPauseLatched: () => false,
transferFile: async () => {
transferCalls += 1;
},
tryStatTarget: async () => ({
size: 10,
lastModified: staleMtime,
type: "file",
}),
tryStatSource: async () => {
sourceStatCalls += 1;
return {
size: 10,
lastModified: freshMtime,
type: "file",
};
},
});
assert.equal(sourceStatCalls, 1, "must re-stat source before skip check");
assert.equal(transferCalls, 1, "stale listing must not skip a changed source");
const child = transfers.find((row) => row.parentTaskId === "root");
assert.ok(child);
assert.equal(child?.status, "completed");
assert.equal(child?.sourceLastModified, freshMtime);
});
test("transferDiscoveredFiles does not skip when source re-stat fails", async () => {
const transfersRef = { current: [rootTask()] } as MutableRefObject<TransferTask[]>;
const cancelledTasksRef = { current: new Set<string>() } as MutableRefObject<Set<string>>;
const activeChildIdsRef = {
current: new Map<string, Set<string>>(),
} as MutableRefObject<Map<string, Set<string>>>;
let transfers = transfersRef.current;
let transferCalls = 0;
let transferredTotalBytes: number | undefined;
let transferredSourceLastModified: number | undefined;
const mtime = 1_700_000_000_000;
await transferDiscoveredFiles({
rootTask: rootTask(),
files: [{
name: "a.txt",
sourcePath: "/source/a.txt",
targetPath: "/target/a.txt",
size: 10,
lastModified: mtime,
}],
sourceSftpId: null,
targetSftpId: null,
sourceIsLocal: true,
targetIsLocal: true,
sourceEncoding: "auto",
targetEncoding: "auto",
rootTaskId: "root",
skipUnchanged: true,
startEntryIndex: 0,
cancelledTasksRef,
activeChildIdsRef,
transfersRef,
setTransfers: (updater) => {
transfers = typeof updater === "function" ? updater(transfers) : updater;
transfersRef.current = transfers;
},
waitWhileTransferPaused: async () => {},
isPauseLatched: () => false,
transferFile: async (task) => {
transferCalls += 1;
transferredTotalBytes = task.totalBytes;
transferredSourceLastModified = task.sourceLastModified;
},
tryStatTarget: async () => ({
size: 10,
lastModified: mtime,
type: "file",
}),
tryStatSource: async () => null,
});
assert.equal(transferCalls, 1, "missing fresh source meta must not skip");
assert.equal(transferredTotalBytes, 0, "must not pass stale listing size after re-stat failure");
assert.equal(
transferredSourceLastModified,
undefined,
"must clear stale listing mtime after re-stat failure",
);
});
test("transferDiscoveredFiles skips only when fresh source still matches", async () => {
const transfersRef = { current: [rootTask()] } as MutableRefObject<TransferTask[]>;
const cancelledTasksRef = { current: new Set<string>() } as MutableRefObject<Set<string>>;
const activeChildIdsRef = {
current: new Map<string, Set<string>>(),
} as MutableRefObject<Map<string, Set<string>>>;
let transfers = transfersRef.current;
let transferCalls = 0;
const mtime = 1_700_000_000_000;
await transferDiscoveredFiles({
rootTask: rootTask(),
files: [{
name: "a.txt",
sourcePath: "/source/a.txt",
targetPath: "/target/a.txt",
size: 10,
lastModified: mtime - 60_000,
}],
sourceSftpId: null,
targetSftpId: null,
sourceIsLocal: true,
targetIsLocal: true,
sourceEncoding: "auto",
targetEncoding: "auto",
rootTaskId: "root",
skipUnchanged: true,
startEntryIndex: 0,
cancelledTasksRef,
activeChildIdsRef,
transfersRef,
setTransfers: (updater) => {
transfers = typeof updater === "function" ? updater(transfers) : updater;
transfersRef.current = transfers;
},
waitWhileTransferPaused: async () => {},
isPauseLatched: () => false,
transferFile: async () => {
transferCalls += 1;
},
tryStatTarget: async () => ({
size: 10,
lastModified: mtime,
type: "file",
}),
tryStatSource: async () => ({
size: 10,
lastModified: mtime,
type: "file",
}),
});
assert.equal(transferCalls, 0, "matching fresh identity must skip transfer");
const child = transfers.find((row) => row.parentTaskId === "root");
assert.equal(child?.status, "completed");
assert.equal(child?.transferredBytes, 10);
});