mirror of
https://github.com/binaricat/Netcatty.git
synced 2026-11-22 16:10:50 +00:00
* fix(sftp): parallel folder discovery and unblock transfer stalls
Address #2712 / #2556 leftovers after the pipelined fail-closed work
(#2449 / #2720 / #2761): keep bulk body I/O on concurrent READ/WRITE only,
surface upload digest prep as verifying instead of a silent hang, make the
fastPut path-gate wait cancelable (#2755), parallelize multi-select enqueue,
and bound-parallelize directory listings so folder totals appear faster.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* feat(sftp): folder pre-scan and skip unchanged files
Absorb rsync strengths without an rsync backend: optionally walk the
full tree before transfer for stable file totals, and skip files whose
size and mtime already match the destination.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): drop unused DiscoveredTransferFile import
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix: address Codex review on PR #2772
* fix(sftp): address Codex review on path gate, aliases, and worker drain
- Invalidate same-transfer fallback when a force-released path gate gets a late truncating OPEN
- Use per-branch ancestors so sibling symlink aliases copy under pre-scan BFS
- Keep interleaved subdirectory walks sequential for deterministic manifest indices
- Drain transfer/bounded workers before propagating errors; ASCII punctuation
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): canonical pre-scan file order and pre-2001 mtime skip
Sort discovered files with compareDirectoryTraversalPaths before index
assignment, and treat timestamps >= 1e11 as milliseconds so year-2000
local mtimes still match SFTP second precision.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): bound multi-select downloads and preserve dest mtime
Address Codex review on PR #2772: serialize local blob reads, bound
multi-select root downloads before pre-scan, replace remaining Unicode
em dashes, and stamp destination mtimes after successful transfers so
skip-unchanged can match on a later pass.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): keep in-place OPEN poison until callback settles
Address Codex review on e2cc8241: do not force-release path gates for
in-place truncating OPENs, fail closed on pending-OPEN wait timeouts
without clearing poison, and replace remaining Unicode em dashes in
ASCII-only transfer modules.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): fail closed when in-place OPEN poison is held
When an isolated in-place truncating OPEN leaves the path gate poisoned
after channel error/cancel, mark the transfer terminal (noTransferFallback)
so fastPut/concurrent-shared cannot wait forever on the unreleased prior
gate.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): omit cyclic dirs from pre-scan plan and lower mtime cutoff
Only record discovered directories after a visit is accepted so symlink
cycles are not mkdir'd as empty targets, and treat magnitudes >= 1e10 as
milliseconds so early-1970s local Date values still match SFTP seconds.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): count max-depth symlink gaps and skip symlink quick-check
Pre-scan now surfaces max-depth symlink directory omissions as transfer
errors (matching interleaved walk), and discovered file symlinks carry
isSymlink so skip-unchanged does not trust link-node size/mtime.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): unit-aware skip mtimes and interleaved symlink bypass
Declare mtimeUnit on skip identities so early-1970 millisecond values
normalize without magnitude cutoffs, and bypass skip-unchanged for
symlinks on the interleaved (non-prescan) directory walk as well.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): bound local pre-scan and preserve SCP destination mtimes
Apply the same directory/entry traversal budget to local folder pre-scan,
and best-effort touch SCP upload targets after commit so skip-unchanged
can match on a later pass.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): address Codex review on pause, skip restat, mtime bound
- Honor transfer pause during folder pre-scan BFS waves/listings
- Re-stat source immediately before skip-unchanged on pre-scanned files
- Bound best-effort destination setStat/utimes so sendComplete cannot hang
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): harden skip restat and pre-transfer mtime capture
- Require a fresh source stat before skip-unchanged (prescan + interleaved)
- Capture source soft identity before bytes move for non-resumable/SCP
- Drop post-commit mtime re-stat fallback; keep ASCII in touched comments
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): propagate fresh re-stat size into interleaved transfers
Use the re-statted source size/mtime for child tasks on the non-prescan
path so grown files are not truncated to the stale listing size.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): omit stale listing size after re-stat failure
When source re-stat fails, clear child totalBytes/mtime so
startStreamTransfer omits an explicit snapshot and the bridge
re-stats on the dedicated session instead of truncating to the
stale listing size.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): make soft-identity capture cancelable
Race pre-transfer source soft-identity capture against cancel and
pass the transfer abort signal into requireSftpChannel so a dead
SFTP reopen cannot block cancel for the channel timeout.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
Co-authored-by: netcatty-bot <308658023+netcatty-bot@users.noreply.github.com>
224 lines
6.8 KiB
TypeScript
224 lines
6.8 KiB
TypeScript
import assert from "node:assert/strict";
|
|
import test from "node:test";
|
|
import type { MutableRefObject } from "react";
|
|
|
|
import type { TransferTask } from "../../../domain/models";
|
|
import { STORAGE_KEY_SFTP_TRANSFER_CONCURRENCY } from "../../../infrastructure/config/storageKeys";
|
|
import { transferDiscoveredFiles } from "./transferDirectoryFileBatch";
|
|
|
|
const memoryStore = new Map<string, string>();
|
|
(globalThis as { localStorage?: Storage }).localStorage = {
|
|
getItem: (key: string) => memoryStore.get(key) ?? null,
|
|
setItem: (key: string, value: string) => {
|
|
memoryStore.set(key, String(value));
|
|
},
|
|
removeItem: (key: string) => {
|
|
memoryStore.delete(key);
|
|
},
|
|
clear: () => memoryStore.clear(),
|
|
key: () => null,
|
|
length: 0,
|
|
} as Storage;
|
|
memoryStore.set(STORAGE_KEY_SFTP_TRANSFER_CONCURRENCY, "1");
|
|
|
|
const rootTask = (): TransferTask => ({
|
|
id: "root",
|
|
fileName: "source",
|
|
sourcePath: "/source",
|
|
targetPath: "/target",
|
|
sourceConnectionId: "local",
|
|
targetConnectionId: "local",
|
|
direction: "upload",
|
|
status: "transferring",
|
|
totalBytes: 1,
|
|
transferredBytes: 0,
|
|
speed: 0,
|
|
startTime: 0,
|
|
isDirectory: true,
|
|
});
|
|
|
|
test("transferDiscoveredFiles re-stats source before skip-unchanged", async () => {
|
|
const transfersRef = { current: [rootTask()] } as MutableRefObject<TransferTask[]>;
|
|
const cancelledTasksRef = { current: new Set<string>() } as MutableRefObject<Set<string>>;
|
|
const activeChildIdsRef = {
|
|
current: new Map<string, Set<string>>(),
|
|
} as MutableRefObject<Map<string, Set<string>>>;
|
|
let transfers = transfersRef.current;
|
|
let transferCalls = 0;
|
|
let sourceStatCalls = 0;
|
|
|
|
const staleMtime = 1_700_000_000_000;
|
|
const freshMtime = 1_700_000_100_000;
|
|
|
|
await transferDiscoveredFiles({
|
|
rootTask: rootTask(),
|
|
files: [{
|
|
name: "a.txt",
|
|
sourcePath: "/source/a.txt",
|
|
targetPath: "/target/a.txt",
|
|
size: 10,
|
|
lastModified: staleMtime,
|
|
}],
|
|
sourceSftpId: null,
|
|
targetSftpId: null,
|
|
sourceIsLocal: true,
|
|
targetIsLocal: true,
|
|
sourceEncoding: "auto",
|
|
targetEncoding: "auto",
|
|
rootTaskId: "root",
|
|
skipUnchanged: true,
|
|
startEntryIndex: 0,
|
|
cancelledTasksRef,
|
|
activeChildIdsRef,
|
|
transfersRef,
|
|
setTransfers: (updater) => {
|
|
transfers = typeof updater === "function" ? updater(transfers) : updater;
|
|
transfersRef.current = transfers;
|
|
},
|
|
waitWhileTransferPaused: async () => {},
|
|
isPauseLatched: () => false,
|
|
transferFile: async () => {
|
|
transferCalls += 1;
|
|
},
|
|
tryStatTarget: async () => ({
|
|
size: 10,
|
|
lastModified: staleMtime,
|
|
type: "file",
|
|
}),
|
|
tryStatSource: async () => {
|
|
sourceStatCalls += 1;
|
|
return {
|
|
size: 10,
|
|
lastModified: freshMtime,
|
|
type: "file",
|
|
};
|
|
},
|
|
});
|
|
|
|
assert.equal(sourceStatCalls, 1, "must re-stat source before skip check");
|
|
assert.equal(transferCalls, 1, "stale listing must not skip a changed source");
|
|
const child = transfers.find((row) => row.parentTaskId === "root");
|
|
assert.ok(child);
|
|
assert.equal(child?.status, "completed");
|
|
assert.equal(child?.sourceLastModified, freshMtime);
|
|
});
|
|
|
|
test("transferDiscoveredFiles does not skip when source re-stat fails", async () => {
|
|
const transfersRef = { current: [rootTask()] } as MutableRefObject<TransferTask[]>;
|
|
const cancelledTasksRef = { current: new Set<string>() } as MutableRefObject<Set<string>>;
|
|
const activeChildIdsRef = {
|
|
current: new Map<string, Set<string>>(),
|
|
} as MutableRefObject<Map<string, Set<string>>>;
|
|
let transfers = transfersRef.current;
|
|
let transferCalls = 0;
|
|
let transferredTotalBytes: number | undefined;
|
|
let transferredSourceLastModified: number | undefined;
|
|
const mtime = 1_700_000_000_000;
|
|
|
|
await transferDiscoveredFiles({
|
|
rootTask: rootTask(),
|
|
files: [{
|
|
name: "a.txt",
|
|
sourcePath: "/source/a.txt",
|
|
targetPath: "/target/a.txt",
|
|
size: 10,
|
|
lastModified: mtime,
|
|
}],
|
|
sourceSftpId: null,
|
|
targetSftpId: null,
|
|
sourceIsLocal: true,
|
|
targetIsLocal: true,
|
|
sourceEncoding: "auto",
|
|
targetEncoding: "auto",
|
|
rootTaskId: "root",
|
|
skipUnchanged: true,
|
|
startEntryIndex: 0,
|
|
cancelledTasksRef,
|
|
activeChildIdsRef,
|
|
transfersRef,
|
|
setTransfers: (updater) => {
|
|
transfers = typeof updater === "function" ? updater(transfers) : updater;
|
|
transfersRef.current = transfers;
|
|
},
|
|
waitWhileTransferPaused: async () => {},
|
|
isPauseLatched: () => false,
|
|
transferFile: async (task) => {
|
|
transferCalls += 1;
|
|
transferredTotalBytes = task.totalBytes;
|
|
transferredSourceLastModified = task.sourceLastModified;
|
|
},
|
|
tryStatTarget: async () => ({
|
|
size: 10,
|
|
lastModified: mtime,
|
|
type: "file",
|
|
}),
|
|
tryStatSource: async () => null,
|
|
});
|
|
|
|
assert.equal(transferCalls, 1, "missing fresh source meta must not skip");
|
|
assert.equal(transferredTotalBytes, 0, "must not pass stale listing size after re-stat failure");
|
|
assert.equal(
|
|
transferredSourceLastModified,
|
|
undefined,
|
|
"must clear stale listing mtime after re-stat failure",
|
|
);
|
|
});
|
|
|
|
test("transferDiscoveredFiles skips only when fresh source still matches", async () => {
|
|
const transfersRef = { current: [rootTask()] } as MutableRefObject<TransferTask[]>;
|
|
const cancelledTasksRef = { current: new Set<string>() } as MutableRefObject<Set<string>>;
|
|
const activeChildIdsRef = {
|
|
current: new Map<string, Set<string>>(),
|
|
} as MutableRefObject<Map<string, Set<string>>>;
|
|
let transfers = transfersRef.current;
|
|
let transferCalls = 0;
|
|
const mtime = 1_700_000_000_000;
|
|
|
|
await transferDiscoveredFiles({
|
|
rootTask: rootTask(),
|
|
files: [{
|
|
name: "a.txt",
|
|
sourcePath: "/source/a.txt",
|
|
targetPath: "/target/a.txt",
|
|
size: 10,
|
|
lastModified: mtime - 60_000,
|
|
}],
|
|
sourceSftpId: null,
|
|
targetSftpId: null,
|
|
sourceIsLocal: true,
|
|
targetIsLocal: true,
|
|
sourceEncoding: "auto",
|
|
targetEncoding: "auto",
|
|
rootTaskId: "root",
|
|
skipUnchanged: true,
|
|
startEntryIndex: 0,
|
|
cancelledTasksRef,
|
|
activeChildIdsRef,
|
|
transfersRef,
|
|
setTransfers: (updater) => {
|
|
transfers = typeof updater === "function" ? updater(transfers) : updater;
|
|
transfersRef.current = transfers;
|
|
},
|
|
waitWhileTransferPaused: async () => {},
|
|
isPauseLatched: () => false,
|
|
transferFile: async () => {
|
|
transferCalls += 1;
|
|
},
|
|
tryStatTarget: async () => ({
|
|
size: 10,
|
|
lastModified: mtime,
|
|
type: "file",
|
|
}),
|
|
tryStatSource: async () => ({
|
|
size: 10,
|
|
lastModified: mtime,
|
|
type: "file",
|
|
}),
|
|
});
|
|
|
|
assert.equal(transferCalls, 0, "matching fresh identity must skip transfer");
|
|
const child = transfers.find((row) => row.parentTaskId === "root");
|
|
assert.equal(child?.status, "completed");
|
|
assert.equal(child?.transferredBytes, 10);
|
|
});
|