Files
Netcatty/domain/sftpDirectoryCheckpoint.test.ts
T
ab3580982d fix(sftp): parallel directory discovery, folder pre-scan, and skip-unchanged (#2712 / #2556) (#2772)
* fix(sftp): parallel folder discovery and unblock transfer stalls

Address #2712 / #2556 leftovers after the pipelined fail-closed work
(#2449 / #2720 / #2761): keep bulk body I/O on concurrent READ/WRITE only,
surface upload digest prep as verifying instead of a silent hang, make the
fastPut path-gate wait cancelable (#2755), parallelize multi-select enqueue,
and bound-parallelize directory listings so folder totals appear faster.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* feat(sftp): folder pre-scan and skip unchanged files

Absorb rsync strengths without an rsync backend: optionally walk the
full tree before transfer for stable file totals, and skip files whose
size and mtime already match the destination.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): drop unused DiscoveredTransferFile import

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix: address Codex review on PR #2772

* fix(sftp): address Codex review on path gate, aliases, and worker drain

- Invalidate same-transfer fallback when a force-released path gate gets a late truncating OPEN
- Use per-branch ancestors so sibling symlink aliases copy under pre-scan BFS
- Keep interleaved subdirectory walks sequential for deterministic manifest indices
- Drain transfer/bounded workers before propagating errors; ASCII punctuation

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): canonical pre-scan file order and pre-2001 mtime skip

Sort discovered files with compareDirectoryTraversalPaths before index
assignment, and treat timestamps >= 1e11 as milliseconds so year-2000
local mtimes still match SFTP second precision.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): bound multi-select downloads and preserve dest mtime

Address Codex review on PR #2772: serialize local blob reads, bound
multi-select root downloads before pre-scan, replace remaining Unicode
em dashes, and stamp destination mtimes after successful transfers so
skip-unchanged can match on a later pass.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): keep in-place OPEN poison until callback settles

Address Codex review on e2cc8241: do not force-release path gates for
in-place truncating OPENs, fail closed on pending-OPEN wait timeouts
without clearing poison, and replace remaining Unicode em dashes in
ASCII-only transfer modules.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): fail closed when in-place OPEN poison is held

When an isolated in-place truncating OPEN leaves the path gate poisoned
after channel error/cancel, mark the transfer terminal (noTransferFallback)
so fastPut/concurrent-shared cannot wait forever on the unreleased prior
gate.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): omit cyclic dirs from pre-scan plan and lower mtime cutoff

Only record discovered directories after a visit is accepted so symlink
cycles are not mkdir'd as empty targets, and treat magnitudes >= 1e10 as
milliseconds so early-1970s local Date values still match SFTP seconds.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): count max-depth symlink gaps and skip symlink quick-check

Pre-scan now surfaces max-depth symlink directory omissions as transfer
errors (matching interleaved walk), and discovered file symlinks carry
isSymlink so skip-unchanged does not trust link-node size/mtime.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): unit-aware skip mtimes and interleaved symlink bypass

Declare mtimeUnit on skip identities so early-1970 millisecond values
normalize without magnitude cutoffs, and bypass skip-unchanged for
symlinks on the interleaved (non-prescan) directory walk as well.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): bound local pre-scan and preserve SCP destination mtimes

Apply the same directory/entry traversal budget to local folder pre-scan,
and best-effort touch SCP upload targets after commit so skip-unchanged
can match on a later pass.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): address Codex review on pause, skip restat, mtime bound

- Honor transfer pause during folder pre-scan BFS waves/listings
- Re-stat source immediately before skip-unchanged on pre-scanned files
- Bound best-effort destination setStat/utimes so sendComplete cannot hang

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): harden skip restat and pre-transfer mtime capture

- Require a fresh source stat before skip-unchanged (prescan + interleaved)
- Capture source soft identity before bytes move for non-resumable/SCP
- Drop post-commit mtime re-stat fallback; keep ASCII in touched comments

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): propagate fresh re-stat size into interleaved transfers

Use the re-statted source size/mtime for child tasks on the non-prescan
path so grown files are not truncated to the stale listing size.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): omit stale listing size after re-stat failure

When source re-stat fails, clear child totalBytes/mtime so
startStreamTransfer omits an explicit snapshot and the bridge
re-stats on the dedicated session instead of truncating to the
stale listing size.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): make soft-identity capture cancelable

Race pre-transfer source soft-identity capture against cancel and
pass the transfer abort signal into requireSftpChannel so a dead
SFTP reopen cannot block cancel for the channel timeout.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
Co-authored-by: netcatty-bot <308658023+netcatty-bot@users.noreply.github.com>
2026-08-06 18:46:03 +08:00

126 lines
5.2 KiB
TypeScript

import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import test from "node:test";
import {
accountSftpDirectoryEntries,
appendDirectoryCheckpointIdentity,
appendDirectoryManifestIdentity,
claimSftpDirectoryVisit,
createDirectoryManifestAccumulator,
createEmptyDirectoryResumeCheckpoint,
createSftpDirectoryBranchAncestors,
createSftpDirectoryTraversalBudget,
EMPTY_DIRECTORY_MANIFEST_HASH,
isValidDirectoryResumeCheckpoint,
MAX_SFTP_FOLLOWED_SYMLINK_DEPTH,
releaseSftpDirectoryVisit,
shouldFollowSftpSymlinkDirectory,
} from "./sftpDirectoryCheckpoint";
test("versioned directory manifests detect changed and reordered entries", () => {
const first = "a".repeat(64);
const second = "b".repeat(64);
const changed = "c".repeat(64);
const build = (identities: string[]) => {
const checkpoint = createEmptyDirectoryResumeCheckpoint();
for (const identity of identities) {
checkpoint.manifestHash = appendDirectoryCheckpointIdentity(checkpoint, identity);
checkpoint.coveredEntries += 1;
}
return checkpoint;
};
const original = build([first, second]);
const batched = createDirectoryManifestAccumulator(createEmptyDirectoryResumeCheckpoint());
batched.append(first);
batched.append(second);
assert.equal(original.version, 2);
assert.equal(original.manifestHash, batched.digest());
assert.equal(original.manifestHash, build([first, second]).manifestHash);
assert.notEqual(original.manifestHash, build([first, changed]).manifestHash);
assert.notEqual(original.manifestHash, build([second, first]).manifestHash);
});
test("legacy chained manifests remain valid and append-compatible", () => {
const legacy = {
version: 1 as const,
coveredEntries: 1,
completedEntries: 1,
manifestHash: appendDirectoryManifestIdentity(
EMPTY_DIRECTORY_MANIFEST_HASH,
"a".repeat(64),
),
};
assert.equal(isValidDirectoryResumeCheckpoint(legacy), true);
assert.equal(
legacy.manifestHash,
createHash("sha256")
.update(`${EMPTY_DIRECTORY_MANIFEST_HASH}:${"a".repeat(64)}`)
.digest("hex"),
);
assert.equal(
appendDirectoryCheckpointIdentity(legacy, "b".repeat(64)),
appendDirectoryManifestIdentity(legacy.manifestHash, "b".repeat(64)),
);
});
test("live and resumed directory transfers share the same symlink depth policy", () => {
assert.equal(MAX_SFTP_FOLLOWED_SYMLINK_DEPTH, 32);
assert.equal(shouldFollowSftpSymlinkDirectory(8), true);
assert.equal(shouldFollowSftpSymlinkDirectory(31), true);
assert.equal(shouldFollowSftpSymlinkDirectory(32), false);
});
test("remote directory traversal rejects ancestor cycles but allows sibling aliases", () => {
const budget = createSftpDirectoryTraversalBudget({ maxDirectories: 3, maxEntries: 3 });
const root = claimSftpDirectoryVisit(budget, "/srv/root");
assert.ok(root);
accountSftpDirectoryEntries(budget, 2);
// Re-entering the same canonical path on the current branch is a cycle.
assert.equal(claimSftpDirectoryVisit(budget, "/srv/root"), null);
// Distinct symlink aliases to one target must both be traversed.
const aliasA = claimSftpDirectoryVisit(budget, "/srv/shared");
assert.ok(aliasA);
releaseSftpDirectoryVisit(budget, aliasA);
const aliasB = claimSftpDirectoryVisit(budget, "/srv/shared");
assert.ok(aliasB);
releaseSftpDirectoryVisit(budget, aliasB);
assert.throws(() => accountSftpDirectoryEntries(budget, 2), /entry limit/i);
assert.throws(() => claimSftpDirectoryVisit(budget, "/srv/third"), /directory limit/i);
releaseSftpDirectoryVisit(budget, root);
assert.equal(budget.activeCanonicalDirectories.size, 0);
});
test("parallel sibling branches may both claim the same canonical alias", () => {
const budget = createSftpDirectoryTraversalBudget({ maxDirectories: 8, maxEntries: 8 });
const rootAncestors = createSftpDirectoryBranchAncestors();
const root = claimSftpDirectoryVisit(budget, "/srv/root", rootAncestors);
assert.ok(root);
const branchA = createSftpDirectoryBranchAncestors(rootAncestors);
const branchB = createSftpDirectoryBranchAncestors(rootAncestors);
const aliasA = claimSftpDirectoryVisit(budget, "/srv/shared", branchA);
const aliasB = claimSftpDirectoryVisit(budget, "/srv/shared", branchB);
assert.ok(aliasA);
assert.ok(aliasB);
assert.equal(claimSftpDirectoryVisit(budget, "/srv/root", branchA), null);
assert.equal(claimSftpDirectoryVisit(budget, "/srv/root", branchB), null);
releaseSftpDirectoryVisit(budget, aliasA, branchA);
releaseSftpDirectoryVisit(budget, aliasB, branchB);
releaseSftpDirectoryVisit(budget, root, rootAncestors);
});
test("remote directory traversal releases sibling visits without retaining history", () => {
const budget = createSftpDirectoryTraversalBudget({ maxDirectories: 50_000 });
const root = claimSftpDirectoryVisit(budget, "/srv/root");
assert.ok(root);
for (let index = 1; index < 50_000; index += 1) {
const sibling = claimSftpDirectoryVisit(budget, `/srv/root/alias-${index}`);
assert.ok(sibling);
releaseSftpDirectoryVisit(budget, sibling);
}
assert.equal(budget.activeCanonicalDirectories.size, 1);
releaseSftpDirectoryVisit(budget, root);
assert.equal(budget.activeCanonicalDirectories.size, 0);
});