mirror of
https://github.com/binaricat/Netcatty.git
synced 2026-11-17 13:01:24 +00:00
* fix(sftp): parallel folder discovery and unblock transfer stalls
Address #2712 / #2556 leftovers after the pipelined fail-closed work
(#2449 / #2720 / #2761): keep bulk body I/O on concurrent READ/WRITE only,
surface upload digest prep as verifying instead of a silent hang, make the
fastPut path-gate wait cancelable (#2755), parallelize multi-select enqueue,
and bound-parallelize directory listings so folder totals appear faster.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* feat(sftp): folder pre-scan and skip unchanged files
Absorb rsync strengths without an rsync backend: optionally walk the
full tree before transfer for stable file totals, and skip files whose
size and mtime already match the destination.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): drop unused DiscoveredTransferFile import
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix: address Codex review on PR #2772
* fix(sftp): address Codex review on path gate, aliases, and worker drain
- Invalidate same-transfer fallback when a force-released path gate gets a late truncating OPEN
- Use per-branch ancestors so sibling symlink aliases copy under pre-scan BFS
- Keep interleaved subdirectory walks sequential for deterministic manifest indices
- Drain transfer/bounded workers before propagating errors; ASCII punctuation
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): canonical pre-scan file order and pre-2001 mtime skip
Sort discovered files with compareDirectoryTraversalPaths before index
assignment, and treat timestamps >= 1e11 as milliseconds so year-2000
local mtimes still match SFTP second precision.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): bound multi-select downloads and preserve dest mtime
Address Codex review on PR #2772: serialize local blob reads, bound
multi-select root downloads before pre-scan, replace remaining Unicode
em dashes, and stamp destination mtimes after successful transfers so
skip-unchanged can match on a later pass.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): keep in-place OPEN poison until callback settles
Address Codex review on e2cc8241: do not force-release path gates for
in-place truncating OPENs, fail closed on pending-OPEN wait timeouts
without clearing poison, and replace remaining Unicode em dashes in
ASCII-only transfer modules.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): fail closed when in-place OPEN poison is held
When an isolated in-place truncating OPEN leaves the path gate poisoned
after channel error/cancel, mark the transfer terminal (noTransferFallback)
so fastPut/concurrent-shared cannot wait forever on the unreleased prior
gate.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): omit cyclic dirs from pre-scan plan and lower mtime cutoff
Only record discovered directories after a visit is accepted so symlink
cycles are not mkdir'd as empty targets, and treat magnitudes >= 1e10 as
milliseconds so early-1970s local Date values still match SFTP seconds.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): count max-depth symlink gaps and skip symlink quick-check
Pre-scan now surfaces max-depth symlink directory omissions as transfer
errors (matching interleaved walk), and discovered file symlinks carry
isSymlink so skip-unchanged does not trust link-node size/mtime.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): unit-aware skip mtimes and interleaved symlink bypass
Declare mtimeUnit on skip identities so early-1970 millisecond values
normalize without magnitude cutoffs, and bypass skip-unchanged for
symlinks on the interleaved (non-prescan) directory walk as well.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): bound local pre-scan and preserve SCP destination mtimes
Apply the same directory/entry traversal budget to local folder pre-scan,
and best-effort touch SCP upload targets after commit so skip-unchanged
can match on a later pass.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): address Codex review on pause, skip restat, mtime bound
- Honor transfer pause during folder pre-scan BFS waves/listings
- Re-stat source immediately before skip-unchanged on pre-scanned files
- Bound best-effort destination setStat/utimes so sendComplete cannot hang
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): harden skip restat and pre-transfer mtime capture
- Require a fresh source stat before skip-unchanged (prescan + interleaved)
- Capture source soft identity before bytes move for non-resumable/SCP
- Drop post-commit mtime re-stat fallback; keep ASCII in touched comments
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): propagate fresh re-stat size into interleaved transfers
Use the re-statted source size/mtime for child tasks on the non-prescan
path so grown files are not truncated to the stale listing size.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): omit stale listing size after re-stat failure
When source re-stat fails, clear child totalBytes/mtime so
startStreamTransfer omits an explicit snapshot and the bridge
re-stats on the dedicated session instead of truncating to the
stale listing size.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
* fix(sftp): make soft-identity capture cancelable
Race pre-transfer source soft-identity capture against cancel and
pass the transfer abort signal into requireSftpChannel so a dead
SFTP reopen cannot block cancel for the channel timeout.
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
Co-authored-by: netcatty-bot <308658023+netcatty-bot@users.noreply.github.com>
126 lines
5.2 KiB
TypeScript
126 lines
5.2 KiB
TypeScript
import assert from "node:assert/strict";
|
|
import { createHash } from "node:crypto";
|
|
import test from "node:test";
|
|
|
|
import {
|
|
accountSftpDirectoryEntries,
|
|
appendDirectoryCheckpointIdentity,
|
|
appendDirectoryManifestIdentity,
|
|
claimSftpDirectoryVisit,
|
|
createDirectoryManifestAccumulator,
|
|
createEmptyDirectoryResumeCheckpoint,
|
|
createSftpDirectoryBranchAncestors,
|
|
createSftpDirectoryTraversalBudget,
|
|
EMPTY_DIRECTORY_MANIFEST_HASH,
|
|
isValidDirectoryResumeCheckpoint,
|
|
MAX_SFTP_FOLLOWED_SYMLINK_DEPTH,
|
|
releaseSftpDirectoryVisit,
|
|
shouldFollowSftpSymlinkDirectory,
|
|
} from "./sftpDirectoryCheckpoint";
|
|
|
|
test("versioned directory manifests detect changed and reordered entries", () => {
|
|
const first = "a".repeat(64);
|
|
const second = "b".repeat(64);
|
|
const changed = "c".repeat(64);
|
|
const build = (identities: string[]) => {
|
|
const checkpoint = createEmptyDirectoryResumeCheckpoint();
|
|
for (const identity of identities) {
|
|
checkpoint.manifestHash = appendDirectoryCheckpointIdentity(checkpoint, identity);
|
|
checkpoint.coveredEntries += 1;
|
|
}
|
|
return checkpoint;
|
|
};
|
|
|
|
const original = build([first, second]);
|
|
const batched = createDirectoryManifestAccumulator(createEmptyDirectoryResumeCheckpoint());
|
|
batched.append(first);
|
|
batched.append(second);
|
|
assert.equal(original.version, 2);
|
|
assert.equal(original.manifestHash, batched.digest());
|
|
assert.equal(original.manifestHash, build([first, second]).manifestHash);
|
|
assert.notEqual(original.manifestHash, build([first, changed]).manifestHash);
|
|
assert.notEqual(original.manifestHash, build([second, first]).manifestHash);
|
|
});
|
|
|
|
test("legacy chained manifests remain valid and append-compatible", () => {
|
|
const legacy = {
|
|
version: 1 as const,
|
|
coveredEntries: 1,
|
|
completedEntries: 1,
|
|
manifestHash: appendDirectoryManifestIdentity(
|
|
EMPTY_DIRECTORY_MANIFEST_HASH,
|
|
"a".repeat(64),
|
|
),
|
|
};
|
|
assert.equal(isValidDirectoryResumeCheckpoint(legacy), true);
|
|
assert.equal(
|
|
legacy.manifestHash,
|
|
createHash("sha256")
|
|
.update(`${EMPTY_DIRECTORY_MANIFEST_HASH}:${"a".repeat(64)}`)
|
|
.digest("hex"),
|
|
);
|
|
assert.equal(
|
|
appendDirectoryCheckpointIdentity(legacy, "b".repeat(64)),
|
|
appendDirectoryManifestIdentity(legacy.manifestHash, "b".repeat(64)),
|
|
);
|
|
});
|
|
|
|
test("live and resumed directory transfers share the same symlink depth policy", () => {
|
|
assert.equal(MAX_SFTP_FOLLOWED_SYMLINK_DEPTH, 32);
|
|
assert.equal(shouldFollowSftpSymlinkDirectory(8), true);
|
|
assert.equal(shouldFollowSftpSymlinkDirectory(31), true);
|
|
assert.equal(shouldFollowSftpSymlinkDirectory(32), false);
|
|
});
|
|
|
|
test("remote directory traversal rejects ancestor cycles but allows sibling aliases", () => {
|
|
const budget = createSftpDirectoryTraversalBudget({ maxDirectories: 3, maxEntries: 3 });
|
|
const root = claimSftpDirectoryVisit(budget, "/srv/root");
|
|
assert.ok(root);
|
|
accountSftpDirectoryEntries(budget, 2);
|
|
// Re-entering the same canonical path on the current branch is a cycle.
|
|
assert.equal(claimSftpDirectoryVisit(budget, "/srv/root"), null);
|
|
// Distinct symlink aliases to one target must both be traversed.
|
|
const aliasA = claimSftpDirectoryVisit(budget, "/srv/shared");
|
|
assert.ok(aliasA);
|
|
releaseSftpDirectoryVisit(budget, aliasA);
|
|
const aliasB = claimSftpDirectoryVisit(budget, "/srv/shared");
|
|
assert.ok(aliasB);
|
|
releaseSftpDirectoryVisit(budget, aliasB);
|
|
assert.throws(() => accountSftpDirectoryEntries(budget, 2), /entry limit/i);
|
|
assert.throws(() => claimSftpDirectoryVisit(budget, "/srv/third"), /directory limit/i);
|
|
releaseSftpDirectoryVisit(budget, root);
|
|
assert.equal(budget.activeCanonicalDirectories.size, 0);
|
|
});
|
|
|
|
test("parallel sibling branches may both claim the same canonical alias", () => {
|
|
const budget = createSftpDirectoryTraversalBudget({ maxDirectories: 8, maxEntries: 8 });
|
|
const rootAncestors = createSftpDirectoryBranchAncestors();
|
|
const root = claimSftpDirectoryVisit(budget, "/srv/root", rootAncestors);
|
|
assert.ok(root);
|
|
const branchA = createSftpDirectoryBranchAncestors(rootAncestors);
|
|
const branchB = createSftpDirectoryBranchAncestors(rootAncestors);
|
|
const aliasA = claimSftpDirectoryVisit(budget, "/srv/shared", branchA);
|
|
const aliasB = claimSftpDirectoryVisit(budget, "/srv/shared", branchB);
|
|
assert.ok(aliasA);
|
|
assert.ok(aliasB);
|
|
assert.equal(claimSftpDirectoryVisit(budget, "/srv/root", branchA), null);
|
|
assert.equal(claimSftpDirectoryVisit(budget, "/srv/root", branchB), null);
|
|
releaseSftpDirectoryVisit(budget, aliasA, branchA);
|
|
releaseSftpDirectoryVisit(budget, aliasB, branchB);
|
|
releaseSftpDirectoryVisit(budget, root, rootAncestors);
|
|
});
|
|
|
|
test("remote directory traversal releases sibling visits without retaining history", () => {
|
|
const budget = createSftpDirectoryTraversalBudget({ maxDirectories: 50_000 });
|
|
const root = claimSftpDirectoryVisit(budget, "/srv/root");
|
|
assert.ok(root);
|
|
for (let index = 1; index < 50_000; index += 1) {
|
|
const sibling = claimSftpDirectoryVisit(budget, `/srv/root/alias-${index}`);
|
|
assert.ok(sibling);
|
|
releaseSftpDirectoryVisit(budget, sibling);
|
|
}
|
|
assert.equal(budget.activeCanonicalDirectories.size, 1);
|
|
releaseSftpDirectoryVisit(budget, root);
|
|
assert.equal(budget.activeCanonicalDirectories.size, 0);
|
|
});
|