Files
Netcatty/application/state/sftp/transferDirectoryFileBatch.ts
T
ab3580982d fix(sftp): parallel directory discovery, folder pre-scan, and skip-unchanged (#2712 / #2556) (#2772)
* fix(sftp): parallel folder discovery and unblock transfer stalls

Address #2712 / #2556 leftovers after the pipelined fail-closed work
(#2449 / #2720 / #2761): keep bulk body I/O on concurrent READ/WRITE only,
surface upload digest prep as verifying instead of a silent hang, make the
fastPut path-gate wait cancelable (#2755), parallelize multi-select enqueue,
and bound-parallelize directory listings so folder totals appear faster.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* feat(sftp): folder pre-scan and skip unchanged files

Absorb rsync strengths without an rsync backend: optionally walk the
full tree before transfer for stable file totals, and skip files whose
size and mtime already match the destination.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): drop unused DiscoveredTransferFile import

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix: address Codex review on PR #2772

* fix(sftp): address Codex review on path gate, aliases, and worker drain

- Invalidate same-transfer fallback when a force-released path gate gets a late truncating OPEN
- Use per-branch ancestors so sibling symlink aliases copy under pre-scan BFS
- Keep interleaved subdirectory walks sequential for deterministic manifest indices
- Drain transfer/bounded workers before propagating errors; ASCII punctuation

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): canonical pre-scan file order and pre-2001 mtime skip

Sort discovered files with compareDirectoryTraversalPaths before index
assignment, and treat timestamps >= 1e11 as milliseconds so year-2000
local mtimes still match SFTP second precision.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): bound multi-select downloads and preserve dest mtime

Address Codex review on PR #2772: serialize local blob reads, bound
multi-select root downloads before pre-scan, replace remaining Unicode
em dashes, and stamp destination mtimes after successful transfers so
skip-unchanged can match on a later pass.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): keep in-place OPEN poison until callback settles

Address Codex review on e2cc8241: do not force-release path gates for
in-place truncating OPENs, fail closed on pending-OPEN wait timeouts
without clearing poison, and replace remaining Unicode em dashes in
ASCII-only transfer modules.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): fail closed when in-place OPEN poison is held

When an isolated in-place truncating OPEN leaves the path gate poisoned
after channel error/cancel, mark the transfer terminal (noTransferFallback)
so fastPut/concurrent-shared cannot wait forever on the unreleased prior
gate.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): omit cyclic dirs from pre-scan plan and lower mtime cutoff

Only record discovered directories after a visit is accepted so symlink
cycles are not mkdir'd as empty targets, and treat magnitudes >= 1e10 as
milliseconds so early-1970s local Date values still match SFTP seconds.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): count max-depth symlink gaps and skip symlink quick-check

Pre-scan now surfaces max-depth symlink directory omissions as transfer
errors (matching interleaved walk), and discovered file symlinks carry
isSymlink so skip-unchanged does not trust link-node size/mtime.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): unit-aware skip mtimes and interleaved symlink bypass

Declare mtimeUnit on skip identities so early-1970 millisecond values
normalize without magnitude cutoffs, and bypass skip-unchanged for
symlinks on the interleaved (non-prescan) directory walk as well.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): bound local pre-scan and preserve SCP destination mtimes

Apply the same directory/entry traversal budget to local folder pre-scan,
and best-effort touch SCP upload targets after commit so skip-unchanged
can match on a later pass.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): address Codex review on pause, skip restat, mtime bound

- Honor transfer pause during folder pre-scan BFS waves/listings
- Re-stat source immediately before skip-unchanged on pre-scanned files
- Bound best-effort destination setStat/utimes so sendComplete cannot hang

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): harden skip restat and pre-transfer mtime capture

- Require a fresh source stat before skip-unchanged (prescan + interleaved)
- Capture source soft identity before bytes move for non-resumable/SCP
- Drop post-commit mtime re-stat fallback; keep ASCII in touched comments

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): propagate fresh re-stat size into interleaved transfers

Use the re-statted source size/mtime for child tasks on the non-prescan
path so grown files are not truncated to the stale listing size.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): omit stale listing size after re-stat failure

When source re-stat fails, clear child totalBytes/mtime so
startStreamTransfer omits an explicit snapshot and the bridge
re-stats on the dedicated session instead of truncating to the
stale listing size.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

* fix(sftp): make soft-identity capture cancelable

Race pre-transfer source soft-identity capture against cancel and
pass the transfer abort signal into requireSftpChannel so a dead
SFTP reopen cannot block cancel for the channel timeout.

Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: 陈大猫 <binaricat@users.noreply.github.com>
Co-authored-by: netcatty-bot <308658023+netcatty-bot@users.noreply.github.com>
2026-08-06 18:46:03 +08:00

355 lines
12 KiB
TypeScript

import type { Dispatch, MutableRefObject, SetStateAction } from "react";
import type {
SftpFilenameEncoding,
TransferStatus,
TransferTask,
} from "../../../domain/models";
import { createDirectoryEntryIdentity } from "../../../domain/sftpDirectoryCheckpoint";
import { isUnchangedTransferCandidate } from "../../../domain/sftpTransferSkip";
import { STORAGE_KEY_SFTP_TRANSFER_CONCURRENCY } from "../../../infrastructure/config/storageKeys";
import { localStorageAdapter } from "../../../infrastructure/persistence/localStorageAdapter";
import { logger } from "../../../lib/logger";
import { isSessionError } from "./errors";
import type { DiscoveredTransferFile } from "./transferDirectoryDiscovery";
import { runSftpTransferWorkers } from "./transferConcurrency";
import { isTransferCancelledError } from "./transferRetry";
type TransferOneFile = (
task: TransferTask,
sourceSftpId: string | null,
targetSftpId: string | null,
sourceIsLocal: boolean,
targetIsLocal: boolean,
sourceEncoding: SftpFilenameEncoding,
targetEncoding: SftpFilenameEncoding,
rootTaskId: string,
sameHost?: boolean,
) => Promise<void>;
export type TransferDiscoveredFilesParams = {
rootTask: TransferTask;
files: DiscoveredTransferFile[];
sourceSftpId: string | null;
targetSftpId: string | null;
sourceIsLocal: boolean;
targetIsLocal: boolean;
sourceEncoding: SftpFilenameEncoding;
targetEncoding: SftpFilenameEncoding;
rootTaskId: string;
sameHost?: boolean;
skipUnchanged: boolean;
startEntryIndex: number;
cancelledTasksRef: MutableRefObject<Set<string>>;
activeChildIdsRef: MutableRefObject<Map<string, Set<string>>>;
transfersRef: MutableRefObject<TransferTask[]>;
setTransfers: Dispatch<SetStateAction<TransferTask[]>>;
waitWhileTransferPaused: (rootTaskId: string, taskId?: string) => Promise<void>;
isPauseLatched: (rootTaskId: string, taskId?: string) => boolean;
transferFile: TransferOneFile;
tryStatTarget: (
targetPath: string,
) => Promise<{ size: number; lastModified: number; type: string } | null>;
/** Fresh source identity immediately before skip-unchanged (Codex P1). */
tryStatSource: (
sourcePath: string,
) => Promise<{ size: number; lastModified: number; type: string } | null>;
};
/**
* Transfer a flat pre-scanned file list with the same child-task bookkeeping as
* the interleaved directory walk.
*/
export async function transferDiscoveredFiles(
params: TransferDiscoveredFilesParams,
): Promise<number> {
const {
rootTask,
files,
sourceSftpId,
targetSftpId,
sourceIsLocal,
targetIsLocal,
sourceEncoding,
targetEncoding,
rootTaskId,
sameHost,
skipUnchanged,
startEntryIndex,
cancelledTasksRef,
activeChildIdsRef,
transfersRef,
setTransfers,
waitWhileTransferPaused,
isPauseLatched,
transferFile,
tryStatTarget,
tryStatSource,
} = params;
if (files.length === 0) return 0;
setTransfers((prev) => prev.map((candidate) => candidate.id === rootTaskId
? { ...candidate, phase: "transferring" }
: candidate));
const errors: Error[] = [];
let sessionLostError: Error | null = null;
await runSftpTransferWorkers(
files,
() => localStorageAdapter.readNumber(STORAGE_KEY_SFTP_TRANSFER_CONCURRENCY),
async (file, fileIndex) => {
if (sessionLostError) throw sessionLostError;
if (cancelledTasksRef.current.has(rootTask.id) || cancelledTasksRef.current.has(rootTaskId)) {
throw new Error("Transfer cancelled");
}
const fileSize = file.size;
const sourcePath = file.sourcePath;
const targetPath = file.targetPath;
let skipSourceSize = fileSize;
let skipSourceLastModified = file.lastModified;
const directoryEntryIndex = startEntryIndex + fileIndex;
let directoryEntryIdentity = createDirectoryEntryIdentity({
sourcePath,
targetPath,
size: fileSize,
lastModified: file.lastModified,
});
const persistedChild = transfersRef.current.find((candidate) => (
candidate.parentTaskId === rootTaskId
&& candidate.sourcePath === sourcePath
&& candidate.targetPath === targetPath
));
if (persistedChild?.status === "completed") return;
await waitWhileTransferPaused(rootTaskId);
if (cancelledTasksRef.current.has(rootTask.id) || cancelledTasksRef.current.has(rootTaskId)) {
throw new Error("Transfer cancelled");
}
if (isPauseLatched(rootTaskId)) {
await waitWhileTransferPaused(rootTaskId);
}
// Symlink listing attrs are the link node; transfer follows target bytes.
// Re-stat regular files so skip and transfer both use current size/mtime
// (Codex P1: pre-scan listing attrs can go stale before transfer starts).
let freshSourceOk = false;
if (!file.isSymlink) {
const freshSource = await tryStatSource(sourcePath);
if (freshSource && freshSource.type !== "directory") {
skipSourceSize = freshSource.size;
skipSourceLastModified = freshSource.lastModified;
directoryEntryIdentity = createDirectoryEntryIdentity({
sourcePath,
targetPath,
size: skipSourceSize,
lastModified: skipSourceLastModified,
});
freshSourceOk = true;
} else {
// Do not pass stale pre-scan size as an explicit snapshot (Codex P2).
// Child totalBytes 0 is omitted at startStreamTransfer via `|| undefined`,
// so the dedicated transfer session re-stats the source itself.
skipSourceSize = 0;
skipSourceLastModified = 0;
}
}
if (skipUnchanged && !file.isSymlink && freshSourceOk) {
const existing = await tryStatTarget(targetPath);
if (
existing
&& existing.type !== "directory"
&& isUnchangedTransferCandidate(
{ size: skipSourceSize, lastModified: skipSourceLastModified, mtimeUnit: "ms" },
{ size: existing.size, lastModified: existing.lastModified, mtimeUnit: "ms" },
)
) {
const skippedId = persistedChild?.id ?? crypto.randomUUID();
setTransfers((prev) => {
const hasChild = prev.some((row) => row.id === skippedId);
const next = hasChild
? prev.map((row) => row.id === skippedId
? {
...row,
status: "completed" as TransferStatus,
transferredBytes: skipSourceSize,
totalBytes: skipSourceSize,
endTime: Date.now(),
speed: 0,
error: undefined,
}
: row)
: [...prev, {
...rootTask,
id: skippedId,
fileName: file.name,
originalFileName: file.name,
sourcePath,
targetPath,
isDirectory: false,
progressMode: "bytes" as const,
parentTaskId: rootTaskId,
totalBytes: skipSourceSize,
transferredBytes: skipSourceSize,
sourceLastModified: skipSourceLastModified,
directoryEntryIndex,
directoryEntryIdentity,
status: "completed" as TransferStatus,
speed: 0,
startTime: Date.now(),
endTime: Date.now(),
}];
return next.map((row) => {
if (row.id !== rootTaskId) return row;
if (row.status === "paused" || row.status === "pausing" || isPauseLatched(rootTaskId)) {
return { ...row, speed: 0 };
}
return { ...row, transferredBytes: row.transferredBytes + 1 };
});
});
return;
}
}
const fileId = persistedChild?.id ?? crypto.randomUUID();
if (!activeChildIdsRef.current.has(rootTaskId)) {
activeChildIdsRef.current.set(rootTaskId, new Set());
}
activeChildIdsRef.current.get(rootTaskId)!.add(fileId);
const childTask: TransferTask = {
...rootTask,
...persistedChild,
id: fileId,
fileName: file.name,
originalFileName: file.name,
sourcePath,
targetPath,
isDirectory: false,
progressMode: "bytes",
parentTaskId: rootTaskId,
totalBytes: skipSourceSize,
sourceLastModified: skipSourceLastModified || undefined,
directoryEntryIndex,
directoryEntryIdentity,
retryable: rootTask.retryable,
lifecycleEpoch: undefined,
phase: undefined,
pauseUnavailableReason: undefined,
};
setTransfers((prev) => persistedChild
? prev.map((candidate) => candidate.id === fileId ? {
...childTask,
status: "queued" as TransferStatus,
speed: 0,
error: undefined,
endTime: undefined,
lifecycleEpoch: undefined,
} : candidate)
: [...prev, {
...childTask,
status: "queued" as TransferStatus,
transferredBytes: 0,
speed: 0,
startTime: Date.now(),
lifecycleEpoch: undefined,
}]);
try {
await transferFile(
childTask,
sourceSftpId,
targetSftpId,
sourceIsLocal,
targetIsLocal,
sourceEncoding,
targetEncoding,
rootTaskId,
sameHost,
);
activeChildIdsRef.current.get(rootTaskId)?.delete(fileId);
setTransfers((prev) => {
const parentRow = prev.find((row) => row.id === rootTaskId);
const parentFrozen = !!parentRow && (
parentRow.status === "paused"
|| parentRow.status === "pausing"
|| isPauseLatched(rootTaskId)
);
return prev.map((t) => {
if (t.id === fileId) {
return {
...t,
status: "completed" as TransferStatus,
endTime: Date.now(),
transferredBytes: t.totalBytes,
};
}
if (t.id === rootTaskId) {
if (parentFrozen) return { ...t, speed: 0 };
return { ...t, transferredBytes: t.transferredBytes + 1, speed: t.speed };
}
return t;
});
});
await waitWhileTransferPaused(rootTaskId);
} catch (err) {
activeChildIdsRef.current.get(rootTaskId)?.delete(fileId);
const message = err instanceof Error ? err.message : String(err);
if (isTransferCancelledError(err)) {
setTransfers((prev) => prev.map((t) => (
t.id === fileId
? { ...t, status: "cancelled" as TransferStatus, error: undefined, endTime: Date.now() }
: t
)));
errors.push(err instanceof Error ? err : new Error(message));
return;
}
setTransfers((prev) => prev.map((t) => (
t.id === fileId
? { ...t, status: "failed" as TransferStatus, error: message }
: t
)));
if (isSessionError(err) && !sessionLostError) {
sessionLostError = err instanceof Error ? err : new Error(message);
setTransfers((prev) => prev.map((t) => (
t.parentTaskId === rootTaskId
&& (t.status === "queued" || t.status === "pending")
? {
...t,
status: "failed" as TransferStatus,
error: "SFTP session lost - reconnect and resume remaining files",
speed: 0,
endTime: Date.now(),
}
: t
)));
}
errors.push(err instanceof Error ? err : new Error(message));
if (sessionLostError) throw sessionLostError;
if (isPauseLatched(rootTaskId)) {
await waitWhileTransferPaused(rootTaskId);
}
}
},
{
beforeClaim: async () => {
if (sessionLostError) return;
await waitWhileTransferPaused(rootTaskId);
},
},
).catch((err) => {
if (sessionLostError || isTransferCancelledError(err)) return;
throw err;
});
if (sessionLostError) {
logger.warn("[SFTP] Directory transfer stopped early: session lost", sessionLostError.message);
} else if (errors.length > 0) {
logger.debug?.("[SFTP] Some files in directory transfer failed", errors);
}
return errors.length;
}