mirror of
https://github.com/binaricat/Netcatty.git
synced 2026-10-29 18:11:40 +00:00
Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 bincxz
|
2269443f42 | ||
|
bincxz
|
a5321157a0 | ||
|
bincxz
|
2454256c3d |
No files matched your search
@@ -0,0 +1,63 @@
|
||||
# Catty prompt cache repair and verification
|
||||
|
||||
## Scope and issue history
|
||||
|
||||
- Original report: [#3545](https://github.com/binaricat/Netcatty/issues/3545), duplicate [#3546](https://github.com/binaricat/Netcatty/issues/3546).
|
||||
- [#3547](https://github.com/binaricat/Netcatty/pull/3547) moved static guidelines before scope/session details. That repair remains valid, but its closing comment did not claim a measured DeepSeek result.
|
||||
- This change starts from main `b5fee9632ea012e5c15fbdae454dec2aa7d32572` and repairs additional cache invalidation and usage-reporting paths.
|
||||
|
||||
## Confirmed defects and changes
|
||||
|
||||
1. Completed terminal results were rewritten when the next user turn rebuilt history. Keep the already bounded result, with secret redaction, and let context budgeting own necessary reductions.
|
||||
2. Live SDK results use JSON while saved history uses text. Apply the same bounded compression to both before sending them, avoiding a second, different representation on the next turn.
|
||||
3. Routine output cleanup inserted the latest goal and pending handles at the start of otherwise intact history. Reserve that reinjection for actual budget pressure or summarization.
|
||||
4. Large-output handle notices were replaced at the front on every step. Results already containing a handle need no duplicate notice. Announce missing handles once at the tail and preserve prior notices; continue to reserve their budget if older results are pruned.
|
||||
5. DeepSeek's documented `prompt_cache_hit_tokens` was lost when the OpenAI-compatible SDK parsed responses. Normalize it to the standard cached-token field before parsing, preserving an existing standard value, including zero. Cover streaming and non-streaming responses.
|
||||
6. Catty omitted cache counts from its usage events and saved assistant messages. Preserve provider counts and attach aggregate usage after the final assistant message has been created.
|
||||
7. Native Anthropic requests lacked cache breakpoints. Cache the static system block and a bounded conversation tail; leave implicit-cache OpenAI/Google requests untouched and preserve signed reasoning content. Enable these vendor fields only for the official HTTPS Anthropic endpoint (including the SDK's default endpoint), not third-party Messages-compatible gateways. Use the shared resolved SDK address, including OpenRouter/Ollama fallbacks when a stored URL is empty. Review follow-ups add endpoint gating and full-driver gateway/fallback regressions; they do not change the measured DeepSeek path.
|
||||
|
||||
## Comparison with mature agents and official guidance
|
||||
|
||||
- [Codex prompt-caching tests](https://github.com/openai/codex/blob/8c076942cb70fba8e5c4732c23f1faabc4aed484/codex-rs/core/tests/suite/prompt_caching.rs): exercise stable tools and append-only request prefixes. The new tests compare the actual serialized requests produced by Catty's driver, SDK and bridge adapter across consecutive turns.
|
||||
- [OpenCode cache policy](https://github.com/anomalyco/opencode/blob/c38d83c0c4f92466ff9c8e12b8a741698e811012/packages/ai/src/cache-policy.ts): use provider-specific explicit cache boundaries where required and leave implicit-cache protocols alone.
|
||||
- [DeepSeek context caching](https://api-docs.deepseek.com/guides/kv_cache/): exact shared prefixes, asynchronous/best-effort cache persistence and provider-reported hit/miss counts. Live probes wait between turns and use the official endpoint.
|
||||
- [Anthropic prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching): explicit boundaries, tool/system/message ordering and a limit of four breakpoints. Catty uses at most three automatic markers.
|
||||
- [OpenAI prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching): stable shared prefixes. No additional explicit cache-control fields are sent on this path.
|
||||
|
||||
## Live verification
|
||||
|
||||
The final fixed Catty driver was bundled from the isolated worktree and executed inside the running Electron renderer, using the app's existing IPC and configured `https://api.deepseek.com` service, model `deepseek-v4-flash`. The API key stayed inside the app's credential handling.
|
||||
|
||||
Each run contains six consecutive turns for each of three scenarios: plain chat, one command per turn, and a command producing 14,000 synthetic log lines followed by a bounded saved-output read. Model requests and usage are real. Tool execution uses an in-memory fixture; no command is sent to the user's connected SSH hosts. The prompts contain only synthetic session/host information.
|
||||
|
||||
The results file records every provider-reported input/cache/output count, per-step counts and errors. The rate is total cached input divided by total input, including all model calls in a turn. It is not an average of rounded percentages.
|
||||
|
||||
Two complete runs of the final source produced 36 turns / 72 real model requests, with zero errors:
|
||||
|
||||
| Scenario | Turns | Weighted cache hit rate | Per-turn range |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| Plain chat | 12 | 98.55% | 98.19%-98.92% |
|
||||
| Command execution | 12 | 98.59% | 98.12%-99.03% |
|
||||
| Large command output + saved-output read | 12 | 97.80% | 97.30%-98.32% |
|
||||
|
||||
Raw per-turn and per-step counters are in [the JSON evidence file](./catty-prompt-cache-live-2026-10-18.json). These are warm-cache measurements; an earlier probe's first request correctly reported zero cached input before subsequent requests reached about 98%. The earlier probe is excluded from the final-source totals.
|
||||
|
||||
The production chat-message component was rendered with the live conversation in Electron. Its usage footer showed input, output, cache and reasoning counts; opening the command entry showed the expected fixture result. No product UI layout was changed.
|
||||
|
||||
Limits: live provider evidence covers the configured DeepSeek model. Native Anthropic request formatting and usage preservation are verified with the real SDK and a protocol fixture, not a paid Claude session. Cache hits remain provider-managed; a new prefix, necessary compaction or expired cache can legitimately miss.
|
||||
|
||||
## Automated verification
|
||||
|
||||
- Full AI/history/UI-related suite after the endpoint review follow-ups: 600 tests passed.
|
||||
- New regression coverage includes six consecutive plain-chat turns, six command turns, six large-file turns, six large-terminal-output turns, long-output handles, streaming/non-streaming DeepSeek usage, aggregate multi-step usage and native Anthropic boundaries.
|
||||
- Production build passed.
|
||||
- Full lint passed with eight existing warnings; final changed-file lint passed without warnings.
|
||||
- Type checking matches the untouched main baseline: 758 existing errors, zero additional errors after normalizing checkout paths and line numbers.
|
||||
|
||||
Full-repository verification also found setup and unrelated timing limits:
|
||||
|
||||
- The first run passed 13,968 tests and skipped 36, with one failed file caused by the fresh worktree missing the nested `plugin-cli/node_modules/yauzl` dependency. Linking the existing dependency directory repaired the setup; all 29 CLI tests then passed.
|
||||
- A repeat full run hit the unchanged `large directory progress coalesces synchronous persistence work` timing assertion (2,638,624 characters persisted against a 588,644-character snapshot). That assertion passed in the prior full run. All 102 tests in its file passed on isolated rerun; the same assertion also passed on untouched main. This is an observed full-run timing fluctuation, not a claimed baseline reproduction or an SFTP repair.
|
||||
- The Homebrew Node 22 binary is broken locally (missing shared library); verification uses an isolated official npm Node 22 runtime. `TMPDIR=/tmp` selects Netcatty's shorter private per-user temporary directory for Unix-socket tests. The default macOS path exceeded the FIDO test socket limit; no product source was changed to accommodate this environment.
|
||||
|
||||
The full suite is therefore not reported as unconditionally green. No Catty/cache test failed in the final implementation.
|
||||
@@ -0,0 +1,955 @@
|
||||
{
|
||||
"date": "2026-10-18",
|
||||
"endpoint": "https://api.deepseek.com",
|
||||
"model": "deepseek-v4-flash",
|
||||
"input": "synthetic prompts and in-memory tool output",
|
||||
"totalTurns": 36,
|
||||
"totalModelCalls": 72,
|
||||
"summary": [
|
||||
{
|
||||
"kind": "chat",
|
||||
"turns": 12,
|
||||
"inputTokens": 162870,
|
||||
"cachedTokens": 160512,
|
||||
"weightedHitRate": 98.55,
|
||||
"minTurnRate": 98.19,
|
||||
"maxTurnRate": 98.92
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"turns": 12,
|
||||
"inputTokens": 341856,
|
||||
"cachedTokens": 337024,
|
||||
"weightedHitRate": 98.59,
|
||||
"minTurnRate": 98.12,
|
||||
"maxTurnRate": 99.03
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"turns": 12,
|
||||
"inputTokens": 598639,
|
||||
"cachedTokens": 585472,
|
||||
"weightedHitRate": 97.8,
|
||||
"minTurnRate": 97.3,
|
||||
"maxTurnRate": 98.32
|
||||
}
|
||||
],
|
||||
"runs": [
|
||||
{
|
||||
"model": "deepseek-v4-flash",
|
||||
"rows": [
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 1,
|
||||
"inputTokens": 13500,
|
||||
"cachedTokens": 13312,
|
||||
"hitRate": 98.61,
|
||||
"outputTokens": 21,
|
||||
"modelCalls": 1,
|
||||
"toolResults": 0,
|
||||
"milliseconds": 690
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 2,
|
||||
"inputTokens": 13529,
|
||||
"cachedTokens": 13312,
|
||||
"hitRate": 98.4,
|
||||
"outputTokens": 4,
|
||||
"modelCalls": 1,
|
||||
"toolResults": 0,
|
||||
"milliseconds": 898
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 3,
|
||||
"inputTokens": 13558,
|
||||
"cachedTokens": 13312,
|
||||
"hitRate": 98.19,
|
||||
"outputTokens": 4,
|
||||
"modelCalls": 1,
|
||||
"toolResults": 0,
|
||||
"milliseconds": 658
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 4,
|
||||
"inputTokens": 13587,
|
||||
"cachedTokens": 13440,
|
||||
"hitRate": 98.92,
|
||||
"outputTokens": 4,
|
||||
"modelCalls": 1,
|
||||
"toolResults": 0,
|
||||
"milliseconds": 982
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 5,
|
||||
"inputTokens": 13616,
|
||||
"cachedTokens": 13440,
|
||||
"hitRate": 98.71,
|
||||
"outputTokens": 4,
|
||||
"modelCalls": 1,
|
||||
"toolResults": 0,
|
||||
"milliseconds": 611
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 6,
|
||||
"inputTokens": 13645,
|
||||
"cachedTokens": 13440,
|
||||
"hitRate": 98.5,
|
||||
"outputTokens": 4,
|
||||
"modelCalls": 1,
|
||||
"toolResults": 0,
|
||||
"milliseconds": 676
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 1,
|
||||
"inputTokens": 27473,
|
||||
"cachedTokens": 27008,
|
||||
"hitRate": 98.31,
|
||||
"outputTokens": 384,
|
||||
"modelCalls": 2,
|
||||
"toolResults": 1,
|
||||
"milliseconds": 5711
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 2,
|
||||
"inputTokens": 28156,
|
||||
"cachedTokens": 27776,
|
||||
"hitRate": 98.65,
|
||||
"outputTokens": 93,
|
||||
"modelCalls": 2,
|
||||
"toolResults": 2,
|
||||
"milliseconds": 4606
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 3,
|
||||
"inputTokens": 28538,
|
||||
"cachedTokens": 28160,
|
||||
"hitRate": 98.68,
|
||||
"outputTokens": 83,
|
||||
"modelCalls": 2,
|
||||
"toolResults": 3,
|
||||
"milliseconds": 4738
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 4,
|
||||
"inputTokens": 28940,
|
||||
"cachedTokens": 28544,
|
||||
"hitRate": 98.63,
|
||||
"outputTokens": 113,
|
||||
"modelCalls": 2,
|
||||
"toolResults": 4,
|
||||
"milliseconds": 5000
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 5,
|
||||
"inputTokens": 29342,
|
||||
"cachedTokens": 29056,
|
||||
"hitRate": 99.03,
|
||||
"outputTokens": 83,
|
||||
"modelCalls": 2,
|
||||
"toolResults": 5,
|
||||
"milliseconds": 5011
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 6,
|
||||
"inputTokens": 29714,
|
||||
"cachedTokens": 29312,
|
||||
"hitRate": 98.65,
|
||||
"outputTokens": 83,
|
||||
"modelCalls": 2,
|
||||
"toolResults": 6,
|
||||
"milliseconds": 4517
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 1,
|
||||
"inputTokens": 42570,
|
||||
"cachedTokens": 41472,
|
||||
"hitRate": 97.42,
|
||||
"outputTokens": 344,
|
||||
"modelCalls": 3,
|
||||
"toolResults": 2,
|
||||
"milliseconds": 6895
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 2,
|
||||
"inputTokens": 45647,
|
||||
"cachedTokens": 44416,
|
||||
"hitRate": 97.3,
|
||||
"outputTokens": 163,
|
||||
"modelCalls": 3,
|
||||
"toolResults": 4,
|
||||
"milliseconds": 5644
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 3,
|
||||
"inputTokens": 48547,
|
||||
"cachedTokens": 47488,
|
||||
"hitRate": 97.82,
|
||||
"outputTokens": 165,
|
||||
"modelCalls": 3,
|
||||
"toolResults": 6,
|
||||
"milliseconds": 5271
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 4,
|
||||
"inputTokens": 51449,
|
||||
"cachedTokens": 50304,
|
||||
"hitRate": 97.77,
|
||||
"outputTokens": 163,
|
||||
"modelCalls": 3,
|
||||
"toolResults": 8,
|
||||
"milliseconds": 5826
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 5,
|
||||
"inputTokens": 54353,
|
||||
"cachedTokens": 53248,
|
||||
"hitRate": 97.97,
|
||||
"outputTokens": 166,
|
||||
"modelCalls": 3,
|
||||
"toolResults": 10,
|
||||
"milliseconds": 6235
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 6,
|
||||
"inputTokens": 57280,
|
||||
"cachedTokens": 56320,
|
||||
"hitRate": 98.32,
|
||||
"outputTokens": 168,
|
||||
"modelCalls": 3,
|
||||
"toolResults": 12,
|
||||
"milliseconds": 5692
|
||||
}
|
||||
],
|
||||
"steps": [
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 1,
|
||||
"promptTokens": 13500,
|
||||
"cachedTokens": 13312,
|
||||
"completionTokens": 21
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 2,
|
||||
"promptTokens": 13529,
|
||||
"cachedTokens": 13312,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 3,
|
||||
"promptTokens": 13558,
|
||||
"cachedTokens": 13312,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 4,
|
||||
"promptTokens": 13587,
|
||||
"cachedTokens": 13440,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 5,
|
||||
"promptTokens": 13616,
|
||||
"cachedTokens": 13440,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 6,
|
||||
"promptTokens": 13645,
|
||||
"cachedTokens": 13440,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 1,
|
||||
"promptTokens": 13531,
|
||||
"cachedTokens": 13312,
|
||||
"completionTokens": 364
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 1,
|
||||
"promptTokens": 13942,
|
||||
"cachedTokens": 13696,
|
||||
"completionTokens": 20
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 2,
|
||||
"promptTokens": 14018,
|
||||
"cachedTokens": 13824,
|
||||
"completionTokens": 73
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 2,
|
||||
"promptTokens": 14138,
|
||||
"cachedTokens": 13952,
|
||||
"completionTokens": 20
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 3,
|
||||
"promptTokens": 14214,
|
||||
"cachedTokens": 14080,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 3,
|
||||
"promptTokens": 14324,
|
||||
"cachedTokens": 14080,
|
||||
"completionTokens": 20
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 4,
|
||||
"promptTokens": 14400,
|
||||
"cachedTokens": 14208,
|
||||
"completionTokens": 93
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 4,
|
||||
"promptTokens": 14540,
|
||||
"cachedTokens": 14336,
|
||||
"completionTokens": 20
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 5,
|
||||
"promptTokens": 14616,
|
||||
"cachedTokens": 14464,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 5,
|
||||
"promptTokens": 14726,
|
||||
"cachedTokens": 14592,
|
||||
"completionTokens": 20
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 6,
|
||||
"promptTokens": 14802,
|
||||
"cachedTokens": 14592,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 6,
|
||||
"promptTokens": 14912,
|
||||
"cachedTokens": 14720,
|
||||
"completionTokens": 20
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 1,
|
||||
"promptTokens": 13558,
|
||||
"cachedTokens": 13312,
|
||||
"completionTokens": 243
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 1,
|
||||
"promptTokens": 14394,
|
||||
"cachedTokens": 13696,
|
||||
"completionTokens": 97
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 1,
|
||||
"promptTokens": 14618,
|
||||
"cachedTokens": 14464,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 2,
|
||||
"promptTokens": 14705,
|
||||
"cachedTokens": 14464,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 2,
|
||||
"promptTokens": 15360,
|
||||
"cachedTokens": 14720,
|
||||
"completionTokens": 96
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 2,
|
||||
"promptTokens": 15582,
|
||||
"cachedTokens": 15232,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 3,
|
||||
"promptTokens": 15669,
|
||||
"cachedTokens": 15488,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 3,
|
||||
"promptTokens": 16326,
|
||||
"cachedTokens": 15616,
|
||||
"completionTokens": 98
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 3,
|
||||
"promptTokens": 16552,
|
||||
"cachedTokens": 16384,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 4,
|
||||
"promptTokens": 16639,
|
||||
"cachedTokens": 16384,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 4,
|
||||
"promptTokens": 17294,
|
||||
"cachedTokens": 16640,
|
||||
"completionTokens": 96
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 4,
|
||||
"promptTokens": 17516,
|
||||
"cachedTokens": 17280,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 5,
|
||||
"promptTokens": 17603,
|
||||
"cachedTokens": 17408,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 5,
|
||||
"promptTokens": 18261,
|
||||
"cachedTokens": 17536,
|
||||
"completionTokens": 99
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 5,
|
||||
"promptTokens": 18489,
|
||||
"cachedTokens": 18304,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 6,
|
||||
"promptTokens": 18576,
|
||||
"cachedTokens": 18432,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 6,
|
||||
"promptTokens": 19236,
|
||||
"cachedTokens": 18560,
|
||||
"completionTokens": 101
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 6,
|
||||
"promptTokens": 19468,
|
||||
"cachedTokens": 19328,
|
||||
"completionTokens": 4
|
||||
}
|
||||
],
|
||||
"errors": []
|
||||
},
|
||||
{
|
||||
"model": "deepseek-v4-flash",
|
||||
"rows": [
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 1,
|
||||
"inputTokens": 13500,
|
||||
"cachedTokens": 13312,
|
||||
"hitRate": 98.61,
|
||||
"outputTokens": 57,
|
||||
"modelCalls": 1,
|
||||
"toolResults": 0,
|
||||
"milliseconds": 1278
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 2,
|
||||
"inputTokens": 13529,
|
||||
"cachedTokens": 13312,
|
||||
"hitRate": 98.4,
|
||||
"outputTokens": 4,
|
||||
"modelCalls": 1,
|
||||
"toolResults": 0,
|
||||
"milliseconds": 658
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 3,
|
||||
"inputTokens": 13558,
|
||||
"cachedTokens": 13312,
|
||||
"hitRate": 98.19,
|
||||
"outputTokens": 4,
|
||||
"modelCalls": 1,
|
||||
"toolResults": 0,
|
||||
"milliseconds": 809
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 4,
|
||||
"inputTokens": 13587,
|
||||
"cachedTokens": 13440,
|
||||
"hitRate": 98.92,
|
||||
"outputTokens": 4,
|
||||
"modelCalls": 1,
|
||||
"toolResults": 0,
|
||||
"milliseconds": 958
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 5,
|
||||
"inputTokens": 13616,
|
||||
"cachedTokens": 13440,
|
||||
"hitRate": 98.71,
|
||||
"outputTokens": 4,
|
||||
"modelCalls": 1,
|
||||
"toolResults": 0,
|
||||
"milliseconds": 857
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 6,
|
||||
"inputTokens": 13645,
|
||||
"cachedTokens": 13440,
|
||||
"hitRate": 98.5,
|
||||
"outputTokens": 4,
|
||||
"modelCalls": 1,
|
||||
"toolResults": 0,
|
||||
"milliseconds": 1061
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 1,
|
||||
"inputTokens": 27300,
|
||||
"cachedTokens": 26880,
|
||||
"hitRate": 98.46,
|
||||
"outputTokens": 195,
|
||||
"modelCalls": 2,
|
||||
"toolResults": 1,
|
||||
"milliseconds": 4773
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 2,
|
||||
"inputTokens": 27785,
|
||||
"cachedTokens": 27264,
|
||||
"hitRate": 98.12,
|
||||
"outputTokens": 84,
|
||||
"modelCalls": 2,
|
||||
"toolResults": 2,
|
||||
"milliseconds": 4529
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 3,
|
||||
"inputTokens": 28142,
|
||||
"cachedTokens": 27776,
|
||||
"hitRate": 98.7,
|
||||
"outputTokens": 67,
|
||||
"modelCalls": 2,
|
||||
"toolResults": 3,
|
||||
"milliseconds": 4584
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 4,
|
||||
"inputTokens": 28482,
|
||||
"cachedTokens": 28032,
|
||||
"hitRate": 98.42,
|
||||
"outputTokens": 67,
|
||||
"modelCalls": 2,
|
||||
"toolResults": 4,
|
||||
"milliseconds": 4706
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 5,
|
||||
"inputTokens": 28822,
|
||||
"cachedTokens": 28416,
|
||||
"hitRate": 98.59,
|
||||
"outputTokens": 67,
|
||||
"modelCalls": 2,
|
||||
"toolResults": 5,
|
||||
"milliseconds": 4558
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 6,
|
||||
"inputTokens": 29162,
|
||||
"cachedTokens": 28800,
|
||||
"hitRate": 98.76,
|
||||
"outputTokens": 67,
|
||||
"modelCalls": 2,
|
||||
"toolResults": 6,
|
||||
"milliseconds": 4496
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 1,
|
||||
"inputTokens": 42428,
|
||||
"cachedTokens": 41344,
|
||||
"hitRate": 97.45,
|
||||
"outputTokens": 270,
|
||||
"modelCalls": 3,
|
||||
"toolResults": 2,
|
||||
"milliseconds": 5945
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 2,
|
||||
"inputTokens": 45447,
|
||||
"cachedTokens": 44416,
|
||||
"hitRate": 97.73,
|
||||
"outputTokens": 164,
|
||||
"modelCalls": 3,
|
||||
"toolResults": 4,
|
||||
"milliseconds": 5125
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 3,
|
||||
"inputTokens": 48356,
|
||||
"cachedTokens": 47232,
|
||||
"hitRate": 97.68,
|
||||
"outputTokens": 166,
|
||||
"modelCalls": 3,
|
||||
"toolResults": 6,
|
||||
"milliseconds": 5238
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 4,
|
||||
"inputTokens": 51279,
|
||||
"cachedTokens": 50176,
|
||||
"hitRate": 97.85,
|
||||
"outputTokens": 167,
|
||||
"modelCalls": 3,
|
||||
"toolResults": 8,
|
||||
"milliseconds": 5535
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 5,
|
||||
"inputTokens": 54195,
|
||||
"cachedTokens": 53120,
|
||||
"hitRate": 98.02,
|
||||
"outputTokens": 164,
|
||||
"modelCalls": 3,
|
||||
"toolResults": 10,
|
||||
"milliseconds": 5977
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 6,
|
||||
"inputTokens": 57088,
|
||||
"cachedTokens": 55936,
|
||||
"hitRate": 97.98,
|
||||
"outputTokens": 162,
|
||||
"modelCalls": 3,
|
||||
"toolResults": 12,
|
||||
"milliseconds": 5391
|
||||
}
|
||||
],
|
||||
"steps": [
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 1,
|
||||
"promptTokens": 13500,
|
||||
"cachedTokens": 13312,
|
||||
"completionTokens": 57
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 2,
|
||||
"promptTokens": 13529,
|
||||
"cachedTokens": 13312,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 3,
|
||||
"promptTokens": 13558,
|
||||
"cachedTokens": 13312,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 4,
|
||||
"promptTokens": 13587,
|
||||
"cachedTokens": 13440,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 5,
|
||||
"promptTokens": 13616,
|
||||
"cachedTokens": 13440,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "chat",
|
||||
"round": 6,
|
||||
"promptTokens": 13645,
|
||||
"cachedTokens": 13440,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 1,
|
||||
"promptTokens": 13531,
|
||||
"cachedTokens": 13312,
|
||||
"completionTokens": 191
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 1,
|
||||
"promptTokens": 13769,
|
||||
"cachedTokens": 13568,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 2,
|
||||
"promptTokens": 13829,
|
||||
"cachedTokens": 13568,
|
||||
"completionTokens": 80
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 2,
|
||||
"promptTokens": 13956,
|
||||
"cachedTokens": 13696,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 3,
|
||||
"promptTokens": 14016,
|
||||
"cachedTokens": 13824,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 3,
|
||||
"promptTokens": 14126,
|
||||
"cachedTokens": 13952,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 4,
|
||||
"promptTokens": 14186,
|
||||
"cachedTokens": 13952,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 4,
|
||||
"promptTokens": 14296,
|
||||
"cachedTokens": 14080,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 5,
|
||||
"promptTokens": 14356,
|
||||
"cachedTokens": 14208,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 5,
|
||||
"promptTokens": 14466,
|
||||
"cachedTokens": 14208,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 6,
|
||||
"promptTokens": 14526,
|
||||
"cachedTokens": 14336,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "command",
|
||||
"round": 6,
|
||||
"promptTokens": 14636,
|
||||
"cachedTokens": 14464,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 1,
|
||||
"promptTokens": 13558,
|
||||
"cachedTokens": 13312,
|
||||
"completionTokens": 166
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 1,
|
||||
"promptTokens": 14320,
|
||||
"cachedTokens": 13696,
|
||||
"completionTokens": 100
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 1,
|
||||
"promptTokens": 14550,
|
||||
"cachedTokens": 14336,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 2,
|
||||
"promptTokens": 14637,
|
||||
"cachedTokens": 14464,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 2,
|
||||
"promptTokens": 15293,
|
||||
"cachedTokens": 14592,
|
||||
"completionTokens": 97
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 2,
|
||||
"promptTokens": 15517,
|
||||
"cachedTokens": 15360,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 3,
|
||||
"promptTokens": 15604,
|
||||
"cachedTokens": 15360,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 3,
|
||||
"promptTokens": 16262,
|
||||
"cachedTokens": 15616,
|
||||
"completionTokens": 99
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 3,
|
||||
"promptTokens": 16490,
|
||||
"cachedTokens": 16256,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 4,
|
||||
"promptTokens": 16577,
|
||||
"cachedTokens": 16384,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 4,
|
||||
"promptTokens": 17236,
|
||||
"cachedTokens": 16512,
|
||||
"completionTokens": 100
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 4,
|
||||
"promptTokens": 17466,
|
||||
"cachedTokens": 17280,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 5,
|
||||
"promptTokens": 17553,
|
||||
"cachedTokens": 17408,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 5,
|
||||
"promptTokens": 18209,
|
||||
"cachedTokens": 17536,
|
||||
"completionTokens": 97
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 5,
|
||||
"promptTokens": 18433,
|
||||
"cachedTokens": 18176,
|
||||
"completionTokens": 4
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 6,
|
||||
"promptTokens": 18520,
|
||||
"cachedTokens": 18304,
|
||||
"completionTokens": 63
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 6,
|
||||
"promptTokens": 19174,
|
||||
"cachedTokens": 18432,
|
||||
"completionTokens": 95
|
||||
},
|
||||
{
|
||||
"kind": "large-output",
|
||||
"round": 6,
|
||||
"promptTokens": 19394,
|
||||
"cachedTokens": 19200,
|
||||
"completionTokens": 4
|
||||
}
|
||||
],
|
||||
"errors": []
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -30,7 +30,7 @@ export interface CattyStreamChunk {
|
||||
stepNumber?: number;
|
||||
}
|
||||
|
||||
const STEP_HANDLE_NOTICE_RE = /^\[step \d+\] Tool output handles available:/;
|
||||
const STEP_HANDLE_NOTICE_RE = /^(?:\[step \d+\] Tool output handles available:|\[Tool output handles available:)/;
|
||||
|
||||
function isStepHandleNoticeMessage(content: unknown): boolean {
|
||||
return typeof content === 'string' && STEP_HANDLE_NOTICE_RE.test(content);
|
||||
|
||||
@@ -0,0 +1,251 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import { CattyTurnDriver } from './turnDrivers/cattyTurnDriver';
|
||||
import { ToolOutputStore } from './toolOutputStore';
|
||||
import { ToolResultDedup } from './toolResultDedup';
|
||||
import { SessionStateStore } from './sessionState';
|
||||
import type { CattyTurnInput, TurnDriverContext } from './turnDrivers/types';
|
||||
import type { AgentEvent } from './types';
|
||||
import type { AISession, ChatMessage, ProviderConfig } from '../types';
|
||||
|
||||
interface WireMessage {
|
||||
role: string;
|
||||
content: unknown;
|
||||
}
|
||||
|
||||
interface WireRequest {
|
||||
messages: WireMessage[];
|
||||
tools?: unknown[];
|
||||
system?: Array<{ type: string; text: string; cache_control?: unknown }>;
|
||||
}
|
||||
|
||||
type Reply = { tool: 'terminal_execute' | 'sftp_read_file' } | { text: string };
|
||||
|
||||
function fixture(t: test.TestContext, replies: Reply[], style: 'openai' | 'anthropic' = 'openai', terminalOutput = 'small output') {
|
||||
const originalWindow = globalThis.window;
|
||||
const originalRaf = globalThis.requestAnimationFrame;
|
||||
const originalCancelRaf = globalThis.cancelAnimationFrame;
|
||||
t.after(() => {
|
||||
globalThis.window = originalWindow;
|
||||
globalThis.requestAnimationFrame = originalRaf;
|
||||
globalThis.cancelAnimationFrame = originalCancelRaf;
|
||||
});
|
||||
globalThis.requestAnimationFrame = callback => setTimeout(() => callback(0), 0) as unknown as number;
|
||||
globalThis.cancelAnimationFrame = id => clearTimeout(id);
|
||||
|
||||
const requests: WireRequest[] = [];
|
||||
const urls: string[] = [];
|
||||
const events: AgentEvent[] = [];
|
||||
const messages: ChatMessage[] = [];
|
||||
const data = new Map<string, (value: string) => void>();
|
||||
const end = new Map<string, () => void>();
|
||||
const bridge = {
|
||||
aiSetChatSessionCancelled: async () => ({ ok: true }),
|
||||
aiMcpUpdateSessions: async () => undefined,
|
||||
aiExec: async () => ({ ok: true, stdout: terminalOutput, stderr: '', exitCode: 0 }),
|
||||
aiCapability: async (_method: string, params: Record<string, unknown>) => ({
|
||||
ok: true,
|
||||
path: params.path,
|
||||
content: Array.from({ length: 2000 }, (_, i) => `log ${i}: synthetic ${String(params.path)}`).join('\n'),
|
||||
}),
|
||||
onAiStreamData: (id: string, callback: (value: string) => void) => {
|
||||
data.set(id, callback);
|
||||
return () => { data.delete(id); };
|
||||
},
|
||||
onAiStreamEnd: (id: string, callback: () => void) => {
|
||||
end.set(id, callback);
|
||||
return () => { end.delete(id); };
|
||||
},
|
||||
onAiStreamError: () => () => undefined,
|
||||
aiChatCancel: async () => true,
|
||||
aiChatStream: async (id: string, url: string, _headers: Record<string, string>, body: string) => {
|
||||
urls.push(url);
|
||||
requests.push(JSON.parse(body) as WireRequest);
|
||||
const reply = replies[requests.length - 1];
|
||||
assert.ok(reply, 'unexpected model request');
|
||||
const send = (chunk: unknown) => data.get(id)?.(JSON.stringify(chunk));
|
||||
if (style === 'anthropic') {
|
||||
send({ type: 'message_start', message: {
|
||||
id: `response-${requests.length}`, type: 'message', role: 'assistant', model: 'claude-sonnet-4-5',
|
||||
content: [], stop_reason: null, stop_sequence: null,
|
||||
usage: { input_tokens: 100, output_tokens: 0, cache_read_input_tokens: 900, cache_creation_input_tokens: 0 },
|
||||
} });
|
||||
send({ type: 'content_block_start', index: 0, content_block: { type: 'text', text: '' } });
|
||||
send({ type: 'content_block_delta', index: 0, delta: { type: 'text_delta', text: 'OK' } });
|
||||
send({ type: 'content_block_stop', index: 0 });
|
||||
send({ type: 'message_delta', delta: { stop_reason: 'end_turn', stop_sequence: null }, usage: { output_tokens: 1 } });
|
||||
send({ type: 'message_stop' });
|
||||
} else {
|
||||
const toolReply = 'tool' in reply;
|
||||
const delta = toolReply ? {
|
||||
role: 'assistant',
|
||||
tool_calls: [{ index: 0, id: `call-${requests.length}`, type: 'function', function: {
|
||||
name: reply.tool,
|
||||
arguments: JSON.stringify(reply.tool === 'terminal_execute'
|
||||
? { sessionId: 'terminal-1', command: `printf audit-${requests.length}` }
|
||||
: { sessionId: 'terminal-1', path: `/fixture/log-${requests.length}` }),
|
||||
} }],
|
||||
} : { role: 'assistant', content: reply.text };
|
||||
send({ id: 'response', object: 'chat.completion.chunk', created: 0, model: 'test-model',
|
||||
choices: [{ index: 0, delta, finish_reason: null }] });
|
||||
send({ id: 'response', object: 'chat.completion.chunk', created: 0, model: 'test-model',
|
||||
choices: [{ index: 0, delta: {}, finish_reason: toolReply ? 'tool_calls' : 'stop' }] });
|
||||
send({ id: 'response', object: 'chat.completion.chunk', created: 0, model: 'test-model', choices: [],
|
||||
usage: { prompt_tokens: 1000, completion_tokens: 1, total_tokens: 1001,
|
||||
prompt_cache_hit_tokens: 900, prompt_cache_miss_tokens: 100 } });
|
||||
data.get(id)?.('[DONE]');
|
||||
}
|
||||
end.get(id)?.();
|
||||
return { ok: true, statusCode: 200 };
|
||||
},
|
||||
};
|
||||
globalThis.window = { netcatty: bridge } as unknown as Window & typeof globalThis;
|
||||
const store = new ToolOutputStore();
|
||||
const dedup = new ToolResultDedup();
|
||||
const state = new SessionStateStore();
|
||||
const provider: ProviderConfig = {
|
||||
id: 'fixture', providerId: 'custom', name: 'Fixture', enabled: true, style,
|
||||
defaultModel: style === 'anthropic' ? 'claude-sonnet-4-5' : 'test-model',
|
||||
baseURL: style === 'anthropic' ? 'https://api.anthropic.com' : 'https://provider.example.test/v1', apiKey: 'not-a-real-key',
|
||||
};
|
||||
let turnNumber = 0;
|
||||
return {
|
||||
requests, urls, events, messages, provider,
|
||||
async turn(maxIterations = 1) {
|
||||
turnNumber += 1;
|
||||
const history = structuredClone(messages);
|
||||
const userText = `Audit round ${turnNumber}`;
|
||||
const assistantId = `assistant-${turnNumber}`;
|
||||
messages.push({ id: `user-${turnNumber}`, role: 'user', content: userText, timestamp: turnNumber },
|
||||
{ id: assistantId, role: 'assistant', content: '', timestamp: turnNumber });
|
||||
const currentSession: AISession = {
|
||||
id: 'chat-1', title: 'Fixture', agentId: 'catty', scope: { type: 'terminal', targetId: 'terminal-1' },
|
||||
createdAt: 0, updatedAt: turnNumber, messages: history,
|
||||
};
|
||||
const signal = new AbortController().signal;
|
||||
const input: CattyTurnInput = {
|
||||
backend: 'catty', chatSessionId: 'chat-1', sendScopeKey: 'chat-1', userText, signal,
|
||||
currentSession, assistantMsgId: assistantId, maxIterations, bridge,
|
||||
context: {
|
||||
activeProvider: provider, activeModelId: provider.defaultModel!, scopeType: 'terminal', scopeLabel: 'Fixture',
|
||||
globalPermissionMode: 'auto', commandBlocklist: [], autoTitleSession: () => undefined,
|
||||
terminalSessions: [{ sessionId: 'terminal-1', hostname: 'fixture.example.test', label: 'Fixture', connected: true }],
|
||||
},
|
||||
ui: {
|
||||
addMessageToSession: (_session, message) => { messages.push(message); },
|
||||
updateMessageById: (_session, id, update) => {
|
||||
const index = messages.findIndex(message => message.id === id);
|
||||
if (index >= 0) messages[index] = update(messages[index]);
|
||||
},
|
||||
updateLastMessage: () => undefined,
|
||||
reportStreamError: (_session, _signal, error) => { throw error; },
|
||||
setStreamingForScope: () => undefined,
|
||||
},
|
||||
};
|
||||
const context: TurnDriverContext = {
|
||||
turnId: `turn-${turnNumber}`, chatSessionId: 'chat-1', sessionId: 'chat-1', backend: 'catty', signal,
|
||||
emit: event => { events.push({
|
||||
...event, sessionId: 'chat-1', chatSessionId: 'chat-1', backend: 'catty', timestamp: Date.now(),
|
||||
} as AgentEvent); }, toolOutputStore: store, toolResultDedup: dedup, sessionStateStore: state,
|
||||
};
|
||||
dedup.beginTurn();
|
||||
await new CattyTurnDriver().run(input, context);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function assertRequestPrefix(previous: WireRequest, next: WireRequest) {
|
||||
assert.deepEqual(next.messages.slice(0, previous.messages.length), previous.messages,
|
||||
'already-sent conversation must remain an exact prefix');
|
||||
assert.deepEqual(next.tools, previous.tools);
|
||||
}
|
||||
|
||||
test('Catty preserves the request prefix across six ordinary turns', async t => {
|
||||
const f = fixture(t, Array.from({ length: 6 }, () => ({ text: 'OK' })));
|
||||
for (let i = 0; i < 6; i += 1) await f.turn();
|
||||
assert.equal(f.requests.length, 6);
|
||||
for (let i = 1; i < f.requests.length; i += 1) assertRequestPrefix(f.requests[i - 1], f.requests[i]);
|
||||
});
|
||||
|
||||
test('Catty replays completed small commands without rewriting their results', async t => {
|
||||
const f = fixture(t, Array.from({ length: 6 }, () => [
|
||||
{ tool: 'terminal_execute' as const }, { text: 'OK' },
|
||||
]).flat());
|
||||
for (let i = 0; i < 6; i += 1) await f.turn(2);
|
||||
assert.equal(f.requests.length, 12);
|
||||
for (let i = 1; i < f.requests.length; i += 1) assertRequestPrefix(f.requests[i - 1], f.requests[i]);
|
||||
assert.equal(f.messages.at(-1)?.usage?.cachedInputTokens, 1800, 'usage includes both model steps');
|
||||
assert.doesNotMatch(JSON.stringify(f.requests), /cache_control/, 'OpenAI-compatible requests use implicit caching');
|
||||
});
|
||||
|
||||
test('Catty keeps large-output handles without replacing the conversation prefix', async t => {
|
||||
const f = fixture(t, [{ tool: 'sftp_read_file' }, { tool: 'sftp_read_file' }, { text: 'OK' }]);
|
||||
await f.turn(3);
|
||||
assert.equal(f.requests.length, 3);
|
||||
for (let i = 1; i < f.requests.length; i += 1) assertRequestPrefix(f.requests[i - 1], f.requests[i]);
|
||||
assert.match(JSON.stringify(f.requests.at(-1)), /tool output handle:/);
|
||||
assert.doesNotMatch(JSON.stringify(f.requests.at(-1)), /Tool output handles available/);
|
||||
});
|
||||
|
||||
test('Catty preserves large-output history across consecutive user turns', async t => {
|
||||
const f = fixture(t, Array.from({ length: 6 }, () => [
|
||||
{ tool: 'sftp_read_file' as const }, { text: 'OK' },
|
||||
]).flat());
|
||||
for (let i = 0; i < 6; i += 1) await f.turn(2);
|
||||
assert.equal(f.requests.length, 12);
|
||||
for (let i = 1; i < f.requests.length; i += 1) assertRequestPrefix(f.requests[i - 1], f.requests[i]);
|
||||
});
|
||||
|
||||
test('Catty preserves large terminal results across consecutive user turns', async t => {
|
||||
const f = fixture(t, Array.from({ length: 6 }, () => [
|
||||
{ tool: 'terminal_execute' as const }, { text: 'OK' },
|
||||
]).flat(), 'openai', Array.from({ length: 14000 }, (_, i) => `synthetic log record ${i}`).join('\n'));
|
||||
for (let i = 0; i < 6; i += 1) await f.turn(2);
|
||||
assert.equal(f.requests.length, 12);
|
||||
for (let i = 1; i < f.requests.length; i += 1) assertRequestPrefix(f.requests[i - 1], f.requests[i]);
|
||||
});
|
||||
|
||||
test('Catty keeps DeepSeek cache reads in trace and persisted assistant usage', async t => {
|
||||
const f = fixture(t, [{ text: 'OK' }]);
|
||||
await f.turn();
|
||||
const event = f.events.find(event => event.type === 'usage');
|
||||
assert.ok(event?.type === 'usage');
|
||||
assert.equal(event.cachedPromptTokens, 900);
|
||||
assert.equal(f.messages.at(-1)?.usage?.cachedInputTokens, 900);
|
||||
});
|
||||
|
||||
test('Catty marks a stable Claude system block and the conversation tail for caching', async t => {
|
||||
const f = fixture(t, [{ text: 'OK' }], 'anthropic');
|
||||
await f.turn();
|
||||
const system = f.requests[0].system!;
|
||||
assert.ok(system.length >= 2, 'static and dynamic system context need separate boundaries');
|
||||
assert.deepEqual(system[0].cache_control, { type: 'ephemeral' });
|
||||
assert.doesNotMatch(system[0].text, /## Current Scope/);
|
||||
assert.match(system[1].text, /## Current Scope/);
|
||||
assert.match(JSON.stringify(f.requests[0].messages.at(-1)), /cache_control/);
|
||||
const event = f.events.find(event => event.type === 'usage');
|
||||
assert.ok(event?.type === 'usage');
|
||||
assert.equal(event.cachedPromptTokens, 900);
|
||||
assert.equal(f.messages.at(-1)?.usage?.cachedInputTokens, 900);
|
||||
});
|
||||
|
||||
test('Catty does not send vendor cache extensions to an Anthropic-compatible gateway', async t => {
|
||||
const f = fixture(t, [{ text: 'OK' }], 'anthropic');
|
||||
f.provider.baseURL = 'https://gateway.example.test/anthropic';
|
||||
await f.turn();
|
||||
assert.equal(f.requests.length, 1);
|
||||
assert.doesNotMatch(JSON.stringify(f.requests), /cache_control/);
|
||||
assert.equal(f.messages.at(-1)?.content, 'OK');
|
||||
});
|
||||
|
||||
for (const providerId of ['openrouter', 'ollama'] as const) {
|
||||
test(`Catty respects the ${providerId} endpoint fallback with Anthropic format`, async t => {
|
||||
const f = fixture(t, [{ text: 'OK' }], 'anthropic');
|
||||
f.provider.providerId = providerId;
|
||||
f.provider.baseURL = '';
|
||||
await f.turn();
|
||||
assert.match(f.urls[0], providerId === 'openrouter' ? /^https:\/\/openrouter\.ai\// : /^http:\/\/localhost:11434\//);
|
||||
assert.doesNotMatch(JSON.stringify(f.requests), /cache_control/);
|
||||
assert.equal(f.messages.at(-1)?.content, 'OK');
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
import type { ModelMessage, SystemModelMessage } from 'ai';
|
||||
import type { ProviderConfig } from '../types';
|
||||
|
||||
const CACHE_CONTROL = { type: 'ephemeral' } as const;
|
||||
|
||||
/** Native Anthropic needs explicit breakpoints; implicit-cache protocols stay untouched. */
|
||||
export function createCattyPromptCache(
|
||||
systemPrompt: string,
|
||||
provider: string,
|
||||
endpoint?: Pick<ProviderConfig, 'baseURL'>,
|
||||
) {
|
||||
let officialEndpoint = false;
|
||||
if (endpoint) {
|
||||
try {
|
||||
// This must be the resolved SDK address, including vendor fallbacks.
|
||||
// Only an unresolved URL after that step uses the SDK's native default.
|
||||
officialEndpoint = new URL(endpoint.baseURL || 'https://api.anthropic.com').origin === 'https://api.anthropic.com';
|
||||
} catch { /* An unknown endpoint must not receive vendor extensions. */ }
|
||||
}
|
||||
const enabled = /^anthropic(?:\.|$)/.test(provider) && officialEndpoint;
|
||||
const boundary = systemPrompt.indexOf('\n\n## Current Scope');
|
||||
const instructions: string | SystemModelMessage[] = enabled
|
||||
? [{
|
||||
role: 'system',
|
||||
content: boundary < 0 ? systemPrompt : systemPrompt.slice(0, boundary),
|
||||
providerOptions: { anthropic: { cacheControl: CACHE_CONTROL } },
|
||||
}, ...(boundary < 0 ? [] : [{
|
||||
role: 'system' as const,
|
||||
content: systemPrompt.slice(boundary),
|
||||
}])]
|
||||
: systemPrompt;
|
||||
|
||||
return {
|
||||
enabled,
|
||||
instructions,
|
||||
prepareMessages(messages: ModelMessage[]): ModelMessage[] {
|
||||
if (!enabled) return messages;
|
||||
// Keep at most two moving conversation breakpoints, plus the static
|
||||
// system boundary. Remove our prior message-level markers as v7 carries
|
||||
// prepared messages forward; never alter signed reasoning content.
|
||||
const targets = new Set<number>();
|
||||
for (let index = messages.length - 1; index >= 0 && targets.size < 2; index -= 1) {
|
||||
if (messages[index].role === 'user' || messages[index].role === 'tool') targets.add(index);
|
||||
}
|
||||
return messages.map((message, index) => {
|
||||
const existing = message.providerOptions?.anthropic;
|
||||
if (!targets.has(index) && !existing?.cacheControl && !existing?.cache_control) return message;
|
||||
const anthropic = { ...existing };
|
||||
delete anthropic.cacheControl;
|
||||
delete anthropic.cache_control;
|
||||
if (targets.has(index)) anthropic.cacheControl = CACHE_CONTROL;
|
||||
return { ...message, providerOptions: { ...message.providerOptions, anthropic } };
|
||||
});
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import type { ModelMessage } from 'ai';
|
||||
import { createCattyPromptCache } from './cattyPromptCache';
|
||||
|
||||
test('leaves implicit-cache protocols unchanged', () => {
|
||||
for (const provider of ['openai.chat', 'openai.responses', 'google.generative-ai']) {
|
||||
const cache = createCattyPromptCache('instructions', provider);
|
||||
const messages: ModelMessage[] = [{ role: 'user', content: 'hello' }];
|
||||
assert.equal(cache.instructions, 'instructions');
|
||||
assert.equal(cache.prepareMessages(messages), messages);
|
||||
}
|
||||
});
|
||||
|
||||
test('Claude markers stay bounded across tool steps without changing signed reasoning', () => {
|
||||
const cache = createCattyPromptCache('static\n\n## Current Scope\nvolatile', 'anthropic.messages', { baseURL: 'https://api.anthropic.com' });
|
||||
const reasoning = { type: 'reasoning' as const, text: 'thinking',
|
||||
providerOptions: { anthropic: { signature: 'signed-value' } } };
|
||||
let messages: ModelMessage[] = [{ role: 'user', content: 'hello' }];
|
||||
for (let step = 0; step < 30; step += 1) {
|
||||
messages = cache.prepareMessages(messages);
|
||||
const marked = messages.filter(message => message.providerOptions?.anthropic?.cacheControl);
|
||||
assert.ok(marked.length <= 2);
|
||||
messages.push({ role: 'assistant', content: [reasoning, { type: 'text', text: 'result' }] });
|
||||
messages.push({ role: 'user', content: `next-${step}` });
|
||||
}
|
||||
for (const message of messages) {
|
||||
if (message.role === 'assistant') assert.equal(message.content[0], reasoning);
|
||||
}
|
||||
assert.ok(Array.isArray(cache.instructions));
|
||||
assert.deepEqual(cache.instructions[0].providerOptions?.anthropic?.cacheControl, { type: 'ephemeral' });
|
||||
});
|
||||
|
||||
test('only the official HTTPS Anthropic endpoint enables vendor cache markers', () => {
|
||||
for (const baseURL of ['https://proxy.example.test/v1', 'http://api.anthropic.com', 'https://api.anthropic.com.example.test', 'invalid']) {
|
||||
const cache = createCattyPromptCache('instructions', 'anthropic.messages', { baseURL });
|
||||
assert.equal(cache.enabled, false);
|
||||
}
|
||||
assert.equal(createCattyPromptCache('instructions', 'anthropic.messages').enabled, false);
|
||||
assert.equal(createCattyPromptCache('instructions', 'anthropic.messages', {}).enabled, true);
|
||||
assert.equal(createCattyPromptCache('instructions', 'anthropic.messages', { baseURL: 'https://api.anthropic.com/v1' }).enabled, true);
|
||||
});
|
||||
@@ -78,7 +78,7 @@ test('prepareTurnContext applies typed compression before LLM summarize threshol
|
||||
assert.ok(prepared.messages.length >= messages.length);
|
||||
const serialized = JSON.stringify(prepared.messages);
|
||||
assert.match(serialized, /Fix only the upstream timeout issue/);
|
||||
assert.match(serialized, /Permission mode: confirm/);
|
||||
assert.doesNotMatch(serialized, /Netcatty session context/);
|
||||
assert.ok(serialized.length < JSON.stringify(messages).length);
|
||||
});
|
||||
|
||||
@@ -163,7 +163,7 @@ test('TraceStore bounds compaction history independently from events', () => {
|
||||
assert.equal(store.getCompactions('chat-bounded-compactions').length, 3);
|
||||
});
|
||||
|
||||
test('prepareStepContext replaces prior step handle notices under v7 carry-forward semantics', async () => {
|
||||
test('prepareStepContext preserves prior notices and appends only newly available handles', async () => {
|
||||
const store = new ToolOutputStore();
|
||||
store.store({
|
||||
chatSessionId: 'chat-4',
|
||||
@@ -194,9 +194,22 @@ test('prepareStepContext replaces prior step handle notices under v7 carry-forwa
|
||||
&& typeof message.content === 'string'
|
||||
&& message.content.includes('Tool output handles available'),
|
||||
);
|
||||
assert.equal(notices.length, 1);
|
||||
assert.match(String(notices[0]?.content), /\[step 2\]/);
|
||||
assert.doesNotMatch(String(notices[0]?.content), /tool-output-old/);
|
||||
assert.equal(notices.length, 2);
|
||||
assert.deepEqual(prepared.messages.slice(0, 2), [priorNotice, { role: 'user', content: 'continue' }]);
|
||||
assert.equal(notices[0], priorNotice);
|
||||
assert.match(String(notices[1]?.content), /^\[Tool output handles available:/);
|
||||
assert.doesNotMatch(String(notices[1]?.content), /tool-output-old|\[step/);
|
||||
assert.equal(prepared.messages.at(-1), notices[1]);
|
||||
|
||||
const next = await prepareStepContext({
|
||||
messages: prepared.messages,
|
||||
stepNumber: 3,
|
||||
sessionId: 'chat-4',
|
||||
chatSessionId: 'chat-4',
|
||||
toolOutputStore: store,
|
||||
runtimeContext,
|
||||
});
|
||||
assert.deepEqual(next.messages, prepared.messages, 'unchanged handles need no per-step rewrite');
|
||||
});
|
||||
|
||||
test('prepareStepContext emits step compaction trace when over budget', async () => {
|
||||
@@ -266,7 +279,8 @@ test('prepareStepContext retains handle notice after step budget guard', async (
|
||||
&& message.content.includes('Tool output handles available'),
|
||||
);
|
||||
assert.equal(notices.length, 1);
|
||||
assert.match(String(notices[0]?.content), /\[step 2\]/);
|
||||
assert.match(String(notices[0]?.content), /^\[Tool output handles available:/);
|
||||
assert.equal(prepared.messages.at(-1), notices[0]);
|
||||
});
|
||||
|
||||
test('prepareStepContext never leaves orphan results from a parallel tool batch', async () => {
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
import type { ModelMessage } from 'ai';
|
||||
import type { ChatMessage, AIPermissionMode } from '../types';
|
||||
import { isStepHandleNoticeMessage } from './agentEventAdapter';
|
||||
import {
|
||||
DEFAULT_CONTEXT_WINDOW_TOKENS,
|
||||
DEFAULT_PROTECT_RECENT_MESSAGES,
|
||||
@@ -265,7 +264,9 @@ export async function prepareTurnContext(
|
||||
}
|
||||
|
||||
const reinjection = buildReinjectionMessages(input.reinjection);
|
||||
if (reinjection.length > 0 && didAdjust) {
|
||||
// Routine output cleanup does not discard the conversation or its goals.
|
||||
// Reinserting the latest goal at the front would invalidate every turn.
|
||||
if (reinjection.length > 0 && didAdjust && (underBudgetPressure || didLlmSummarize)) {
|
||||
const reinjectionTokens = estimateModelMessagesTokensWithKind({
|
||||
messages: reinjection,
|
||||
providerId: input.providerId,
|
||||
@@ -414,22 +415,6 @@ export async function prepareStepContext(
|
||||
const typed = compressMessagesForRequestTooLargeRetry(working);
|
||||
working = typed.messages;
|
||||
|
||||
const pendingHandles = input.toolOutputStore?.listPendingHandles(input.chatSessionId ?? input.sessionId) ?? [];
|
||||
let didHandleNotice = false;
|
||||
if (pendingHandles.length > 0 && input.stepNumber > 0) {
|
||||
working = working.filter((message) => {
|
||||
if (message.role !== 'user') return true;
|
||||
const content = typeof message.content === 'string' ? message.content : '';
|
||||
return !isStepHandleNoticeMessage(content);
|
||||
});
|
||||
const notice: ModelMessage = {
|
||||
role: 'user',
|
||||
content: `[step ${input.stepNumber}] Tool output handles available: ${pendingHandles.map(h => h.id).join(', ')}`,
|
||||
};
|
||||
working = [notice, ...working];
|
||||
didHandleNotice = true;
|
||||
}
|
||||
|
||||
const budgetGuard = applyStepBudgetGuard(working, {
|
||||
contextWindow,
|
||||
reservedTokens: input.reservedTokens ?? 0,
|
||||
@@ -438,36 +423,42 @@ export async function prepareStepContext(
|
||||
protectRecentMessages: protectRecent,
|
||||
});
|
||||
working = budgetGuard.messages;
|
||||
if (didHandleNotice && pendingHandles.length > 0) {
|
||||
const hasHandleNotice = working.some(
|
||||
(message) => message.role === 'user' && isStepHandleNoticeMessage(message.content),
|
||||
);
|
||||
if (!hasHandleNotice) {
|
||||
working = [{
|
||||
role: 'user',
|
||||
content: `[step ${input.stepNumber}] Tool output handles available: ${pendingHandles.map(h => h.id).join(', ')}`,
|
||||
}, ...working];
|
||||
}
|
||||
}
|
||||
if (didHandleNotice) {
|
||||
const notices = working.filter(
|
||||
message => message.role === 'user' && isStepHandleNoticeMessage(message.content),
|
||||
);
|
||||
const body = working.filter(
|
||||
message => !(message.role === 'user' && isStepHandleNoticeMessage(message.content)),
|
||||
);
|
||||
const pendingHandles = input.stepNumber > 0
|
||||
? input.toolOutputStore?.listPendingHandles(input.chatSessionId ?? input.sessionId) ?? []
|
||||
: [];
|
||||
const unannouncedHandles = () => {
|
||||
// Bounded tool results already carry their read handles. Announce only
|
||||
// handles whose result was pruned, so normal history needs no transient
|
||||
// user message that would disappear when the next turn is rebuilt.
|
||||
const visibleContent = working.map(message => typeof message.content === 'string'
|
||||
? message.content : JSON.stringify(message.content));
|
||||
return pendingHandles.filter(handle => !visibleContent.some(content => content.includes(handle.id)));
|
||||
};
|
||||
let didHandleNotice = false;
|
||||
if (unannouncedHandles().length > 0) {
|
||||
// Reserve the complete list: the guard may remove an older announcement.
|
||||
// New notices belong at the tail; replacing a step-numbered prefix would
|
||||
// invalidate the entire cached conversation on every model call.
|
||||
const noticeTokens = estimateModelMessagesTokensWithKind({
|
||||
messages: notices,
|
||||
messages: [{ role: 'user', content: `[Tool output handles available: ${pendingHandles.map(h => h.id).join(', ')}]` }],
|
||||
providerId: input.providerId,
|
||||
}).tokens;
|
||||
const finalGuard = applyStepBudgetGuard(body, {
|
||||
const finalGuard = applyStepBudgetGuard(working, {
|
||||
contextWindow,
|
||||
reservedTokens: (input.reservedTokens ?? 0) + noticeTokens,
|
||||
maxOutputTokens,
|
||||
providerId: input.providerId,
|
||||
protectRecentMessages: protectRecent,
|
||||
});
|
||||
working = [...notices, ...finalGuard.messages];
|
||||
working = finalGuard.messages;
|
||||
const handles = unannouncedHandles();
|
||||
if (handles.length > 0) {
|
||||
working = [...working, {
|
||||
role: 'user',
|
||||
content: `[Tool output handles available: ${handles.map(h => h.id).join(', ')}]`,
|
||||
}];
|
||||
didHandleNotice = true;
|
||||
}
|
||||
budgetGuard.didAdjust = budgetGuard.didAdjust || finalGuard.didAdjust;
|
||||
}
|
||||
const didBudgetAdjust = integrity.didAdjust || stale.didAdjust || typed.didAdjust || budgetGuard.didAdjust;
|
||||
|
||||
@@ -27,7 +27,7 @@ import {
|
||||
type AssistantContentPart,
|
||||
type CattyProviderContinuationContext,
|
||||
} from '../../aiChatStreamingSupport';
|
||||
import { redactSecretsInValueForModel } from '../modelSecretRedaction';
|
||||
import { redactSecretsForModel, redactSecretsInValueForModel } from '../modelSecretRedaction';
|
||||
import { fitLargeUserInputForModel } from '../largeUserInput';
|
||||
import type { ToolOutputStore } from '../toolOutputStore';
|
||||
|
||||
@@ -198,7 +198,6 @@ export interface BuildCattySdkMessagesInput {
|
||||
trimmed: string;
|
||||
attachments?: ChatMessageAttachment[];
|
||||
continuationContext: CattyProviderContinuationContext;
|
||||
preserveTerminalToolResults?: ReadonlySet<ToolResult>;
|
||||
chatSessionId: string;
|
||||
toolOutputStore: ToolOutputStore;
|
||||
fieldsByMessage: Map<ModelMessage, OpenAIChatAssistantFields | undefined>;
|
||||
@@ -212,7 +211,6 @@ export function buildCattySdkMessages(input: BuildCattySdkMessagesInput): ModelM
|
||||
trimmed,
|
||||
attachments,
|
||||
continuationContext,
|
||||
preserveTerminalToolResults = new Set<ToolResult>(),
|
||||
chatSessionId,
|
||||
toolOutputStore,
|
||||
fieldsByMessage,
|
||||
@@ -394,9 +392,12 @@ export function buildCattySdkMessages(input: BuildCattySdkMessagesInput): ModelM
|
||||
toolName: toolCall?.name ?? 'unknown',
|
||||
output: {
|
||||
type: 'text' as const,
|
||||
value: buildHistoricalToolResultReplayText(tr, toolCall, {
|
||||
preserveTerminalOutput: preserveTerminalToolResults.has(tr),
|
||||
}),
|
||||
// Live results are already bounded. Reformatting them merely
|
||||
// because another user turn arrived breaks the cached prefix.
|
||||
// Context-budget pruning still owns necessary reductions.
|
||||
value: redactSecretsForModel(buildHistoricalToolResultReplayText(tr, toolCall, {
|
||||
preserveTerminalOutput: true,
|
||||
})),
|
||||
},
|
||||
};
|
||||
}),
|
||||
|
||||
@@ -8,13 +8,14 @@ import {
|
||||
} from '../../cattyRequestTooLargeRetry';
|
||||
import { mapCattyStreamChunkToAgentEvents } from '../agentEventAdapter';
|
||||
import type { AgentEvent } from '../types';
|
||||
import type { ProviderAdvancedParams } from '../../types';
|
||||
import { resolveProviderStyle, type ProviderAdvancedParams } from '../../types';
|
||||
import type { CattyReasoningProviderOptions } from '../../cattyReasoning';
|
||||
import { createModelFromConfig } from '../../sdk/providers';
|
||||
import { createModelFromConfig, resolveProviderEndpoint } from '../../sdk/providers';
|
||||
import type { CattyToolsBundle } from '../capabilityTools';
|
||||
import { buildCattyToolApproval } from '../cattyToolApproval';
|
||||
import type { CattyRuntimeContext } from '../cattyRuntimeContext';
|
||||
import { buildCattyStreamTimeouts } from '../streamTimeouts';
|
||||
import { createCattyPromptCache } from '../cattyPromptCache';
|
||||
import {
|
||||
extractProviderContinuationFromRawChunk,
|
||||
mergeProviderContinuation,
|
||||
@@ -39,7 +40,7 @@ import {
|
||||
type ToolOutputDeniedChunk,
|
||||
type ToolResultChunk,
|
||||
} from '../../aiChatStreamingSupport';
|
||||
import type { ChatMessage } from '../../types';
|
||||
import type { ChatMessage, ProviderConfig } from '../../types';
|
||||
|
||||
export type CattyModel = ReturnType<typeof createModelFromConfig>;
|
||||
|
||||
@@ -51,6 +52,7 @@ export interface CattyStreamUiSink {
|
||||
export interface ProcessCattyStreamInput {
|
||||
streamSessionId: string;
|
||||
model: CattyModel;
|
||||
providerConfig?: ProviderConfig;
|
||||
systemPrompt: string;
|
||||
toolsBundle: CattyToolsBundle;
|
||||
sdkMessages: ModelMessage[];
|
||||
@@ -76,6 +78,7 @@ export interface ProcessCattyStreamInput {
|
||||
export interface ProcessCattyStreamResult {
|
||||
usage?: {
|
||||
promptTokens?: number;
|
||||
cachedPromptTokens?: number;
|
||||
completionTokens?: number;
|
||||
totalTokens?: number;
|
||||
};
|
||||
@@ -129,11 +132,15 @@ export async function processCattyStream(input: ProcessCattyStreamInput): Promis
|
||||
|
||||
let runtimeContext = initialRuntimeContext;
|
||||
const { tools, toolsContext } = toolsBundle;
|
||||
const endpoint = input.providerConfig
|
||||
? resolveProviderEndpoint(input.providerConfig, resolveProviderStyle(input.providerConfig), undefined)
|
||||
: undefined;
|
||||
const promptCache = createCattyPromptCache(systemPrompt, model.provider, endpoint);
|
||||
|
||||
const result = streamText({
|
||||
model,
|
||||
messages: sdkMessages,
|
||||
instructions: systemPrompt,
|
||||
messages: promptCache.prepareMessages(sdkMessages),
|
||||
instructions: promptCache.instructions,
|
||||
tools,
|
||||
toolsContext,
|
||||
runtimeContext,
|
||||
@@ -186,6 +193,7 @@ export async function processCattyStream(input: ProcessCattyStreamInput): Promis
|
||||
modelId: step.model.modelId,
|
||||
finishReason: step.finishReason,
|
||||
promptTokens: usage.inputTokens ?? 0,
|
||||
cachedPromptTokens: usage.inputTokenDetails?.cacheReadTokens,
|
||||
completionTokens: usage.outputTokens ?? 0,
|
||||
totalTokens: (usage.inputTokens ?? 0) + (usage.outputTokens ?? 0),
|
||||
} as AgentEvent);
|
||||
@@ -201,15 +209,17 @@ export async function processCattyStream(input: ProcessCattyStreamInput): Promis
|
||||
timestamp: Date.now(),
|
||||
turnId,
|
||||
promptTokens: usage.inputTokens ?? 0,
|
||||
cachedPromptTokens: usage.inputTokenDetails?.cacheReadTokens,
|
||||
completionTokens: usage.outputTokens ?? 0,
|
||||
reasoningTokens: usage.outputTokenDetails?.reasoningTokens,
|
||||
totalTokens: (usage.inputTokens ?? 0) + (usage.outputTokens ?? 0),
|
||||
estimated: false,
|
||||
} as AgentEvent);
|
||||
}
|
||||
},
|
||||
...(prepareStep ? {
|
||||
...(prepareStep || promptCache.enabled ? {
|
||||
prepareStep: async ({ stepNumber, messages, runtimeContext: stepRuntimeContext }) => {
|
||||
const prepared = await prepareStep({
|
||||
const prepared = await prepareStep?.({
|
||||
stepNumber,
|
||||
messages,
|
||||
runtimeContext: stepRuntimeContext as CattyRuntimeContext,
|
||||
@@ -217,7 +227,7 @@ export async function processCattyStream(input: ProcessCattyStreamInput): Promis
|
||||
if (prepared?.runtimeContext) {
|
||||
runtimeContext = prepared.runtimeContext;
|
||||
}
|
||||
return prepared ?? { messages };
|
||||
return { ...prepared, messages: promptCache.prepareMessages(prepared?.messages ?? messages) };
|
||||
},
|
||||
} : {}),
|
||||
...(advancedParams?.maxTokens != null && { maxOutputTokens: advancedParams.maxTokens }),
|
||||
@@ -552,6 +562,21 @@ export async function processCattyStream(input: ProcessCattyStreamInput): Promis
|
||||
const usage = await result.usage;
|
||||
const finalStep = await result.finalStep;
|
||||
const performance = finalStep?.performance;
|
||||
// onEnd can run before this consumer flushes its final text and creates the
|
||||
// post-tool assistant message. Attach turn totals only after that flush.
|
||||
if (usage) {
|
||||
ui.updateMessageById(streamSessionId, activeMsgId, message => ({
|
||||
...message,
|
||||
usage: {
|
||||
inputTokens: usage.inputTokens ?? 0,
|
||||
cachedInputTokens: usage.inputTokenDetails?.cacheReadTokens,
|
||||
outputTokens: usage.outputTokens ?? 0,
|
||||
reasoningTokens: usage.outputTokenDetails?.reasoningTokens,
|
||||
totalTokens: (usage.inputTokens ?? 0) + (usage.outputTokens ?? 0),
|
||||
estimated: false,
|
||||
},
|
||||
}));
|
||||
}
|
||||
|
||||
if (performance) {
|
||||
onAgentEvent?.({
|
||||
@@ -571,6 +596,7 @@ export async function processCattyStream(input: ProcessCattyStreamInput): Promis
|
||||
return {
|
||||
usage: usage ? {
|
||||
promptTokens: usage.inputTokens,
|
||||
cachedPromptTokens: usage.inputTokenDetails?.cacheReadTokens,
|
||||
completionTokens: usage.outputTokens,
|
||||
totalTokens: (usage.inputTokens ?? 0) + (usage.outputTokens ?? 0),
|
||||
} : undefined,
|
||||
|
||||
@@ -37,7 +37,6 @@ import {
|
||||
buildCattySdkMessages,
|
||||
collectOpenAIChatAssistantFieldsForMessages,
|
||||
collectPreservedTerminalWriteFingerprints,
|
||||
collectToolResultsAfterMessage,
|
||||
createContinuationContext,
|
||||
} from './cattyMessageBuilder';
|
||||
import { hadToolProgressBeforeRequestTooLarge, processCattyStream } from './cattyStreamProcessor';
|
||||
@@ -218,7 +217,6 @@ async function runCattyTurn(input: CattyTurnInput, ctx: TurnDriverContext): Prom
|
||||
const buildSdkMessages = (
|
||||
allMessages: import('../../types').ChatMessage[],
|
||||
includeCurrentUserMessage: boolean,
|
||||
options: { preserveTerminalToolResults?: ReadonlySet<import('../../types').ToolResult> } = {},
|
||||
sessionContextCompaction = currentSession?.contextCompaction,
|
||||
) => buildCattySdkMessages({
|
||||
allMessages,
|
||||
@@ -227,7 +225,6 @@ async function runCattyTurn(input: CattyTurnInput, ctx: TurnDriverContext): Prom
|
||||
trimmed: modelUserText,
|
||||
attachments: includeCurrentUserMessage ? attachments : undefined,
|
||||
continuationContext,
|
||||
preserveTerminalToolResults: options.preserveTerminalToolResults,
|
||||
chatSessionId: sessionId,
|
||||
toolOutputStore: ctx.toolOutputStore,
|
||||
fieldsByMessage: openAIChatAssistantFieldsByMessage,
|
||||
@@ -399,7 +396,6 @@ async function runCattyTurn(input: CattyTurnInput, ctx: TurnDriverContext): Prom
|
||||
const headSdkMessages = buildSdkMessages(
|
||||
uiMessages.slice(0, compactedMessageCount),
|
||||
false,
|
||||
{},
|
||||
undefined,
|
||||
);
|
||||
const compacted = await compactMessages(headSdkMessages, {
|
||||
@@ -414,7 +410,7 @@ async function runCattyTurn(input: CattyTurnInput, ctx: TurnDriverContext): Prom
|
||||
};
|
||||
ui.persistContextCompaction?.(sessionId, nextCompaction);
|
||||
emitContextSnapshot(prepareMessagesForStream(
|
||||
buildSdkMessages(uiMessages, false, {}, nextCompaction),
|
||||
buildSdkMessages(uiMessages, false, nextCompaction),
|
||||
));
|
||||
} else {
|
||||
// Non-durable outcome: keep the meter honest (full current context).
|
||||
@@ -452,6 +448,7 @@ async function runCattyTurn(input: CattyTurnInput, ctx: TurnDriverContext): Prom
|
||||
await processCattyStream({
|
||||
streamSessionId: sessionId,
|
||||
model,
|
||||
providerConfig: context.activeProvider,
|
||||
systemPrompt,
|
||||
toolsBundle,
|
||||
sdkMessages: streamMessages,
|
||||
@@ -514,12 +511,7 @@ async function runCattyTurn(input: CattyTurnInput, ctx: TurnDriverContext): Prom
|
||||
assistantMsgId,
|
||||
sessionId,
|
||||
);
|
||||
retryBaseMessages = buildSdkMessages(latestSession.messages, false, {
|
||||
preserveTerminalToolResults: collectToolResultsAfterMessage(
|
||||
latestSession.messages,
|
||||
assistantMsgId,
|
||||
),
|
||||
}, latestSession.contextCompaction);
|
||||
retryBaseMessages = buildSdkMessages(latestSession.messages, false, latestSession.contextCompaction);
|
||||
}
|
||||
retryAssistantMsgId = generateId();
|
||||
ui.addMessageToSession(sessionId, {
|
||||
|
||||
@@ -180,6 +180,7 @@ export interface StepEndEvent extends AgentEventBase {
|
||||
modelId?: string;
|
||||
finishReason?: string;
|
||||
promptTokens: number;
|
||||
cachedPromptTokens?: number;
|
||||
completionTokens: number;
|
||||
totalTokens: number;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import { normalizeOpenAIChatCacheUsage } from './openAIChatCacheUsage';
|
||||
|
||||
test('adapts documented DeepSeek cache usage while preserving the rest of the response', () => {
|
||||
const response = { choices: [], usage: {
|
||||
prompt_tokens: 1000, prompt_cache_hit_tokens: 900, prompt_cache_miss_tokens: 100,
|
||||
prompt_tokens_details: { other: 1 }, completion_tokens: 5,
|
||||
} };
|
||||
const parsed = JSON.parse(normalizeOpenAIChatCacheUsage(JSON.stringify(response)));
|
||||
assert.deepEqual(parsed, {
|
||||
...response, usage: { ...response.usage, prompt_tokens_details: { other: 1, cached_tokens: 900 } },
|
||||
});
|
||||
});
|
||||
|
||||
test('preserves standard usage and rejects invalid cache counts without breaking stream data', () => {
|
||||
for (const hit of [-1, 1.5, '900', 1001, null]) {
|
||||
const data = JSON.stringify({ usage: { prompt_tokens: 1000, prompt_cache_hit_tokens: hit } });
|
||||
assert.equal(normalizeOpenAIChatCacheUsage(data), data);
|
||||
}
|
||||
for (const data of ['[DONE]', 'not JSON', '{"usage":null}', '{"type":"message_start"}']) {
|
||||
assert.equal(normalizeOpenAIChatCacheUsage(data), data);
|
||||
}
|
||||
const data = JSON.stringify({ usage: {
|
||||
prompt_tokens: 1000, prompt_cache_hit_tokens: 900, prompt_tokens_details: { cached_tokens: 0 },
|
||||
} });
|
||||
assert.equal(normalizeOpenAIChatCacheUsage(data), data);
|
||||
});
|
||||
@@ -0,0 +1,17 @@
|
||||
/** Adapt DeepSeek's documented usage fields before the OpenAI SDK validates them. */
|
||||
export function normalizeOpenAIChatCacheUsage(data: string): string {
|
||||
if (!data.includes('prompt_cache_hit_tokens')) return data;
|
||||
try {
|
||||
const body = JSON.parse(data);
|
||||
const usage = body?.usage;
|
||||
const hit = usage?.prompt_cache_hit_tokens;
|
||||
if (!Number.isSafeInteger(hit) || hit < 0) return data;
|
||||
if (Number.isFinite(usage.prompt_tokens) && hit > usage.prompt_tokens) return data;
|
||||
// Preserve a provider's standard field, including an explicit zero.
|
||||
if (usage.prompt_tokens_details?.cached_tokens != null) return data;
|
||||
usage.prompt_tokens_details = { ...usage.prompt_tokens_details, cached_tokens: hit };
|
||||
return JSON.stringify(body);
|
||||
} catch {
|
||||
return data;
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import { isStepCount, streamText, tool } from 'ai';
|
||||
import { generateText, isStepCount, streamText, tool } from 'ai';
|
||||
import { z } from 'zod';
|
||||
import { createBridgeFetchForSDK, createModelFromConfig } from './sdk/providers';
|
||||
import type { OpenAIChatAssistantFields } from './providerContinuation';
|
||||
@@ -1305,3 +1305,25 @@ for (const indexes of [[0], [1], [1, 4], [4, 1]]) {
|
||||
indexes.map(index => `call_${index}`).sort());
|
||||
});
|
||||
}
|
||||
|
||||
test('non-streaming DeepSeek usage reaches the SDK as cached input', async t => {
|
||||
const originalWindow = globalThis.window;
|
||||
t.after(() => { globalThis.window = originalWindow; });
|
||||
globalThis.window = { netcatty: {
|
||||
aiFetch: async () => ({ ok: true, status: 200, data: JSON.stringify({
|
||||
id: 'response', object: 'chat.completion', created: 0, model: 'deepseek-chat',
|
||||
choices: [{ index: 0, message: { role: 'assistant', content: 'OK' }, finish_reason: 'stop' }],
|
||||
usage: { prompt_tokens: 1000, completion_tokens: 1, total_tokens: 1001,
|
||||
prompt_cache_hit_tokens: 900, prompt_cache_miss_tokens: 100 },
|
||||
}) }),
|
||||
} } as unknown as Window & typeof globalThis;
|
||||
const result = await generateText({
|
||||
model: createModelFromConfig({
|
||||
id: 'fixture', providerId: 'custom', name: 'Fixture', enabled: true,
|
||||
baseURL: 'https://provider.example.test/v1', apiKey: 'fixture', defaultModel: 'deepseek-chat',
|
||||
}),
|
||||
prompt: 'hello',
|
||||
});
|
||||
assert.equal(result.usage.inputTokenDetails.cacheReadTokens, 900);
|
||||
assert.equal(result.usage.inputTokenDetails.noCacheTokens, 100);
|
||||
});
|
||||
@@ -158,13 +158,19 @@ function compressContentPartForRequestRetry(part: unknown): unknown {
|
||||
const output = record.output;
|
||||
if (output && typeof output === "object") {
|
||||
const outputRecord = output as Record<string, unknown>;
|
||||
if (outputRecord.type === "text" && typeof outputRecord.value === "string") {
|
||||
const value = compressAndTruncateText(outputRecord.value, RETRY_MAX_TOOL_RESULT_CHARS);
|
||||
if (value === outputRecord.value) return part;
|
||||
const text = outputRecord.type === "text" && typeof outputRecord.value === "string"
|
||||
? outputRecord.value
|
||||
: outputRecord.type === "json" ? JSON.stringify(outputRecord.value) : undefined;
|
||||
if (text !== undefined) {
|
||||
// Live SDK tool results are JSON, while persisted results replay as
|
||||
// text. Apply the same reduction before their first model request.
|
||||
const value = compressAndTruncateText(text, RETRY_MAX_TOOL_RESULT_CHARS);
|
||||
if (value === text) return part;
|
||||
return {
|
||||
...record,
|
||||
output: {
|
||||
...outputRecord,
|
||||
type: "text",
|
||||
value,
|
||||
},
|
||||
};
|
||||
|
||||
@@ -6,6 +6,7 @@ import { resolveOpenAIApi, resolveProviderStyle } from '../types';
|
||||
import { normalizeAnthropicSdkBaseURL } from '../anthropicCompatBaseUrl';
|
||||
import { buildSdkRequestHeaders } from '../providerRequestHeaders';
|
||||
import { normalizeOllamaSdkBaseURL } from '../ollamaCompatBaseUrl';
|
||||
import { normalizeOpenAIChatCacheUsage } from '../openAIChatCacheUsage';
|
||||
|
||||
export { normalizeOllamaSdkBaseURL };
|
||||
import {
|
||||
@@ -529,7 +530,7 @@ export function createBridgeFetchForSDK(
|
||||
};
|
||||
|
||||
const unsubData = bridge.onAiStreamData(requestId, (data: string) => {
|
||||
const normalizedData = normalizeOpenAIChatToolCalls(data);
|
||||
const normalizedData = normalizeOpenAIChatCacheUsage(normalizeOpenAIChatToolCalls(data));
|
||||
captureOpenAIChatFields(normalizedData);
|
||||
// Re-wrap as SSE so the SDK can parse it
|
||||
enqueueChunk(encoder.encode(`data: ${normalizedData}\n\n`));
|
||||
@@ -627,7 +628,7 @@ export function createBridgeFetchForSDK(
|
||||
// Non-streaming path
|
||||
const result = await bridge.aiFetch(url, method, headers, requestBody, providerId);
|
||||
|
||||
return new Response(result.data, {
|
||||
return new Response(normalizeOpenAIChatCacheUsage(result.data), {
|
||||
status: result.status,
|
||||
statusText: result.ok ? 'OK' : 'Error',
|
||||
headers: { 'content-type': 'application/json' },
|
||||
|
||||
Reference in new issue
Block a user