Files
Netcatty/docs/plugin-platform/assistant-native-qa.md
T

99 lines
5.4 KiB
Markdown

# Test the selected Note assistant on an isolated desktop
This recipe tests a sandbox plugin against a deterministic loopback provider.
The provider does not run a model or contact a paid service. The profile contains
only synthetic data. The parent owns native screenshots and video on the working
cloud GUI machine. The local executor's sandbox-helper failure remains a separate
failed launch record.
## Prepare the integrated head
Use a head containing the market source API, #3985's malformed-context repair,
#3986, and the transport task's canonical AI/context registration. Record the exact
commit and artifact hashes. Keep all PRs Draft. Do not merge or release them.
Run the build and tests with the repository's supported Node version:
```sh
npm run build:plugin-runtime-deps
node examples/plugins/assistant/build.mjs
node packages/plugin-cli/dist/cli.js pack examples/plugins/assistant/.package --out /tmp/assistant-example.ncpkg
node packages/plugin-cli/dist/cli.js validate /tmp/assistant-example.ncpkg
sha256sum /tmp/assistant-example.ncpkg
```
The package contains six files: its manifest, README, three panel files, and one
browser runtime entry. It has no Node entry, networking, filesystem, secrets,
storage, terminal, or command-execution permission. Its `commands` permission
registers its own view callbacks. It cannot execute terminal commands.
## Start the deterministic fixture
Start `node scripts/plugin-assistant-provider-fixture.mjs` from the repository.
The server prints its loopback URL. It listens only on `127.0.0.1`; use a fresh
unused port. Its response is fixed synthetic text, split into delayed SSE chunks.
The response includes fixed usage metadata. It accepts no request tools or system
messages. It keeps only request counts and prints no request content or headers.
Launch the exact integrated app on the working desktop with a fresh disposable
user-data directory. Preserve Chromium sandboxing. Do not use a real app profile,
cloud account, credential, host, Note, or chat history.
In AI Settings, create a custom OpenAI-compatible provider with the fixture's
`/v1` URL, model `deterministic-fixture`, and the literal synthetic credential marker
`fixture-only`. This marker exercises the existing credential injection path
without a real API credential. Select that
provider and model through the host UI. Do not enable an external Agent or tools.
Create one Note named `Synthetic retry plan` containing:
```text
Synthetic retry plan: add a retry button and show a connection status label.
```
## Record the installed-plugin path
1. Import the package. Record its ID `com.netcatty.assistant`, version `0.1.0`,
source, SHA-256, browser runtime, required permissions, and optional permissions.
Installation must not grant optional access. Source labels do not imply that
an author or package is officially maintained.
2. Enable the plugin and open its `Selected Note Assistant` modal from the plugin
view catalog. Allow only its required view and command registration permissions.
3. Grant `ai.invoke` and `context.selected.list` when the host prompts. Record the
host-selected model label. The host's Note chooser must show app-owned chrome.
Select only `Synthetic retry plan` and click **Share selected summaries**.
4. Verify that the plugin reference dropdown starts at **No reference**. Select
`Synthetic retry plan` explicitly, then click **Send**.
5. Deny `context.selected.read`. No fixture request may be sent. The optional
model permission must not authorize Note access.
6. Retry. Allow the single selected-context read. The separate host text dialog
must show the exact synthetic text, scope, and version, with **Cancel** selected
by default. Cancel once and confirm that no fixture request is sent.
7. Retry and click **Share this text**. Record streamed text, completion, and token
usage. Capture the full host window, not only the plugin iframe.
8. Click **Send** again, share the same synthetic text, and click **Stop** while
the delayed fixture is streaming. The text must stop changing; the fixture's
aborted-connection counter must increase. Record this sequence as video.
9. Change the Note during a pending review and attempt to share the old dialog.
The expected-version check must reject the result. Do not send stale content.
10. Revoke the plugin's AI/context permission while a turn is active. Disable,
re-enable, and close its view. Old handles and late results must not resume.
Lock/unlock the app and navigate or reload the main window in the isolated
profile. Record a fresh successful run after reauthorization.
11. Keep the fixture request counter and aborted-connection counter in the
evidence. Do not include prompt bodies, headers, credentials, or Note content
other than the synthetic text above in logs.
The automated suites cover source substitution, runtime identity, two-handle
revocation after malformed input, late replies, byte/token/time bounds, and
no-key serialization. Native execution must still verify the real prompts,
installed runtime, configured provider transport, and UI lifecycle.
## Record remaining limits
Independent review found the malformed-context revocation bug. The regression
reproduced and the repair passed focused tests. Further independent review was
blocked by the review tool/model restriction. That review remains incomplete and
has not been retried through another route. CI and native evidence do not replace
that missing review.