Compare commits

..
44 Commits
Author SHA1 Message Date
Akizon f18ad72c17 Merge pull request #669 from MousseC/fix/upgrade-request-detection
fix(plugin): 仅当 Connection 与 Upgrade 头同时存在时判定协议升级请求
2026-09-14 13:44:49 +08:00
Akizon 8e4a3f8709 Merge pull request #672 from MiaM1ku/fix/latest-status-gpu-usage
fix: getNodesLatestStatus 写入实时 GPU 占用
2026-09-14 13:43:40 +08:00
Nya 80add06a12 fix: getNodeRecentStatus 同样写入 GPU 占用
该接口扁平化近期上报时 Gpu 也被写死为 0,与 getNodesLatestStatus 同类问题。
2026-09-11 23:56:18 +08:00
Nya 1333d63ce0 fix: getNodesLatestStatus 写入实时 GPU 占用
此前 recordLike.Gpu 被写死为 0,主题实时 GPU 一直显示 0,而历史 gpu.usage 正常。改为使用 report.GPU.AverageUsage,并附带 gpu_count / gpu_average_usage / gpu_detailed_info。
2026-09-11 23:52:18 +08:00
MousseC 44ed3035f1 chore(plugin): shorten isUpgradeRequest doc comment 2026-09-08 20:34:23 +08:00
MousseC 7ec36157d5 test(plugin): cover upgrade detection and injection behind bare Connection header 2026-09-08 18:41:58 +08:00
MousseC 360fd5f50b fix(plugin): detect protocol upgrades only when both headers are present
isUpgradeRequest previously flagged any request carrying a bare
`Connection: upgrade` header -- or a bare `Upgrade: websocket` header --
as a protocol upgrade, bypassing plugin HTML injection and
request/response hooks. Some reverse-proxy templates (e.g. sites created
by 1Panel) set `proxy_set_header Connection upgrade;` unconditionally, so
every proxied page request skipped the plugin middleware silently while
plugins still reported as loaded.

Per RFC 9110 section 7.8 a valid upgrade request carries both headers:
Connection lists the upgrade token and Upgrade names the target protocol.
Require the Upgrade header to be present and the Connection list to
contain the upgrade token, and accept any upgrade protocol rather than
only websocket.
2026-09-08 18:41:55 +08:00
Akizon77 b11ffd3aa7 使用zstd压缩前端,节约空间! 2026-09-04 16:16:41 +08:00
Akizon77 d7d6135c49 让SSRF策略可以调整1下! 2026-09-04 13:59:37 +08:00
Akizon77 ac037f2177 换了1个更好的按安装脚本! 2026-09-04 11:43:42 +08:00
Akizon77 5405bdf377 使用人类友好的时间格式 2026-09-02 16:09:16 +08:00
Akizon f754f01f53 Merge pull request #652 from YZoiu/fix/metric-store-remote-write-timeout
fix(metricstore): break report counter restore retry loop on slow remote databases
2026-09-02 15:41:27 +08:00
Akizon77 434ba108e1 后续版本将不再支持v1协议了捏 2026-09-02 15:30:30 +08:00
Yuzhong ZhangandAkizon77 24254749b5 现在只剩v2了 (#653)
Co-authored-by: Akizon77 <hytf233g@gmail.com>
2026-09-02 15:01:04 +08:00
Akizon c6b02af000 感谢新的金主喵 2026-09-01 15:50:56 +08:00
Akizon b28ac9d42c 感谢新的金主喵 2026-09-01 15:50:03 +08:00
Akizon77 2ff7c11db0 refactor: remove v1 protocol and stabilize file transfer 2026-08-29 19:03:53 +08:00
Akizon77 46fd2af44c feat: 自动推断分片大小 2026-08-29 15:03:14 +08:00
Akizon77 4d2eb70c62 fix: 分片文件操作过早关闭io导致502 2026-08-29 12:38:32 +08:00
Akizon77 7a6692b79a pref: 文件上传下载链路 2026-08-29 11:24:22 +08:00
Akizon77 63d78b6ed4 pref: 下载速度优化 2026-08-28 23:41:27 +08:00
Akizon77 2319ea77f6 fix: 上传大小限制 2026-08-28 22:02:27 +08:00
Akizon77 0c39a52e0b feat: file access / editor 2026-08-28 20:11:18 +08:00
Akizon77 e31a032d89 feat(terminal): support session reattachment on WebSocket reconnect 2026-08-26 14:40:01 +08:00
Akizon77 da4d5187c1 docs: update Rainyun deployment descriptions 2026-08-17 14:59:36 +08:00
Akizon77 bf6b45ec3a fix(metricstore): #646 skip traffic deltas on agent restart or counter reset 2026-08-13 17:10:59 +08:00
Akizon77 4a3400bee6 feat(notifier): warn users to migrate before traffic report removal 2026-08-13 15:45:56 +08:00
Akizon77 4caa067556 feat(plugin): reload plugin after config update 2026-08-13 14:28:32 +08:00
Akizon77 2f3959adde fix(child_process): stop inheriting runtime timeout for child processes
Replace default runtime timeout with explicit timeout only, avoiding unexpected termination of long-running child processes when a runtime-level timeout is configured.
2026-08-13 14:13:26 +08:00
Akizon77 63a7ffddcf fix(plugin): normalize RPC results before resolving promises
RPC results were passed directly to goja's resolve, which could fail for
non-JSON-compatible Go types. Add normalizeRPCValue to round-trip the
result through JSON, ensuring it becomes plain data structures. Handle
serialization errors properly by rejecting the promise.
2026-08-13 13:05:12 +08:00
Akizon77 92d679ffdb feat(console): format values as JSON and error messages
Improve console output by replacing fmt.Sprint with a dedicated
value formatter that renders null, strings, errors, and JSON-serializable
objects more intuitively for JavaScript runtime users.
2026-08-13 11:45:10 +08:00
Akizon77 7c30fc4dc9 feat: #635 add historical backup cleanup and harden upgrade rollback 2026-08-13 09:43:05 +08:00
Akizon77 773a3715dc feat(upload): add chunked upload store for themes/plugins 2026-08-12 22:38:07 +08:00
Akizon77 75d7b6c7a3 feat(migration): cap structure upgrade progress at 80% 2026-08-12 20:38:31 +08:00
Akizon77 c294c17642 feat: add managed configuration selectors 2026-08-12 16:46:05 +08:00
Akizon77 2883bf6295 feat: add raw SQL query and exec RPC methods 2026-08-11 17:41:46 +08:00
Akizon77 14bf8bde7c build: use COPY --chmod in Dockerfile 2026-08-07 16:57:13 +08:00
Akizon77 f043f2890c docs: update README 2026-08-07 16:23:31 +08:00
Akizon77 057108a2f3 docs: update README 2026-08-07 16:13:50 +08:00
Akizon77 5879d4b3da pref: mem alloc 2026-08-07 13:31:05 +08:00
Akizon77 16769d5687 pref: reduce cpu usage while querying metrics 2026-08-06 23:14:31 +08:00
Akizon77 5ea32c48b6 remove undefined metrics before reclaiming space 2026-08-06 20:25:36 +08:00
Akizon77 a06d922d78 fix(metricstore): do not delete metric history during startup 2026-08-06 20:06:38 +08:00
Akizon77 0ed14fc73d pref: 减少重复查询 2026-08-06 19:53:19 +08:00
113 changed files with 8566 additions and 2944 deletions

No files matched your search

+15 -3
View File
@@ -23,6 +23,15 @@ runs:
with:
node-version: ${{ inputs.node-version }}
- name: Install zstd
shell: bash
run: |
set -euxo pipefail
if ! command -v zstd >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y zstd
fi
- name: Clone and build frontend
shell: bash
run: |
@@ -38,9 +47,12 @@ runs:
npm run build
cd ..
mkdir -p web/public/defaultTheme/dist
rm -rf web/public/defaultTheme/dist/*
cp -r komari-web/dist/* web/public/defaultTheme/dist/
mkdir -p web/public/defaultTheme
rm -rf web/public/defaultTheme/dist web/public/defaultTheme/dist.tar.zst
test -f komari-web/dist/index.html
tar -cf "$RUNNER_TEMP/komari-default-dist.tar" -C komari-web/dist .
zstd -19 -T0 -q -f "$RUNNER_TEMP/komari-default-dist.tar" -o web/public/defaultTheme/dist.tar.zst
rm -f "$RUNNER_TEMP/komari-default-dist.tar"
cp -f komari-web/komari-theme.json web/public/defaultTheme/
if [ -f komari-web/preview.png ]; then cp -f komari-web/preview.png web/public/defaultTheme/; fi
if [ -f komari-web/perview.png ]; then cp -f komari-web/perview.png web/public/defaultTheme/; fi
@@ -22,3 +22,4 @@ runs:
mkdir -p web/public/defaultTheme
rm -rf web/public/defaultTheme/*
tar -xzf "$RUNNER_TEMP/frontend-artifact/${{ inputs.artifact-name }}.tar.gz" -C web/public/defaultTheme
test -s web/public/defaultTheme/dist.tar.zst
-5
View File
@@ -14,11 +14,6 @@ jobs:
- name: Checkout code
uses: actions/checkout@v4
#- name: Create dummy index.html
# run: |
# mkdir -p web/public/defaultTheme/dist
# echo "<html><body><h1>This is komari development server.</h1></body></html>" > web/public/defaultTheme/dist/index.html
- name: Clone and build frontend
uses: ./.github/actions/build-frontend
+2
View File
@@ -36,3 +36,5 @@ go.work.sum
komari
/dist/
.codex
/web/filemanager/data
+1 -4
View File
@@ -8,10 +8,7 @@ ARG TARGETARCH
RUN apk add --no-cache ca-certificates curl tzdata
COPY komari-${TARGETOS}-${TARGETARCH} /app/komari
RUN chmod +x /app/komari
COPY --chmod=755 komari-${TARGETOS}-${TARGETARCH} /app/komari
ENV GIN_MODE=release
ENV KOMARI_LISTEN=0.0.0.0:25774
+43 -23
View File
@@ -5,43 +5,63 @@
![komari](https://socialify.git.ci/komari-monitor/komari/image?description=1&font=Inter&forks=1&issues=1&language=1&logo=https%3A%2F%2Fraw.githubusercontent.com%2Fkomari-monitor%2Fkomari-web%2Fd54ce1288df41ead08aa19f8700186e68028a889%2Fpublic%2Ffavicon.png&name=1&owner=1&pattern=Plus&pulls=1&stargazers=1&theme=Auto)
[简体中文](./README_zh-cn.md)
[English](./README.md) | [简体中文](./README_zh-cn.md)
Komari is a lightweight, self-hosted server monitoring tool designed to provide a simple and efficient solution for monitoring server performance. It supports viewing server status through a web interface and collects data through a lightweight agent.
Komari is a lightweight, self-hosted server monitoring solution. It provides a simple and efficient way to track server performance through a web interface, with metrics collected by a lightweight agent.
> [!WARNING]
> Komari is a self-hosted monitoring and control program intended only for systems you own or are authorized to administer. Do not weaponize Komari, deploy it without consent, or use it for unauthorized access, persistence, command execution, or other abusive activity. For context on real-world abuse risks, see Huntress' analysis: [Komari C2 agent abuse](https://www.huntress.com/blog/komari-c2-agent-abuse).
> Users are solely responsible for how they deploy and operate Komari. The developers do not accept responsibility for unauthorized or abusive use, or for any resulting consequences.
> On Windows, when remote control is enabled, the client displays a Windows notification at each user login to remind the user that Komari is remote control software.
> Komari is a self-hosted monitoring and control application. Deploy it only on systems you own or are authorized to manage. You are solely responsible for how you deploy and use Komari. The developers accept no liability for unauthorized access, persistence, command execution, other misuse, or any resulting consequences.
[文档/Documentation](https://komari-document.pages.dev/) | [文档(镜像站 By Geekertao)](https://www.komari.wiki) | [Telegram Group](https://t.me/komari_monitor)
[Documentation](https://www.komari.wiki/) | [Telegram Group](https://t.me/komari_monitor)
## Features
- **Lightweight and Efficient**: Low resource consumption, suitable for servers of all sizes.
- **Self-hosted**: Complete control over data privacy, easy to deploy.
- **Web Interface**: Intuitive monitoring dashboard, easy to use.
- **Real-time monitoring**: Displays monitoring data at one-second intervals.
- **Lightweight and efficient**: Uses minimal system resources and works well on servers of any size.
- **Self-hosted**: Keeps you in control of your data and privacy.
- **Web interface**: Provides an intuitive, easy-to-use monitoring dashboard.
- **Extensible**: Supports custom themes and plugins.
## One-click Deployment
## Quick Start
- Rainyun - CNY 4.5/month
| Platform | Description |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <a href="https://app.rainyun.com/apps/rca/store/6780/NzYxNzAz_"><img src="https://rainyun-apps.cn-nb1.rains3.com/materials/deploy-on-rainyun-cn.svg" alt="Rainyun" width="180"></a> | Deploy websites, databases, and hundreds of popular apps in seconds with flexible hourly billing. [Get started for just ¥5/month. Deploy now!](https://app.rainyun.com/apps/rca/store/6780/NzYxNzAz_) |
| <a href="https://apps.fit2cloud.com/1panel/komari"><img src="https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/1panel-logo-blue.png" alt="1Panel App Store" width="180"></a> | A modern, open-source Linux server management panel for websites, databases, containers, files, backups, security, and AI, with one-click deployment from its app store. |
[![](https://rainyun-apps.cn-nb1.rains3.com/materials/deploy-on-rainyun-cn.svg)](https://app.rainyun.com/apps/rca/store/6780/NzYxNzAz_)
For instructions on Docker deployment, binary installation, building from source, and updates, see the [installation guide](https://www.komari.wiki/en/install/quick-start).
- 1Panel App Store
## Screenshots
Available on 1Panel App Store. Install via **App Store > Utilities > Komari**.
| Page | Screenshot |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Home Dashboard | <img src="https://b2.akz.moe/awesome-pictures/komari-screenshot/%E4%B8%BB%E9%A1%B5%E4%BB%AA%E8%A1%A8%E7%9B%98-en.webp" width="800" alt="Home Dashboard"> |
| Admin Dashboard | <img src="https://b2.akz.moe/awesome-pictures/komari-screenshot/%E5%90%8E%E5%8F%B0%E4%BB%AA%E8%A1%A8%E7%9B%98-en.webp" width="800" alt="Admin Dashboard"> |
| History Charts | <img src="https://b2.akz.moe/awesome-pictures/komari-screenshot/%E5%8E%86%E5%8F%B2%E5%9B%BE%E8%A1%A8-en.webp" width="800" alt="History Charts"> |
| Web Terminal | <img src="https://b2.akz.moe/awesome-pictures/komari-screenshot/%E7%BD%91%E9%A1%B5%E7%BB%88%E7%AB%AF.webp" width="800" alt="Web Terminal"> |
| Customizable Themes | <img src="https://b2.akz.moe/awesome-pictures/komari-screenshot/%E4%B8%BB%E9%A2%98%E5%8F%AF%E8%87%AA%E5%AE%9A%E4%B9%89-en.webp" width="800" alt="Customizable Themes"> |
| Theme Market | <img src="https://b2.akz.moe/awesome-pictures/komari-screenshot/%E4%B8%BB%E9%A2%98%E5%B8%82%E5%9C%BA-en.webp" width="800" alt="Theme Market"> |
For Docker, binary, source-build, and update instructions, see the [installation guide](https://komari-document.pages.dev/en/install/quick-start).
## Sponsors
## Contributing
Interested in sponsoring Komari? Contact the developer via [email](mailto:komari@akz.moe) or [Telegram](https://t.me/mamomoe).
Issues and Pull Requests are welcome!
| Sponsor | Description |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <a href="https://axisnow.io/zh?utm=komari"><img src="https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/AxisNow.jpg" alt="AxisNow" width="180"></a> | [Self-Hosted Private CDN \| Subscription-Based CDN-Like Service \| A Fully Controlled, Flexible, Modular CDN Network](https://axisnow.io/zh?utm=komari) |
| <a href="https://whmcs.as211392.com/aff.php?aff=110"><img src="https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/dreamcloud.png" alt="Dream Cloud" width="180"></a> | Cost-effective Asia-Pacific hosting with direct connectivity and robust DDoS protection, backed by transparent capacity claims. |
| <a href="https://sharon.io"><img src="https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/sharon-networks.webp" alt="Sharon Networks" width="180"></a> | Premium China-optimized connectivity from Asia-Pacific data centers, featuring low latency, high bandwidth, and Tbps-scale local DDoS mitigation. Join the [Telegram community](https://t.me/SharonNetwork) to participate in charitable initiatives and community giveaways. |
## Acknowledgements
## Contributors
| Acknowledgement | Details |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <a href="https://whmcs.as211392.com/aff.php?aff=110"><img src="https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/dreamcloud.png" alt="Dream Cloud" width="180"></a> | 极高性价比解锁直连亚太高防,真高防,不虚标,打死退款 |
| <a href="https://sharon.io"><img src="https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/sharon-networks.webp" alt="Sharon Networks" width="180"></a> | 亚太数据中心提供顶级的中国优化网络接入 · 低延时&高带宽&提供Tbps级本地清洗高防服务, 为您的业务保驾护航, 为您的客户提供极致体验. 加入社区 [Telegram群组](https://t.me/SharonNetwork) 可参与公益募捐或群内抽奖免费使用 |
| Open-source community | Everyone who has submitted pull requests or created themes. |
Thanks to everyone who has contributed code, themes, plugins, documentation, translations, bug reports, or feedback to Komari.
<a href="https://github.com/komari-monitor/komari/graphs/contributors"><img src="https://contributors-img.web.app/image?repo=komari-monitor/komari" alt="Komari contributors" width="600"></a>
## Support the Project
If Komari has been useful to you, consider buying me a coffee. Thank you for your support!
| WeChat Pay | TRON Network |
| ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- |
| ![WeChat Pay QR code](https://b2.akz.moe/awesome-pictures/%E5%BE%AE%E4%BF%A1%E8%B5%9E%E8%B5%8F%E7%A0%81.png) | ![TRON Network QR code](https://b2.akz.moe/awesome-pictures/PixPin_2026-08-07_15-16-52.png) |
+36 -18
View File
@@ -5,45 +5,63 @@
![komari](https://socialify.git.ci/komari-monitor/komari/image?description=1&font=Inter&forks=1&issues=1&language=1&logo=https%3A%2F%2Fraw.githubusercontent.com%2Fkomari-monitor%2Fkomari-web%2Fd54ce1288df41ead08aa19f8700186e68028a889%2Fpublic%2Ffavicon.png&name=1&owner=1&pattern=Plus&pulls=1&stargazers=1&theme=Auto)
[简体中文](./README_zh-cn.md)
[English](./README.md) | [简体中文](./README_zh-cn.md)
Komari 是一款轻量级的自托管服务器监控工具,旨在提供简单、高效的服务器性能监控解决方案。它支持通过 Web 界面查看服务器状态,并通过轻量级 Agent 收集数据。
> [!WARNING]
> Komari 是一款自托管的监控/控制程序,仅应部署在你拥有或已获得授权管理的系统上。请勿将 Komari 武器化,或在未获授权的情况下部署、访问、持久化、执行命令及从事其他滥用行为。关于现实中的滥用风险,可参考 Huntress 的分析:[Komari C2 agent abuse](https://www.huntress.com/blog/komari-c2-agent-abuse)。
> 用户需要自行承担部署和使用 Komari 的责任。开发者不对未经授权或滥用行为及其后果承担责任。
> 在 Windows 端开启远程控制后,客户端会在每次用户登录时通过 Windows 通知提醒用户 Komari 是一款远程控制软件。
> Komari 是一款自托管的监控/控制程序,仅应部署在你拥有或已获得授权管理的系统上。在未获授权的情况下部署、访问、持久化、执行命令及从事其他滥用行为,用户需要自行承担部署和使用 Komari 的责任。开发者不对未经授权或滥用行为及其后果承担责任。
[文档/Documentation](https://www.komari.wiki/) | [Telegram 群](https://t.me/komari_monitor)
[文档](https://www.komari.wiki/) | [Telegram 群](https://t.me/komari_monitor)
## 特性
- **实时监控**: 秒级实时数据展示。
- **轻量高效**:低资源占用,适合各种规模的服务器。
- **自托管**:完全掌控数据隐私,部署简单。
- **Web 界面**:直观的监控仪表盘,易于使用。
- **极强的可扩展性**: 支持自定义主题和插件。
## 快速开始
### 0. 容器云一键部署
- 雨云云应用 - CNY 4.5/月
[![](https://rainyun-apps.cn-nb1.rains3.com/materials/deploy-on-rainyun-cn.svg)](https://app.rainyun.com/apps/rca/store/6780/NzYxNzAz_)
- 1Panel 应用商店
已上架1Panel应用商店,应用商店-实用工具-Komari 即可安装
| 平台 | 介绍 |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| <a href="https://app.rainyun.com/apps/rca/store/6780/NzYxNzAz_"><img src="https://rainyun-apps.cn-nb1.rains3.com/materials/deploy-on-rainyun-cn.svg" alt="Rainyun" width="180"></a> | 秒级部署网站、数据库及数百款热门 App,并采用按小时灵活计费。[每月5元,立即部署](https://app.rainyun.com/apps/rca/store/6780/NzYxNzAz_) |
| <a href="https://apps.fit2cloud.com/1panel/komari"><img src="https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/1panel-logo-blue.png" alt="1Panel Appstore" width="180"></a> | 现代化、开源的Linux 服务器运维管理面板,提供网站、数据库、容器、文件、备份、安全与AI 管理能力,支持应用商店一键部署。 |
Docker、二进制文件、源码构建和更新说明,请参阅 [安装指南](https://www.komari.wiki/install/quick-start).
## 贡献
## 截图
欢迎提交 Issue 或 Pull Request!
| 页面 | 截图 |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 主页仪表盘 | <img src="https://b2.akz.moe/awesome-pictures/komari-screenshot/%E4%B8%BB%E9%A1%B5%E4%BB%AA%E8%A1%A8%E7%9B%98.webp" width="800" alt="主页仪表盘"> |
| 后台仪表盘 | <img src="https://b2.akz.moe/awesome-pictures/komari-screenshot/%E5%90%8E%E5%8F%B0%E4%BB%AA%E8%A1%A8%E7%9B%98.webp" width="800" alt="后台仪表盘"> |
| 历史图表 | <img src="https://b2.akz.moe/awesome-pictures/komari-screenshot/%E5%8E%86%E5%8F%B2%E5%9B%BE%E8%A1%A8.webp" width="800" alt="历史图表"> |
| 网页终端 | <img src="https://b2.akz.moe/awesome-pictures/komari-screenshot/%E7%BD%91%E9%A1%B5%E7%BB%88%E7%AB%AF.webp" width="800" alt="网页终端"> |
| 主题可自定义 | <img src="https://b2.akz.moe/awesome-pictures/komari-screenshot/%E4%B8%BB%E9%A2%98%E5%8F%AF%E8%87%AA%E5%AE%9A%E4%B9%89.webp" width="800" alt="主题可自定义"> |
| 主题市场 | <img src="https://b2.akz.moe/awesome-pictures/komari-screenshot/%E4%B8%BB%E9%A2%98%E5%B8%82%E5%9C%BA.webp" width="800" alt="主题市场"> |
## 鸣谢
## 赞助商
有意赞助 Komari?请通过 [电子邮箱](mailto:komari@akz.moe) 或 [Telegram](https://t.me/mamomoe) 联系开发者。
| 赞助商 | 描述 |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <a href="https://axisnow.io/zh?utm=komari"><img src="https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/AxisNow.jpg" alt="AxisNow" width="180"></a> | [自建私有部署CDN \| 订阅式高仿CDN \| 自主可控、灵活组合的CDN网络](https://axisnow.io/zh?utm=komari) |
| <a href="https://whmcs.as211392.com/aff.php?aff=110"><img src="https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/dreamcloud.png" alt="Dream Cloud" width="180"></a> | 极高性价比解锁直连亚太高防,真高防,不虚标,打死退款 |
| <a href="https://sharon.io"><img src="https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/sharon-networks.webp" alt="Sharon Networks" width="180"></a> | 亚太数据中心提供顶级的中国优化网络接入 · 低延时&高带宽&提供Tbps级本地清洗高防服务, 为您的业务保驾护航, 为您的客户提供极致体验. 加入社区 [Telegram群组](https://t.me/SharonNetwork) 可参与公益募捐或群内抽奖免费使用 |
| 开源社区 | 提交 PR、制作主题的各位开发者 |
## 贡献者
感谢所有为 Komari 贡献代码、主题、插件、文档、翻译、问题报告或反馈的朋友。
<a href="https://github.com/komari-monitor/komari/graphs/contributors"><img src="https://contributors-img.web.app/image?repo=komari-monitor/komari" alt="Komari 贡献者" width="600"></a>
## 支持项目
如果 Komari 对你有所帮助,欢迎请作者喝一杯奶茶。感谢你的支持!
| 微信赞赏码 | TRON Network |
| ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------- |
| ![wechat](https://b2.akz.moe/awesome-pictures/%E5%BE%AE%E4%BF%A1%E8%B5%9E%E8%B5%8F%E7%A0%81.png) | ![TRON](https://b2.akz.moe/awesome-pictures/PixPin_2026-08-07_15-16-52.png) |
+2 -2
View File
@@ -6,7 +6,7 @@ import (
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
v1 "github.com/komari-monitor/komari/protocol/v1"
v2 "github.com/komari-monitor/komari/protocol/v2"
)
func GetClientUUIDByToken(token string) (clientUUID string, err error) {
@@ -20,7 +20,7 @@ func GetClientUUIDByToken(token string) (clientUUID string, err error) {
}
// 检查数据防止异常数据导致数据库损坏
func ReportVerify(report v1.Report) error {
func ReportVerify(report v2.Report) error {
// 防止输入不合理范围
if report.CPU.Usage < 0 || report.CPU.Usage > 100 {
return fmt.Errorf("CPU.Usage must be between 0 and 100")
+25 -4
View File
@@ -15,16 +15,37 @@ const (
// Theme represents a komari theme information
type Theme struct {
Name string `json:"name"` // 主题名称
Name any `json:"name"` // 主题名称,支持字符串或多语言对象
Short string `json:"short"` // 短名称,用作文件夹名
Description string `json:"description"` // 主题描述
Description any `json:"description"` // 主题描述,支持字符串或多语言对象
Version string `json:"version"` // 版本号
Author string `json:"author"` // 作者
Author any `json:"author"` // 作者,支持字符串或多语言对象
URL string `json:"url"` // 主题URL
Preview string `json:"preview"` // 预览图片相对路径
Configuration Configuration `json:"configuration"` // 声明配置项
}
// IsLocalizedText reports whether a manifest text field has a usable value.
func IsLocalizedText(value any) bool {
switch text := value.(type) {
case string:
return strings.TrimSpace(text) != ""
case map[string]any:
for _, item := range text {
if itemText, ok := item.(string); ok && strings.TrimSpace(itemText) != "" {
return true
}
}
case map[string]string:
for _, item := range text {
if strings.TrimSpace(item) != "" {
return true
}
}
}
return false
}
type Configuration struct {
Type string `json:"type"` // managed raw redirect
Icon string `json:"icon"` // 图标
@@ -36,7 +57,7 @@ type ManagedThemeConfigurationItem struct {
Key string `json:"key"`
Name any `json:"name"`
Required bool `json:"required"`
Type string `json:"type"` // string number select switch title
Type string `json:"type"` // string number select switch title textbox richtext nodes pingtasks
Options string `json:"options"`
Default any `json:"default"`
Help any `json:"help"`
+31 -52
View File
@@ -3,16 +3,17 @@ package database
import (
"context"
"encoding/json"
logger "github.com/komari-monitor/komari/utils/log"
"os"
"path/filepath"
"strings"
"github.com/gin-gonic/gin"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/internal/metricstore"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/internal/config"
"github.com/komari-monitor/komari/internal/managedconfig"
"github.com/komari-monitor/komari/internal/metricstore"
logger "github.com/komari-monitor/komari/utils/log"
"github.com/komari-monitor/komari/web/public"
)
func GetPublicInfo() (map[string]interface{}, error) {
@@ -69,58 +70,20 @@ func GetPublicInfo() (map[string]interface{}, error) {
if err != nil {
logger.Infof("database", "%v", err)
}
// Try to load theme declaration file and merge defaults for managed configuration
// Theme declarations live in ./data/theme/<short>/komari-theme.json
if cst.Theme != "" && cst.Theme != "default" {
themeConfigPath := filepath.Join("./data/theme", cst.Theme, "komari-theme.json")
if _, err := os.Stat(themeConfigPath); err == nil {
b, err := os.ReadFile(themeConfigPath)
if err == nil {
var themeDecl struct {
Configuration struct {
Type string `json:"type"`
Data []models.ManagedThemeConfigurationItem `json:"data"`
} `json:"configuration"`
}
if err := json.Unmarshal(b, &themeDecl); err == nil {
if themeDecl.Configuration.Type == "managed" {
for _, item := range themeDecl.Configuration.Data {
if item.Key == "" {
continue
}
// missing
if _, exists := tc_data[item.Key]; !exists {
var def any = item.Default
// select
if item.Type == "select" {
if def == nil || def == "" {
if item.Options != "" {
opts := strings.Split(item.Options, ",")
if len(opts) > 0 {
def = strings.TrimSpace(opts[0])
}
}
}
}
// number->0, string->"", switch->false
if def == nil {
switch item.Type {
case "number":
def = 0
case "switch":
def = false
default:
def = ""
}
}
tc_data[item.Key] = def
}
}
}
}
items := themeConfigurationItems(cst.Theme)
if cst.Theme != "default" {
for _, item := range items {
if item.Key == "" {
continue
}
if _, exists := tc_data[item.Key]; !exists {
tc_data[item.Key] = managedconfig.DefaultValue(item)
}
}
}
if err := managedconfig.ResolveForOutput(tc_data, items); err != nil {
return nil, err
}
return gin.H{
"sitename": cst.Sitename,
@@ -140,3 +103,19 @@ func GetPublicInfo() (map[string]interface{}, error) {
"theme_settings": tc_data,
}, nil
}
func themeConfigurationItems(short string) []models.ManagedThemeConfigurationItem {
var manifest models.Theme
if short == "default" {
data, err := public.PublicFS.ReadFile("defaultTheme/komari-theme.json")
if err != nil || json.Unmarshal(data, &manifest) != nil {
return nil
}
} else {
data, err := os.ReadFile(filepath.Join("./data/theme", short, "komari-theme.json"))
if err != nil || json.Unmarshal(data, &manifest) != nil {
return nil
}
}
return managedconfig.Items(manifest.Configuration)
}
+1059 -224
View File
File diff suppressed because it is too large. Load diff
+2
View File
@@ -17,6 +17,7 @@ type Settings struct {
ScriptDomain string `json:"script_domain" default:""` // 自定义脚本域名
SendIpAddrToGuest bool `json:"send_ip_addr_to_guest" default:"false"` // 是否向访客页面发送 IP 地址,默认 false
VisitorAuditEnabled bool `json:"visitor_audit_enabled" default:"false"` // 是否允许公开访客事件写入审计日志,默认 false
SSRFProtectionEnabled bool `json:"ssrf_protection_enabled" default:"false"` // 是否启用 SSRF 防护,默认关闭
EulaAccepted bool `json:"eula_accepted" default:"false"`
BaseScriptsURLKey string `json:"base_scripts_url" default:""`
// GeoIP 配置
@@ -55,6 +56,7 @@ const (
ScriptDomainKey = "script_domain"
SendIpAddrToGuestKey = "send_ip_addr_to_guest"
VisitorAuditEnabledKey = "visitor_audit_enabled"
SSRFProtectionEnabledKey = "ssrf_protection_enabled"
EulaAcceptedKey = "eula_accepted"
BaseScriptsURLKey = "base_scripts_url"
GeoIpEnabledKey = "geo_ip_enabled"
+144
View File
@@ -0,0 +1,144 @@
// Package managedconfig implements the shared managed-configuration behavior
// used by themes and plugins.
package managedconfig
import (
"encoding/json"
"strings"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
)
const (
TypeNodes = "nodes"
TypePingTasks = "pingtasks"
)
// Items returns the declared managed configuration items. An omitted type is
// managed, matching the theme configuration default.
func Items(configuration models.Configuration) []models.ManagedThemeConfigurationItem {
if configuration.Type != "" && !strings.EqualFold(configuration.Type, models.ThemeConfigurationManaged) {
return nil
}
raw, err := json.Marshal(configuration.Data)
if err != nil {
return nil
}
var items []models.ManagedThemeConfigurationItem
if err := json.Unmarshal(raw, &items); err != nil {
return nil
}
return items
}
func DefaultValue(item models.ManagedThemeConfigurationItem) any {
value := item.Default
if item.Type == "select" && (value == nil || value == "") && item.Options != "" {
options := strings.Split(item.Options, ",")
if len(options) > 0 {
return strings.TrimSpace(options[0])
}
}
if value != nil {
return value
}
switch item.Type {
case "number":
return float64(0)
case "switch":
return false
case TypeNodes, TypePingTasks:
return "[]"
default:
return ""
}
}
// ResolveForOutput decodes declared selectors, drops deleted references, and
// returns typed arrays suitable for public theme settings and plugin config.
func ResolveForOutput(values map[string]any, items []models.ManagedThemeConfigurationItem) error {
hasNodes := false
hasPingTasks := false
for _, item := range items {
switch item.Type {
case TypeNodes:
hasNodes = true
case TypePingTasks:
hasPingTasks = true
}
}
db := dbcore.GetDBInstance()
liveNodes := map[string]struct{}{}
if hasNodes {
var nodes []models.Client
if err := db.Select("uuid").Find(&nodes).Error; err != nil {
return err
}
for _, node := range nodes {
liveNodes[node.UUID] = struct{}{}
}
}
livePingTasks := map[uint]struct{}{}
if hasPingTasks {
var tasks []models.PingTask
if err := db.Select("id").Find(&tasks).Error; err != nil {
return err
}
for _, task := range tasks {
livePingTasks[task.Id] = struct{}{}
}
}
for _, item := range items {
if item.Key == "" {
continue
}
switch item.Type {
case TypeNodes:
selected := NodeIDs(values[item.Key])
filtered := make([]string, 0, len(selected))
for _, id := range selected {
if _, ok := liveNodes[id]; ok {
filtered = append(filtered, id)
}
}
values[item.Key] = filtered
case TypePingTasks:
selected := PingTaskIDs(values[item.Key])
filtered := make([]uint, 0, len(selected))
for _, id := range selected {
if _, ok := livePingTasks[id]; ok {
filtered = append(filtered, id)
}
}
values[item.Key] = filtered
}
}
return nil
}
func NodeIDs(value any) []string {
raw, ok := value.(string)
if !ok {
return nil
}
var ids []string
if json.Unmarshal([]byte(raw), &ids) != nil {
return nil
}
return ids
}
func PingTaskIDs(value any) []uint {
raw, ok := value.(string)
if !ok {
return nil
}
var ids []uint
if json.Unmarshal([]byte(raw), &ids) != nil {
return nil
}
return ids
}
-9
View File
@@ -92,9 +92,6 @@ func createMetricDefinitionsWithDefaultRetention(ctx context.Context, s *metric.
}
if err == nil {
if existing.RetentionDays == 0 {
if _, err := s.SetMetricRetention(ctx, def.Name, 0); err != nil {
return fmt.Errorf("failed to preserve disabled metric %s: %w", def.Name, err)
}
continue
}
def.RetentionDays = existing.RetentionDays
@@ -103,11 +100,5 @@ func createMetricDefinitionsWithDefaultRetention(ctx context.Context, s *metric.
return fmt.Errorf("failed to create metric %s: %w", def.Name, err)
}
}
for _, name := range obsoleteBuiltinMetricNames {
if err := s.DeleteMetric(ctx, name); err != nil {
return fmt.Errorf("failed to remove obsolete metric %s: %w", name, err)
}
}
return nil
}
+25 -1
View File
@@ -84,7 +84,31 @@ func ReclaimSpace(ctx context.Context) (MaintenanceResult, error) {
}
maintenanceCtx := context.Background()
result.Before, result.BeforeSizeError = activeStore.StorageSize(maintenanceCtx)
maintenanceErr := activeStore.ReclaimSpace(maintenanceCtx)
maintenanceErr := deleteUndefinedMetrics(maintenanceCtx, activeStore)
if maintenanceErr == nil {
maintenanceErr = activeStore.ReclaimSpace(maintenanceCtx)
}
result.After, result.AfterSizeError = activeStore.StorageSize(maintenanceCtx)
return result, maintenanceErr
}
func deleteUndefinedMetrics(ctx context.Context, s *metric.Store) error {
definitions, err := s.ListMetrics(ctx)
if err != nil {
return fmt.Errorf("list metric definitions before reclaim: %w", err)
}
builtin := make(map[string]struct{}, len(builtinMetricNames))
for _, name := range builtinMetricNames {
builtin[name] = struct{}{}
}
for _, definition := range definitions {
if _, ok := builtin[definition.Name]; ok {
continue
}
if err := s.DeleteMetric(ctx, definition.Name); err != nil {
return fmt.Errorf("delete undefined metric %q before reclaim: %w", definition.Name, err)
}
}
return nil
}
+2 -7
View File
@@ -32,13 +32,6 @@ var loadRecordMetricNames = []string{
MetricProcess, MetricConnections, MetricConnectionsUDP,
}
var obsoleteBuiltinMetricNames = []string{
"memory.total",
"swap.total",
"temperature",
"disk.total",
}
// gpuDeviceRecordMetricNames are stored separately from the entity-level GPU
// average and are included when deleting all system records.
var gpuDeviceRecordMetricNames = []string{
@@ -50,6 +43,8 @@ var recordMetricNames = joinMetricNames(loadRecordMetricNames, gpuDeviceRecordMe
// Ping has an independent retention and cleanup boundary.
var pingMetricNames = []string{MetricPingLatency, MetricPingLoss}
var builtinMetricNames = joinMetricNames(recordMetricNames, pingMetricNames)
func metricNameForRecordField(name string) (string, bool) {
switch name {
case "cpu":
+17 -14
View File
@@ -3,14 +3,13 @@ package metricstore
import (
"context"
"database/sql"
"errors"
"path/filepath"
"testing"
"time"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/pkg/metric"
v1 "github.com/komari-monitor/komari/protocol/v1"
v2 "github.com/komari-monitor/komari/protocol/v2"
)
func TestDefaultRollupPolicy(t *testing.T) {
@@ -283,7 +282,7 @@ func TestCreateMetricDefinitionsUsesExplicitRetentionAndPreservesOverrides(t *te
}
}
func TestCreateMetricDefinitionsRemovesObsoleteMetrics(t *testing.T) {
func TestCreateMetricDefinitionsKeepsExistingMetrics(t *testing.T) {
ctx := context.Background()
s, err := metric.Open(ctx, metric.SQLite(":memory:", metric.WithMaxOpenConns(1)))
if err != nil {
@@ -299,15 +298,19 @@ func TestCreateMetricDefinitionsRemovesObsoleteMetrics(t *testing.T) {
if err := createMetricDefinitions(ctx, s); err != nil {
t.Fatalf("refresh built-in definitions: %v", err)
}
if _, err := s.GetMetric(ctx, "memory.total"); !errors.Is(err, metric.ErrNotFound) {
t.Fatalf("obsolete definition error = %v, want ErrNotFound", err)
definition, err := s.GetMetric(ctx, "memory.total")
if err != nil {
t.Fatalf("existing definition was removed: %v", err)
}
if definition.RetentionDays != 1 {
t.Fatalf("existing retention = %d, want 1", definition.RetentionDays)
}
points, err := s.Query(ctx, metric.Query{MetricName: "memory.total", EntityID: "node-a", Start: time.Now().UTC().Add(-time.Hour), End: time.Now().UTC().Add(time.Hour)})
if err != nil {
t.Fatalf("query obsolete points: %v", err)
t.Fatalf("query existing points: %v", err)
}
if len(points) != 0 {
t.Fatalf("obsolete points remain: %#v", points)
if len(points) != 1 || points[0].Value != 1024 {
t.Fatalf("existing points were removed: %#v", points)
}
}
@@ -569,15 +572,15 @@ func TestGetRecordsByClientAndTimeReadsRollupsAfterRawCompaction(t *testing.T) {
Connections: 321,
ConnectionsUdp: 12,
}
if _, err := WriteReport(ctx, v1.Report{
if _, err := WriteReport(ctx, v2.Report{
UUID: rec.Client,
UpdatedAt: ts,
CPU: v1.CPUReport{Usage: float64(rec.Cpu)},
Ram: v1.RamReport{Used: rec.Ram, Total: rec.RamTotal},
Load: v1.LoadReport{Load1: float64(rec.Load)},
Disk: v1.DiskReport{Used: rec.Disk, Total: rec.DiskTotal},
CPU: v2.CPUReport{Usage: float64(rec.Cpu)},
Ram: v2.RamReport{Used: rec.Ram, Total: rec.RamTotal},
Load: v2.LoadReport{Load1: float64(rec.Load)},
Disk: v2.DiskReport{Used: rec.Disk, Total: rec.DiskTotal},
Process: rec.Process,
Connections: v1.ConnectionsReport{
Connections: v2.ConnectionsReport{
TCP: rec.Connections,
UDP: rec.ConnectionsUdp,
},
+63
View File
@@ -0,0 +1,63 @@
package metricstore
import (
"context"
"database/sql"
"fmt"
"github.com/komari-monitor/komari/pkg/metric"
)
// QueryContext runs a raw read query against the active metric store. The
// caller must invoke the returned release function after closing rows.
func QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, metric.Driver, func(), error) {
return QueryForDriver(ctx, func(metric.Driver) (string, error) {
return query, nil
}, args...)
}
// QueryForDriver runs a raw read query whose SQL depends on the active store's
// driver. The caller must invoke the returned release function after closing rows.
func QueryForDriver(ctx context.Context, query func(metric.Driver) (string, error), args ...any) (*sql.Rows, metric.Driver, func(), error) {
if err := storeOperations.AcquireShared(ctx); err != nil {
return nil, "", nil, fmt.Errorf("wait for metric store operations before query: %w", err)
}
storeMu.RLock()
activeStore := store
storeMu.RUnlock()
if activeStore == nil {
storeOperations.ReleaseShared()
return nil, "", nil, ErrStoreNotInitialized
}
statement, err := query(activeStore.Driver())
if err != nil {
storeOperations.ReleaseShared()
return nil, "", nil, err
}
rows, err := activeStore.QueryContext(ctx, statement, args...)
if err != nil {
storeOperations.ReleaseShared()
return nil, "", nil, err
}
return rows, activeStore.Driver(), storeOperations.ReleaseShared, nil
}
// ExecContext runs a raw statement against the active metric store.
func ExecContext(ctx context.Context, query string, args ...any) (sql.Result, metric.Driver, error) {
if err := storeOperations.Acquire(ctx); err != nil {
return nil, "", fmt.Errorf("wait for metric store operations before execution: %w", err)
}
defer storeOperations.Release()
storeMu.RLock()
activeStore := store
storeMu.RUnlock()
if activeStore == nil {
return nil, "", ErrStoreNotInitialized
}
result, err := activeStore.ExecContext(ctx, query, args...)
return result, activeStore.Driver(), err
}
+30 -29
View File
@@ -11,7 +11,7 @@ import (
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/pkg/metric"
v1 "github.com/komari-monitor/komari/protocol/v1"
v2 "github.com/komari-monitor/komari/protocol/v2"
)
type reportTrafficState struct {
@@ -57,7 +57,7 @@ type reportBatchRequest struct {
type reportBatchWorker struct {
mu sync.Mutex
queue chan v1.Report
queue chan v2.Report
pingQueue chan models.PingRecord
requests chan reportBatchRequest
done chan struct{}
@@ -73,7 +73,7 @@ func StartReportBatcher() {
return
}
worker := &reportBatchWorker{
queue: make(chan v1.Report, reportBatchQueueSize),
queue: make(chan v2.Report, reportBatchQueueSize),
pingQueue: make(chan models.PingRecord, reportBatchQueueSize),
requests: make(chan reportBatchRequest, 1),
done: make(chan struct{}),
@@ -150,16 +150,16 @@ func FlushReportBatch(ctx context.Context) error {
// WriteReport persists one agent report and adds it to in-memory minute
// summaries using the same server receive time. Traffic deltas remain summable
// after rollup.
func WriteReport(ctx context.Context, report v1.Report) (v1.Report, error) {
func WriteReport(ctx context.Context, report v2.Report) (v2.Report, error) {
if report.UUID == "" {
return v1.Report{}, fmt.Errorf("report UUID is required")
return v2.Report{}, fmt.Errorf("report UUID is required")
}
if report.UpdatedAt.IsZero() {
return v1.Report{}, fmt.Errorf("report receive time is required")
return v2.Report{}, fmt.Errorf("report receive time is required")
}
report.UpdatedAt = report.UpdatedAt.UTC()
if GetStore() == nil {
return v1.Report{}, fmt.Errorf("metric store not enabled")
return v2.Report{}, fmt.Errorf("metric store not enabled")
}
reportBatcherMu.Lock()
@@ -167,19 +167,19 @@ func WriteReport(ctx context.Context, report v1.Report) (v1.Report, error) {
reportBatcherMu.Unlock()
if worker != nil {
if err := worker.enqueue(ctx, report); err != nil {
return v1.Report{}, err
return v2.Report{}, err
}
return report, nil
}
saved, err := writeReportBatch(ctx, []v1.Report{report})
saved, err := writeReportBatch(ctx, []v2.Report{report})
if err != nil {
return v1.Report{}, err
return v2.Report{}, err
}
return saved[0], nil
}
func (w *reportBatchWorker) enqueue(ctx context.Context, report v1.Report) error {
func (w *reportBatchWorker) enqueue(ctx context.Context, report v2.Report) error {
w.mu.Lock()
defer w.mu.Unlock()
if w.stopping {
@@ -200,7 +200,7 @@ func (w *reportBatchWorker) run() {
ticker := time.NewTicker(reportBatchInterval)
defer ticker.Stop()
var pending []v1.Report
var pending []v2.Report
var pendingPings []models.PingRecord
for {
select {
@@ -250,7 +250,7 @@ func (w *reportBatchWorker) enqueuePing(ctx context.Context, record models.PingR
}
}
func drainReportQueue(queue <-chan v1.Report, limit int) []v1.Report {
func drainReportQueue(queue <-chan v2.Report, limit int) []v2.Report {
if limit <= 0 {
return nil
}
@@ -261,7 +261,7 @@ func drainReportQueue(queue <-chan v1.Report, limit int) []v1.Report {
if capacity == 0 {
return nil
}
reports := make([]v1.Report, 0, capacity)
reports := make([]v2.Report, 0, capacity)
for len(reports) < limit {
select {
case report := <-queue:
@@ -296,7 +296,7 @@ func drainPingQueue(queue <-chan models.PingRecord, limit int) []models.PingReco
return records
}
func writePendingReports(ctx context.Context, pending *[]v1.Report) error {
func writePendingReports(ctx context.Context, pending *[]v2.Report) error {
if len(*pending) == 0 {
return nil
}
@@ -333,7 +333,7 @@ func writePendingPingRecords(ctx context.Context, pending *[]models.PingRecord)
return nil
}
func writeReportBatch(ctx context.Context, reports []v1.Report) ([]v1.Report, error) {
func writeReportBatch(ctx context.Context, reports []v2.Report) ([]v2.Report, error) {
if len(reports) == 0 {
return nil, nil
}
@@ -347,7 +347,7 @@ func writeReportBatch(ctx context.Context, reports []v1.Report) ([]v1.Report, er
return nil, fmt.Errorf("metric store not enabled")
}
prepared := make([]v1.Report, len(reports))
prepared := make([]v2.Report, len(reports))
copy(prepared, reports)
points := make([]metric.Point, 0, len(reports)*20)
pendingStates := make(map[*reportTrafficState]reportTrafficValues)
@@ -362,21 +362,18 @@ func writeReportBatch(ctx context.Context, reports []v1.Report) ([]v1.Report, er
}
if !values.initialized {
totalUp, hasUp, err := latestReportCounter(ctx, s, MetricNetTotalUp, report.UUID, report.UpdatedAt)
if err == nil {
if err != nil {
logger.Errorf("metricstore", "failed to restore previous upload counter for %s: %v", report.UUID, err)
} else {
values.totalUp = totalUp
values.hasUp = hasUp
} else if ctx.Err() == nil {
logger.Errorf("metricstore", "failed to restore previous upload counter for %s: %v", report.UUID, err)
}
totalDown, hasDown, err := latestReportCounter(ctx, s, MetricNetTotalDown, report.UUID, report.UpdatedAt)
if err == nil {
if err != nil {
logger.Errorf("metricstore", "failed to restore previous download counter for %s: %v", report.UUID, err)
} else {
values.totalDown = totalDown
values.hasDown = hasDown
} else if ctx.Err() == nil {
logger.Errorf("metricstore", "failed to restore previous download counter for %s: %v", report.UUID, err)
}
if err := ctx.Err(); err != nil {
return nil, err
}
values.initialized = true
}
@@ -402,13 +399,17 @@ func writeReportBatch(ctx context.Context, reports []v1.Report) ([]v1.Report, er
prepared[i] = report
}
if err := s.WriteBatch(ctx, points); err != nil {
return nil, err
}
// Persist the restored per-report traffic state even if the write below
// fails, so a slow or failing database does not re-issue the previous-
// counter queries on every batch.
for state, values := range pendingStates {
state.mu.Lock()
state.reportTrafficValues = values
state.mu.Unlock()
}
if err := s.WriteBatch(ctx, points); err != nil {
return nil, err
}
return prepared, nil
}
+13 -3
View File
@@ -8,10 +8,10 @@ import (
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/pkg/metric"
v1 "github.com/komari-monitor/komari/protocol/v1"
v2 "github.com/komari-monitor/komari/protocol/v2"
)
func reportMetricPoints(report v1.Report, trafficUp, trafficDown int64) []metric.Point {
func reportMetricPoints(report v2.Report, trafficUp, trafficDown int64) []metric.Point {
entityID := report.UUID
ts := report.UpdatedAt
points := []metric.Point{
@@ -93,8 +93,15 @@ func GetLatestTrafficBefore(ctx context.Context, entityIDs []string, before time
return result, nil
}
// maxResetAwareDelta is the largest per-sample increase accepted when a
// counter appears to reset. A "reset" whose current value is larger than this
// is treated as jitter of a still-huge counter, not a wrap or reboot leftover.
const maxResetAwareDelta int64 = 64 << 30
// TrafficCounterDelta returns a reset-aware increase between two cumulative
// traffic counters. After a reset, the current counter is the new increase.
// traffic counters. After a wrap or reboot, the current counter is the new
// increase. A tiny dip of a TB-scale counter is not a reset: counting the
// still-huge current value would inject a multi-terabyte spike.
func TrafficCounterDelta(current, previous int64) int64 {
if current < 0 || previous < 0 {
return 0
@@ -102,6 +109,9 @@ func TrafficCounterDelta(current, previous int64) int64 {
if current >= previous {
return current - previous
}
if current > maxResetAwareDelta {
return 0
}
return current
}
+352 -26
View File
@@ -2,14 +2,18 @@ package metricstore
import (
"context"
"database/sql"
"database/sql/driver"
"errors"
"fmt"
"sync/atomic"
"testing"
"time"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/pkg/metric"
v1 "github.com/komari-monitor/komari/protocol/v1"
v2 "github.com/komari-monitor/komari/protocol/v2"
sqlite3 "github.com/mattn/go-sqlite3"
)
func useReportTestStore(t *testing.T, policy *metric.RollupPolicy) *metric.Store {
@@ -44,6 +48,124 @@ func useReportTestStore(t *testing.T, policy *metric.RollupPolicy) *metric.Store
return s
}
type reportCounterFault struct {
remaining atomic.Int32
denied atomic.Int32
}
func (f *reportCounterFault) denyRollupRead() bool {
for {
remaining := f.remaining.Load()
if remaining <= 0 {
return false
}
if f.remaining.CompareAndSwap(remaining, remaining-1) {
f.denied.Add(1)
return true
}
}
}
type reportSQLiteConnector struct {
driver *sqlite3.SQLiteDriver
dsn string
}
func (c *reportSQLiteConnector) Connect(context.Context) (driver.Conn, error) {
return c.driver.Open(c.dsn)
}
func (c *reportSQLiteConnector) Driver() driver.Driver {
return c.driver
}
// useReportCounterFailureStore denies exactly two rollup reads after schema
// setup, exercising failed counter restoration while leaving writes usable.
func useReportCounterFailureStore(t *testing.T) (*metric.Store, *reportCounterFault) {
t.Helper()
fault := &reportCounterFault{}
dsn := fmt.Sprintf("file:report-counter-fault-%d?mode=memory&cache=shared", time.Now().UnixNano())
driver := &sqlite3.SQLiteDriver{
ConnectHook: func(conn *sqlite3.SQLiteConn) error {
conn.RegisterAuthorizer(func(op int, arg1, _, _ string) int {
if op == sqlite3.SQLITE_READ && arg1 == "metric_rollups" && fault.denyRollupRead() {
return sqlite3.SQLITE_DENY
}
return sqlite3.SQLITE_OK
})
return nil
},
}
db := sql.OpenDB(&reportSQLiteConnector{driver: driver, dsn: dsn})
s, err := metric.Open(context.Background(), metric.SQLite("", metric.WithDB(db), metric.WithMaxOpenConns(1)))
if err != nil {
_ = db.Close()
t.Fatalf("open metric store: %v", err)
}
if err := createMetricDefinitions(context.Background(), s); err != nil {
_ = s.Close()
_ = db.Close()
t.Fatalf("create metric definitions: %v", err)
}
fault.remaining.Store(2)
storeMu.Lock()
previous := store
store = s
storeMu.Unlock()
t.Cleanup(func() {
clearReportTrafficStates()
storeMu.Lock()
store = previous
storeMu.Unlock()
_ = s.Close()
_ = db.Close()
})
return s, fault
}
func TestReportBatchCounterRestoreFailureInitializesStateOnce(t *testing.T) {
s, fault := useReportCounterFailureStore(t)
ctx := context.Background()
base := time.Now().UTC().Truncate(time.Second)
first := v2.Report{
UUID: "counter-restore-failure",
UpdatedAt: base,
CPU: v2.CPUReport{Usage: 10},
Network: v2.NetworkReport{TotalUp: 100, TotalDown: 200},
}
if _, err := writeReportBatch(ctx, []v2.Report{first}); err != nil {
t.Fatalf("write first report after counter restore failure: %v", err)
}
if fault.denied.Load() != 2 {
t.Fatalf("counter restore queries = %d, want 2", fault.denied.Load())
}
stateValue, ok := reportTrafficStates.Load(first.UUID)
if !ok {
t.Fatal("report traffic state was not persisted")
}
state := stateValue.(*reportTrafficState)
state.mu.Lock()
initialized := state.initialized
state.mu.Unlock()
if !initialized {
t.Fatal("report traffic state was not initialized after restore failure")
}
second := first
second.UpdatedAt = base.Add(time.Second)
second.Network.TotalUp = 150
second.Network.TotalDown = 260
if _, err := writeReportBatch(ctx, []v2.Report{second}); err != nil {
t.Fatalf("write second report: %v", err)
}
if fault.denied.Load() != 2 {
t.Fatalf("counter restore queries after second batch = %d, want no repeat", fault.denied.Load())
}
assertMetricValues(t, s, MetricTrafficUp, first.UUID, base.Add(-time.Second), base.Add(2*time.Second), []float64{0, 50})
assertMetricValues(t, s, MetricTrafficDown, first.UUID, base.Add(-time.Second), base.Add(2*time.Second), []float64{0, 60})
}
func TestWriteReportStoresMinuteMetricsAndResetAwareTraffic(t *testing.T) {
ctx := context.Background()
policy := defaultRollupPolicy()
@@ -51,20 +173,20 @@ func TestWriteReportStoresMinuteMetricsAndResetAwareTraffic(t *testing.T) {
base := time.Now().UTC().Truncate(time.Minute).Add(5 * time.Second)
now := base.Add(45 * time.Second)
report := v1.Report{
report := v2.Report{
UUID: "node-a",
UpdatedAt: base,
CPU: v1.CPUReport{Usage: 12.5},
Ram: v1.RamReport{Used: 100, Total: 1000},
Swap: v1.RamReport{Used: 20, Total: 200},
Load: v1.LoadReport{Load1: 0.5},
Disk: v1.DiskReport{Used: 300, Total: 3000},
Network: v1.NetworkReport{Up: 3, Down: 4, TotalUp: 100, TotalDown: 200},
CPU: v2.CPUReport{Usage: 12.5},
Ram: v2.RamReport{Used: 100, Total: 1000},
Swap: v2.RamReport{Used: 20, Total: 200},
Load: v2.LoadReport{Load1: 0.5},
Disk: v2.DiskReport{Used: 300, Total: 3000},
Network: v2.NetworkReport{Up: 3, Down: 4, TotalUp: 100, TotalDown: 200},
Process: 7,
Connections: v1.ConnectionsReport{TCP: 8, UDP: 9},
GPU: &v1.GPUDetailReport{
Connections: v2.ConnectionsReport{TCP: 8, UDP: 9},
GPU: &v2.GPUDetailReport{
AverageUsage: 25,
DetailedInfo: []v1.GPUDeviceInfo{{
DetailedInfo: []v2.GPUDeviceInfo{{
Name: "GPU 0", MemoryUsed: 400, MemoryTotal: 800, Utilization: 30, Temperature: 55,
}},
},
@@ -126,7 +248,7 @@ func TestWriteReportSkipsMetricsWithoutAgentData(t *testing.T) {
ctx := context.Background()
s := useReportTestStore(t, nil)
timestamp := time.Now().UTC()
if _, err := WriteReport(ctx, v1.Report{
if _, err := WriteReport(ctx, v2.Report{
UUID: "node-without-gpu", UpdatedAt: timestamp,
}); err != nil {
t.Fatalf("write report: %v", err)
@@ -154,11 +276,11 @@ func TestReportBatcherFlushesQueuedReports(t *testing.T) {
})
base := time.Now().UTC().Truncate(time.Minute).Add(10 * time.Second)
first := v1.Report{
first := v2.Report{
UUID: "batched-node",
UpdatedAt: base,
CPU: v1.CPUReport{Usage: 10},
Network: v1.NetworkReport{TotalUp: 100, TotalDown: 200},
CPU: v2.CPUReport{Usage: 10},
Network: v2.NetworkReport{TotalUp: 100, TotalDown: 200},
}
second := first
second.UpdatedAt = base.Add(3 * time.Second)
@@ -271,11 +393,11 @@ func TestReportBatchKeepsEverySample(t *testing.T) {
ctx := context.Background()
s := useReportTestStore(t, nil)
base := time.Now().UTC().Truncate(time.Second)
pending := []v1.Report{
{UUID: "node-a", UpdatedAt: base, CPU: v1.CPUReport{Usage: 10}, Network: v1.NetworkReport{TotalUp: 100}},
{UUID: "node-a", UpdatedAt: base, CPU: v1.CPUReport{Usage: 20}, Network: v1.NetworkReport{TotalUp: 150}},
{UUID: "node-b", UpdatedAt: base, CPU: v1.CPUReport{Usage: 30}, Network: v1.NetworkReport{TotalUp: 200}},
{UUID: "node-b", UpdatedAt: base.Add(time.Second), CPU: v1.CPUReport{Usage: 40}, Network: v1.NetworkReport{TotalUp: 260}},
pending := []v2.Report{
{UUID: "node-a", UpdatedAt: base, CPU: v2.CPUReport{Usage: 10}, Network: v2.NetworkReport{TotalUp: 100}},
{UUID: "node-a", UpdatedAt: base, CPU: v2.CPUReport{Usage: 20}, Network: v2.NetworkReport{TotalUp: 150}},
{UUID: "node-b", UpdatedAt: base, CPU: v2.CPUReport{Usage: 30}, Network: v2.NetworkReport{TotalUp: 200}},
{UUID: "node-b", UpdatedAt: base.Add(time.Second), CPU: v2.CPUReport{Usage: 40}, Network: v2.NetworkReport{TotalUp: 260}},
}
if err := writePendingReports(ctx, &pending); err != nil {
@@ -292,11 +414,11 @@ func TestReportQueueFullReturnsError(t *testing.T) {
ctx := context.Background()
useReportTestStore(t, nil)
worker := &reportBatchWorker{
queue: make(chan v1.Report, 1),
queue: make(chan v2.Report, 1),
requests: make(chan reportBatchRequest, 1),
done: make(chan struct{}),
}
worker.queue <- v1.Report{UUID: "already-queued"}
worker.queue <- v2.Report{UUID: "already-queued"}
reportBatcherMu.Lock()
reportBatcher = worker
reportBatcherMu.Unlock()
@@ -308,7 +430,7 @@ func TestReportQueueFullReturnsError(t *testing.T) {
reportBatcherMu.Unlock()
})
report := v1.Report{
report := v2.Report{
UUID: "realtime-node",
UpdatedAt: time.Now().UTC(),
}
@@ -348,6 +470,10 @@ func TestRecordReconstructionUsesMetricSpecificAggregation(t *testing.T) {
}
func TestTrafficCounterDelta(t *testing.T) {
const (
oneGB = int64(1_000_000_000)
oneTB = int64(1_000_000_000_000)
)
tests := []struct {
name string
current int64
@@ -360,6 +486,11 @@ func TestTrafficCounterDelta(t *testing.T) {
{name: "counter reset", current: 15, previous: 250, want: 15},
{name: "negative current", current: -1, previous: 100, want: 0},
{name: "negative previous", current: 15, previous: -1, want: 0},
{name: "tb-scale monotonic", current: 2*oneTB + 5*oneGB, previous: 2 * oneTB, want: 5 * oneGB},
{name: "32-bit wrap leftover", current: 800_000_000, previous: 3*oneGB + 900_000_000, want: 800_000_000},
{name: "reboot leftover below cap", current: 8 * oneGB, previous: 2 * oneTB, want: 8 * oneGB},
{name: "tb-scale tiny dip", current: 2*oneTB - 100, previous: 2 * oneTB, want: 0},
{name: "tb-scale false reset", current: 2*oneTB + 5*oneGB - 1000, previous: 2*oneTB + 5*oneGB, want: 0},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
@@ -370,16 +501,211 @@ func TestTrafficCounterDelta(t *testing.T) {
}
}
func TestWriteReportRebasesTrafficAfterAgentRestart(t *testing.T) {
ctx := context.Background()
s := useReportTestStore(t, nil)
base := time.Now().UTC().Truncate(time.Minute).Add(5 * time.Second)
report := v2.Report{
UUID: "restarted-node",
UpdatedAt: base,
Uptime: 1000,
Network: v2.NetworkReport{TotalUp: 100, TotalDown: 200},
}
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write first report: %v", err)
}
report.UpdatedAt = base.Add(3 * time.Second)
report.Uptime = 1003
report.Network.TotalUp = 150
report.Network.TotalDown = 260
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write continuous report: %v", err)
}
report.UpdatedAt = base.Add(6 * time.Second)
report.Uptime = 1
report.Network.TotalUp = 155
report.Network.TotalDown = 265
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write report after agent restart: %v", err)
}
report.UpdatedAt = base.Add(9 * time.Second)
report.Uptime = 4
report.Network.TotalUp = 180
report.Network.TotalDown = 300
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write report after new baseline: %v", err)
}
assertMetricValues(t, s, MetricTrafficUp, report.UUID, base.Add(-time.Second), base.Add(time.Minute), []float64{0, 50, 5, 25})
assertMetricValues(t, s, MetricTrafficDown, report.UUID, base.Add(-time.Second), base.Add(time.Minute), []float64{0, 60, 5, 35})
}
func TestWriteReportCountsTBScaleTrafficDeltas(t *testing.T) {
const (
oneGB = int64(1_000_000_000)
oneTB = int64(1_000_000_000_000)
)
ctx := context.Background()
s := useReportTestStore(t, nil)
base := time.Now().UTC().Truncate(time.Minute).Add(5 * time.Second)
report := v2.Report{
UUID: "tb-node",
UpdatedAt: base,
Uptime: 10_000,
Network: v2.NetworkReport{TotalUp: 2 * oneTB, TotalDown: 3 * oneTB},
}
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write first report: %v", err)
}
report.UpdatedAt = base.Add(3 * time.Second)
report.Uptime = 10_003
report.Network.TotalUp = 2*oneTB + 5*oneGB
report.Network.TotalDown = 3*oneTB + 7*oneGB
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write growing report: %v", err)
}
assertMetricValues(t, s, MetricTrafficUp, report.UUID, base.Add(-time.Second), base.Add(time.Minute), []float64{0, float64(5 * oneGB)})
assertMetricValues(t, s, MetricTrafficDown, report.UUID, base.Add(-time.Second), base.Add(time.Minute), []float64{0, float64(7 * oneGB)})
}
func TestWriteReportRestoresTBScaleCountersFromStore(t *testing.T) {
const (
oneGB = int64(1_000_000_000)
oneTB = int64(1_000_000_000_000)
)
ctx := context.Background()
policy := defaultRollupPolicy()
s := useReportTestStore(t, &policy)
base := time.Now().UTC().Truncate(time.Minute).Add(5 * time.Second)
now := base.Add(45 * time.Second)
report := v2.Report{
UUID: "tb-restore-node",
UpdatedAt: base,
Uptime: 10_000,
Network: v2.NetworkReport{TotalUp: 2*oneTB + 123, TotalDown: 4*oneTB + 456},
}
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write first report: %v", err)
}
restoredUp, hasUp, err := latestReportCounter(ctx, s, MetricNetTotalUp, report.UUID, now)
if err != nil {
t.Fatalf("restore upload counter: %v", err)
}
if !hasUp || restoredUp != report.Network.TotalUp {
t.Fatalf("restored upload counter = %d (has=%v), want %d", restoredUp, hasUp, report.Network.TotalUp)
}
restoredDown, hasDown, err := latestReportCounter(ctx, s, MetricNetTotalDown, report.UUID, now)
if err != nil {
t.Fatalf("restore download counter: %v", err)
}
if !hasDown || restoredDown != report.Network.TotalDown {
t.Fatalf("restored download counter = %d (has=%v), want %d", restoredDown, hasDown, report.Network.TotalDown)
}
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact reports: %v", err)
}
deleteReportTrafficState(report.UUID)
report.UpdatedAt = now
report.Uptime = 10_045
report.Network.TotalUp = 2*oneTB + 123 + 5*oneGB
report.Network.TotalDown = 4*oneTB + 456 + 7*oneGB
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write after restoring tb-scale baseline: %v", err)
}
assertMetricValues(t, s, MetricTrafficUp, report.UUID, now.Add(-time.Second), now.Add(time.Second), []float64{float64(5 * oneGB)})
assertMetricValues(t, s, MetricTrafficDown, report.UUID, now.Add(-time.Second), now.Add(time.Second), []float64{float64(7 * oneGB)})
}
func TestWriteReportCountsTrafficAfterHighRateCounterWrap(t *testing.T) {
const oneGB = int64(1_000_000_000)
ctx := context.Background()
s := useReportTestStore(t, nil)
base := time.Now().UTC().Truncate(time.Minute).Add(5 * time.Second)
report := v2.Report{
UUID: "wrap-node",
UpdatedAt: base,
Uptime: 10_000,
Network: v2.NetworkReport{TotalUp: 3*oneGB + 900_000_000, TotalDown: 3*oneGB + 800_000_000},
}
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write first report: %v", err)
}
report.UpdatedAt = base.Add(3 * time.Second)
report.Uptime = 10_003
report.Network.TotalUp = 800_000_000
report.Network.TotalDown = 900_000_000
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write wrapped report: %v", err)
}
assertMetricValues(t, s, MetricTrafficUp, report.UUID, base.Add(-time.Second), base.Add(time.Minute), []float64{0, 800_000_000})
assertMetricValues(t, s, MetricTrafficDown, report.UUID, base.Add(-time.Second), base.Add(time.Minute), []float64{0, 900_000_000})
}
func TestWriteReportIgnoresTinyDipOfTBScaleCounter(t *testing.T) {
const (
oneGB = int64(1_000_000_000)
oneTB = int64(1_000_000_000_000)
)
ctx := context.Background()
s := useReportTestStore(t, nil)
base := time.Now().UTC().Truncate(time.Minute).Add(5 * time.Second)
report := v2.Report{
UUID: "tb-jitter-node",
UpdatedAt: base,
Uptime: 10_000,
Network: v2.NetworkReport{TotalUp: 2 * oneTB, TotalDown: 3 * oneTB},
}
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write first report: %v", err)
}
report.UpdatedAt = base.Add(3 * time.Second)
report.Uptime = 10_003
report.Network.TotalUp = 2*oneTB + 5*oneGB
report.Network.TotalDown = 3*oneTB + 7*oneGB
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write growing report: %v", err)
}
report.UpdatedAt = base.Add(6 * time.Second)
report.Uptime = 10_006
report.Network.TotalUp = 2*oneTB + 5*oneGB - 1000
report.Network.TotalDown = 3*oneTB + 7*oneGB - 2000
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write jitter report: %v", err)
}
report.UpdatedAt = base.Add(9 * time.Second)
report.Uptime = 10_009
report.Network.TotalUp = 2*oneTB + 6*oneGB
report.Network.TotalDown = 3*oneTB + 8*oneGB
if _, err := WriteReport(ctx, report); err != nil {
t.Fatalf("write recovered report: %v", err)
}
assertMetricValues(t, s, MetricTrafficUp, report.UUID, base.Add(-time.Second), base.Add(time.Minute), []float64{0, float64(5 * oneGB), 0, float64(oneGB + 1000)})
assertMetricValues(t, s, MetricTrafficDown, report.UUID, base.Add(-time.Second), base.Add(time.Minute), []float64{0, float64(7 * oneGB), 0, float64(oneGB + 2000)})
}
func TestWriteReportNormalizesReceiveTimeToUTC(t *testing.T) {
ctx := context.Background()
s := useReportTestStore(t, nil)
local := time.FixedZone("UTC+8", 8*60*60)
receiveTime := time.Now().In(local).Add(-10 * time.Second)
report := v1.Report{
report := v2.Report{
UUID: "utc-report",
UpdatedAt: receiveTime,
CPU: v1.CPUReport{Usage: 10},
Network: v1.NetworkReport{TotalUp: 1, TotalDown: 2},
CPU: v2.CPUReport{Usage: 10},
Network: v2.NetworkReport{TotalUp: 1, TotalDown: 2},
}
saved, err := WriteReport(ctx, report)
+8 -43
View File
@@ -5,10 +5,10 @@ import (
"errors"
"fmt"
"path/filepath"
"strings"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/internal/managedconfig"
"gorm.io/gorm"
)
@@ -26,7 +26,11 @@ func GetConfiguration(short string) (map[string]any, error) {
if err != nil {
return values, nil // not installed (or unreadable): keep saved values
}
mergeConfigurationDefaults(values, info.Configuration)
items := managedconfig.Items(info.Configuration)
mergeConfigurationDefaults(values, items)
if err := managedconfig.ResolveForOutput(values, items); err != nil {
return nil, fmt.Errorf("resolve plugin configuration: %w", err)
}
return values, nil
}
@@ -48,18 +52,7 @@ func savedConfiguration(short string) (map[string]any, error) {
// mergeConfigurationDefaults fills manifest-declared defaults for keys that
// are missing from the saved values, following the theme merge rules.
func mergeConfigurationDefaults(values map[string]any, configuration models.Configuration) {
if configuration.Type != models.ThemeConfigurationManaged {
return
}
raw, err := json.Marshal(configuration.Data)
if err != nil {
return
}
var items []models.ManagedThemeConfigurationItem
if err := json.Unmarshal(raw, &items); err != nil {
return
}
func mergeConfigurationDefaults(values map[string]any, items []models.ManagedThemeConfigurationItem) {
for _, item := range items {
if item.Key == "" {
continue
@@ -67,38 +60,10 @@ func mergeConfigurationDefaults(values map[string]any, configuration models.Conf
if _, exists := values[item.Key]; exists {
continue
}
values[item.Key] = configurationDefault(item)
values[item.Key] = managedconfig.DefaultValue(item)
}
}
// configurationDefault resolves one item's effective default, mirroring the
// theme rules: select falls back to its first option; nil defaults become
// 0 for number, false for switch and "" otherwise.
func configurationDefault(item models.ManagedThemeConfigurationItem) any {
def := item.Default
if item.Type == "select" {
if def == nil || def == "" {
if item.Options != "" {
opts := strings.Split(item.Options, ",")
if len(opts) > 0 {
return strings.TrimSpace(opts[0])
}
}
}
}
if def == nil {
switch item.Type {
case "number":
return float64(0) // JSON numbers decode as float64
case "switch":
return false
default:
return ""
}
}
return def
}
// SaveConfiguration upserts the configuration values of one plugin.
func SaveConfiguration(short string, data map[string]any) error {
if data == nil {
+54
View File
@@ -14,6 +14,8 @@ import (
"time"
"github.com/gin-gonic/gin"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/pkg/rpc"
)
@@ -398,6 +400,58 @@ func TestGetConfigurationMergesManifestDefaults(t *testing.T) {
t.Errorf("mode still default = %v", values["mode"])
}
}
func TestGetConfigurationResolvesSelectorValues(t *testing.T) {
withTempDataDir(t)
if err := dbcore.GetDBInstance().Create(&models.Client{UUID: "node-a"}).Error; err != nil {
t.Fatal(err)
}
if err := dbcore.GetDBInstance().Create(&models.PingTask{Id: 8}).Error; err != nil {
t.Fatal(err)
}
zipPath := writePluginZip(t, map[string]string{
"komari-plugin.json": `{"name":"Cfg","short":"cfg","version":"1.0.0","configuration":{"type":"managed","data":[
{"key":"nodes","name":"Nodes","type":"nodes"},
{"key":"tasks","name":"Tasks","type":"pingtasks"}
]}}`,
"script.js": `function load() {}`,
})
if _, err := InstallZip(zipPath); err != nil {
t.Fatal(err)
}
if err := SaveConfiguration("cfg", map[string]any{
"nodes": `["node-a","deleted-node"]`,
"tasks": `[8,7]`,
}); err != nil {
t.Fatal(err)
}
values, err := GetConfiguration("cfg")
if err != nil {
t.Fatal(err)
}
nodes, ok := values["nodes"].([]string)
if !ok || len(nodes) != 1 || nodes[0] != "node-a" {
t.Fatalf("nodes = %#v", values["nodes"])
}
tasks, ok := values["tasks"].([]uint)
if !ok || len(tasks) != 1 || tasks[0] != 8 {
t.Fatalf("tasks = %#v", values["tasks"])
}
var stored models.PluginConfiguration
if err := dbcore.GetDBInstance().Where("short = ?", "cfg").First(&stored).Error; err != nil {
t.Fatal(err)
}
var raw map[string]any
if err := json.Unmarshal([]byte(stored.Data), &raw); err != nil {
t.Fatal(err)
}
if raw["nodes"] != `["node-a","deleted-node"]` || raw["tasks"] != `[8,7]` {
t.Fatalf("stored = %#v", raw)
}
}
func TestGetConfigFromScript(t *testing.T) {
withTempDataDir(t)
gin.SetMode(gin.TestMode)
+11 -2
View File
@@ -392,9 +392,18 @@ func headerToMap(header http.Header) map[string]any {
return out
}
// isUpgradeRequest reports whether r is a genuine protocol-upgrade request:
// both Connection: upgrade and an Upgrade header must be present (RFC 9110 §7.8).
func isUpgradeRequest(r *http.Request) bool {
return strings.EqualFold(strings.TrimSpace(r.Header.Get("Connection")), "upgrade") ||
strings.EqualFold(r.Header.Get("Upgrade"), "websocket")
if r.Header.Get("Upgrade") == "" {
return false
}
for _, token := range strings.Split(r.Header.Get("Connection"), ",") {
if strings.EqualFold(strings.TrimSpace(token), "upgrade") {
return true
}
}
return false
}
// bufferedResponseWriter captures status/headers/body so response hooks can
+2 -22
View File
@@ -41,7 +41,7 @@ func validateManifest(info *models.Plugin) error {
if !validShort(info.Short) {
return fmt.Errorf("plugin short %q is invalid: only letters, digits, '_' and '-' are allowed", info.Short)
}
if !localizedText(info.Name) {
if !models.IsLocalizedText(info.Name) {
return fmt.Errorf("plugin name is required")
}
if info.Entry == "" {
@@ -90,7 +90,7 @@ func validatePage(page *models.PluginPage) error {
default:
return fmt.Errorf("plugin page type %q is invalid: use \"iframe\" or \"redirect\"", page.Type)
}
if !localizedText(page.Title) {
if !models.IsLocalizedText(page.Title) {
return fmt.Errorf("plugin page %q requires a title", page.Title)
}
return nil
@@ -113,23 +113,3 @@ func isSafeInternalPath(target string) bool {
}
return true
}
// localizedText reports whether v is a usable name/description/author value:
// a non-empty string, or an i18n object with at least one non-empty value.
func localizedText(v any) bool {
switch t := v.(type) {
case nil:
return false
case string:
return strings.TrimSpace(t) != ""
case map[string]any:
for _, value := range t {
if s, ok := value.(string); ok && strings.TrimSpace(s) != "" {
return true
}
}
return false
default:
return false
}
}
+17
View File
@@ -153,6 +153,13 @@ func SetEnabled(short string, enabled, approved bool) error {
return global.setEnabled(short, enabled, approved)
}
// Reload unloads and loads an enabled plugin. The load hook completes before
// this function returns; a failed load is persisted as the plugin's last
// error and disables the plugin, matching the normal enable/startup path.
func Reload(short string) error {
return global.reload(short)
}
// CloseAll unloads every loaded plugin. Used during server shutdown.
func CloseAll() error {
return global.closeAll()
@@ -521,6 +528,16 @@ func (m *Manager) restartPlugin(short string) error {
return nil
}
func (m *Manager) reload(short string) error {
if !m.stateStore().get(short).Enabled {
return nil
}
if err := m.unload(short); err != nil && !errors.Is(err, errNotLoaded) {
return err
}
return m.restartPlugin(short)
}
// disableWithError persists the auto-disable state for one plugin and wraps
// the load failure for the startup error summary.
func (m *Manager) disableWithError(short string, st PluginState, err error) error {
+20 -1
View File
@@ -3,6 +3,7 @@ package plugin
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
@@ -143,7 +144,12 @@ func (m *Manager) registerServerModule(host *jsruntime.Host, registry *require.R
_ = reject(jsRPCError(vm, resp.Error))
return
}
_ = resolve(resp.Result)
result, err := normalizeRPCValue(resp.Result)
if err != nil {
_ = reject(vm.NewGoError(err))
return
}
_ = resolve(result)
})
}()
return vm.ToValue(promise)
@@ -266,6 +272,19 @@ func jsRPCError(vm *goja.Runtime, e *rpc.JsonRpcError) *goja.Object {
return obj
}
func normalizeRPCValue(value any) (any, error) {
data, err := json.Marshal(value)
if err != nil {
return nil, fmt.Errorf("serialize RPC result: %w", err)
}
var normalized any
if err := json.Unmarshal(data, &normalized); err != nil {
return nil, fmt.Errorf("decode RPC result: %w", err)
}
return normalized, nil
}
// registerRoute registers (or reuses) a gin route slot for one plugin. It is
// called from the plugin's own event loop during script evaluation and takes
// the manager lock itself. The critical section is short and never waits on
+99
View File
@@ -0,0 +1,99 @@
package plugin
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/gin-gonic/gin"
)
// TestIsUpgradeRequest 覆盖升级请求判定:真正的协议升级必须同时携带
// Connection: upgrade 记号与 Upgrade 目标协议头(RFC 9110 §7.8);仅带
// 其中一个头的普通请求(如反代模板给每个请求无条件附加
// Connection: upgrade)不得被视为升级。
func TestIsUpgradeRequest(t *testing.T) {
cases := []struct {
name string
connection string
upgrade string
want bool
}{
{"websocket upgrade with both headers", "upgrade", "websocket", true},
{"case insensitive", "Upgrade", "WebSocket", true},
{"connection list keeps other tokens", "keep-alive, upgrade", "websocket", true},
{"bare connection upgrade is not an upgrade", "upgrade", "", false},
{"upgrade header alone is not an upgrade", "", "websocket", false},
{"plain request", "", "", false},
{"keep-alive only", "keep-alive", "", false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, "/", nil)
if tc.connection != "" {
req.Header.Set("Connection", tc.connection)
}
if tc.upgrade != "" {
req.Header.Set("Upgrade", tc.upgrade)
}
if got := isUpgradeRequest(req); got != tc.want {
t.Fatalf("isUpgradeRequest(Connection=%q, Upgrade=%q) = %v, want %v",
tc.connection, tc.upgrade, got, tc.want)
}
})
}
}
// TestInjectSurvivesBareConnectionUpgradeHeader 回归:部分反代模板(如 1Panel
// 的 OpenResty 站点模板)会给每个代理请求无条件附加 Connection: upgrade。
// 这类普通页面请求必须照常获得插件 HTML 注入;只有同时携带两个头的真实升级
// 才走放行路径。
func TestInjectSurvivesBareConnectionUpgradeHeader(t *testing.T) {
withTempDataDir(t)
gin.SetMode(gin.TestMode)
engine := gin.New()
engine.GET("/admin/dashboard", func(c *gin.Context) {
c.Data(http.StatusOK, "text/html; charset=utf-8",
[]byte("<!DOCTYPE html><html><head></head><body></body></html>"))
})
Init(engine)
zipPath := writePluginZip(t, map[string]string{
"komari-plugin.json": `{"name":"Guard","short":"guard","version":"1.0.0","permissions":{"allowHTMLInject":true,"timeout":5}}`,
"script.js": `
const server = require("server");
function load() {
server.injectHTML("", "<script>window.__guard = 1;</script>");
}
`,
})
if _, err := InstallZip(zipPath); err != nil {
t.Fatal(err)
}
if err := SetEnabled("guard", true, true); err != nil {
t.Fatal(err)
}
defer func() { _ = SetEnabled("guard", false, false) }()
handler := HTMLInjectHandler(engine)
// 仅 Connection: upgrade(反代常见行为)→ 注入照常生效
req := httptest.NewRequest(http.MethodGet, "/admin/dashboard", nil)
req.Header.Set("Connection", "upgrade")
rec := httptest.NewRecorder()
handler.ServeHTTP(rec, req)
if !strings.Contains(rec.Body.String(), "__guard") {
t.Fatalf("bare Connection: upgrade must not skip injection, body = %q", rec.Body.String())
}
// 两个头齐全的真实升级 → 放行,不注入
req2 := httptest.NewRequest(http.MethodGet, "/admin/dashboard", nil)
req2.Header.Set("Connection", "upgrade")
req2.Header.Set("Upgrade", "websocket")
rec2 := httptest.NewRecorder()
handler.ServeHTTP(rec2, req2)
if strings.Contains(rec2.Body.String(), "__guard") {
t.Fatalf("real upgrade must skip injection, body = %q", rec2.Body.String())
}
}
+1 -1
View File
@@ -22,7 +22,7 @@ const (
)
// wsFrameHookTimeout bounds one frame-level callback. The agent read pumps
// run on deadlines (v1: 11s, v2: readWait); a frame hook must never stall
// run on the agent read deadline; a frame hook must never stall
// them, so the wait is cut short and the frame passes through untouched.
const wsFrameHookTimeout = time.Second
+2 -1
View File
@@ -705,7 +705,8 @@ function runtimeInfo() {
| ChildProcess `kill/ref/unref/disconnect/send` | `部分实现` 或调用抛错,见下文 |
支持的 options:`cwd`、`env`、`shell`、`timeout`、`encoding`、`maxBuffer`。
`cwd` 受 `BaseDir` 限制;请求的 timeout 和 maxBuffer 只能收紧 Go 侧全局上限,不能放大。
`cwd` 受 `BaseDir` 限制;子进程默认不继承 runtime 的 `Timeout`,只有显式传入 `timeout` 时才会自动结束。
请求的 `maxBuffer` 只能收紧 Go 侧全局上限,不能放大。
`stdio/input/detached/uid/gid/windowsHide/windowsVerbatimArguments/serialization` 等未实现。
- `spawn()` 的 `stdin` 是 `stream.Writable`(写入经后台写队列串行执行,`end()` 后关闭管道),
+2 -2
View File
@@ -163,7 +163,7 @@ func (m *Module) childCommandArguments(vm *goja.Runtime, call goja.FunctionCall,
}
func (m *Module) childOptions(vm *goja.Runtime, value goja.Value) childCommandOptions {
options := childCommandOptions{cwd: m.filesystem.Cwd(), env: os.Environ(), timeout: m.runtime.Timeout(), maxBuffer: m.maxOutput}
options := childCommandOptions{cwd: m.filesystem.Cwd(), env: os.Environ(), maxBuffer: m.maxOutput}
if value == nil || goja.IsUndefined(value) || goja.IsNull(value) {
return options
}
@@ -190,7 +190,7 @@ func (m *Module) childOptions(vm *goja.Runtime, value goja.Value) childCommandOp
}
if timeoutValue := object.Get("timeout"); timeoutValue != nil && timeoutValue.ToInteger() > 0 {
timeout := timeoutValue.ToInteger()
options.timeout = min(options.timeout, time.Duration(timeout)*time.Millisecond)
options.timeout = time.Duration(timeout) * time.Millisecond
}
if encoding := object.Get("encoding"); encoding != nil && !goja.IsUndefined(encoding) && !goja.IsNull(encoding) {
options.encoding = encoding.String()
+20 -3
View File
@@ -5,6 +5,7 @@ package console
// substitutions. assert(false, ...) and trace() include a stack trace.
import (
"encoding/json"
"fmt"
"io"
"strings"
@@ -121,11 +122,27 @@ func formatConsoleValues(values []goja.Value) string {
}
parts := make([]string, 0, len(args))
for _, arg := range args {
parts = append(parts, fmt.Sprint(arg))
parts = append(parts, formatConsoleValue(arg))
}
return strings.Join(parts, " ")
}
func formatConsoleValue(value any) string {
switch value := value.(type) {
case nil:
return "null"
case string:
return value
case error:
return value.Error()
}
if data, err := json.Marshal(value); err == nil {
return string(data)
}
return fmt.Sprint(value)
}
func formatConsoleString(format string, args []any) string {
var builder strings.Builder
argIndex := 0
@@ -154,7 +171,7 @@ func formatConsoleString(format string, args []any) string {
argIndex++
switch format[i] {
case 's', 'o', 'O':
builder.WriteString(fmt.Sprint(arg))
builder.WriteString(formatConsoleValue(arg))
case 'd', 'i':
builder.WriteString(fmt.Sprintf("%d", toConsoleInteger(arg)))
case 'f':
@@ -168,7 +185,7 @@ func formatConsoleString(format string, args []any) string {
if argIndex < len(args) {
remaining := make([]string, 0, len(args)-argIndex)
for _, arg := range args[argIndex:] {
remaining = append(remaining, fmt.Sprint(arg))
remaining = append(remaining, formatConsoleValue(arg))
}
if builder.Len() > 0 {
builder.WriteByte(' ')
+609
View File
@@ -0,0 +1,609 @@
package metric
import (
"context"
"database/sql"
"fmt"
"sort"
"strings"
"time"
)
type batchSeriesGroupKey struct {
resolution time.Duration
needDigest bool
}
type batchSeriesGroup struct {
key batchSeriesGroupKey
metricNames map[string]struct{}
fields rollupReadFields
}
type seriesIdentity struct {
metricName string
entityID string
tagsHash string
}
type seriesReadMeta struct {
id int64
metricName string
entityID string
tagsHash string
tagsJSON string
tags map[string]string
}
type rollupRateSample struct {
timestamp int64
value float64
}
type rollupAggregateState struct {
summary *rollupBucket
rateSamples []rollupRateSample
tags map[string]string
}
type metricSeriesAccumulator struct {
spec BatchSeriesSpec
compression float64
needDigest bool
needSummary bool
needRate bool
groups map[rollupKey]*rollupAggregateState
rateGroups map[rollupKey]*rollupAggregateState
}
// SeriesBatch is the single rollup query implementation. It resolves metric
// metadata once, scans each selected backing tier once, and streams rows into
// the final output buckets.
func (s *Store) SeriesBatch(ctx context.Context, query BatchSeriesQuery, now time.Time) (BatchSeriesResult, error) {
if err := s.ensureOpen(); err != nil {
return BatchSeriesResult{}, err
}
if err := query.Validate(); err != nil {
return BatchSeriesResult{}, err
}
return s.seriesBatchAt(ctx, query.normalized(), now.UTC(), nil)
}
// GetMetrics loads the requested metric definitions with one database query.
func (s *Store) GetMetrics(ctx context.Context, metricNames []string) (map[string]Definition, error) {
if err := s.ensureOpen(); err != nil {
return nil, err
}
if len(metricNames) == 0 {
return map[string]Definition{}, nil
}
return s.loadBatchDefinitions(ctx, metricNames)
}
func (s *Store) seriesBatchAt(ctx context.Context, query BatchSeriesQuery, now time.Time, forcedResolutions map[string]time.Duration) (BatchSeriesResult, error) {
metricNames := make([]string, 0, len(query.Specs))
for _, spec := range query.Specs {
metricNames = append(metricNames, spec.MetricName)
}
definitions, err := s.loadBatchDefinitions(ctx, metricNames)
if err != nil {
return BatchSeriesResult{}, err
}
result := BatchSeriesResult{
Definitions: definitions,
Values: make(map[string]map[Aggregation][]AggregatePoint, len(query.Specs)),
}
accumulators := make(map[string]*metricSeriesAccumulator, len(query.Specs))
groups := make(map[batchSeriesGroupKey]*batchSeriesGroup)
for _, spec := range query.Specs {
values := make(map[Aggregation][]AggregatePoint, len(spec.Aggregations))
accumulator := &metricSeriesAccumulator{
spec: spec, compression: s.cfg.RollupPolicy.compression(),
groups: make(map[rollupKey]*rollupAggregateState), rateGroups: make(map[rollupKey]*rollupAggregateState),
}
for _, aggregation := range spec.Aggregations {
values[aggregation] = []AggregatePoint{}
if aggregation == AggRate {
accumulator.needRate = true
} else {
accumulator.needSummary = true
}
if isPercentile(aggregation) {
accumulator.needDigest = true
}
}
result.Values[spec.MetricName] = values
accumulators[spec.MetricName] = accumulator
policy := s.cfg.RollupPolicy
if definition, ok := definitions[spec.MetricName]; ok {
if definition.RetentionDays <= 0 {
continue
}
policy = policy.withMetricRetention(time.Duration(definition.RetentionDays) * 24 * time.Hour)
}
if len(policy.Tiers) == 0 {
continue
}
resolution := seriesResolutionForPolicy(query.Start, spec.Interval, now, policy)
if forced, ok := forcedResolutions[spec.MetricName]; ok {
resolution = forced
}
key := batchSeriesGroupKey{resolution: resolution, needDigest: accumulator.needDigest}
group := groups[key]
if group == nil {
group = &batchSeriesGroup{key: key, metricNames: make(map[string]struct{})}
groups[key] = group
}
group.metricNames[spec.MetricName] = struct{}{}
group.fields |= rollupFieldsForAggregations(spec.Aggregations)
}
seriesByID, seriesByIdentity, err := s.loadSeriesDictionary(ctx, seriesDictionaryPlan{
MetricNames: metricNames,
EntityIDs: query.EntityIDs,
Tags: query.Tags,
})
if err != nil {
return BatchSeriesResult{}, err
}
orderedGroups := make([]*batchSeriesGroup, 0, len(groups))
for _, group := range groups {
orderedGroups = append(orderedGroups, group)
}
sort.Slice(orderedGroups, func(i, j int) bool {
if orderedGroups[i].key.resolution != orderedGroups[j].key.resolution {
return orderedGroups[i].key.resolution < orderedGroups[j].key.resolution
}
return !orderedGroups[i].key.needDigest && orderedGroups[j].key.needDigest
})
s.rollupViewMu.RLock()
defer s.rollupViewMu.RUnlock()
for _, group := range orderedGroups {
if err := s.scanPersistedRollupGroup(ctx, query, group, seriesByID, accumulators); err != nil {
return BatchSeriesResult{}, err
}
if err := s.scanMemoryRollupGroup(query, group, seriesByIdentity, accumulators); err != nil {
return BatchSeriesResult{}, err
}
}
for metricName, accumulator := range accumulators {
values, err := accumulator.points(query.Order)
if err != nil {
return BatchSeriesResult{}, err
}
result.Values[metricName] = values
}
return result, nil
}
func (s *Store) loadBatchDefinitions(ctx context.Context, metricNames []string) (map[string]Definition, error) {
args := make([]any, 0, len(metricNames))
placeholders := make([]string, 0, len(metricNames))
for _, metricName := range metricNames {
args = append(args, metricName)
placeholders = append(placeholders, s.dialect.placeholder(len(args)))
}
rows, err := s.reader().QueryContext(ctx, fmt.Sprintf(
"SELECT name, type, unit, description, retention_days, metadata, created_at_milli, updated_at_milli FROM %s WHERE name IN (%s) ORDER BY name ASC",
s.tables.definitions, strings.Join(placeholders, ", "),
), args...)
if err != nil {
return nil, err
}
defer rows.Close()
definitions := make(map[string]Definition, len(metricNames))
for rows.Next() {
definition, err := scanDefinition(rows)
if err != nil {
return nil, err
}
definitions[definition.Name] = definition
}
return definitions, rows.Err()
}
func (s *Store) loadSeriesDictionary(ctx context.Context, plan seriesDictionaryPlan) (map[int64]*seriesReadMeta, map[seriesIdentity]*seriesReadMeta, error) {
rendered := s.dialect.renderSeriesDictionary(s.tables, s.cfg.TablePrefix+"series_metric_entity_idx", plan)
rows, err := s.reader().QueryContext(ctx, rendered.Query, rendered.Args...)
if err != nil {
return nil, nil, err
}
defer rows.Close()
byID := make(map[int64]*seriesReadMeta)
byIdentity := make(map[seriesIdentity]*seriesReadMeta)
for rows.Next() {
var meta seriesReadMeta
var rawTags any
if err := rows.Scan(&meta.id, &meta.metricName, &meta.entityID, &meta.tagsHash, &rawTags); err != nil {
return nil, nil, err
}
meta.tagsJSON, err = rawJSONToString(rawTags)
if err != nil {
return nil, nil, err
}
meta.tags, err = rollupTagsFromJSON(meta.tagsJSON)
if err != nil {
return nil, nil, err
}
byID[meta.id] = &meta
byIdentity[seriesIdentity{metricName: meta.metricName, entityID: meta.entityID, tagsHash: meta.tagsHash}] = &meta
}
return byID, byIdentity, rows.Err()
}
func (s *Store) scanPersistedRollupGroup(ctx context.Context, query BatchSeriesQuery, group *batchSeriesGroup, seriesByID map[int64]*seriesReadMeta, accumulators map[string]*metricSeriesAccumulator) error {
seriesIDs := make([]int64, 0)
for seriesID, meta := range seriesByID {
if _, ok := group.metricNames[meta.metricName]; ok {
seriesIDs = append(seriesIDs, seriesID)
}
}
if len(seriesIDs) == 0 {
return nil
}
sort.Slice(seriesIDs, func(i, j int) bool { return seriesIDs[i] < seriesIDs[j] })
resolutionID, found, err := s.resolveResolutionID(ctx, group.key.resolution)
if err != nil || !found {
return err
}
rendered := s.dialect.renderRollupRead(s.tables, s.cfg.TablePrefix+"rollups_resolution_bucket_idx", rollupReadPlan{
SeriesIDs: seriesIDs,
ResolutionID: resolutionID,
StartMilli: bucketStartMillis(query.Start.UnixMilli(), group.key.resolution.Milliseconds()),
EndMilli: query.End.UnixMilli(),
Fields: group.fields,
})
rows, err := s.reader().QueryContext(ctx, rendered.Query, rendered.Args...)
if err != nil {
return err
}
defer rows.Close()
var seriesID, bucket, count, firstTS, lastTS int64
var sum, sumSq, min, max, firstVal, lastVal float64
var digestBlob []byte
destinations := []any{&seriesID, &bucket, &count}
if group.fields&rollupReadSum != 0 {
destinations = append(destinations, &sum)
}
if group.fields&rollupReadSumSq != 0 {
destinations = append(destinations, &sumSq)
}
if group.fields&rollupReadMin != 0 {
destinations = append(destinations, &min)
}
if group.fields&rollupReadMax != 0 {
destinations = append(destinations, &max)
}
if group.fields&rollupReadFirst != 0 {
destinations = append(destinations, &firstVal, &firstTS)
}
if group.fields&rollupReadLast != 0 {
destinations = append(destinations, &lastVal, &lastTS)
}
if group.fields&rollupReadDigest != 0 {
destinations = append(destinations, &digestBlob)
}
for rows.Next() {
err = rows.Scan(destinations...)
if err != nil {
return err
}
meta := seriesByID[seriesID]
accumulator := accumulators[meta.metricName]
state := accumulator.consume(meta, bucket, count, sum, sumSq, min, max, firstVal, firstTS, lastVal, lastTS, nil)
if accumulator.needDigest {
if err := mergeEncodedRollupDigest(state.summary.digest, count, min, max, digestBlob); err != nil {
return err
}
}
}
return rows.Err()
}
func rollupFieldsForAggregations(aggregations []Aggregation) rollupReadFields {
var fields rollupReadFields
for _, aggregation := range aggregations {
switch aggregation {
case AggAvg, AggSum:
fields |= rollupReadSum
case AggMin:
fields |= rollupReadMin
case AggMax:
fields |= rollupReadMax
case AggFirst:
fields |= rollupReadFirst
case AggLast, AggRate:
fields |= rollupReadLast
case AggStdDev:
fields |= rollupReadSum | rollupReadSumSq
default:
if isPercentile(aggregation) {
fields |= rollupReadMin | rollupReadMax | rollupReadDigest
}
}
}
return fields
}
func (s *Store) resolveResolutionID(ctx context.Context, resolution time.Duration) (int64, bool, error) {
var resolutionID int64
err := s.reader().QueryRowContext(ctx,
"SELECT id FROM "+s.tables.resolutions+" WHERE resolution_milli = "+s.dialect.placeholder(1),
resolution.Milliseconds(),
).Scan(&resolutionID)
if err == sql.ErrNoRows {
return 0, false, nil
}
return resolutionID, err == nil, err
}
func (s *Store) scanMemoryRollupGroup(query BatchSeriesQuery, group *batchSeriesGroup, seriesByIdentity map[seriesIdentity]*seriesReadMeta, accumulators map[string]*metricSeriesAccumulator) error {
if group.key.resolution == time.Minute {
return s.scanHotRollupGroup(query, group, seriesByIdentity, accumulators)
}
return s.scanCoarseRollupGroup(query, group, seriesByIdentity, accumulators)
}
func (s *Store) scanHotRollupGroup(query BatchSeriesQuery, group *batchSeriesGroup, seriesByIdentity map[seriesIdentity]*seriesReadMeta, accumulators map[string]*metricSeriesAccumulator) error {
start := bucketStartMillis(query.Start.UnixMilli(), time.Minute.Milliseconds())
end := query.End.UnixMilli()
entityIDs := stringSet(query.EntityIDs)
s.hotMu.RLock()
defer s.hotMu.RUnlock()
for key, bucket := range s.hot {
if _, ok := group.metricNames[key.metricName]; !ok || key.bucket < start || key.bucket > end || bucket.count == 0 {
continue
}
if len(entityIDs) > 0 {
if _, ok := entityIDs[key.entityID]; !ok {
continue
}
}
meta, matched, err := memorySeriesMeta(seriesByIdentity, key.metricName, key.entityID, key.tagsHash, bucket.tagsJSON, query.Tags)
if err != nil {
return err
}
if !matched {
continue
}
accumulators[key.metricName].consume(meta, key.bucket, bucket.count, bucket.sum, bucket.sumSq, bucket.min, bucket.max, bucket.firstVal, bucket.firstTS, bucket.lastVal, bucket.lastTS, bucket.digest)
}
return nil
}
func (s *Store) scanCoarseRollupGroup(query BatchSeriesQuery, group *batchSeriesGroup, seriesByIdentity map[seriesIdentity]*seriesReadMeta, accumulators map[string]*metricSeriesAccumulator) error {
start := bucketStartMillis(query.Start.UnixMilli(), group.key.resolution.Milliseconds())
end := query.End.UnixMilli()
entityIDs := stringSet(query.EntityIDs)
s.coarseMu.RLock()
defer s.coarseMu.RUnlock()
for key, parent := range s.coarse {
if key.interval != group.key.resolution || key.bucket < start || key.bucket > end {
continue
}
if _, ok := group.metricNames[key.metricName]; !ok {
continue
}
if len(entityIDs) > 0 {
if _, ok := entityIDs[key.entityID]; !ok {
continue
}
}
childKeys := make([]rollupKey, 0, len(parent.children))
for childKey := range parent.children {
childKeys = append(childKeys, childKey)
}
if len(childKeys) == 0 {
continue
}
sortRollupKeys(childKeys)
firstChild := parent.children[childKeys[0]]
meta, matched, err := memorySeriesMeta(seriesByIdentity, key.metricName, key.entityID, key.tagsHash, firstChild.tagsJSON, query.Tags)
if err != nil {
return err
}
if !matched {
continue
}
summary := newRollupBucketWithDigest(s.cfg.RollupPolicy.compression(), group.key.needDigest)
for _, childKey := range childKeys {
child := parent.children[childKey]
mergeRollupValues(summary, child.count, child.sum, child.sumSq, child.min, child.max, child.firstVal, child.firstTS, child.lastVal, child.lastTS, child.digest, group.key.needDigest)
}
accumulators[key.metricName].consume(meta, key.bucket, summary.count, summary.sum, summary.sumSq, summary.min, summary.max, summary.firstVal, summary.firstTS, summary.lastVal, summary.lastTS, summary.digest)
}
return nil
}
func memorySeriesMeta(seriesByIdentity map[seriesIdentity]*seriesReadMeta, metricName, entityID, tagsHash, tagsJSON string, filter map[string]string) (*seriesReadMeta, bool, error) {
identity := seriesIdentity{metricName: metricName, entityID: entityID, tagsHash: tagsHash}
if meta := seriesByIdentity[identity]; meta != nil {
return meta, true, nil
}
tags, err := rollupTagsFromJSON(tagsJSON)
if err != nil {
return nil, false, err
}
for key, value := range filter {
if tags[key] != value {
return nil, false, nil
}
}
meta := &seriesReadMeta{metricName: metricName, entityID: entityID, tagsHash: tagsHash, tagsJSON: tagsJSON, tags: tags}
seriesByIdentity[identity] = meta
return meta, true, nil
}
func stringSet(values []string) map[string]struct{} {
set := make(map[string]struct{}, len(values))
for _, value := range values {
set[value] = struct{}{}
}
return set
}
func (a *metricSeriesAccumulator) consume(meta *seriesReadMeta, sourceBucket, count int64, sum, sumSq, min, max, firstVal float64, firstTS int64, lastVal float64, lastTS int64, digest *TDigest) *rollupAggregateState {
key := rollupKey{bucket: bucketStartMillis(sourceBucket, a.spec.Interval.Milliseconds())}
if a.spec.PreserveSeries {
key.entityID = meta.entityID
key.tagsHash = meta.tagsHash
}
if a.needSummary {
state := a.groups[key]
if state == nil {
state = &rollupAggregateState{summary: newRollupBucketWithDigest(a.compression, a.needDigest)}
if a.spec.PreserveSeries {
state.tags = meta.tags
}
a.groups[key] = state
}
mergeRollupValues(state.summary, count, sum, sumSq, min, max, firstVal, firstTS, lastVal, lastTS, digest, a.needDigest)
if !a.needRate {
return state
}
}
if a.needRate {
rateKey := rollupKey{
bucket: bucketStartMillis(sourceBucket, a.spec.Interval.Milliseconds()),
entityID: meta.entityID,
tagsHash: meta.tagsHash,
}
state := a.rateGroups[rateKey]
if state == nil {
state = &rollupAggregateState{tags: meta.tags}
a.rateGroups[rateKey] = state
}
state.rateSamples = append(state.rateSamples, rollupRateSample{timestamp: lastTS, value: lastVal})
}
return a.groups[key]
}
func mergeEncodedRollupDigest(target *TDigest, count int64, min, max float64, blob []byte) error {
if count == 0 {
return nil
}
if len(blob) == 0 {
if min != max {
return fmt.Errorf("metric: missing t-digest for rollup with %d samples", count)
}
target.Add(min, float64(count))
return nil
}
digest, err := DecodeTDigest(blob)
if err != nil {
return err
}
target.Merge(digest)
return nil
}
func mergeRollupValues(target *rollupBucket, count int64, sum, sumSq, min, max, firstVal float64, firstTS int64, lastVal float64, lastTS int64, digest *TDigest, includeDigest bool) {
if count == 0 {
return
}
if target.count == 0 {
target.min, target.max = min, max
target.firstVal, target.firstTS = firstVal, firstTS
target.lastVal, target.lastTS = lastVal, lastTS
} else {
if min < target.min {
target.min = min
}
if max > target.max {
target.max = max
}
if firstTS < target.firstTS {
target.firstVal, target.firstTS = firstVal, firstTS
}
if lastTS > target.lastTS {
target.lastVal, target.lastTS = lastVal, lastTS
}
}
target.count += count
target.sum += sum
target.sumSq += sumSq
if includeDigest && digest != nil {
target.digest.Merge(digest)
}
}
func (a *metricSeriesAccumulator) points(order Order) (map[Aggregation][]AggregatePoint, error) {
values := make(map[Aggregation][]AggregatePoint, len(a.spec.Aggregations))
var summaryKeys []rollupKey
if a.needSummary {
summaryKeys = make([]rollupKey, 0, len(a.groups))
for key := range a.groups {
summaryKeys = append(summaryKeys, key)
}
sortRollupKeys(summaryKeys)
}
var rateKeys []rollupKey
if a.needRate {
rateKeys = make([]rollupKey, 0, len(a.rateGroups))
for key, state := range a.rateGroups {
rateKeys = append(rateKeys, key)
sort.SliceStable(state.rateSamples, func(i, j int) bool { return state.rateSamples[i].timestamp < state.rateSamples[j].timestamp })
}
sortRollupKeys(rateKeys)
}
for _, aggregation := range a.spec.Aggregations {
keys := summaryKeys
if aggregation == AggRate {
keys = rateKeys
}
start, end, step := 0, len(keys), 1
if order == OrderDesc {
start, end, step = len(keys)-1, -1, -1
}
points := make([]AggregatePoint, 0, len(keys))
for keyIndex := start; keyIndex != end; keyIndex += step {
key := keys[keyIndex]
state := a.groups[key]
count, value := 0, 0.0
if aggregation == AggRate {
state = a.rateGroups[key]
value = rollupCounterRate(state.rateSamples)
count = len(state.rateSamples)
} else {
var ok bool
value, ok = state.summary.value(aggregation)
if !ok {
return nil, fmt.Errorf("%w: aggregation %q requires raw samples", ErrInvalidArgument, aggregation)
}
count = int(state.summary.count)
}
points = append(points, AggregatePoint{
MetricName: a.spec.MetricName,
EntityID: key.entityID,
Bucket: fromMillis(key.bucket),
Value: value,
Count: count,
Tags: state.tags,
})
}
values[aggregation] = points
}
return values, nil
}
func rollupCounterRate(samples []rollupRateSample) float64 {
if len(samples) < 2 {
return 0
}
seconds := float64(samples[len(samples)-1].timestamp-samples[0].timestamp) / 1000
if seconds <= 0 {
return 0
}
increase := 0.0
for i := 1; i < len(samples); i++ {
if delta := samples[i].value - samples[i-1].value; delta > 0 {
increase += delta
}
}
return increase / seconds
}
+6
View File
@@ -104,6 +104,12 @@ func (s *Store) addCoarseChildren(metricName string, interval time.Duration, hig
// late-arrival window. It flushes one tier at a time so a sealed child can feed
// the next parent without any periodic rewrite of that parent.
func (s *Store) flushClosedCoarseRollups(ctx context.Context, now time.Time) (int, error) {
s.rollupViewMu.Lock()
defer s.rollupViewMu.Unlock()
return s.flushClosedCoarseRollupsUnderView(ctx, now)
}
func (s *Store) flushClosedCoarseRollupsUnderView(ctx context.Context, now time.Time) (int, error) {
now = now.UTC()
written := 0
for {
+2 -62
View File
@@ -35,6 +35,8 @@ type dialect interface {
//
// blobType 是 rollup t-digest sketch 的列类型。
blobType() string
renderSeriesDictionary(tables tables, indexName string, plan seriesDictionaryPlan) renderedSQL
renderRollupRead(tables tables, indexName string, plan rollupReadPlan) renderedSQL
}
// tables stores the physical table names used by a Store.
@@ -287,65 +289,3 @@ func (postgresDialect) jsonExtractEquals(column, key, placeholder string) string
// escaping.
return fmt.Sprintf("%s->>'%s' = %s", column, sqlSingleQuote(key), placeholder)
}
// sqlAggValueExpr returns the SQL value expression that computes agg over the
// "value" column for the given driver, and whether that aggregation can be
// pushed down to this backend. Aggregations that need the ordered raw series
// (first/last/rate) are never pushed down here. Percentiles and population
// standard deviation are pushed down only where the backend has a portable,
// semantics-matching function:
//
// - STDDEV_POP: MySQL and PostgreSQL (matches the in-memory population stddev,
// which divides by N). SQLite has no built-in, so it falls back to memory.
// - percentile_cont(p) WITHIN GROUP (ORDER BY value): PostgreSQL only. This
// matches the in-memory linear-interpolation percentile for ANY p (p50,
// p95, p99, p99.9, ...). MySQL lacks a portable continuous-percentile
// function and SQLite has none, so both fall back to memory.
//
// The simple reductions (avg/min/max/sum/count) are portable everywhere.
//
// sqlAggValueExpr 返回给定驱动上对 "value" 列计算 agg 的 SQL 值表达式,
// 以及该聚合是否能下推到该后端。需要有序原始序列的聚合(first/last/rate)
// 永远不会在这里下推。百分位和总体标准差只会在后端提供可移植且语义匹配的函数时
// 下推:
//
// - STDDEV_POP:MySQL 和 PostgreSQL 支持(匹配内存中的总体标准差,
// 即除以 N)。SQLite 没有内置函数,因此回退到内存计算。
// - percentile_cont(p) WITHIN GROUP (ORDER BY value):仅 PostgreSQL 支持。
// 它匹配内存中任意 p(p50、p95、p99、p99.9 等)的线性插值百分位。
// MySQL 没有可移植的连续百分位函数,SQLite 也没有,因此两者都回退到内存计算。
//
// 简单归约(avg/min/max/sum/count)在所有后端都可移植。
func sqlAggValueExpr(driver Driver, agg Aggregation) (string, bool) {
switch agg {
case AggAvg:
return "AVG(value)", true
case AggMin:
return "MIN(value)", true
case AggMax:
return "MAX(value)", true
case AggSum:
return "SUM(value)", true
case AggCount:
return "COUNT(*)", true
case AggStdDev:
switch driver {
case DriverMySQL, DriverPostgreSQL:
return "STDDEV_POP(value)", true
default:
return "", false
}
default:
// Any percentile (fixed p50/p95/p99 and arbitrary pXX / pXX.X all land
// here). PostgreSQL has a portable continuous percentile that matches the
// in-memory linear-interpolation percentile; other backends fall back to
// the in-memory path.
if frac, ok := percentileFractionString(agg); ok {
if driver == DriverPostgreSQL {
return fmt.Sprintf("percentile_cont(%s) WITHIN GROUP (ORDER BY value)", frac), true
}
return "", false
}
return "", false
}
}
+24 -3
View File
@@ -60,7 +60,11 @@ func (s *Store) writePreparedHotRollups(ctx context.Context, prepared []prepared
if err != nil {
return err
}
_, err = s.flushClosedCoarseRollups(ctx, now)
if len(rebuild) > 0 {
_, err = s.flushClosedCoarseRollupsUnderView(ctx, now)
} else {
_, err = s.flushClosedCoarseRollups(ctx, now)
}
return err
}
@@ -78,8 +82,25 @@ func (s *Store) rebuildHotRollupsLocked(keys map[hotRollupKey]struct{}, compress
if labelID, ok := series.labelIDs[key.labelsHash]; ok {
bucket.labelsJSON = series.labelsJSON[labelID]
}
for _, sample := range series.allSamples() {
if bucketStartMillis(sample.timestamp, minuteMillis) != key.bucket || series.labelHashes[sample.labelID] != key.labelsHash {
bucketEnd := key.bucket + minuteMillis
decoder := newRawSampleDecoder(series.compressed)
for sample, more := decoder.next(); more; sample, more = decoder.next() {
if sample.timestamp < key.bucket {
continue
}
if sample.timestamp >= bucketEnd {
break
}
if series.labelHashes[sample.labelID] != key.labelsHash {
continue
}
bucket.labelsJSON = series.labelsJSON[sample.labelID]
bucket.addPoint(sample.value, sample.timestamp)
}
directStart := sort.Search(len(series.samples), func(i int) bool { return series.samples[i].timestamp >= key.bucket })
directEnd := sort.Search(len(series.samples), func(i int) bool { return series.samples[i].timestamp >= bucketEnd })
for _, sample := range series.samples[directStart:directEnd] {
if series.labelHashes[sample.labelID] != key.labelsHash {
continue
}
bucket.labelsJSON = series.labelsJSON[sample.labelID]
+32 -17
View File
@@ -634,7 +634,7 @@ func TestSeriesMergesLabelsWithoutCollapsingRawShape(t *testing.T) {
}
}
func TestCoveringSeriesResolutionUsesFinestTierCoveringStart(t *testing.T) {
func TestSeriesResolutionDoesNotInspectStoredCoverage(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
Tiers: []RollupTier{
@@ -686,16 +686,13 @@ func TestCoveringSeriesResolutionUsesFinestTierCoveringStart(t *testing.T) {
t.Fatalf("commit seed tiers: %v", err)
}
resolution, err := s.coveringSeriesResolution(ctx, Query{MetricName: "coverage-tier", EntityID: "n1", Start: now.Add(-6 * time.Hour), End: now}, time.Minute)
if err != nil {
t.Fatalf("select covering tier: %v", err)
}
if resolution != 5*time.Minute {
t.Fatalf("covering tier = %s, want 5m", resolution)
resolution := seriesResolutionForPolicy(now.Add(-6*time.Hour), time.Minute, now, policy)
if resolution != time.Minute {
t.Fatalf("selected tier = %s, want 1m", resolution)
}
}
func TestSeriesFallsBackWhenPreferredPolicyTierHasNoRows(t *testing.T) {
func TestSeriesReturnsEmptyWhenSelectedTierHasNoRows(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
Tiers: []RollupTier{
@@ -743,8 +740,8 @@ func TestSeriesFallsBackWhenPreferredPolicyTierHasNoRows(t *testing.T) {
if err != nil {
t.Fatalf("query series: %v", err)
}
if len(got) != 1 || got[0].Value != 42 || got[0].Count != 1 {
t.Fatalf("series did not fall back to existing hourly tier: %#v", got)
if len(got) != 0 {
t.Fatalf("series read a non-selected hourly tier: %#v", got)
}
}
@@ -931,15 +928,33 @@ func TestSeriesHasNoTrailingEmptyBucketAcrossDashboardRanges(t *testing.T) {
t.Fatalf("create metric: %v", err)
}
now := time.Date(2026, 7, 24, 12, 0, 0, 0, time.UTC)
points := make([]Point, 0, 60*24*60)
for at := now.Add(-60 * 24 * time.Hour); at.Before(now); at = at.Add(time.Minute) {
points = append(points, Point{MetricName: "coverage", EntityID: "n1", Timestamp: at, Value: 1})
tagsHash, tagsJSON, err := tagsFingerprint(nil)
if err != nil {
t.Fatalf("fingerprint tags: %v", err)
}
if err := s.WriteBatch(ctx, points); err != nil {
t.Fatalf("write history: %v", err)
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
t.Fatalf("begin history transaction: %v", err)
}
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact history: %v", err)
metricPolicy := policy.withMetricRetention(90 * 24 * time.Hour)
for _, tierIndex := range []int{0, 2, 3} {
tier := metricPolicy.Tiers[tierIndex]
buckets := make(map[rollupKey]*rollupBucket)
start := bucketStartMillis(now.Add(-tier.Retention).UnixMilli(), tier.Interval.Milliseconds())
for bucketStart := start; bucketStart < now.UnixMilli(); bucketStart += tier.Interval.Milliseconds() {
bucket := newRollupBucket(policy.compression())
bucket.tagsHash, bucket.tagsJSON = tagsHash, tagsJSON
bucket.labelsHash, bucket.labelsJSON = emptyLabelsHash, "{}"
bucket.addPoint(1, bucketStart)
buckets[rollupKey{entityID: "n1", tagsHash: tagsHash, labelsHash: emptyLabelsHash, bucket: bucketStart}] = bucket
}
if _, err := s.writeRollupBucketsTx(ctx, "coverage", tier.Interval, buckets, tx); err != nil {
_ = tx.Rollback()
t.Fatalf("seed %s history: %v", tier.Interval, err)
}
}
if err := tx.Commit(); err != nil {
t.Fatalf("commit history: %v", err)
}
for _, duration := range []time.Duration{time.Hour, 6 * time.Hour, 7 * 24 * time.Hour, 30 * 24 * time.Hour, 60 * 24 * time.Hour} {
interval := s.CompatibleSeriesInterval(now.Add(-duration), now, time.Minute)
+33 -64
View File
@@ -104,10 +104,11 @@ func TestTagFilterPushdownWithPaging(t *testing.T) {
}
}
// TestSQLAggregateMatchesInMemory compares SQL and in-memory aggregation.
// TestAggregateMatchesAggregatePoints compares Store aggregation with the
// standalone aggregation helper.
//
// TestSQLAggregateMatchesInMemory 对比 SQL 下推聚合和内存聚合结果。
func TestSQLAggregateMatchesInMemory(t *testing.T) {
// TestAggregateMatchesAggregatePoints 对比 Store 聚合和独立聚合 helper 的结果。
func TestAggregateMatchesAggregatePoints(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "g", Type: TypeGauge, RetentionDays: 30}); err != nil {
@@ -131,10 +132,9 @@ func TestSQLAggregateMatchesInMemory(t *testing.T) {
Aggregation: AggAvg,
Interval: 15 * time.Second,
}
// SQL pushdown path.
sqlRes, err := s.Aggregate(ctx, q)
storeRes, err := s.Aggregate(ctx, q)
if err != nil {
t.Fatalf("sql aggregate: %v", err)
t.Fatalf("store aggregate: %v", err)
}
// In-memory reference over the same raw points.
pts, err := s.Query(ctx, q.Query)
@@ -145,12 +145,12 @@ func TestSQLAggregateMatchesInMemory(t *testing.T) {
if err != nil {
t.Fatalf("mem aggregate: %v", err)
}
if len(sqlRes) != len(memRes) {
t.Fatalf("bucket count mismatch: sql=%d mem=%d", len(sqlRes), len(memRes))
if len(storeRes) != len(memRes) {
t.Fatalf("bucket count mismatch: store=%d helper=%d", len(storeRes), len(memRes))
}
for i := range sqlRes {
if !sqlRes[i].Bucket.Equal(memRes[i].Bucket) || sqlRes[i].Value != memRes[i].Value || sqlRes[i].Count != memRes[i].Count {
t.Fatalf("bucket %d mismatch: sql=%#v mem=%#v", i, sqlRes[i], memRes[i])
for i := range storeRes {
if !storeRes[i].Bucket.Equal(memRes[i].Bucket) || storeRes[i].Value != memRes[i].Value || storeRes[i].Count != memRes[i].Count {
t.Fatalf("bucket %d mismatch: store=%#v helper=%#v", i, storeRes[i], memRes[i])
}
}
}
@@ -260,10 +260,10 @@ func TestStdDevPopMatchesCalculateStats(t *testing.T) {
}
}
// TestAggregateStdDevSQLiteUsesMemoryPath verifies SQLite stddev fallback.
// TestAggregateStdDev verifies raw population standard deviation aggregation.
//
// TestAggregateStdDevSQLiteUsesMemoryPath 验证 SQLite 标准差聚合会回退到内存路径。
func TestAggregateStdDevSQLiteUsesMemoryPath(t *testing.T) {
// TestAggregateStdDev 验证原始样本总体标准差聚合。
func TestAggregateStdDev(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "sd", Type: TypeGauge, RetentionDays: 30}); err != nil {
@@ -294,36 +294,6 @@ func TestAggregateStdDevSQLiteUsesMemoryPath(t *testing.T) {
}
}
// TestSQLAggValueExprPushdownMatrix verifies aggregation pushdown support.
//
// TestSQLAggValueExprPushdownMatrix 验证各后端支持的聚合下推矩阵。
func TestSQLAggValueExprPushdownMatrix(t *testing.T) {
cases := []struct {
driver Driver
agg Aggregation
ok bool
}{
{DriverSQLite, AggAvg, true},
{DriverSQLite, AggStdDev, false}, // no STDDEV_POP in sqlite -> memory
{DriverSQLite, AggP95, false}, // no percentile in sqlite -> memory
{DriverMySQL, AggStdDev, true}, // STDDEV_POP
{DriverMySQL, AggP95, false}, // no portable continuous percentile
{DriverPostgreSQL, AggStdDev, true},
{DriverPostgreSQL, AggP50, true}, // percentile_cont
{DriverPostgreSQL, AggP99, true},
{DriverPostgreSQL, AggFirst, false}, // needs ordered raw series
}
for _, c := range cases {
expr, ok := sqlAggValueExpr(c.driver, c.agg)
if ok != c.ok {
t.Fatalf("%s/%s: pushdown=%v want %v (expr=%q)", c.driver, c.agg, ok, c.ok, expr)
}
if ok && expr == "" {
t.Fatalf("%s/%s: pushdown ok but empty expr", c.driver, c.agg)
}
}
}
// TestSQLiteReadPoolOpens verifies SQLite read-pool creation.
//
// TestSQLiteReadPoolOpens 验证 SQLite 只读连接池会按配置打开。
@@ -403,10 +373,10 @@ func TestWriteBatchAtomicAcrossChunks(t *testing.T) {
}
}
// TestAggregateBucketPagingSQLPath verifies bucket paging in SQL aggregation.
// TestAggregateBucketPaging verifies bucket paging.
//
// TestAggregateBucketPagingSQLPath 验证 SQL 聚合路径按桶分页。
func TestAggregateBucketPagingSQLPath(t *testing.T) {
// TestAggregateBucketPaging 验证聚合桶分页。
func TestAggregateBucketPaging(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "bp", Type: TypeGauge, RetentionDays: 30}); err != nil {
@@ -441,10 +411,11 @@ func TestAggregateBucketPagingSQLPath(t *testing.T) {
}
}
// TestAggregateBucketPagingMemoryPathMatchesSQL compares bucket paging paths.
// TestAggregateBucketPagingMatchesAcrossAggregations compares bucket paging
// across aggregation types.
//
// TestAggregateBucketPagingMemoryPathMatchesSQL 对比内存聚合和 SQL 聚合的桶分页语义。
func TestAggregateBucketPagingMemoryPathMatchesSQL(t *testing.T) {
// TestAggregateBucketPagingMatchesAcrossAggregations 对比不同聚合类型的桶分页语义。
func TestAggregateBucketPagingMatchesAcrossAggregations(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "bp2", Type: TypeGauge, RetentionDays: 30}); err != nil {
@@ -459,28 +430,26 @@ func TestAggregateBucketPagingMemoryPathMatchesSQL(t *testing.T) {
t.Fatalf("write: %v", err)
}
q := Query{MetricName: "bp2", EntityID: "n1", Start: base, End: base.Add(time.Minute)}
// AggMin (SQL pushdown) and AggP50 (memory fallback on sqlite) with identical
// bucket paging must select the same bucket window (the same timestamps).
sqlPaged, err := s.Aggregate(ctx, AggregateQuery{Query: q, Aggregation: AggMin, Interval: 10 * time.Second, BucketLimit: 3, BucketOffset: 2})
minPaged, err := s.Aggregate(ctx, AggregateQuery{Query: q, Aggregation: AggMin, Interval: 10 * time.Second, BucketLimit: 3, BucketOffset: 2})
if err != nil {
t.Fatalf("sql aggregate: %v", err)
t.Fatalf("min aggregate: %v", err)
}
memPaged, err := s.Aggregate(ctx, AggregateQuery{Query: q, Aggregation: AggP50, Interval: 10 * time.Second, BucketLimit: 3, BucketOffset: 2})
p50Paged, err := s.Aggregate(ctx, AggregateQuery{Query: q, Aggregation: AggP50, Interval: 10 * time.Second, BucketLimit: 3, BucketOffset: 2})
if err != nil {
t.Fatalf("mem aggregate: %v", err)
t.Fatalf("p50 aggregate: %v", err)
}
if len(sqlPaged) != 3 || len(memPaged) != 3 {
t.Fatalf("expected 3 buckets each, got sql=%d mem=%d", len(sqlPaged), len(memPaged))
if len(minPaged) != 3 || len(p50Paged) != 3 {
t.Fatalf("expected 3 buckets each, got min=%d p50=%d", len(minPaged), len(p50Paged))
}
for i := range sqlPaged {
if !sqlPaged[i].Bucket.Equal(memPaged[i].Bucket) {
t.Fatalf("bucket window mismatch at %d: sql=%v mem=%v", i, sqlPaged[i].Bucket, memPaged[i].Bucket)
for i := range minPaged {
if !minPaged[i].Bucket.Equal(p50Paged[i].Bucket) {
t.Fatalf("bucket window mismatch at %d: min=%v p50=%v", i, minPaged[i].Bucket, p50Paged[i].Bucket)
}
}
// Per-bucket single point, so min == p50 == the value.
for i := range sqlPaged {
if sqlPaged[i].Value != memPaged[i].Value {
t.Fatalf("value mismatch at %d: sql=%v mem=%v", i, sqlPaged[i].Value, memPaged[i].Value)
for i := range minPaged {
if minPaged[i].Value != p50Paged[i].Value {
t.Fatalf("value mismatch at %d: min=%v p50=%v", i, minPaged[i].Value, p50Paged[i].Value)
}
}
}
+3
View File
@@ -125,6 +125,9 @@ func (s *Store) ReclaimSpace(ctx context.Context) error {
if _, err := s.db.ExecContext(ctx, sqliteVacuumSQL); err != nil {
return fmt.Errorf("metric: vacuum sqlite database: %w", err)
}
if _, err := s.db.ExecContext(ctx, "PRAGMA optimize"); err != nil {
return fmt.Errorf("metric: optimize sqlite database: %w", err)
}
// VACUUM itself can populate the WAL; truncate it again so the reported
// physical size reflects the completed reclamation.
return sqliteCheckpoint(ctx, s.db)
+8 -1
View File
@@ -21,7 +21,14 @@ func (s *Store) Migrate(ctx context.Context) error {
return err
}
}
return s.createNormalizedIndexes(ctx)
if err := s.createNormalizedIndexes(ctx); err != nil {
return err
}
if s.cfg.Driver == DriverSQLite {
_, err := s.db.ExecContext(ctx, "PRAGMA optimize")
return err
}
return nil
}
func (s *Store) normalizedSchemaStatements() []string {
+13 -7
View File
@@ -18,7 +18,7 @@ func TestPostgreSQLIntegration(t *testing.T) {
t.Skip("METRIC_POSTGRES_DSN is not set")
}
runSQLIntegration(t, "postgres", PostgreSQL(dsn), true)
runSQLIntegration(t, "postgres", PostgreSQL(dsn))
runSQLRestructureIntegration(t, "postgres", PostgreSQL(dsn))
}
@@ -31,14 +31,24 @@ func TestMySQLIntegration(t *testing.T) {
t.Skip("METRIC_MYSQL_DSN is not set")
}
runSQLIntegration(t, "mysql", MySQL(dsn), false)
runSQLIntegration(t, "mysql", MySQL(dsn))
runSQLRestructureIntegration(t, "mysql", MySQL(dsn))
}
func TestMariaDBIntegration(t *testing.T) {
dsn := os.Getenv("METRIC_MARIADB_DSN")
if dsn == "" {
t.Skip("METRIC_MARIADB_DSN is not set")
}
runSQLIntegration(t, "mariadb", MySQL(dsn))
runSQLRestructureIntegration(t, "mariadb", MySQL(dsn))
}
// runSQLIntegration exercises the SQL store against an external database.
//
// runSQLIntegration 在外部数据库上执行通用 SQL 集成测试流程。
func runSQLIntegration(t *testing.T, name string, cfg Config, expectSQLPercentile bool) {
func runSQLIntegration(t *testing.T, name string, cfg Config) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 6*time.Minute)
@@ -160,10 +170,6 @@ func runSQLIntegration(t *testing.T, name string, cfg Config, expectSQLPercentil
if len(p95Buckets) != 1 || p95Buckets[0].Count != 3 || p95Buckets[0].Value <= 28 {
t.Fatalf("unexpected p95 aggregate: %#v", p95Buckets)
}
if _, ok := sqlAggValueExpr(cfg.Driver, AggP95); ok != expectSQLPercentile {
t.Fatalf("%s p95 pushdown expectation mismatch: got %v want %v", name, ok, expectSQLPercentile)
}
latest, err := store.Latest(ctx, "http.latency", "api-1", 1)
if err != nil {
t.Fatalf("latest: %v", err)
+308 -173
View File
@@ -38,6 +38,61 @@ type compressedRawSamples struct {
lastStamp int64
}
type rawSampleDecoder struct {
data []byte
offset int
index int
count int
previous int64
}
func newRawSampleDecoder(samples compressedRawSamples) rawSampleDecoder {
return rawSampleDecoder{data: samples.data, count: samples.count}
}
func (d *rawSampleDecoder) next() (rawSample, bool) {
if d.index == d.count {
return rawSample{}, false
}
var timestamp int64
if d.index == 0 {
value, size := binary.Varint(d.data[d.offset:])
d.offset += size
timestamp = value
} else {
delta, size := binary.Uvarint(d.data[d.offset:])
d.offset += size
timestamp = d.previous + int64(delta)
}
value := math.Float64frombits(binary.LittleEndian.Uint64(d.data[d.offset:]))
d.offset += 8
labelID, size := binary.Uvarint(d.data[d.offset:])
d.offset += size
d.index++
d.previous = timestamp
return rawSample{timestamp: timestamp, value: value, labelID: uint32(labelID)}, true
}
func newCompressedRawSamples(capacity int) compressedRawSamples {
return compressedRawSamples{data: make([]byte, 0, capacity)}
}
func (c *compressedRawSamples) appendSample(sample rawSample) {
if c.count == 0 {
c.data = appendVarint(c.data, sample.timestamp)
} else {
c.data = appendUvarint(c.data, uint64(sample.timestamp-c.lastStamp))
}
c.data = appendRawSample(c.data, sample)
c.count++
c.lastStamp = sample.timestamp
}
func (c compressedRawSamples) firstStamp() int64 {
value, _ := binary.Varint(c.data)
return value
}
// rawSeries stores exact samples compactly for one logical series. Tags are
// shared by the series and labels are interned into small integer ids.
type rawSeries struct {
@@ -170,31 +225,98 @@ func markRawRebuild(rebuild map[hotRollupKey]struct{}, point preparedMetricPoint
rebuild[hotRollupKey{metricName: point.metricName, entityID: point.entityID, tagsHash: point.tagsHash, labelsHash: labelsHash, bucket: bucketStartMillis(point.timestamp, time.Minute.Milliseconds())}] = struct{}{}
}
type rawBatchPoints struct {
points []Point
tagsHashes []string
order Order
}
func (p *rawBatchPoints) Len() int { return len(p.points) }
func (p *rawBatchPoints) Less(i, j int) bool {
if p.order == OrderDesc {
return rawPointLess(p.points[j], p.tagsHashes[j], p.points[i], p.tagsHashes[i])
}
return rawPointLess(p.points[i], p.tagsHashes[i], p.points[j], p.tagsHashes[j])
}
func (p *rawBatchPoints) Swap(i, j int) {
p.points[i], p.points[j] = p.points[j], p.points[i]
p.tagsHashes[i], p.tagsHashes[j] = p.tagsHashes[j], p.tagsHashes[i]
}
func (p *rawBatchPoints) grow(additional int) {
if additional <= cap(p.points)-len(p.points) {
return
}
pointCount := len(p.points) + additional
points := make([]Point, len(p.points), pointCount)
copy(points, p.points)
p.points = points
tagsHashes := make([]string, len(p.tagsHashes), pointCount)
copy(tagsHashes, p.tagsHashes)
p.tagsHashes = tagsHashes
}
func (s *Store) queryRawPoints(ctx context.Context, query Query) ([]Point, error) {
entityIDs := []string(nil)
if query.EntityID != "" {
entityIDs = []string{query.EntityID}
}
batch, err := s.queryRawPointsBatch(ctx, BatchQuery{
MetricNames: []string{query.MetricName},
EntityIDs: entityIDs,
Start: query.Start,
End: query.End,
Tags: query.Tags,
Order: query.Order,
})
if err != nil {
return nil, err
}
points := batch[query.MetricName]
startIndex := query.Offset
if startIndex > len(points) {
startIndex = len(points)
}
endIndex := len(points)
if query.Limit > 0 && startIndex+query.Limit < endIndex {
endIndex = startIndex + query.Limit
}
return points[startIndex:endIndex], nil
}
func (s *Store) queryRawPointsBatch(ctx context.Context, query BatchQuery) (map[string][]Point, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
start, end := query.Start.UnixMilli(), query.End.UnixMilli()
s.rawMu.Lock()
defer s.rawMu.Unlock()
now := time.Now().UTC()
visibleCutoff := now.Add(-rawMemoryRetention).UnixMilli()
visibleCutoff := time.Now().UTC().Add(-rawMemoryRetention).UnixMilli()
if start < visibleCutoff {
start = visibleCutoff
}
s.compressRawBeforeLocked("", now.Add(-directRawRetention).UnixMilli())
s.pruneRawBeforeLocked("", rawMemoryCutoff(now))
type resultPoint struct {
point Point
entityID string
tagsHash string
metricNames := make(map[string]struct{}, len(query.MetricNames))
matched := make(map[string]*rawBatchPoints, len(query.MetricNames))
for _, metricName := range query.MetricNames {
metricNames[metricName] = struct{}{}
matched[metricName] = &rawBatchPoints{order: query.Order}
}
matched := make([]resultPoint, 0)
entityIDs := make(map[string]struct{}, len(query.EntityIDs))
for _, entityID := range query.EntityIDs {
entityIDs[entityID] = struct{}{}
}
s.rawMu.RLock()
defer s.rawMu.RUnlock()
for key, series := range s.raw {
if key.metricName != query.MetricName || (query.EntityID != "" && key.entityID != query.EntityID) {
if _, ok := metricNames[key.metricName]; !ok {
continue
}
if len(entityIDs) > 0 {
if _, ok := entityIDs[key.entityID]; !ok {
continue
}
}
tags, ok, err := matchRawTags(series.tagsJSON, query.Tags)
if err != nil {
return nil, err
@@ -202,68 +324,88 @@ func (s *Store) queryRawPoints(ctx context.Context, query Query) ([]Point, error
if !ok {
continue
}
all := series.allSamples()
labels := make(map[uint32]map[string]string)
first := sort.Search(len(all), func(i int) bool { return all[i].timestamp >= start })
for _, sample := range all[first:] {
if sample.timestamp > end {
break
}
labelMap := labels[sample.labelID]
if labelMap == nil {
labelMap, err = decodeMapString(series.labelsJSON[sample.labelID])
directStart := sort.Search(len(series.samples), func(i int) bool { return series.samples[i].timestamp >= start })
directEnd := sort.Search(len(series.samples), func(i int) bool { return series.samples[i].timestamp > end })
additional := directEnd - directStart
if series.compressed.count > 0 && start <= series.compressed.firstStamp() && end >= series.compressed.lastStamp {
additional += series.compressed.count
}
metricPoints := matched[key.metricName]
metricPoints.grow(additional)
if series.compressed.count > 0 && start <= series.compressed.lastStamp && end >= series.compressed.firstStamp() {
decoder := newRawSampleDecoder(series.compressed)
for sample, more := decoder.next(); more; sample, more = decoder.next() {
if sample.timestamp < start {
continue
}
if sample.timestamp > end {
break
}
labelMap, err := rawSampleLabels(series, labels, sample.labelID)
if err != nil {
return nil, err
}
labels[sample.labelID] = labelMap
appendRawQueryPoint(metricPoints, key, tags, labelMap, sample)
}
matched = append(matched, resultPoint{
point: Point{MetricName: key.metricName, EntityID: key.entityID, Timestamp: fromMillis(sample.timestamp), Value: sample.value, Tags: cloneStringMap(tags), Labels: cloneStringMap(labelMap)},
entityID: key.entityID,
tagsHash: key.tagsHash,
})
}
for _, sample := range series.samples[directStart:directEnd] {
labelMap, err := rawSampleLabels(series, labels, sample.labelID)
if err != nil {
return nil, err
}
appendRawQueryPoint(metricPoints, key, tags, labelMap, sample)
}
}
sort.Slice(matched, func(i, j int) bool {
left, right := matched[i], matched[j]
less := left.point.Timestamp.Before(right.point.Timestamp)
if left.point.Timestamp.Equal(right.point.Timestamp) {
less = left.entityID < right.entityID || (left.entityID == right.entityID && left.tagsHash < right.tagsHash)
}
if query.Order == OrderDesc {
return !less && !(left.point.Timestamp.Equal(right.point.Timestamp) && left.entityID == right.entityID && left.tagsHash == right.tagsHash)
}
return less
})
startIndex := query.Offset
if startIndex > len(matched) {
startIndex = len(matched)
}
endIndex := len(matched)
if query.Limit > 0 && startIndex+query.Limit < endIndex {
endIndex = startIndex + query.Limit
}
result := make([]Point, 0, endIndex-startIndex)
for _, item := range matched[startIndex:endIndex] {
result = append(result, item.point)
result := make(map[string][]Point, len(matched))
for metricName, metricPoints := range matched {
sort.Sort(metricPoints)
result[metricName] = metricPoints.points
}
return result, nil
}
func rawSampleLabels(series *rawSeries, labels map[uint32]map[string]string, labelID uint32) (map[string]string, error) {
if labelMap := labels[labelID]; labelMap != nil {
return labelMap, nil
}
labelMap, err := decodeMapString(series.labelsJSON[labelID])
if err != nil {
return nil, err
}
labels[labelID] = labelMap
return labelMap, nil
}
func appendRawQueryPoint(points *rawBatchPoints, key rawSeriesKey, tags, labels map[string]string, sample rawSample) {
points.points = append(points.points, Point{
MetricName: key.metricName, EntityID: key.entityID,
Timestamp: fromMillis(sample.timestamp), Value: sample.value, Tags: tags, Labels: labels,
})
points.tagsHashes = append(points.tagsHashes, key.tagsHash)
}
func rawPointLess(left Point, leftTagsHash string, right Point, rightTagsHash string) bool {
if !left.Timestamp.Equal(right.Timestamp) {
return left.Timestamp.Before(right.Timestamp)
}
if left.EntityID != right.EntityID {
return left.EntityID < right.EntityID
}
return leftTagsHash < rightTagsHash
}
func (s *Store) rawEntityIDs(ctx context.Context, query Query) ([]string, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
start, end := query.Start.UnixMilli(), query.End.UnixMilli()
s.rawMu.Lock()
defer s.rawMu.Unlock()
now := time.Now().UTC()
visibleCutoff := now.Add(-rawMemoryRetention).UnixMilli()
s.rawMu.RLock()
defer s.rawMu.RUnlock()
visibleCutoff := time.Now().UTC().Add(-rawMemoryRetention).UnixMilli()
if start < visibleCutoff {
start = visibleCutoff
}
s.compressRawBeforeLocked("", now.Add(-directRawRetention).UnixMilli())
s.pruneRawBeforeLocked("", rawMemoryCutoff(now))
seen := make(map[string]struct{})
for key, series := range s.raw {
if key.metricName != query.MetricName || (query.EntityID != "" && key.entityID != query.EntityID) {
@@ -276,9 +418,25 @@ func (s *Store) rawEntityIDs(ctx context.Context, query Query) ([]string, error)
if !ok {
continue
}
all := series.allSamples()
index := sort.Search(len(all), func(i int) bool { return all[i].timestamp >= start })
if index < len(all) && all[index].timestamp <= end {
found := false
if series.compressed.count > 0 && start <= series.compressed.lastStamp && end >= series.compressed.firstStamp() {
decoder := newRawSampleDecoder(series.compressed)
for sample, more := decoder.next(); more; sample, more = decoder.next() {
if sample.timestamp < start {
continue
}
if sample.timestamp > end {
break
}
found = true
break
}
}
if !found {
index := sort.Search(len(series.samples), func(i int) bool { return series.samples[i].timestamp >= start })
found = index < len(series.samples) && series.samples[index].timestamp <= end
}
if found {
seen[key.entityID] = struct{}{}
}
}
@@ -355,32 +513,48 @@ func (s *Store) pruneRawBeforeLocked(metricName string, beforeMilli int64) int64
if metricName != "" && key.metricName != metricName {
continue
}
compressed := series.decodeCompressed()
compressedIndex := sort.Search(len(compressed), func(i int) bool { return compressed[i].timestamp >= beforeMilli })
if compressedIndex > 0 {
compressed = compressed[compressedIndex:]
series.compressed.set(compressed)
deleted += int64(compressedIndex)
}
if len(compressed) == 0 {
series.compressed = compressedRawSamples{}
}
compressedDeleted := pruneCompressedRawSamples(&series.compressed, beforeMilli)
deleted += int64(compressedDeleted)
directIndex := sort.Search(len(series.samples), func(i int) bool { return series.samples[i].timestamp >= beforeMilli })
if directIndex > 0 {
deleted += int64(directIndex)
series.samples = append([]rawSample(nil), series.samples[directIndex:]...)
copy(series.samples, series.samples[directIndex:])
series.samples = series.samples[:len(series.samples)-directIndex]
}
if series.compressed.count == 0 && len(series.samples) == 0 {
delete(s.raw, key)
continue
}
if compressedIndex > 0 || directIndex > 0 {
if compressedDeleted > 0 || directIndex > 0 {
compactRawLabels(series)
}
}
return deleted
}
func pruneCompressedRawSamples(samples *compressedRawSamples, beforeMilli int64) int {
if samples.count == 0 || samples.firstStamp() >= beforeMilli {
return 0
}
if samples.lastStamp < beforeMilli {
deleted := samples.count
*samples = compressedRawSamples{}
return deleted
}
decoder := newRawSampleDecoder(*samples)
kept := newCompressedRawSamples(len(samples.data))
deleted := 0
for sample, more := decoder.next(); more; sample, more = decoder.next() {
if sample.timestamp < beforeMilli {
deleted++
continue
}
kept.appendSample(sample)
}
*samples = kept
return deleted
}
func (s *Store) compressRawBeforeLocked(metricName string, beforeMilli int64) {
for key, series := range s.raw {
if metricName != "" && key.metricName != metricName {
@@ -391,70 +565,38 @@ func (s *Store) compressRawBeforeLocked(metricName string, beforeMilli int64) {
continue
}
series.compressed.append(series.samples[:index])
series.samples = append([]rawSample(nil), series.samples[index:]...)
}
}
func (s *rawSeries) decodeCompressed() []rawSample {
if s.compressed.count == 0 {
return nil
}
result := make([]rawSample, 0, s.compressed.count)
for offset, previous := 0, int64(0); offset < len(s.compressed.data); {
var timestamp int64
if len(result) == 0 {
first, n := binary.Varint(s.compressed.data[offset:])
if n <= 0 {
return result
}
offset += n
timestamp = first
} else {
delta, dn := binary.Uvarint(s.compressed.data[offset:])
if dn <= 0 {
return result
}
offset += dn
timestamp = previous + int64(delta)
copy(series.samples, series.samples[index:])
series.samples = series.samples[:len(series.samples)-index]
if len(series.samples) == 0 {
series.samples = nil
}
if offset+8 > len(s.compressed.data) {
return result
}
value := math.Float64frombits(binary.LittleEndian.Uint64(s.compressed.data[offset:]))
offset += 8
labelID, ln := binary.Uvarint(s.compressed.data[offset:])
if ln <= 0 {
return result
}
offset += ln
result = append(result, rawSample{timestamp: timestamp, value: value, labelID: uint32(labelID)})
previous = timestamp
}
return result
}
func (s *rawSeries) allSamples() []rawSample {
compressed := s.decodeCompressed()
result := make([]rawSample, 0, len(compressed)+len(s.samples))
result = append(result, compressed...)
result = append(result, s.samples...)
return result
}
func (s *rawSeries) replaceCompressed(sample rawSample) (rawSample, bool) {
samples := s.decodeCompressed()
index := sort.Search(len(samples), func(i int) bool { return samples[i].timestamp >= sample.timestamp })
var old rawSample
replaced := index < len(samples) && samples[index].timestamp == sample.timestamp
if replaced {
old = samples[index]
samples[index] = sample
} else {
samples = append(samples, rawSample{})
copy(samples[index+1:], samples[index:])
samples[index] = sample
replaced := false
inserted := false
decoder := newRawSampleDecoder(s.compressed)
encoded := newCompressedRawSamples(len(s.compressed.data) + binary.MaxVarintLen64 + 9)
for current, more := decoder.next(); more; current, more = decoder.next() {
if !inserted && sample.timestamp < current.timestamp {
encoded.appendSample(sample)
inserted = true
}
if !inserted && sample.timestamp == current.timestamp {
old = current
replaced = true
encoded.appendSample(sample)
inserted = true
continue
}
encoded.appendSample(current)
}
s.compressed.set(samples)
if !inserted {
encoded.appendSample(sample)
}
s.compressed = encoded
return old, replaced
}
@@ -463,51 +605,39 @@ func (c *compressedRawSamples) append(samples []rawSample) {
return
}
if c.count > 0 && samples[0].timestamp <= c.lastStamp {
merged := make([]rawSample, 0, c.count+len(samples))
merged = append(merged, c.decode()...)
for _, sample := range samples {
index := sort.Search(len(merged), func(i int) bool { return merged[i].timestamp >= sample.timestamp })
if index < len(merged) && merged[index].timestamp == sample.timestamp {
merged[index] = sample
decoder := newRawSampleDecoder(*c)
merged := newCompressedRawSamples(len(c.data) + len(samples)*10)
current, more := decoder.next()
for sampleIndex := 0; more || sampleIndex < len(samples); {
if !more {
merged.appendSample(samples[sampleIndex])
sampleIndex++
continue
}
merged = append(merged, rawSample{})
copy(merged[index+1:], merged[index:])
merged[index] = sample
if sampleIndex == len(samples) {
merged.appendSample(current)
current, more = decoder.next()
continue
}
next := samples[sampleIndex]
switch {
case next.timestamp < current.timestamp:
merged.appendSample(next)
sampleIndex++
case next.timestamp == current.timestamp:
merged.appendSample(next)
sampleIndex++
current, more = decoder.next()
default:
merged.appendSample(current)
current, more = decoder.next()
}
}
c.set(merged)
*c = merged
return
}
for _, sample := range samples {
if c.count == 0 {
c.data = appendVarint(c.data, sample.timestamp)
} else {
c.data = appendUvarint(c.data, uint64(sample.timestamp-c.lastStamp))
}
c.data = appendRawSample(c.data, sample)
c.count++
c.lastStamp = sample.timestamp
}
}
func (c *compressedRawSamples) decode() []rawSample {
series := rawSeries{compressed: *c}
return series.decodeCompressed()
}
func (c *compressedRawSamples) set(samples []rawSample) {
c.data = nil
c.count = 0
c.lastStamp = 0
for _, sample := range samples {
if c.count == 0 {
c.data = appendVarint(c.data, sample.timestamp)
} else {
c.data = appendUvarint(c.data, uint64(sample.timestamp-c.lastStamp))
}
c.data = appendRawSample(c.data, sample)
c.count++
c.lastStamp = sample.timestamp
c.appendSample(sample)
}
}
@@ -534,9 +664,12 @@ func compactRawLabels(series *rawSeries) {
if len(series.labelHashes) <= 1 {
return
}
all := series.allSamples()
used := make([]bool, len(series.labelHashes))
for _, sample := range all {
decoder := newRawSampleDecoder(series.compressed)
for sample, more := decoder.next(); more; sample, more = decoder.next() {
used[sample.labelID] = true
}
for _, sample := range series.samples {
used[sample.labelID] = true
}
usedCount := 0
@@ -563,11 +696,13 @@ func compactRawLabels(series *rawSeries) {
labelsJSON = append(labelsJSON, series.labelsJSON[oldID])
labelIDs[hash] = newID
}
compressed := series.decodeCompressed()
for i := range compressed {
compressed[i].labelID = remap[compressed[i].labelID]
decoder = newRawSampleDecoder(series.compressed)
compressed := newCompressedRawSamples(len(series.compressed.data))
for sample, more := decoder.next(); more; sample, more = decoder.next() {
sample.labelID = remap[sample.labelID]
compressed.appendSample(sample)
}
series.compressed.set(compressed)
series.compressed = compressed
for i := range series.samples {
series.samples[i].labelID = remap[series.samples[i].labelID]
}
+113 -6
View File
@@ -124,14 +124,121 @@ func TestCompressedRawSamplesRoundTripExactly(t *testing.T) {
}
var compressed compressedRawSamples
compressed.append(samples)
decoded := compressed.decode()
if len(decoded) != len(samples) {
t.Fatalf("decoded sample count = %d, want %d", len(decoded), len(samples))
}
decoder := newRawSampleDecoder(compressed)
for i := range samples {
if decoded[i].timestamp != samples[i].timestamp || decoded[i].labelID != samples[i].labelID || math.Float64bits(decoded[i].value) != math.Float64bits(samples[i].value) {
t.Fatalf("decoded sample %d = %#v, want %#v", i, decoded[i], samples[i])
decoded, more := decoder.next()
if !more {
t.Fatalf("decoded sample count = %d, want %d", i, len(samples))
}
if decoded.timestamp != samples[i].timestamp || decoded.labelID != samples[i].labelID || math.Float64bits(decoded.value) != math.Float64bits(samples[i].value) {
t.Fatalf("decoded sample %d = %#v, want %#v", i, decoded, samples[i])
}
}
if _, more := decoder.next(); more {
t.Fatal("decoder returned extra sample")
}
}
func TestCompressedRawSamplesMergeInsertionAndReplacement(t *testing.T) {
base := time.Now().UTC().Add(-5 * time.Minute).UnixMilli()
var compressed compressedRawSamples
compressed.append([]rawSample{
{timestamp: base, value: 1, labelID: 0},
{timestamp: base + 2000, value: 3, labelID: 2},
})
compressed.append([]rawSample{
{timestamp: base + 1000, value: math.Copysign(0, -1), labelID: 1},
{timestamp: base + 2000, value: 33, labelID: 3},
})
want := []rawSample{
{timestamp: base, value: 1, labelID: 0},
{timestamp: base + 1000, value: math.Copysign(0, -1), labelID: 1},
{timestamp: base + 2000, value: 33, labelID: 3},
}
decoder := newRawSampleDecoder(compressed)
for i := range want {
got, more := decoder.next()
if !more || got.timestamp != want[i].timestamp || got.labelID != want[i].labelID || math.Float64bits(got.value) != math.Float64bits(want[i].value) {
t.Fatalf("sample %d = %#v, want %#v", i, got, want[i])
}
}
if _, more := decoder.next(); more {
t.Fatal("decoder returned extra merged sample")
}
}
func TestRawQueryCompressedDirectBoundaryClosedRange(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "raw-boundary", RetentionDays: 1}); err != nil {
t.Fatal(err)
}
now := time.Now().UTC().Truncate(time.Millisecond)
compressedAt := now.Add(-time.Minute)
directAt := compressedAt.Add(time.Millisecond)
series := &rawSeries{
tagsJSON: "{}", labelIDs: map[string]uint32{emptyLabelsHash: 0},
labelHashes: []string{emptyLabelsHash}, labelsJSON: []string{"{}"},
samples: []rawSample{{timestamp: directAt.UnixMilli(), value: 2}},
}
series.compressed.append([]rawSample{{timestamp: compressedAt.UnixMilli(), value: 1}})
s.raw[rawSeriesKey{metricName: "raw-boundary", entityID: "n1", tagsHash: emptyLabelsHash}] = series
ascending, err := s.Query(ctx, Query{MetricName: "raw-boundary", Start: compressedAt, End: directAt, Order: OrderAsc})
if err != nil {
t.Fatal(err)
}
if len(ascending) != 2 || ascending[0].Value != 1 || ascending[1].Value != 2 {
t.Fatalf("closed ascending range = %#v", ascending)
}
descending, err := s.Query(ctx, Query{MetricName: "raw-boundary", Start: compressedAt, End: directAt, Order: OrderDesc, Offset: 1, Limit: 1})
if err != nil {
t.Fatal(err)
}
if len(descending) != 1 || descending[0].Value != 1 {
t.Fatalf("paged descending range = %#v", descending)
}
}
func TestRawPruneStreamsAndReclaimsLabels(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "raw-prune", RetentionDays: 1}); err != nil {
t.Fatal(err)
}
now := time.Now().UTC().Truncate(time.Millisecond)
cutoff := now.Add(-3 * time.Minute)
series := &rawSeries{
tagsJSON: "{}",
labelIDs: map[string]uint32{"old": 0, "middle": 1, "keep": 2},
labelHashes: []string{"old", "middle", "keep"},
labelsJSON: []string{`{"label":"old"}`, `{"label":"middle"}`, `{"label":"keep"}`},
samples: []rawSample{{timestamp: now.Add(-30 * time.Second).UnixMilli(), value: 4, labelID: 2}},
}
series.compressed.append([]rawSample{
{timestamp: now.Add(-5 * time.Minute).UnixMilli(), value: 1, labelID: 0},
{timestamp: now.Add(-4 * time.Minute).UnixMilli(), value: 2, labelID: 1},
{timestamp: cutoff.UnixMilli(), value: 3, labelID: 2},
})
s.raw[rawSeriesKey{metricName: "raw-prune", entityID: "n1", tagsHash: emptyLabelsHash}] = series
if deleted := s.deleteRawBefore("raw-prune", cutoff.UnixMilli()); deleted != 2 {
t.Fatalf("deleted %d samples, want 2", deleted)
}
if len(series.labelsJSON) != 1 || series.labelsJSON[0] != `{"label":"keep"}` || series.compressed.count != 1 || series.samples[0].labelID != 0 {
t.Fatalf("compacted series = %#v", series)
}
decoder := newRawSampleDecoder(series.compressed)
kept, more := decoder.next()
if !more || kept.timestamp != cutoff.UnixMilli() || kept.labelID != 0 || kept.value != 3 {
t.Fatalf("kept compressed sample = %#v", kept)
}
points, err := s.Query(ctx, Query{MetricName: "raw-prune", Start: cutoff, End: now})
if err != nil {
t.Fatal(err)
}
if len(points) != 2 || points[0].Labels["label"] != "keep" || points[1].Labels["label"] != "keep" {
t.Fatalf("pruned query = %#v", points)
}
}
+2 -85
View File
@@ -832,7 +832,7 @@ func TestValidateNormalizedRestructureRejectsMissingDigest(t *testing.T) {
if err := s.CreateMetric(ctx, Definition{Name: "invalid", Type: TypeGauge, RetentionDays: 1}); err != nil {
t.Fatal(err)
}
at := time.Now().UTC().Add(-2 * time.Minute)
at := time.Now().UTC().Truncate(time.Minute).Add(-2 * time.Minute)
if err := s.WriteBatch(ctx, []Point{
{MetricName: "invalid", EntityID: "n1", Timestamp: at, Value: 1},
{MetricName: "invalid", EntityID: "n1", Timestamp: at.Add(time.Second), Value: 2},
@@ -857,7 +857,7 @@ func TestValidateNormalizedRestructureRejectsWrongDigestCompression(t *testing.T
if err := s.CreateMetric(ctx, Definition{Name: "invalid-compression", Type: TypeGauge, RetentionDays: 1}); err != nil {
t.Fatal(err)
}
at := time.Now().UTC().Add(-2 * time.Minute)
at := time.Now().UTC().Truncate(time.Minute).Add(-2 * time.Minute)
if err := s.WriteBatch(ctx, []Point{
{MetricName: "invalid-compression", EntityID: "n1", Timestamp: at, Value: 1},
{MetricName: "invalid-compression", EntityID: "n1", Timestamp: at.Add(time.Second), Value: 2},
@@ -929,7 +929,6 @@ func TestRestructureDump(t *testing.T) {
if err := s.ReclaimSpace(ctx); err != nil {
t.Fatalf("reclaim rebuilt store: %v", err)
}
assertRestructuredDumpQueryContinuity(t, ctx, s)
after, err := s.StorageSize(ctx)
if err != nil {
t.Fatalf("measure rebuilt storage: %v", err)
@@ -944,85 +943,3 @@ func TestRestructureDump(t *testing.T) {
}
t.Logf("restructured rows=%d metrics=%d before=%d after=%d saved=%d percent=%.2f", result.RowsCopied, result.Metrics, before, after, saved, percent)
}
func assertRestructuredDumpQueryContinuity(t *testing.T, ctx context.Context, s *Store) {
t.Helper()
var metricName, entityID string
var latestMilli int64
err := s.db.QueryRowContext(ctx, fmt.Sprintf(`SELECT series.metric_name, series.entity_id,
(SELECT MAX(recent.last_ts_milli) FROM %s recent
JOIN %s recent_series ON recent_series.id = recent.series_id
WHERE recent_series.metric_name = series.metric_name AND recent_series.entity_id = series.entity_id)
FROM %s daily
JOIN %s series ON series.id = daily.series_id
JOIN %s daily_resolution ON daily_resolution.id = daily.resolution_id
WHERE daily_resolution.resolution_milli = ?
GROUP BY series.metric_name, series.entity_id
ORDER BY COUNT(DISTINCT daily.bucket_milli) DESC, series.metric_name, series.entity_id LIMIT 1`,
s.tables.rollups, s.tables.series,
s.tables.rollups, s.tables.series, s.tables.resolutions),
(24*time.Hour).Milliseconds()).Scan(&metricName, &entityID, &latestMilli)
if err != nil {
t.Fatalf("select migrated series: %v", err)
}
now := fromMillis(latestMilli).Add(time.Millisecond)
for _, duration := range []time.Duration{time.Hour, 6 * time.Hour, 7 * 24 * time.Hour, 30 * 24 * time.Hour, 60 * 24 * time.Hour} {
start := now.Add(-duration)
interval := s.CompatibleSeriesInterval(start, now, time.Minute)
resolution, err := s.coveringSeriesResolution(ctx, Query{MetricName: metricName, EntityID: entityID, Start: start, End: now}, interval)
if err != nil {
t.Fatalf("select %s backing tier: %v", duration, err)
}
series, err := s.Series(ctx, AggregateQuery{
Query: Query{MetricName: metricName, EntityID: entityID, Start: start, End: now},
Aggregation: AggAvg,
Interval: interval,
}, now)
if err != nil {
t.Fatalf("query %s: %v", duration, err)
}
if len(series) == 0 || series[len(series)-1].Count == 0 {
t.Fatalf("query %s has no final data bucket: %#v", duration, series)
}
expectedStep := max(interval, resolution)
maxGap := time.Duration(0)
for i := 1; i < len(series); i++ {
gap := series[i].Bucket.Sub(series[i-1].Bucket)
if gap > maxGap {
maxGap = gap
}
}
if maxGap > 2*expectedStep {
t.Fatalf("query %s has a %s discontinuity larger than two %s buckets", duration, maxGap, expectedStep)
}
tailGap := now.Sub(series[len(series)-1].Bucket)
if tailGap < 0 || tailGap > 2*expectedStep {
t.Fatalf("query %s final data is %s from the query end, expected at most %s", duration, tailGap, 2*expectedStep)
}
t.Logf("range=%s metric=%s entity=%s requested=%s backing=%s points=%d first=%s last=%s max_gap=%s tail_gap=%s",
duration, metricName, entityID, interval, resolution, len(series), series[0].Bucket, series[len(series)-1].Bucket, maxGap, tailGap)
}
}
func TestRestructuredDumpQueryContinuity(t *testing.T) {
path := os.Getenv("KOMARI_METRIC_RESTRUCTURE_RESULT")
if path == "" {
t.Skip("set KOMARI_METRIC_RESTRUCTURE_RESULT to validate a rebuilt dump")
}
ctx := context.Background()
s, err := Open(ctx, SQLite(path, WithAutoMigrate(false), WithMaxOpenConns(1), WithRollupPolicy(RollupPolicy{
RawRetention: 15 * time.Minute,
Tiers: []RollupTier{
{Interval: time.Minute, Retention: 600 * time.Minute},
{Interval: 5 * time.Minute, Retention: 600 * 5 * time.Minute},
{Interval: time.Hour, Retention: 600 * time.Hour},
{Interval: 24 * time.Hour, Retention: 100 * 365 * 24 * time.Hour},
},
Compression: 30,
})))
if err != nil {
t.Fatalf("open rebuilt dump: %v", err)
}
defer s.Close()
assertRestructuredDumpQueryContinuity(t, ctx, s)
}
+8 -8
View File
@@ -348,12 +348,10 @@ func TestSeriesHybridIncludesPartialCoarseBucketAtRawCutoff(t *testing.T) {
now := time.Date(2026, 6, 18, 12, 50, 0, 0, time.UTC)
old := now.Add(-40 * time.Minute)
recent := now.Add(-5 * time.Minute)
if err := s.WriteBatch(ctx, []Point{
{MetricName: "coarse-cutoff", EntityID: "n1", Timestamp: old, Value: 10},
{MetricName: "coarse-cutoff", EntityID: "n1", Timestamp: recent, Value: 100},
}); err != nil {
t.Fatalf("write: %v", err)
}
writeRollupPointsAt(t, ctx, s, now,
Point{MetricName: "coarse-cutoff", EntityID: "n1", Timestamp: old, Value: 10},
Point{MetricName: "coarse-cutoff", EntityID: "n1", Timestamp: recent, Value: 100},
)
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact: %v", err)
}
@@ -367,7 +365,7 @@ func TestSeriesHybridIncludesPartialCoarseBucketAtRawCutoff(t *testing.T) {
t.Fatalf("series: %v", err)
}
if len(got) != 1 || got[0].Count != 2 || math.Abs(got[0].Value-55) > 1e-9 {
t.Fatalf("expected coarse rollup and recent raw point, got %#v", got)
t.Fatalf("expected persisted and in-memory hourly data, got %#v", got)
}
}
@@ -411,7 +409,9 @@ func TestSeriesAndAggregateRollupBridgeSealedCoarseTail(t *testing.T) {
read func() ([]AggregatePoint, error)
}{
{name: "series", read: func() ([]AggregatePoint, error) { return s.Series(ctx, query, base.Add(16*time.Minute)) }},
{name: "aggregate", read: func() ([]AggregatePoint, error) { return s.aggregateRollupAt(ctx, query, 5*time.Minute, base.Add(16*time.Minute)) }},
{name: "aggregate", read: func() ([]AggregatePoint, error) {
return s.aggregateRollupAt(ctx, query, 5*time.Minute, base.Add(16*time.Minute))
}},
} {
points, err := result.read()
if err != nil {
+193 -353
View File
@@ -7,292 +7,118 @@ import (
"time"
)
// AggregateRollup reduces one materialized tier into the requested output
// interval. The hot minute is folded in separately so recent charts never wait
// for another report before their final real bucket becomes visible.
// AggregateRollup reduces one explicitly selected materialized tier into the
// requested output interval.
func (s *Store) AggregateRollup(ctx context.Context, query AggregateQuery, resolution time.Duration) ([]AggregatePoint, error) {
return s.aggregateRollupAt(ctx, query, resolution, time.Now().UTC())
}
// aggregateRollupAt reads a sealed rollup prefix at the requested backing
// resolution, then recursively falls back through finer tiers for the
// ten-minute late-arrival tail. Series passes its supplied now value here so
// callers receive a stable view while AggregateRollup keeps its public API.
func (s *Store) aggregateRollupAt(ctx context.Context, query AggregateQuery, resolution time.Duration, now time.Time) ([]AggregatePoint, error) {
if err := s.ensureOpen(); err != nil {
return nil, err
}
if err := query.Validate(); err != nil {
return nil, err
}
if resolution <= 0 {
return nil, fmt.Errorf("%w: rollup resolution must be positive", ErrInvalidArgument)
}
s.rollupViewMu.RLock()
defer s.rollupViewMu.RUnlock()
q := query.Query.normalized()
needDigest := isPercentile(query.Aggregation)
policy, err := s.metricRollupPolicy(ctx, q.MetricName)
entityIDs := []string(nil)
if query.EntityID != "" {
entityIDs = []string{query.EntityID}
}
batch := BatchSeriesQuery{
Specs: []BatchSeriesSpec{{
MetricName: query.MetricName,
Aggregations: []Aggregation{query.Aggregation},
Interval: query.Interval,
PreserveSeries: query.PreserveSeries,
}},
EntityIDs: entityIDs,
Start: query.Start,
End: query.End,
Tags: query.Tags,
Order: query.Order,
}
if err := batch.Validate(); err != nil {
return nil, err
}
result, err := s.seriesBatchAt(ctx, batch.normalized(), now.UTC(), map[string]time.Duration{query.MetricName: resolution})
if err != nil {
return nil, err
}
rows, err := s.hybridRollupRows(ctx, q, policy, resolution, now.UTC(), needDigest)
return pageBuckets(result.Values[query.MetricName][query.Aggregation], query.BucketLimit, query.BucketOffset), nil
}
// Series selects one deterministic backing tier and delegates to SeriesBatch.
func (s *Store) Series(ctx context.Context, query AggregateQuery, now time.Time) ([]AggregatePoint, error) {
if err := s.ensureOpen(); err != nil {
return nil, err
}
if err := query.Validate(); err != nil {
return nil, err
}
entityIDs := []string(nil)
if query.EntityID != "" {
entityIDs = []string{query.EntityID}
}
result, err := s.SeriesBatch(ctx, BatchSeriesQuery{
Specs: []BatchSeriesSpec{{
MetricName: query.MetricName,
Aggregations: []Aggregation{query.Aggregation},
Interval: query.Interval,
PreserveSeries: query.PreserveSeries,
}},
EntityIDs: entityIDs,
Start: query.Start,
End: query.End,
Tags: query.Tags,
Order: query.Order,
}, now)
if err != nil {
return nil, err
}
if query.Aggregation == AggRate {
points := representativePoints(q.MetricName, rows)
return pageBuckets(mustAggregatePoints(points, query), query.BucketLimit, query.BucketOffset), nil
return pageBuckets(result.Values[query.MetricName][query.Aggregation], query.BucketLimit, query.BucketOffset), nil
}
// SeriesAggregates derives multiple aggregations from the same streamed scan.
func (s *Store) SeriesAggregates(ctx context.Context, query Query, aggregations []Aggregation, interval time.Duration, preserveSeries bool, now time.Time) (map[Aggregation][]AggregatePoint, error) {
entityIDs := []string(nil)
if query.EntityID != "" {
entityIDs = []string{query.EntityID}
}
groups := foldRollupRows(nil, rows, query.Interval, s.cfg.RollupPolicy.compression(), query.PreserveSeries, false, needDigest)
result, err := rollupGroupsToPoints(groups, query)
result, err := s.SeriesBatch(ctx, BatchSeriesQuery{
Specs: []BatchSeriesSpec{{
MetricName: query.MetricName,
Aggregations: aggregations,
Interval: interval,
PreserveSeries: preserveSeries,
}},
EntityIDs: entityIDs,
Start: query.Start,
End: query.End,
Tags: query.Tags,
Order: query.Order,
}, now)
if err != nil {
return nil, err
}
return pageBuckets(result, query.BucketLimit, query.BucketOffset), nil
return result.Values[query.MetricName], nil
}
// hybridRollupRows reads each part of a range once. A parent bucket becomes
// eligible only after its end is outside coarseRollupGrace; the remainder is
// represented by the immediate finer tier until minute buckets, where hot
// in-memory observations complete the view.
func (s *Store) hybridRollupRows(ctx context.Context, query Query, policy RollupPolicy, resolution time.Duration, now time.Time, needDigest bool) ([]storedRollup, error) {
if len(policy.Tiers) == 0 {
return nil, nil
func seriesResolutionForPolicy(start time.Time, interval time.Duration, now time.Time, policy RollupPolicy) time.Duration {
preferred := policy.Tiers[0].Interval
if tier := bestRollupTier(policy, interval, start.UTC(), now.UTC()); tier != nil {
return tier.Interval
}
index := -1
for i, tier := range policy.Tiers {
if tier.Interval <= resolution && resolution%tier.Interval == 0 {
index = i
for i := len(policy.Tiers) - 1; i >= 0; i-- {
tier := policy.Tiers[i]
if interval >= tier.Interval && interval%tier.Interval == 0 {
preferred = tier.Interval
break
}
}
if index < 0 {
return s.scanRollupRowsBetween(ctx, query.MetricName, query.EntityID, query.Tags, resolution.Milliseconds(), bucketStartMillis(query.Start.UnixMilli(), resolution.Milliseconds()), query.End.UnixMilli(), needDigest)
}
return s.hybridRollupRowsAtTier(ctx, query, policy, index, query.Start.UTC().UnixMilli(), query.End.UTC().UnixMilli(), now, needDigest)
}
func (s *Store) hybridRollupRowsAtTier(ctx context.Context, query Query, policy RollupPolicy, index int, startMilli, endMilli int64, now time.Time, needDigest bool) ([]storedRollup, error) {
if startMilli > endMilli {
return nil, nil
}
tier := policy.Tiers[index]
intervalMilli := tier.Interval.Milliseconds()
// The greatest bucket start whose end is at least coarseRollupGrace behind
// now. Buckets after it are still mutable in the in-memory parent map.
sealedThrough := bucketStartMillis(now.Add(-coarseRollupGrace-tier.Interval).UnixMilli(), intervalMilli)
rows := make([]storedRollup, 0)
if startMilli <= sealedThrough {
upper := sealedThrough
if upper > endMilli {
upper = endMilli
}
sealed, err := s.scanRollupRowsBetween(ctx, query.MetricName, query.EntityID, query.Tags, intervalMilli, bucketStartMillis(startMilli, intervalMilli), upper, needDigest)
if err != nil {
return nil, err
}
rows = append(rows, sealed...)
}
tailStart := startMilli
if tailStart <= sealedThrough {
tailStart = sealedThrough + intervalMilli
}
if tailStart > endMilli {
return rows, nil
}
if index == 0 {
minute, err := s.scanRollupRowsBetween(ctx, query.MetricName, query.EntityID, query.Tags, intervalMilli, bucketStartMillis(tailStart, intervalMilli), endMilli, needDigest)
if err != nil {
return nil, err
}
hot, err := s.hotRollupRows(query.MetricName, query.EntityID, query.Tags, fromMillis(tailStart), fromMillis(endMilli), needDigest)
if err != nil {
return nil, err
}
return append(rows, append(minute, hot...)...), nil
}
tail, err := s.hybridRollupRowsAtTier(ctx, query, policy, index-1, tailStart, endMilli, now, needDigest)
if err != nil {
return nil, err
}
return append(rows, tail...), nil
}
func mustAggregatePoints(points []Point, query AggregateQuery) []AggregatePoint {
result, err := AggregatePoints(points, query)
if err != nil {
return []AggregatePoint{}
}
return result
}
func representativePoints(metricName string, rows []storedRollup) []Point {
points := make([]Point, 0, len(rows))
for _, row := range rows {
tags, err := rollupTagsFromJSON(row.bucketData.tagsJSON)
if err != nil {
continue
}
labels, err := rollupTagsFromJSON(row.bucketData.labelsJSON)
if err != nil {
continue
}
points = append(points, Point{MetricName: metricName, EntityID: row.entityID, Timestamp: fromMillis(row.bucketData.lastTS), Value: row.bucketData.lastVal, Tags: tags, Labels: labels})
}
return points
}
func foldRollupRows(groups map[rollupKey]*rollupBucket, rows []storedRollup, interval time.Duration, compression float64, preserveSeries, preserveLabels, needDigest bool) map[rollupKey]*rollupBucket {
if groups == nil {
groups = make(map[rollupKey]*rollupBucket)
}
for _, row := range rows {
key := rollupKey{bucket: bucketStartMillis(row.bucket, interval.Milliseconds())}
if preserveSeries {
key.entityID, key.tagsHash = row.entityID, row.bucketData.tagsHash
if preserveLabels {
key.labelsHash = row.bucketData.labelsHash
}
}
bucket := groups[key]
if bucket == nil {
bucket = newRollupBucketWithDigest(compression, needDigest)
bucket.tagsHash, bucket.tagsJSON = row.bucketData.tagsHash, row.bucketData.tagsJSON
bucket.labelsHash, bucket.labelsJSON = row.bucketData.labelsHash, row.bucketData.labelsJSON
groups[key] = bucket
}
bucket.mergeStored(row.bucketData)
}
return groups
}
func rollupGroupsToPoints(groups map[rollupKey]*rollupBucket, query AggregateQuery) ([]AggregatePoint, error) {
keys := make([]rollupKey, 0, len(groups))
for key := range groups {
keys = append(keys, key)
}
sortRollupKeys(keys)
result := make([]AggregatePoint, 0, len(keys))
for _, key := range keys {
bucket := groups[key]
value, ok := bucket.value(query.Aggregation)
if !ok {
return nil, fmt.Errorf("%w: aggregation %q requires raw samples", ErrInvalidArgument, query.Aggregation)
}
tags, err := rollupTagsFromJSON(bucket.tagsJSON)
if err != nil {
return nil, err
}
if !query.PreserveSeries {
tags = map[string]string{}
}
result = append(result, AggregatePoint{MetricName: query.MetricName, EntityID: key.entityID, Bucket: fromMillis(key.bucket), Value: value, Count: int(bucket.count), Tags: tags})
}
return result, nil
}
func (s *Store) scanRollupRowsBetween(ctx context.Context, metricName, entityID string, tags map[string]string, resolutionMilli, lowerBucket, upperBucket int64, needDigest bool) ([]storedRollup, error) {
return s.scanRollupRowsBetweenWith(ctx, s.reader(), metricName, entityID, tags, time.Duration(resolutionMilli)*time.Millisecond, lowerBucket, upperBucket, needDigest)
}
func (s *Store) scanRollupRowsBetweenWith(ctx context.Context, q querier, metricName, entityID string, tags map[string]string, resolution time.Duration, lowerBucket, upperBucket int64, needDigest bool) ([]storedRollup, error) {
args := []any{metricName, resolution.Milliseconds(), lowerBucket, upperBucket}
parts := []string{
"s.metric_name = " + s.dialect.placeholder(1),
"d.resolution_milli = " + s.dialect.placeholder(2),
"r.bucket_milli >= " + s.dialect.placeholder(3),
"r.bucket_milli <= " + s.dialect.placeholder(4),
}
if entityID != "" {
args = append(args, entityID)
parts = append(parts, "s.entity_id = "+s.dialect.placeholder(len(args)))
}
for _, key := range sortedKeys(tags) {
args = append(args, tags[key])
parts = append(parts, s.dialect.jsonExtractEquals("s.tags", key, s.dialect.placeholder(len(args))))
}
columns := "s.entity_id, s.tags_hash, s.tags, l.labels_hash, l.labels, r.bucket_milli, r.count, r.sum, r.sum_sq, r.min_val, r.max_val, r.first_val, r.first_ts_milli, r.last_val, r.last_ts_milli"
if needDigest {
columns += ", r.digest"
}
sqlText := fmt.Sprintf("SELECT %s FROM %s r JOIN %s s ON s.id = r.series_id JOIN %s d ON d.id = r.resolution_id JOIN %s l ON l.id = r.label_id WHERE %s ORDER BY r.bucket_milli ASC", columns, s.tables.rollups, s.tables.series, s.tables.resolutions, s.tables.labels, joinSQLWith(parts, " AND "))
rows, err := q.QueryContext(ctx, sqlText, args...)
if err != nil {
return nil, err
}
defer rows.Close()
return scanStoredRollups(rows, needDigest, s.cfg.RollupPolicy.compression())
}
func joinSQLWith(parts []string, separator string) string {
if len(parts) == 0 {
return ""
}
out := parts[0]
for _, part := range parts[1:] {
out += separator + part
}
return out
}
func scanStoredRollups(rows *sql.Rows, needDigest bool, compression float64) ([]storedRollup, error) {
result := make([]storedRollup, 0)
for rows.Next() {
var entityID, tagsHash, labelsHash string
var rawTags, rawLabels any
var bucket, count, firstTS, lastTS int64
var sum, sumSq, min, max, firstVal, lastVal float64
var digest []byte
args := []any{&entityID, &tagsHash, &rawTags, &labelsHash, &rawLabels, &bucket, &count, &sum, &sumSq, &min, &max, &firstVal, &firstTS, &lastVal, &lastTS}
if needDigest {
args = append(args, &digest)
}
if err := rows.Scan(args...); err != nil {
return nil, err
}
tagsJSON, err := rawJSONToString(rawTags)
if err != nil {
return nil, err
}
labelsJSON, err := rawJSONToString(rawLabels)
if err != nil {
return nil, err
}
var d *TDigest
if needDigest {
d, err = digestFromRollup(count, min, max, digest, compression)
if err != nil {
return nil, err
}
}
result = append(result, storedRollup{entityID: entityID, bucket: bucket, bucketData: &rollupBucket{count: count, sum: sum, sumSq: sumSq, min: min, max: max, firstVal: firstVal, firstTS: firstTS, lastVal: lastVal, lastTS: lastTS, digest: d, tagsHash: tagsHash, tagsJSON: tagsJSON, labelsHash: labelsHash, labelsJSON: labelsJSON}})
}
return result, rows.Err()
}
func rawJSONToString(value any) (string, error) {
switch v := value.(type) {
case nil:
return "{}", nil
case string:
if v == "" {
return "{}", nil
}
return v, nil
case []byte:
if len(v) == 0 {
return "{}", nil
}
return string(v), nil
default:
return "", fmt.Errorf("unsupported JSON column type %T", value)
}
}
func rollupTagsFromJSON(raw string) (map[string]string, error) {
values, err := decodeMapString(raw)
if err != nil {
return nil, err
}
return cloneStringMap(values), nil
return preferred
}
func (s *Store) CompatibleSeriesInterval(start, now time.Time, interval time.Duration) time.Duration {
@@ -324,115 +150,129 @@ func bestRollupTier(policy RollupPolicy, interval time.Duration, start, now time
for i := range policy.Tiers {
tier := &policy.Tiers[i]
if interval >= tier.Interval && interval%tier.Interval == 0 && !now.Add(-tier.Retention).After(start) {
// Prefer the coarsest backing tier compatible with the requested
// output width. hybridRollupRows fills its unsealed tail from finer
// tiers, so this reduces reads without sacrificing recent data.
best = tier
}
}
return best
}
func (s *Store) Series(ctx context.Context, query AggregateQuery, now time.Time) ([]AggregatePoint, error) {
if err := s.ensureOpen(); err != nil {
return nil, err
// scanRollupRowsBetween is retained for compaction maintenance and internal
// verification. User-facing aggregation goes through SeriesBatch.
func (s *Store) scanRollupRowsBetween(ctx context.Context, metricName, entityID string, tags map[string]string, resolutionMilli, lowerBucket, upperBucket int64, needDigest bool) ([]storedRollup, error) {
return s.scanRollupRowsBetweenWith(ctx, s.reader(), metricName, entityID, tags, time.Duration(resolutionMilli)*time.Millisecond, lowerBucket, upperBucket, needDigest)
}
func (s *Store) scanRollupRowsBetweenWith(ctx context.Context, q querier, metricName, entityID string, tags map[string]string, resolution time.Duration, lowerBucket, upperBucket int64, needDigest bool) ([]storedRollup, error) {
args := []any{metricName, resolution.Milliseconds(), lowerBucket, upperBucket}
parts := []string{
"s.metric_name = " + s.dialect.placeholder(1),
"d.resolution_milli = " + s.dialect.placeholder(2),
"r.bucket_milli >= " + s.dialect.placeholder(3),
"r.bucket_milli <= " + s.dialect.placeholder(4),
}
if err := query.Validate(); err != nil {
return nil, err
if entityID != "" {
args = append(args, entityID)
parts = append(parts, "s.entity_id = "+s.dialect.placeholder(len(args)))
}
policy, err := s.metricRollupPolicy(ctx, query.MetricName)
for _, key := range sortedKeys(tags) {
args = append(args, tags[key])
parts = append(parts, s.dialect.jsonExtractEquals("s.tags", key, s.dialect.placeholder(len(args))))
}
columns := "s.entity_id, s.tags_hash, s.tags, l.labels_hash, l.labels, r.bucket_milli, r.count, r.sum, r.sum_sq, r.min_val, r.max_val, r.first_val, r.first_ts_milli, r.last_val, r.last_ts_milli"
if needDigest {
columns += ", r.digest"
}
sqlText := fmt.Sprintf("SELECT %s FROM %s r JOIN %s s ON s.id = r.series_id JOIN %s d ON d.id = r.resolution_id JOIN %s l ON l.id = r.label_id WHERE %s ORDER BY r.bucket_milli ASC", columns, s.tables.rollups, s.tables.series, s.tables.resolutions, s.tables.labels, joinSQLWith(parts, " AND "))
rows, err := q.QueryContext(ctx, sqlText, args...)
if err != nil {
return nil, err
}
if len(policy.Tiers) == 0 {
return []AggregatePoint{}, nil
defer rows.Close()
return scanStoredRollupsForMaintenance(rows, needDigest, s.cfg.RollupPolicy.compression())
}
func joinSQLWith(parts []string, separator string) string {
if len(parts) == 0 {
return ""
}
preferred := policy.Tiers[0].Interval
if tier := bestRollupTier(policy, query.Interval, query.Start.UTC(), now.UTC()); tier != nil {
preferred = tier.Interval
} else {
for i := len(policy.Tiers) - 1; i >= 0; i-- {
tier := policy.Tiers[i]
if query.Interval >= tier.Interval && query.Interval%tier.Interval == 0 {
preferred = tier.Interval
break
out := parts[0]
for _, part := range parts[1:] {
out += separator + part
}
return out
}
func scanStoredRollupsForMaintenance(rows *sql.Rows, needDigest bool, compression float64) ([]storedRollup, error) {
result := make([]storedRollup, 0)
for rows.Next() {
var entityID, tagsHash, tagsJSON, labelsHash, labelsJSON string
var bucket, count, firstTS, lastTS int64
var sum, sumSq, min, max, firstVal, lastVal float64
var digest []byte
var err error
if needDigest {
err = rows.Scan(&entityID, &tagsHash, &tagsJSON, &labelsHash, &labelsJSON, &bucket, &count, &sum, &sumSq, &min, &max, &firstVal, &firstTS, &lastVal, &lastTS, &digest)
} else {
err = rows.Scan(&entityID, &tagsHash, &tagsJSON, &labelsHash, &labelsJSON, &bucket, &count, &sum, &sumSq, &min, &max, &firstVal, &firstTS, &lastVal, &lastTS)
}
if err != nil {
return nil, err
}
var decoded *TDigest
if needDigest {
decoded, err = digestFromRollup(count, min, max, digest, compression)
if err != nil {
return nil, err
}
}
result = append(result, storedRollup{entityID: entityID, bucket: bucket, bucketData: &rollupBucket{
count: count, sum: sum, sumSq: sumSq, min: min, max: max,
firstVal: firstVal, firstTS: firstTS, lastVal: lastVal, lastTS: lastTS,
digest: decoded, tagsHash: tagsHash, tagsJSON: tagsJSON, labelsHash: labelsHash, labelsJSON: labelsJSON,
}})
}
resolution, err := s.coveringSeriesResolutionForPolicy(ctx, query.Query.normalized(), preferred, policy)
return result, rows.Err()
}
func representativePoints(metricName string, rows []storedRollup) []Point {
points := make([]Point, 0, len(rows))
for _, row := range rows {
tags, err := rollupTagsFromJSON(row.bucketData.tagsJSON)
if err != nil {
continue
}
labels, err := rollupTagsFromJSON(row.bucketData.labelsJSON)
if err != nil {
continue
}
points = append(points, Point{MetricName: metricName, EntityID: row.entityID, Timestamp: fromMillis(row.bucketData.lastTS), Value: row.bucketData.lastVal, Tags: tags, Labels: labels})
}
return points
}
func rawJSONToString(value any) (string, error) {
switch v := value.(type) {
case nil:
return "{}", nil
case string:
if v == "" {
return "{}", nil
}
return v, nil
case []byte:
if len(v) == 0 {
return "{}", nil
}
return string(v), nil
default:
return "", fmt.Errorf("unsupported JSON column type %T", value)
}
}
func rollupTagsFromJSON(raw string) (map[string]string, error) {
values, err := decodeMapString(raw)
if err != nil {
return nil, err
}
return s.aggregateRollupAt(ctx, query, resolution, now.UTC())
}
// coveringSeriesResolution falls back to a coarser retained tier only when
// the preferred tier starts after the requested range. This happens directly
// after a legacy migration: historical 5m/hour rollups predate the newly
// written 1m tier. Returning the coarser summary preserves the entire query
// rather than presenting a false gap at that boundary.
func (s *Store) coveringSeriesResolution(ctx context.Context, query Query, preferred time.Duration) (time.Duration, error) {
return s.coveringSeriesResolutionForPolicy(ctx, query, preferred, s.cfg.RollupPolicy)
}
func (s *Store) coveringSeriesResolutionForPolicy(ctx context.Context, query Query, preferred time.Duration, policy RollupPolicy) (time.Duration, error) {
selected := preferred
earliest, latest, found, err := s.rollupTimeBounds(ctx, query, preferred)
if err != nil {
return 0, err
}
startMilli := query.Start.UTC().UnixMilli()
endMilli := query.End.UTC().UnixMilli()
if found && earliest <= startMilli && latest >= startMilli {
return selected, nil
}
selectedFound := found && earliest <= endMilli && latest >= startMilli
for _, tier := range policy.Tiers {
if tier.Interval <= preferred {
continue
}
candidateEarliest, candidateLatest, candidateFound, err := s.rollupTimeBounds(ctx, query, tier.Interval)
if err != nil {
return 0, err
}
if !candidateFound || candidateEarliest > endMilli || candidateLatest < startMilli {
continue
}
if candidateEarliest <= startMilli && candidateLatest >= startMilli {
return tier.Interval, nil
}
if !selectedFound || candidateEarliest < earliest {
selected, earliest, selectedFound = tier.Interval, candidateEarliest, true
}
}
return selected, nil
}
func (s *Store) rollupTimeBounds(ctx context.Context, query Query, interval time.Duration) (int64, int64, bool, error) {
args := []any{query.MetricName}
seriesParts := []string{"s.metric_name = " + s.dialect.placeholder(1)}
if query.EntityID != "" {
args = append(args, query.EntityID)
seriesParts = append(seriesParts, "s.entity_id = "+s.dialect.placeholder(len(args)))
}
for _, key := range sortedKeys(query.Tags) {
args = append(args, query.Tags[key])
seriesParts = append(seriesParts, s.dialect.jsonExtractEquals("s.tags", key, s.dialect.placeholder(len(args))))
}
args = append(args, interval.Milliseconds())
resolutionPlaceholder := s.dialect.placeholder(len(args))
args = append(args, query.End.UTC().UnixMilli())
endPlaceholder := s.dialect.placeholder(len(args))
var earliest, latest sql.NullInt64
err := s.reader().QueryRowContext(ctx, fmt.Sprintf(`SELECT MIN(r.first_ts_milli), MAX(r.last_ts_milli) FROM %s r
WHERE r.series_id IN (SELECT s.id FROM %s s WHERE %s)
AND r.resolution_id IN (SELECT d.id FROM %s d WHERE d.resolution_milli = %s)
AND r.first_ts_milli <= %s`, s.tables.rollups, s.tables.series, joinSQLWith(seriesParts, " AND "),
s.tables.resolutions, resolutionPlaceholder, endPlaceholder),
args...).Scan(&earliest, &latest)
if err != nil {
return 0, 0, false, err
}
return earliest.Int64, latest.Int64, earliest.Valid && latest.Valid, nil
return cloneStringMap(values), nil
}
+152
View File
@@ -0,0 +1,152 @@
package metric
import (
"encoding/json"
"fmt"
"strings"
)
type seriesDictionaryPlan struct {
MetricNames []string
EntityIDs []string
Tags map[string]string
}
type rollupReadPlan struct {
SeriesIDs []int64
ResolutionID int64
StartMilli int64
EndMilli int64
Fields rollupReadFields
}
type rollupReadFields uint16
const (
rollupReadSum rollupReadFields = 1 << iota
rollupReadSumSq
rollupReadMin
rollupReadMax
rollupReadFirst
rollupReadLast
rollupReadDigest
)
type renderedSQL struct {
Query string
Args []any
}
func (d sqliteDialect) renderSeriesDictionary(tables tables, indexName string, plan seriesDictionaryPlan) renderedSQL {
return renderExpandedSeriesDictionary(d, tables, " INDEXED BY "+indexName, plan)
}
func (d mysqlDialect) renderSeriesDictionary(tables tables, indexName string, plan seriesDictionaryPlan) renderedSQL {
return renderExpandedSeriesDictionary(d, tables, " FORCE INDEX ("+indexName+")", plan)
}
func (d postgresDialect) renderSeriesDictionary(tables tables, _ string, plan seriesDictionaryPlan) renderedSQL {
args := []any{plan.MetricNames}
parts := []string{"s.metric_name = ANY(" + d.placeholder(1) + "::text[])"}
if len(plan.EntityIDs) > 0 {
args = append(args, plan.EntityIDs)
parts = append(parts, "s.entity_id = ANY("+d.placeholder(len(args))+"::text[])")
}
for _, key := range sortedKeys(plan.Tags) {
args = append(args, plan.Tags[key])
parts = append(parts, d.jsonExtractEquals("s.tags", key, d.placeholder(len(args))))
}
return renderedSQL{
Query: fmt.Sprintf("SELECT s.id, s.metric_name, s.entity_id, s.tags_hash, s.tags FROM %s s WHERE %s ORDER BY s.id ASC", tables.series, strings.Join(parts, " AND ")),
Args: args,
}
}
func renderExpandedSeriesDictionary(d dialect, tables tables, indexHint string, plan seriesDictionaryPlan) renderedSQL {
args := make([]any, 0, len(plan.MetricNames)+len(plan.EntityIDs)+len(plan.Tags))
metricPlaceholders := appendPlaceholders(d, &args, plan.MetricNames)
parts := []string{"s.metric_name IN (" + strings.Join(metricPlaceholders, ", ") + ")"}
if len(plan.EntityIDs) > 0 {
entityPlaceholders := appendPlaceholders(d, &args, plan.EntityIDs)
parts = append(parts, "s.entity_id IN ("+strings.Join(entityPlaceholders, ", ")+")")
}
for _, key := range sortedKeys(plan.Tags) {
args = append(args, plan.Tags[key])
parts = append(parts, d.jsonExtractEquals("s.tags", key, d.placeholder(len(args))))
}
return renderedSQL{
Query: fmt.Sprintf("SELECT s.id, s.metric_name, s.entity_id, s.tags_hash, s.tags FROM %s s%s WHERE %s ORDER BY s.id ASC", tables.series, indexHint, strings.Join(parts, " AND ")),
Args: args,
}
}
func appendPlaceholders[T ~string](d dialect, args *[]any, values []T) []string {
placeholders := make([]string, 0, len(values))
for _, value := range values {
*args = append(*args, string(value))
placeholders = append(placeholders, d.placeholder(len(*args)))
}
return placeholders
}
func (d sqliteDialect) renderRollupRead(tables tables, indexName string, plan rollupReadPlan) renderedSQL {
seriesJSON, _ := json.Marshal(plan.SeriesIDs)
args := []any{plan.ResolutionID, plan.StartMilli, plan.EndMilli, string(seriesJSON)}
return renderedSQL{
Query: fmt.Sprintf("SELECT %s FROM %s r INDEXED BY %s WHERE r.resolution_id = %s AND r.bucket_milli >= %s AND r.bucket_milli <= %s AND r.series_id IN (SELECT CAST(value AS INTEGER) FROM json_each(%s)) ORDER BY r.bucket_milli ASC, r.series_id ASC, r.label_id ASC",
rollupReadColumns(plan.Fields), tables.rollups, indexName,
d.placeholder(1), d.placeholder(2), d.placeholder(3), d.placeholder(4)),
Args: args,
}
}
func (d mysqlDialect) renderRollupRead(tables tables, indexName string, plan rollupReadPlan) renderedSQL {
args := []any{plan.ResolutionID, plan.StartMilli, plan.EndMilli}
seriesPlaceholders := make([]string, 0, len(plan.SeriesIDs))
for _, seriesID := range plan.SeriesIDs {
args = append(args, seriesID)
seriesPlaceholders = append(seriesPlaceholders, d.placeholder(len(args)))
}
return renderedSQL{
Query: fmt.Sprintf("SELECT %s FROM %s r FORCE INDEX (%s) WHERE r.resolution_id = %s AND r.bucket_milli >= %s AND r.bucket_milli <= %s AND r.series_id IN (%s) ORDER BY r.bucket_milli ASC, r.series_id ASC, r.label_id ASC",
rollupReadColumns(plan.Fields), tables.rollups, indexName,
d.placeholder(1), d.placeholder(2), d.placeholder(3), strings.Join(seriesPlaceholders, ", ")),
Args: args,
}
}
func (d postgresDialect) renderRollupRead(tables tables, _ string, plan rollupReadPlan) renderedSQL {
args := []any{plan.ResolutionID, plan.StartMilli, plan.EndMilli, plan.SeriesIDs}
return renderedSQL{
Query: fmt.Sprintf("SELECT %s FROM %s r WHERE r.resolution_id = %s AND r.bucket_milli >= %s AND r.bucket_milli <= %s AND r.series_id = ANY(%s::bigint[]) ORDER BY r.bucket_milli ASC, r.series_id ASC, r.label_id ASC",
rollupReadColumns(plan.Fields), tables.rollups,
d.placeholder(1), d.placeholder(2), d.placeholder(3), d.placeholder(4)),
Args: args,
}
}
func rollupReadColumns(fields rollupReadFields) string {
columns := []string{"r.series_id", "r.bucket_milli", "r.count"}
if fields&rollupReadSum != 0 {
columns = append(columns, "r.sum")
}
if fields&rollupReadSumSq != 0 {
columns = append(columns, "r.sum_sq")
}
if fields&rollupReadMin != 0 {
columns = append(columns, "r.min_val")
}
if fields&rollupReadMax != 0 {
columns = append(columns, "r.max_val")
}
if fields&rollupReadFirst != 0 {
columns = append(columns, "r.first_val", "r.first_ts_milli")
}
if fields&rollupReadLast != 0 {
columns = append(columns, "r.last_val", "r.last_ts_milli")
}
if fields&rollupReadDigest != 0 {
columns = append(columns, "r.digest")
}
return strings.Join(columns, ", ")
}
+1 -1
View File
@@ -146,7 +146,7 @@ func (s *Store) rebuildRollupBucketTx(ctx context.Context, metricName string, fi
if err != nil {
return nil, err
}
stored, err := scanStoredRollups(rows, true, s.cfg.RollupPolicy.compression())
stored, err := scanStoredRollupsForMaintenance(rows, true, s.cfg.RollupPolicy.compression())
_ = rows.Close()
if err != nil {
return nil, err
+2 -2
View File
@@ -1104,8 +1104,8 @@ func TestCompactWithRawRetentionOnlyWritesChangedBuckets(t *testing.T) {
if err != nil {
t.Fatalf("rollup: %v", err)
}
if len(got) != 1 || got[0].Count != 2 || got[0].Value != 3 {
t.Fatalf("late delta was not merged exactly once: %#v", got)
if len(got) != 1 || got[0].Count != 1 || got[0].Value != 1 {
t.Fatalf("explicit 5m query read a finer tier: %#v", got)
}
}
+29 -1
View File
@@ -74,7 +74,7 @@ type Store struct {
// coarseMu protects in-memory parent buckets. Coarser tiers are only
// materialized once their source window has remained closed long enough for
// the raw late-arrival window to pass.
coarseMu sync.Mutex
coarseMu sync.RWMutex
coarse map[coarseRollupKey]*coarseRollup
// mu protects closed state.
//
@@ -413,6 +413,22 @@ func (s *Store) Ping(ctx context.Context) error {
return s.db.PingContext(ctx)
}
// QueryContext executes a raw read query against the Store's read pool.
func (s *Store) QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error) {
if err := s.ensureOpen(); err != nil {
return nil, err
}
return s.reader().QueryContext(ctx, query, args...)
}
// ExecContext executes a raw statement against the Store's primary write pool.
func (s *Store) ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error) {
if err := s.ensureOpen(); err != nil {
return nil, err
}
return s.db.ExecContext(ctx, query, args...)
}
// ensureOpen verifies that the Store is not closed.
//
// ensureOpen 检查 Store 是否仍处于打开状态。
@@ -934,6 +950,18 @@ func (s *Store) Query(ctx context.Context, query Query) ([]Point, error) {
return s.queryRawPoints(ctx, query)
}
// QueryBatch loads exact raw samples for multiple metrics with one scan of the
// in-memory raw store.
func (s *Store) QueryBatch(ctx context.Context, query BatchQuery) (map[string][]Point, error) {
if err := s.ensureOpen(); err != nil {
return nil, err
}
if err := query.Validate(); err != nil {
return nil, err
}
return s.queryRawPointsBatch(ctx, query.normalized())
}
// EntityIDs returns distinct entity ids that have exact in-memory samples or
// persisted/hot rollups.
func (s *Store) EntityIDs(ctx context.Context, query Query) ([]string, error) {
+10 -7
View File
@@ -139,11 +139,15 @@ func (t *TDigest) Add(x, w float64) {
t.processed = false
// Bound the unprocessed buffer so a long stream cannot grow memory without
// limit; process() collapses it back to ~compression centroids.
if len(t.centroids) > int(8*t.compression)+16 {
if len(t.centroids) > t.processThreshold() {
t.process()
}
}
func (t *TDigest) processThreshold() int {
return int(8*t.compression) + 16
}
// Merge folds every centroid of other into t. This is the operation rollup
// composition relies on: a coarse bucket merges the digests of the finer
// buckets it spans.
@@ -154,10 +158,7 @@ func (t *TDigest) Merge(other *TDigest) {
if other == nil || other.count == 0 {
return
}
other.process()
for _, c := range other.centroids {
t.centroids = append(t.centroids, c)
}
t.centroids = append(t.centroids, other.centroids...)
t.count += other.count
if other.min < t.min {
t.min = other.min
@@ -166,7 +167,9 @@ func (t *TDigest) Merge(other *TDigest) {
t.max = other.max
}
t.processed = false
t.process()
if len(t.centroids) > t.processThreshold() {
t.process()
}
}
// Count returns the total weight observed.
@@ -194,7 +197,7 @@ func (t *TDigest) process() {
return t.centroids[i].mean < t.centroids[j].mean
})
total := t.count
merged := t.centroids[:0:0] // fresh backing array; don't alias input mid-merge
merged := t.centroids[:0]
cur := t.centroids[0]
weightBefore := 0.0
for i := 1; i < len(t.centroids); i++ {
+89
View File
@@ -73,6 +73,95 @@ func TestTDigestMergeMatchesCombined(t *testing.T) {
}
}
func TestTDigestMergeGroupingAccuracy(t *testing.T) {
rng := rand.New(rand.NewSource(17))
all := make([]float64, 0, 24*2000)
fine := make([]*TDigest, 24)
for bucket := range fine {
fine[bucket] = NewTDigest(100)
for i := 0; i < 2000; i++ {
value := math.Abs(rng.NormFloat64())*70 + rng.ExpFloat64()*30
fine[bucket].Add(value, 1)
all = append(all, value)
}
}
flat := NewTDigest(100)
grouped := NewTDigest(100)
for _, digest := range fine {
flat.Merge(digest)
}
for groupStart := 0; groupStart < len(fine); groupStart += 4 {
group := NewTDigest(100)
for _, digest := range fine[groupStart : groupStart+4] {
group.Merge(digest)
}
grouped.Merge(group)
}
for name, digest := range map[string]*TDigest{"flat": flat, "grouped": grouped} {
if digest.Count() != float64(len(all)) || digest.min != exactQuantile(all, 0) || digest.max != exactQuantile(all, 1) {
t.Fatalf("%s exact aggregates changed: count=%v min=%v max=%v", name, digest.Count(), digest.min, digest.max)
}
for _, q := range []float64{0.5, 0.95, 0.99, 0.999} {
exact := exactQuantile(all, q)
relErr := math.Abs(digest.Quantile(q)-exact) / math.Abs(exact)
if relErr > 0.02 {
t.Fatalf("%s q=%v relative error %v exceeds 2%%", name, q, relErr)
}
}
}
}
func TestTDigestMergeDefersProcessingUntilThreshold(t *testing.T) {
source := NewTDigest(30)
source.Add(42, 1)
if source.processed {
t.Fatal("source should have buffered centroid before merge")
}
merged := NewTDigest(30)
for i := 0; i <= merged.processThreshold(); i++ {
merged.Merge(source)
}
if !merged.processed {
t.Fatal("merge did not process after exceeding threshold")
}
if source.processed {
t.Fatal("merge processed the source digest")
}
merged.Merge(source)
if merged.processed {
t.Fatal("merge below threshold should remain buffered")
}
if merged.Count() != float64(merged.processThreshold()+2) {
t.Fatalf("count = %v", merged.Count())
}
if merged.processed {
t.Fatal("Count processed buffered centroids")
}
if value := merged.Quantile(0.5); value != 42 || !merged.processed {
t.Fatalf("Quantile did not finalize digest: value=%v processed=%v", value, merged.processed)
}
merged.Merge(source)
if merged.processed {
t.Fatal("merge below threshold should remain buffered before Encode")
}
encoded := merged.Encode()
if len(encoded) == 0 || !merged.processed {
t.Fatal("Encode did not finalize digest")
}
decoded, err := DecodeTDigest(encoded)
if err != nil {
t.Fatal(err)
}
if decoded.Count() != merged.Count() || decoded.min != merged.min || decoded.max != merged.max {
t.Fatalf("round trip changed exact aggregates: %#v vs %#v", decoded, merged)
}
}
// TestTDigestEncodeRoundTrip verifies t-digest serialization.
//
// TestTDigestEncodeRoundTrip 验证 t-digest 编码和解码往返。
+107
View File
@@ -291,6 +291,43 @@ type Query struct {
Order Order `json:"order,omitempty"`
}
// BatchQuery loads exact raw samples for multiple metrics and entities from one
// shared time window.
type BatchQuery struct {
MetricNames []string `json:"metric_names"`
EntityIDs []string `json:"entity_ids,omitempty"`
Start time.Time `json:"start"`
End time.Time `json:"end"`
Tags map[string]string `json:"tags,omitempty"`
Order Order `json:"order,omitempty"`
}
// BatchSeriesSpec describes the rollup output requested for one metric.
type BatchSeriesSpec struct {
MetricName string
Aggregations []Aggregation
Interval time.Duration
PreserveSeries bool
}
// BatchSeriesQuery combines per-metric rollup specifications that share the
// same time, entity, tag, and ordering filters.
type BatchSeriesQuery struct {
Specs []BatchSeriesSpec
EntityIDs []string
Start time.Time
End time.Time
Tags map[string]string
Order Order
}
// BatchSeriesResult contains definitions and aggregate values keyed by metric
// and aggregation.
type BatchSeriesResult struct {
Definitions map[string]Definition
Values map[string]map[Aggregation][]AggregatePoint
}
// Validate checks whether the value is well formed.
//
// Validate 检查原始查询条件是否合法。
@@ -327,6 +364,76 @@ func (q Query) normalized() Query {
return q
}
// Validate checks whether a batch raw query is well formed.
func (q BatchQuery) Validate() error {
if len(q.MetricNames) == 0 {
return fmt.Errorf("%w: at least one metric name is required", ErrInvalidArgument)
}
for _, name := range q.MetricNames {
if strings.TrimSpace(name) == "" {
return fmt.Errorf("%w: metric name is required", ErrInvalidArgument)
}
}
if q.Start.IsZero() || q.End.IsZero() {
return fmt.Errorf("%w: start and end time are required", ErrInvalidArgument)
}
if q.End.Before(q.Start) {
return fmt.Errorf("%w: end time cannot be before start time", ErrInvalidArgument)
}
switch q.Order {
case "", OrderAsc, OrderDesc:
default:
return fmt.Errorf("%w: unsupported order %q", ErrInvalidArgument, q.Order)
}
return nil
}
func (q BatchQuery) normalized() BatchQuery {
q.Start = q.Start.UTC()
q.End = q.End.UTC()
if q.Order == "" {
q.Order = OrderAsc
}
return q
}
// Validate checks whether a batch rollup query is well formed.
func (q BatchSeriesQuery) Validate() error {
if len(q.Specs) == 0 {
return fmt.Errorf("%w: at least one series specification is required", ErrInvalidArgument)
}
seen := make(map[string]struct{}, len(q.Specs))
for _, spec := range q.Specs {
if _, ok := seen[spec.MetricName]; ok {
return fmt.Errorf("%w: duplicate series specification for metric %q", ErrInvalidArgument, spec.MetricName)
}
seen[spec.MetricName] = struct{}{}
if len(spec.Aggregations) == 0 {
return fmt.Errorf("%w: at least one aggregation is required for metric %q", ErrInvalidArgument, spec.MetricName)
}
for _, aggregation := range spec.Aggregations {
if err := (AggregateQuery{
Query: Query{MetricName: spec.MetricName, Start: q.Start, End: q.End, Tags: q.Tags, Order: q.Order},
Aggregation: aggregation,
Interval: spec.Interval,
PreserveSeries: spec.PreserveSeries,
}).Validate(); err != nil {
return err
}
}
}
return nil
}
func (q BatchSeriesQuery) normalized() BatchSeriesQuery {
q.Start = q.Start.UTC()
q.End = q.End.UTC()
if q.Order == "" {
q.Order = OrderAsc
}
return q
}
// AggregateQuery describes a bucketed aggregate query.
//
// AggregateQuery 描述按固定时间间隔分桶后的聚合查询。
-87
View File
@@ -1,87 +0,0 @@
package v1
import "time"
type Message struct {
Type string `json:"type"`
Content string `json:"content"`
Sender string `json:"sender"`
Timestamp int64 `json:"timestamp"`
}
type IPAddress struct {
Ipv4 string `json:"ipv4"`
Ipv6 string `json:"ipv6"`
}
type Report struct {
UUID string `json:"uuid,omitempty"`
CPU CPUReport `json:"cpu"`
Ram RamReport `json:"ram"`
Swap RamReport `json:"swap"`
Load LoadReport `json:"load"`
Disk DiskReport `json:"disk"`
Network NetworkReport `json:"network"`
Connections ConnectionsReport `json:"connections"`
GPU *GPUDetailReport `json:"gpu,omitempty"`
Uptime int64 `json:"uptime"`
Process int `json:"process"`
Message string `json:"message"`
Method string `json:"method,omitempty"`
UpdatedAt time.Time `json:"updated_at"`
}
type CPUReport struct {
Name string `json:"name,omitempty"`
Cores int `json:"cores,omitempty"`
Arch string `json:"arch,omitempty"`
Usage float64 `json:"usage,omitempty"`
}
type GPUDetailReport struct {
Count int `json:"count"`
AverageUsage float64 `json:"average_usage"`
DetailedInfo []GPUDeviceInfo `json:"detailed_info"`
}
type GPUDeviceInfo struct {
Name string `json:"name"`
MemoryTotal int64 `json:"memory_total"`
MemoryUsed int64 `json:"memory_used"`
Utilization float64 `json:"utilization"`
Temperature int `json:"temperature"`
}
// GPUReport is the legacy v1 single-GPU shape kept for wire compatibility.
type GPUReport struct {
Name string `json:"name,omitempty"`
Usage float64 `json:"usage,omitempty"`
}
type RamReport struct {
Total int64 `json:"total"`
Used int64 `json:"used"`
}
type LoadReport struct {
Load1 float64 `json:"load1"`
Load5 float64 `json:"load5"`
Load15 float64 `json:"load15"`
}
type DiskReport struct {
Total int64 `json:"total"`
Used int64 `json:"used"`
}
type NetworkReport struct {
Up int64 `json:"up"`
Down int64 `json:"down"`
TotalUp int64 `json:"totalUp"`
TotalDown int64 `json:"totalDown"`
}
type ConnectionsReport struct {
TCP int `json:"tcp"`
UDP int `json:"udp"`
}
+107 -4
View File
@@ -1,9 +1,8 @@
package v2
import (
"encoding/json"
"time"
v1 "github.com/komari-monitor/komari/protocol/v1"
)
const (
@@ -18,6 +17,8 @@ const (
MethodAgentEvent = "agent.event"
MethodAgentTerminal = "agent.terminal.request"
MethodAgentPull = "agent.pull"
MethodAgentFile = "agent.file"
MethodAgentFileResult = "agent.file.result"
)
type Request struct {
@@ -49,8 +50,86 @@ type RPCError struct {
}
type ReportParams struct {
Report v1.Report `json:"report"`
AckEventIDs []string `json:"ack_event_ids,omitempty"`
Report Report `json:"report"`
AckEventIDs []string `json:"ack_event_ids,omitempty"`
}
type Message struct {
Type string `json:"type"`
Content string `json:"content"`
Sender string `json:"sender"`
Timestamp int64 `json:"timestamp"`
}
type IPAddress struct {
Ipv4 string `json:"ipv4"`
Ipv6 string `json:"ipv6"`
}
type Report struct {
UUID string `json:"uuid,omitempty"`
CPU CPUReport `json:"cpu"`
Ram RamReport `json:"ram"`
Swap RamReport `json:"swap"`
Load LoadReport `json:"load"`
Disk DiskReport `json:"disk"`
Network NetworkReport `json:"network"`
Connections ConnectionsReport `json:"connections"`
GPU *GPUDetailReport `json:"gpu,omitempty"`
Uptime int64 `json:"uptime"`
Process int `json:"process"`
Message string `json:"message"`
Method string `json:"method,omitempty"`
UpdatedAt time.Time `json:"updated_at"`
}
type CPUReport struct {
Name string `json:"name,omitempty"`
Cores int `json:"cores,omitempty"`
Arch string `json:"arch,omitempty"`
Usage float64 `json:"usage,omitempty"`
}
type GPUDetailReport struct {
Count int `json:"count"`
AverageUsage float64 `json:"average_usage"`
DetailedInfo []GPUDeviceInfo `json:"detailed_info"`
}
type GPUDeviceInfo struct {
Name string `json:"name"`
MemoryTotal int64 `json:"memory_total"`
MemoryUsed int64 `json:"memory_used"`
Utilization float64 `json:"utilization"`
Temperature int `json:"temperature"`
}
type RamReport struct {
Total int64 `json:"total"`
Used int64 `json:"used"`
}
type LoadReport struct {
Load1 float64 `json:"load1"`
Load5 float64 `json:"load5"`
Load15 float64 `json:"load15"`
}
type DiskReport struct {
Total int64 `json:"total"`
Used int64 `json:"used"`
}
type NetworkReport struct {
Up int64 `json:"up"`
Down int64 `json:"down"`
TotalUp int64 `json:"totalUp"`
TotalDown int64 `json:"totalDown"`
}
type ConnectionsReport struct {
TCP int `json:"tcp"`
UDP int `json:"udp"`
}
type BasicInfoParams struct {
@@ -64,6 +143,13 @@ type PingResultParams struct {
FinishedAt time.Time `json:"finished_at"`
}
type TaskResultParams struct {
TaskID string `json:"task_id"`
Result string `json:"result"`
ExitCode int `json:"exit_code"`
FinishedAt time.Time `json:"finished_at"`
}
type PullParams struct {
Capabilities []string `json:"capabilities,omitempty"`
AckEventIDs []string `json:"ack_event_ids,omitempty"`
@@ -96,6 +182,23 @@ type TerminalRequestParams struct {
RequestID string `json:"request_id"`
}
// FileOperation is metadata-only. File contents travel through the dedicated
// HTTP transfer endpoint rather than through JSON-RPC.
type FileOperation struct {
UUID string `json:"uuid"`
RequestID string `json:"request_id"`
Op string `json:"op"`
Args map[string]any `json:"args,omitempty"`
}
type FileResult struct {
UUID string `json:"uuid"`
RequestID string `json:"request_id"`
OK bool `json:"ok"`
Result json.RawMessage `json:"result,omitempty"`
Error string `json:"error,omitempty"`
}
func Success(id any, result any) Response {
return Response{JSONRPC: Version, ID: id, Result: result}
}
+2 -3
View File
@@ -4,7 +4,6 @@ import (
"encoding/json"
"errors"
"fmt"
"time"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/pkg/jsruntime"
@@ -88,7 +87,7 @@ func (j *JavaScriptSender) fallbackToTextMessage(event models.EventMessage) erro
event.Emoji, event.Emoji, event.Emoji,
event.Event,
event.Message,
event.Time.UTC().Format(time.RFC3339Nano))
event.Time.UTC().Format("2006-01-02 15:04:05"))
if len(event.Clients) > 0 {
clientNames := make([]string, 0, len(event.Clients))
for _, client := range event.Clients {
@@ -103,7 +102,7 @@ func (j *JavaScriptSender) fallbackToTextMessage(event models.EventMessage) erro
event.Event,
clientNames,
event.Message,
event.Time.UTC().Format(time.RFC3339Nano))
event.Time.UTC().Format("2006-01-02 15:04:05"))
}
return j.SendTextMessage(message, fmt.Sprint(event.Event))
}
+1 -1
View File
@@ -15,7 +15,7 @@ func TestParseTemplateFormatsEventTimeInLocalTimezone(t *testing.T) {
eventTime := time.Date(2026, 7, 17, 1, 30, 0, 123456789, time.UTC)
got := parseTemplate("{{time}}", models.EventMessage{Time: eventTime})
want := "2026-07-17T09:30:00.123456789+08:00"
want := "2026-07-17 09:30:00"
if got != want {
t.Fatalf("formatted event time = %q, want %q", got, want)
}
+1 -1
View File
@@ -232,7 +232,7 @@ func formatTemplateField(fieldName string, v reflect.Value) string {
}
case reflect.Struct:
if t, ok := v.Interface().(time.Time); ok {
return t.In(time.Local).Format(time.RFC3339Nano)
return t.In(time.Local).Format("2006-01-02 15:04:05")
}
}
return ""
+1 -1
View File
@@ -130,7 +130,7 @@ func sendTrafficReport(daily, weekly, monthly bool) {
return
}
message := strings.Join(lines, "\n")
message := strings.Join(lines, "\nKomari 内置的流量报告即将在 1.5.0 版本移除,请尽快迁移至插件“流量定期报告”。\nKomari's built-in traffic reports will be removed in version 1.5.0. Please migrate to the Scheduled Traffic Reports plugin as soon as possible.")
var emoji string
switch {
case daily:
+1 -13
View File
@@ -57,18 +57,6 @@ func (m *PingTaskManager) Reload(pingTasks []models.PingTask) error {
// executePingTask 执行单个PingTask
func executePingTask(ctx context.Context, task models.PingTask) {
var message struct {
TaskID uint `json:"ping_task_id"`
Message string `json:"message"`
Type string `json:"ping_type"`
Target string `json:"ping_target"`
}
message.Message = "ping"
message.TaskID = task.Id
message.Type = task.Type
message.Target = task.Target
for _, clientUUID := range targetPingClientUUIDs(task) {
select {
case <-ctx.Done():
@@ -78,7 +66,7 @@ func executePingTask(ctx context.Context, task models.PingTask) {
// Context is still active, continue.
}
agent_runtime.DispatchPing(clientUUID, message, v2.PingParams{TaskID: task.Id, Type: task.Type, Target: task.Target})
agent_runtime.DispatchPing(clientUUID, v2.PingParams{TaskID: task.Id, Type: task.Type, Target: task.Target})
}
}
+20 -19
View File
@@ -5,15 +5,15 @@ import (
"sync"
"time"
v1 "github.com/komari-monitor/komari/protocol/v1"
v2 "github.com/komari-monitor/komari/protocol/v2"
"github.com/komari-monitor/komari/web/connection"
)
var (
connectedClients = make(map[string]*connection.SafeConn)
connectedClientV2 = make(map[string]bool)
latestReport = make(map[string]*v1.Report)
recentReports = make(map[string][]v1.Report)
connectedClients = make(map[string]*connection.SafeConn)
v2Clients = make(map[string]struct{})
latestReport = make(map[string]*v2.Report)
recentReports = make(map[string][]v2.Report)
// presenceOnly stores online state for non-WebSocket agents.
// value keeps connectionID and a soft expiration to avoid flicker
presenceOnly = make(map[string]struct {
@@ -41,16 +41,17 @@ func SetConnectedClients(uuid string, conn *connection.SafeConn) {
connectedClients[uuid] = conn
}
func SetClientProtocolVersion(uuid string, version int) {
func MarkV2Client(uuid string) {
mu.Lock()
defer mu.Unlock()
connectedClientV2[uuid] = version >= 2
v2Clients[uuid] = struct{}{}
}
func IsV2Client(uuid string) bool {
mu.RLock()
defer mu.RUnlock()
return connectedClientV2[uuid]
_, ok := v2Clients[uuid]
return ok
}
func DeleteClientConditionally(uuid string, connToRemove *connection.SafeConn) {
@@ -60,7 +61,7 @@ func DeleteClientConditionally(uuid string, connToRemove *connection.SafeConn) {
// 检查当前 map 里的 conn 是否就是要删除的这一个
if currentConn, exists := connectedClients[uuid]; exists && currentConn == connToRemove {
delete(connectedClients, uuid)
delete(connectedClientV2, uuid)
delete(v2Clients, uuid)
}
}
func DeleteConnectedClients(uuid string) {
@@ -68,7 +69,7 @@ func DeleteConnectedClients(uuid string) {
defer mu.Unlock()
// 只从 map 中删除,不再负责关闭连接
delete(connectedClients, uuid)
delete(connectedClientV2, uuid)
delete(v2Clients, uuid)
}
// SetPresence sets or clears presence for non-WebSocket agents.
@@ -121,10 +122,10 @@ func GetAllOnlineUUIDs() []string {
}
return res
}
func GetLatestReport() map[string]*v1.Report {
func GetLatestReport() map[string]*v2.Report {
mu.RLock()
defer mu.RUnlock()
reportCopy := make(map[string]*v1.Report)
reportCopy := make(map[string]*v2.Report)
for k, v := range latestReport {
if v == nil {
continue
@@ -137,7 +138,7 @@ func GetLatestReport() map[string]*v1.Report {
// RecordReport updates the latest runtime state and keeps only the short raw
// window used by recent-status compatibility endpoints.
func RecordReport(report v1.Report) {
func RecordReport(report v2.Report) {
if report.UUID == "" {
return
}
@@ -161,30 +162,30 @@ func RecordReport(report v1.Report) {
insertAt := sort.Search(len(reports), func(i int) bool {
return reports[i].UpdatedAt.After(report.UpdatedAt)
})
reports = append(reports, v1.Report{})
reports = append(reports, v2.Report{})
copy(reports[insertAt+1:], reports[insertAt:])
reports[insertAt] = report
recentReports[report.UUID] = reports
}
func GetRecentReports(uuid string) []v1.Report {
func GetRecentReports(uuid string) []v2.Report {
mu.Lock()
defer mu.Unlock()
reports := reportsAfter(recentReports[uuid], time.Now().UTC().Add(-recentReportRetention))
if len(reports) == 0 {
delete(recentReports, uuid)
return []v1.Report{}
return []v2.Report{}
}
recentReports[uuid] = reports
return append([]v1.Report(nil), reports...)
return append([]v2.Report(nil), reports...)
}
func reportsAfter(reports []v1.Report, cutoff time.Time) []v1.Report {
func reportsAfter(reports []v2.Report, cutoff time.Time) []v2.Report {
first := 0
for first < len(reports) && reports[first].UpdatedAt.Before(cutoff) {
first++
}
out := make([]v1.Report, len(reports)-first)
out := make([]v2.Report, len(reports)-first)
copy(out, reports[first:])
return out
}
+6 -6
View File
@@ -4,15 +4,15 @@ import (
"testing"
"time"
v1 "github.com/komari-monitor/komari/protocol/v1"
v2 "github.com/komari-monitor/komari/protocol/v2"
)
func TestRecordReportKeepsLatestAndShortRecentWindow(t *testing.T) {
mu.Lock()
previousLatest := latestReport
previousRecent := recentReports
latestReport = make(map[string]*v1.Report)
recentReports = make(map[string][]v1.Report)
latestReport = make(map[string]*v2.Report)
recentReports = make(map[string][]v2.Report)
mu.Unlock()
t.Cleanup(func() {
mu.Lock()
@@ -22,9 +22,9 @@ func TestRecordReportKeepsLatestAndShortRecentWindow(t *testing.T) {
})
now := time.Now().UTC()
RecordReport(v1.Report{UUID: "node-a", UpdatedAt: now.Add(-2 * time.Minute), CPU: v1.CPUReport{Usage: 10}})
RecordReport(v1.Report{UUID: "node-a", UpdatedAt: now.Add(-30 * time.Second), CPU: v1.CPUReport{Usage: 20}})
RecordReport(v1.Report{UUID: "node-a", UpdatedAt: now.Add(-45 * time.Second), CPU: v1.CPUReport{Usage: 15}})
RecordReport(v2.Report{UUID: "node-a", UpdatedAt: now.Add(-2 * time.Minute), CPU: v2.CPUReport{Usage: 10}})
RecordReport(v2.Report{UUID: "node-a", UpdatedAt: now.Add(-30 * time.Second), CPU: v2.CPUReport{Usage: 20}})
RecordReport(v2.Report{UUID: "node-a", UpdatedAt: now.Add(-45 * time.Second), CPU: v2.CPUReport{Usage: 15}})
recent := GetRecentReports("node-a")
if len(recent) != 2 || recent[0].CPU.Usage != 15 || recent[1].CPU.Usage != 20 {
+15 -5
View File
@@ -15,6 +15,10 @@ const (
v2EventQueueLimit = 128
v2EventTTL = 5 * time.Minute
v2PingEventTTL = 3 * time.Second
// File operations may include a remote read/search with a 90 second
// deadline, so queued file commands need a little headroom while an agent
// reconnects.
v2FileEventTTL = 2 * time.Minute
)
type v2EventQueue struct {
@@ -50,12 +54,9 @@ func DispatchV2Event(uuid, method string, params any) bool {
return true
}
func DispatchPing(uuid string, legacy any, params v2.PingParams) bool {
func DispatchPing(uuid string, params v2.PingParams) bool {
if conn := GetConnectedClients()[uuid]; conn != nil {
payload := legacy
if IsV2Client(uuid) {
payload = v2.Request{JSONRPC: v2.Version, Method: v2.MethodAgentPing, Params: params}
}
payload := v2.Request{JSONRPC: v2.Version, Method: v2.MethodAgentPing, Params: params}
if conn.WriteJSON(payload) == nil {
return true
}
@@ -79,6 +80,8 @@ func EnqueueV2Event(uuid, method string, params any) v2.Event {
ttl := v2EventTTL
if method == v2.MethodAgentPing {
ttl = v2PingEventTTL
} else if method == v2.MethodAgentFile {
ttl = v2FileEventTTL
}
event := v2.Event{
ID: newV2EventID(),
@@ -126,6 +129,13 @@ func coalesceV2EventLocked(q *v2EventQueue, event v2.Event) {
}
func v2EventCoalesceKey(event v2.Event) string {
if event.Method == v2.MethodAgentTerminal {
var params v2.TerminalRequestParams
if err := bindV2EventParams(event.Params, &params); err == nil && params.RequestID != "" {
return event.Method + ":" + params.RequestID
}
return ""
}
if event.Method != v2.MethodAgentPing {
return ""
}
+1
View File
@@ -94,6 +94,7 @@ var publicPaths = []string{
"/api/recent",
"/api/admin", // 由 RequireRole 处理
"/api/clients/", // 由 RequireRole 处理
"/api/preview/", // 预览令牌校验后放行
}
// PrivateSiteMiddleware 私有站点访问控制。
+70
View File
@@ -0,0 +1,70 @@
package admin
import (
"fmt"
"os"
"path/filepath"
"time"
"github.com/komari-monitor/komari/internal/plugin"
logger "github.com/komari-monitor/komari/utils/log"
"github.com/komari-monitor/komari/web/backup"
"github.com/komari-monitor/komari/web/upload"
)
func NewArchiveUploadHandler() *upload.Handler {
return upload.NewHandler(upload.DefaultStore, map[upload.Purpose]upload.Finalizer{
upload.PurposeBackup: finalizeBackupUpload,
upload.PurposePlugin: finalizePluginUpload,
upload.PurposeTheme: finalizeThemeUpload,
})
}
func finalizeBackupUpload(session upload.Session) (upload.Result, error) {
restoreLock, err := backup.AcquireRestoreLock()
if err != nil {
return upload.Result{}, err
}
archive, err := os.Open(session.ArchivePath)
if err != nil {
restoreLock.Release()
return upload.Result{}, fmt.Errorf("open merged backup: %w", err)
}
if err := restoreLock.SaveUploadedBackup(archive, session.Metadata.Filename); err != nil {
_ = archive.Close()
restoreLock.Release()
return upload.Result{}, err
}
if err := archive.Close(); err != nil {
restoreLock.Release()
return upload.Result{}, fmt.Errorf("close merged backup: %w", err)
}
go func() {
logger.InfoArgs("admin-api", "Backup uploaded, restarting service in 2 seconds to apply on startup...")
time.Sleep(2 * time.Second)
restoreLock.Release()
os.Exit(0)
}()
return upload.Result{
Message: "Backup uploaded successfully. The service will restart and apply the backup.",
Data: map[string]string{"path": filepath.Join(".", "data", "backup.zip")},
}, nil
}
func finalizePluginUpload(session upload.Session) (upload.Result, error) {
info, err := plugin.InstallZip(session.ArchivePath)
if err != nil {
return upload.Result{}, err
}
return upload.Result{Message: "插件上传成功", Data: info}, nil
}
func finalizeThemeUpload(session upload.Session) (upload.Result, error) {
info, err := extractAndValidateTheme(session.ArchivePath)
if err != nil {
return upload.Result{}, err
}
return upload.Result{Message: "主题上传成功", Data: info}, nil
}
-111
View File
@@ -1,111 +0,0 @@
package admin
import (
"encoding/json"
"os"
"path/filepath"
"testing"
)
func TestIsValidUploadID(t *testing.T) {
for _, uploadID := range []string{chunkUploadID, "../backup", "", "another-upload"} {
want := uploadID == chunkUploadID
if got := isValidUploadID(uploadID); got != want {
t.Errorf("isValidUploadID(%q) = %v, want %v", uploadID, got, want)
}
}
}
func TestChunkUploadDirRejectsTraversal(t *testing.T) {
if _, err := chunkUploadDir(".."); err == nil {
t.Fatal("chunkUploadDir accepted traversal upload_id")
}
}
func TestClearChunkUploadCache(t *testing.T) {
root := t.TempDir()
if err := os.WriteFile(filepath.Join(root, "interrupted.part"), []byte("stale"), 0600); err != nil {
t.Fatalf("write cache: %v", err)
}
if err := clearChunkUploadCache(root); err != nil {
t.Fatalf("clearChunkUploadCache: %v", err)
}
entries, err := os.ReadDir(root)
if err != nil {
t.Fatalf("read cache: %v", err)
}
if len(entries) != 0 {
t.Fatalf("cache entries = %d, want 0", len(entries))
}
}
func TestCopyWhitelistedFilesIncludesFont(t *testing.T) {
dataDir := t.TempDir()
backupDir := t.TempDir()
fontPath := filepath.Join(dataDir, "font.ttf")
if err := os.WriteFile(fontPath, []byte("font-data"), 0644); err != nil {
t.Fatalf("write font: %v", err)
}
if err := copyWhitelistedFilesFrom(dataDir, backupDir); err != nil {
t.Fatalf("copyWhitelistedFilesFrom: %v", err)
}
content, err := os.ReadFile(filepath.Join(backupDir, "font.ttf"))
if err != nil {
t.Fatalf("read copied font: %v", err)
}
if string(content) != "font-data" {
t.Fatalf("font content = %q, want %q", content, "font-data")
}
}
func TestChunkUploadMetadataAndSize(t *testing.T) {
dir := t.TempDir()
metadata, err := json.Marshal(chunkUploadMetadata{Size: 7})
if err != nil {
t.Fatalf("marshal metadata: %v", err)
}
if err := os.WriteFile(filepath.Join(dir, uploadMetadataName), metadata, 0600); err != nil {
t.Fatalf("write metadata: %v", err)
}
for name, content := range map[string]string{"0.part": "abc", "1.part": "defg", ".1-staging.part": "staging", "ignored.txt": "ignored"} {
if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0600); err != nil {
t.Fatalf("write %s: %v", name, err)
}
}
gotMetadata, err := readChunkUploadMetadata(dir)
if err != nil {
t.Fatalf("read metadata: %v", err)
}
if gotMetadata.Size != 7 {
t.Fatalf("metadata size = %d, want 7", gotMetadata.Size)
}
gotSize, err := chunkUploadSize(dir)
if err != nil {
t.Fatalf("chunkUploadSize: %v", err)
}
if gotSize != 7 {
t.Fatalf("chunk upload size = %d, want 7", gotSize)
}
}
func TestMergeChunksIgnoresTemporaryParts(t *testing.T) {
dir := t.TempDir()
for name, content := range map[string]string{"0.part": "first", "1.part": "second", ".1-staging.part": "temporary"} {
if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0600); err != nil {
t.Fatalf("write %s: %v", name, err)
}
}
destination := filepath.Join(dir, "merged.zip")
if err := mergeChunks(dir, destination); err != nil {
t.Fatalf("mergeChunks: %v", err)
}
content, err := os.ReadFile(destination)
if err != nil {
t.Fatalf("read merged chunks: %v", err)
}
if string(content) != "firstsecond" {
t.Fatalf("merged chunks = %q, want %q", content, "firstsecond")
}
}
+2
View File
@@ -15,6 +15,8 @@ var backupWhitelist = []string{
"favicon.ico",
"font.ttf",
"theme/",
"plugin/",
"plguin-data/",
"metrics.db",
}
-381
View File
@@ -1,381 +0,0 @@
package admin
import (
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"time"
"github.com/gin-gonic/gin"
"github.com/komari-monitor/komari/web/api"
"github.com/komari-monitor/komari/web/backup"
)
const (
defaultChunkSize = 5 * 1024 * 1024 // 5 MiB
maxChunkRequestSize = defaultChunkSize + 1*1024*1024
chunkUploadID = "backup"
maxChunkIndex = int(backup.MaxArchiveSize / defaultChunkSize)
)
var chunkUploadRoot = filepath.Join(".", "data", "backup", ".uploading")
const uploadMetadataName = "upload.json"
type chunkUploadMetadata struct {
Size int64 `json:"size"`
}
func isValidUploadID(uploadID string) bool {
return uploadID == chunkUploadID
}
func chunkUploadDir(uploadID string) (string, error) {
if !isValidUploadID(uploadID) {
return "", fmt.Errorf("invalid upload_id")
}
return filepath.Join(chunkUploadRoot, uploadID), nil
}
func readChunkUploadMetadata(chunkDir string) (chunkUploadMetadata, error) {
var metadata chunkUploadMetadata
data, err := os.ReadFile(filepath.Join(chunkDir, uploadMetadataName))
if err != nil {
return metadata, err
}
if err := json.Unmarshal(data, &metadata); err != nil {
return metadata, err
}
if metadata.Size <= 0 || metadata.Size > backup.MaxArchiveSize {
return metadata, fmt.Errorf("invalid upload size")
}
return metadata, nil
}
func chunkPartIndex(name string) (int, bool) {
index, err := strconv.Atoi(strings.TrimSuffix(name, ".part"))
return index, err == nil && index >= 0 && strings.HasSuffix(name, ".part")
}
func chunkUploadSize(chunkDir string) (int64, error) {
entries, err := os.ReadDir(chunkDir)
if err != nil {
return 0, err
}
var total int64
for _, entry := range entries {
if entry.IsDir() {
continue
}
if _, ok := chunkPartIndex(entry.Name()); !ok {
continue
}
info, err := entry.Info()
if err != nil {
return 0, err
}
if total > backup.MaxArchiveSize-info.Size() {
return 0, fmt.Errorf("uploaded chunks exceed the archive limit")
}
total += info.Size()
}
return total, nil
}
func clearChunkUploadCache(root string) error {
if err := os.RemoveAll(root); err != nil {
return err
}
return os.MkdirAll(root, 0755)
}
// InitChunkUpload 初始化分块上传,返回 upload_id 和 chunk_size。
// Komari 使用单一上传槽位,初始化时会清除上次中断留下的全部缓存。
func InitChunkUpload(c *gin.Context) {
var req struct {
Size int64 `json:"size"`
}
if err := c.ShouldBindJSON(&req); err != nil || req.Size <= 0 || req.Size > backup.MaxArchiveSize {
api.RespondError(c, http.StatusBadRequest, fmt.Sprintf("size must be between 1 and %d bytes", backup.MaxArchiveSize))
return
}
if err := clearChunkUploadCache(chunkUploadRoot); err != nil {
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error clearing upload cache: %v", err))
return
}
chunkDir, err := chunkUploadDir(chunkUploadID)
if err != nil {
api.RespondError(c, http.StatusInternalServerError, "Error preparing upload directory")
return
}
if err := os.MkdirAll(chunkDir, 0755); err != nil {
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error creating upload directory: %v", err))
return
}
metadata, err := json.Marshal(chunkUploadMetadata{Size: req.Size})
if err != nil {
clearChunkUploadCache(chunkUploadRoot)
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error encoding upload metadata: %v", err))
return
}
if err := os.WriteFile(filepath.Join(chunkDir, uploadMetadataName), metadata, 0600); err != nil {
clearChunkUploadCache(chunkUploadRoot)
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error writing upload metadata: %v", err))
return
}
c.JSON(http.StatusOK, gin.H{
"upload_id": chunkUploadID,
"chunk_size": defaultChunkSize,
})
}
// UploadChunk 接收单个分块,保存到临时目录。
func UploadChunk(c *gin.Context) {
defer func() {
if c.Request.Context().Err() != nil {
_ = clearChunkUploadCache(chunkUploadRoot)
}
}()
c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, maxChunkRequestSize)
uploadID := c.PostForm("upload_id")
chunkDir, err := chunkUploadDir(uploadID)
if err != nil {
api.RespondError(c, http.StatusBadRequest, "invalid upload_id")
return
}
chunkIndexStr := c.PostForm("chunk_index")
if chunkIndexStr == "" {
api.RespondError(c, http.StatusBadRequest, "chunk_index is required")
return
}
chunkIndex, err := strconv.Atoi(chunkIndexStr)
if err != nil || chunkIndex < 0 || chunkIndex > maxChunkIndex {
api.RespondError(c, http.StatusBadRequest, "chunk_index must be a non-negative integer")
return
}
if info, err := os.Stat(chunkDir); err != nil || !info.IsDir() {
api.RespondError(c, http.StatusNotFound, "upload_id not found or expired")
return
}
file, _, err := c.Request.FormFile("chunk_data")
if err != nil {
api.RespondError(c, http.StatusBadRequest, fmt.Sprintf("Error getting chunk data: %v", err))
return
}
defer file.Close()
metadata, err := readChunkUploadMetadata(chunkDir)
if err != nil {
api.RespondError(c, http.StatusNotFound, "upload_id not found or expired")
return
}
chunkPath := filepath.Join(chunkDir, fmt.Sprintf("%d.part", chunkIndex))
tempChunk, err := os.CreateTemp(chunkDir, fmt.Sprintf(".%d-*.part", chunkIndex))
if err != nil {
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error saving chunk: %v", err))
return
}
tempChunkPath := tempChunk.Name()
defer os.Remove(tempChunkPath)
written, err := io.Copy(tempChunk, io.LimitReader(file, defaultChunkSize+1))
if closeErr := tempChunk.Close(); err == nil {
err = closeErr
}
if err != nil || written > defaultChunkSize {
if err != nil {
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error writing chunk: %v", err))
} else {
api.RespondError(c, http.StatusBadRequest, fmt.Sprintf("Chunk too large: %d bytes (max %d)", written, defaultChunkSize))
}
return
}
currentSize, err := chunkUploadSize(chunkDir)
if err != nil {
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error checking uploaded chunks: %v", err))
return
}
oldSize := int64(0)
if info, err := os.Stat(chunkPath); err == nil {
oldSize = info.Size()
} else if !os.IsNotExist(err) {
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error checking existing chunk: %v", err))
return
}
if currentSize-oldSize+written > metadata.Size {
api.RespondError(c, http.StatusBadRequest, "uploaded chunks exceed the declared backup size")
return
}
if err := os.Remove(chunkPath); err != nil && !os.IsNotExist(err) {
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error replacing chunk: %v", err))
return
}
if err := os.Rename(tempChunkPath, chunkPath); err != nil {
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error publishing chunk: %v", err))
return
}
c.JSON(http.StatusOK, gin.H{
"received": true,
"chunk_index": chunkIndex,
})
}
// MergeChunkUpload 合并分块到 backup.zip 后触发恢复。
func MergeChunkUpload(c *gin.Context) {
restoreLock, err := backup.AcquireRestoreLock()
if err != nil {
api.RespondError(c, http.StatusConflict, err.Error())
return
}
keepRestoreLock := false
defer func() {
if !keepRestoreLock {
restoreLock.Release()
}
}()
var req struct {
UploadID string `json:"upload_id" binding:"required"`
}
if err := c.ShouldBindJSON(&req); err != nil {
api.RespondError(c, http.StatusBadRequest, fmt.Sprintf("Invalid request: %v", err))
return
}
chunkDir, err := chunkUploadDir(req.UploadID)
if err != nil {
api.RespondError(c, http.StatusBadRequest, "invalid upload_id")
return
}
if info, err := os.Stat(chunkDir); err != nil || !info.IsDir() {
api.RespondError(c, http.StatusNotFound, "upload_id not found or expired")
return
}
metadata, err := readChunkUploadMetadata(chunkDir)
if err != nil {
api.RespondError(c, http.StatusNotFound, "upload_id not found or expired")
return
}
if uploadedSize, err := chunkUploadSize(chunkDir); err != nil || uploadedSize != metadata.Size {
_ = clearChunkUploadCache(chunkUploadRoot)
if err != nil {
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error checking uploaded chunks: %v", err))
} else {
api.RespondError(c, http.StatusBadRequest, "uploaded chunk size does not match the declared backup size")
}
return
}
mergedPath := filepath.Join(chunkDir, "merged.zip")
if err := mergeChunks(chunkDir, mergedPath); err != nil {
clearChunkUploadCache(chunkUploadRoot)
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error merging chunks: %v", err))
return
}
if err := validateBackupZip(mergedPath); err != nil {
clearChunkUploadCache(chunkUploadRoot)
api.RespondError(c, http.StatusBadRequest, fmt.Sprintf("Invalid backup: %v", err))
return
}
archive, err := os.Open(mergedPath)
if err != nil {
clearChunkUploadCache(chunkUploadRoot)
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error opening merged backup: %v", err))
return
}
if err := restoreLock.SaveUploadedBackup(archive, "backup.zip"); err != nil {
archive.Close()
clearChunkUploadCache(chunkUploadRoot)
api.RespondError(c, http.StatusBadRequest, fmt.Sprintf("Error preparing restore: %v", err))
return
}
if err := archive.Close(); err != nil {
clearChunkUploadCache(chunkUploadRoot)
api.RespondError(c, http.StatusInternalServerError, fmt.Sprintf("Error closing merged backup: %v", err))
return
}
clearChunkUploadCache(chunkUploadRoot)
c.JSON(http.StatusOK, gin.H{
"status": "success",
"message": "Backup uploaded successfully. The service will restart and apply the backup.",
"path": filepath.Join(".", "data", "backup.zip"),
})
keepRestoreLock = true
go func() {
log.Println("Backup uploaded (chunk), restarting service in 2 seconds to apply on startup...")
time.Sleep(2 * time.Second)
restoreLock.Release()
os.Exit(0)
}()
}
// mergeChunks 按分块索引数值排序后顺序合并所有 .part 文件。
func mergeChunks(chunkDir, destPath string) error {
parts, err := filepath.Glob(filepath.Join(chunkDir, "*.part"))
if err != nil {
return err
}
numericParts := parts[:0]
for _, partPath := range parts {
if _, ok := chunkPartIndex(filepath.Base(partPath)); ok {
numericParts = append(numericParts, partPath)
}
}
parts = numericParts
if len(parts) == 0 {
return fmt.Errorf("no chunks found")
}
// 按分块索引数值排序(非字典序),避免 10.part 排在 2.part 前面
sort.Slice(parts, func(i, j int) bool {
idxI, _ := chunkPartIndex(filepath.Base(parts[i]))
idxJ, _ := chunkPartIndex(filepath.Base(parts[j]))
return idxI < idxJ
})
out, err := os.Create(destPath)
if err != nil {
return err
}
defer out.Close()
for _, partPath := range parts {
f, err := os.Open(partPath)
if err != nil {
return fmt.Errorf("error opening chunk %s: %v", filepath.Base(partPath), err)
}
if _, err := io.Copy(out, f); err != nil {
f.Close()
return fmt.Errorf("error reading chunk %s: %v", filepath.Base(partPath), err)
}
f.Close()
}
return nil
}
// validateBackupZip 校验 ZIP 结构完整性及 komari-backup-markup 标记文件。
func validateBackupZip(zipPath string) error {
if err := backup.ValidateArchive(zipPath); err != nil {
return fmt.Errorf("not a valid backup archive: %w", err)
}
return nil
}
+208
View File
@@ -0,0 +1,208 @@
package admin
import (
"context"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/netip"
"net/url"
"time"
"github.com/komari-monitor/komari/internal/config"
"gorm.io/gorm"
)
// marketDownloadMaxBytes limits the response size of a market download.
const marketDownloadMaxBytes = 100 << 20
const marketDownloadTimeout = 45 * time.Second
var blockedMarketIPPrefixes = []netip.Prefix{
netip.MustParsePrefix("0.0.0.0/8"),
netip.MustParsePrefix("100.64.0.0/10"), // RFC 6598 shared address space.
netip.MustParsePrefix("192.0.0.0/24"),
netip.MustParsePrefix("192.0.2.0/24"), // TEST-NET-1.
netip.MustParsePrefix("198.18.0.0/15"), // Benchmarking.
netip.MustParsePrefix("198.51.100.0/24"), // TEST-NET-2.
netip.MustParsePrefix("203.0.113.0/24"), // TEST-NET-3.
netip.MustParsePrefix("240.0.0.0/4"), // Reserved, including broadcast.
netip.MustParsePrefix("2001:db8::/32"), // Documentation.
netip.MustParsePrefix("fec0::/10"), // Deprecated site-local unicast.
}
// protectedMarketDownloadTransport resolves a hostname once per connection,
// checks every result, and dials the validated IP directly. Proxies are
// disabled so the destination checked here is the destination actually used.
var protectedMarketDownloadTransport = newSSRFProtectedTransport()
// IsSSRFProtectionEnabled reports whether SSRF protection is enabled for
// market downloads. An unreadable setting is treated as enabled so callers
// using this status helper fail closed.
func IsSSRFProtectionEnabled() bool {
enabled, err := ssrfProtectionEnabled()
return err != nil || enabled
}
// DownloadMarketURL performs a market download. When SSRF protection is
// enabled, URLs resolving to private or internal addresses are rejected
// (including on every redirect hop); when disabled, downloads proceed
// without address filtering.
func DownloadMarketURL(rawURL string, maxSize int64) ([]byte, error) {
if err := validateMarketDownloadURL(rawURL); err != nil {
return nil, err
}
protectionEnabled, err := ssrfProtectionEnabled()
if err != nil {
return nil, fmt.Errorf("unable to determine SSRF protection state: %w", err)
}
transport := http.DefaultTransport
if protectionEnabled {
transport = protectedMarketDownloadTransport
}
client := &http.Client{
Timeout: marketDownloadTimeout,
Transport: transport,
CheckRedirect: func(req *http.Request, via []*http.Request) error {
if len(via) >= 10 {
return errors.New("too many redirects")
}
return validateMarketDownloadURL(req.URL.String())
},
}
resp, err := client.Get(rawURL)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("HTTP status %d", resp.StatusCode)
}
data, err := io.ReadAll(io.LimitReader(resp.Body, maxSize+1))
if err != nil {
return nil, err
}
if int64(len(data)) > maxSize {
return nil, fmt.Errorf("response exceeds the %d byte limit", maxSize)
}
if len(data) == 0 {
return nil, errors.New("empty response")
}
return data, nil
}
func validateMarketDownloadURL(rawURL string) error {
parsed, err := url.Parse(rawURL)
if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") || parsed.Hostname() == "" || parsed.User != nil {
return errors.New("only HTTP and HTTPS URLs are allowed")
}
return nil
}
func newSSRFProtectedTransport() *http.Transport {
base, ok := http.DefaultTransport.(*http.Transport)
if !ok {
base = &http.Transport{}
}
transport := base.Clone()
transport.Proxy = nil
transport.Dial = nil
transport.DialTLS = nil
transport.DialTLSContext = nil
transport.DialContext = dialMarketAddress
return transport
}
func dialMarketAddress(ctx context.Context, network, address string) (net.Conn, error) {
host, port, err := net.SplitHostPort(address)
if err != nil {
return nil, fmt.Errorf("invalid market download address %q: %w", address, err)
}
ips, err := resolveMarketIPs(ctx, host)
if err != nil {
return nil, fmt.Errorf("failed to resolve market download host %q: %w", host, err)
}
for _, ip := range ips {
if isBlockedMarketIP(ip) {
return nil, errors.New("requests to private or internal addresses are not allowed")
}
}
dialer := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}
var lastErr error
for _, ip := range ips {
if network == "tcp4" && ip.To4() == nil {
continue
}
if network == "tcp6" && ip.To4() != nil {
continue
}
conn, dialErr := dialer.DialContext(ctx, network, net.JoinHostPort(ip.String(), port))
if dialErr == nil {
return conn, nil
}
lastErr = dialErr
}
if lastErr == nil {
return nil, fmt.Errorf("no compatible address found for market download host %q", host)
}
return nil, fmt.Errorf("failed to connect to market download host %q: %w", host, lastErr)
}
func resolveMarketIPs(ctx context.Context, host string) ([]net.IP, error) {
if ip := net.ParseIP(host); ip != nil {
return []net.IP{ip}, nil
}
addresses, err := net.DefaultResolver.LookupIPAddr(ctx, host)
if err != nil {
return nil, err
}
ips := make([]net.IP, 0, len(addresses))
for _, address := range addresses {
if address.IP != nil {
ips = append(ips, address.IP)
}
}
if len(ips) == 0 {
return nil, errors.New("host did not resolve to an IP address")
}
return ips, nil
}
func isBlockedMarketIP(ip net.IP) bool {
if ip == nil || !ip.IsGlobalUnicast() || ip.IsPrivate() {
return true
}
address, ok := netip.AddrFromSlice(ip)
if !ok {
return true
}
address = address.Unmap()
for _, prefix := range blockedMarketIPPrefixes {
if prefix.Contains(address) {
return true
}
}
return false
}
func ssrfProtectionEnabled() (bool, error) {
enabled, err := config.GetAs[bool](config.SSRFProtectionEnabledKey)
if err == nil {
return enabled, nil
}
if !errors.Is(err, gorm.ErrRecordNotFound) {
return false, err
}
if err := config.Set(config.SSRFProtectionEnabledKey, false); err != nil {
return false, err
}
return false, nil
}
-37
View File
@@ -1,50 +1,13 @@
package admin
import (
"io"
"net/http"
"os"
"strings"
"github.com/gin-gonic/gin"
"github.com/komari-monitor/komari/internal/plugin"
"github.com/komari-monitor/komari/web/api"
)
// UploadPlugin installs a plugin ZIP package. The request body is the raw
// ZIP, mirroring UploadTheme.
func UploadPlugin(c *gin.Context) {
data, err := io.ReadAll(c.Request.Body)
if err != nil || len(data) == 0 {
api.RespondError(c, http.StatusBadRequest, "请选择要上传的插件文件")
return
}
tempFile, err := os.CreateTemp("", "komari-plugin-*.zip")
if err != nil {
api.RespondError(c, http.StatusInternalServerError, "保存文件失败: "+err.Error())
return
}
tempPath := tempFile.Name()
defer func() { _ = os.Remove(tempPath) }()
if _, err := tempFile.Write(data); err != nil {
_ = tempFile.Close()
api.RespondError(c, http.StatusInternalServerError, "保存文件失败: "+err.Error())
return
}
if err := tempFile.Close(); err != nil {
api.RespondError(c, http.StatusInternalServerError, "保存文件失败: "+err.Error())
return
}
info, err := plugin.InstallZip(tempPath)
if err != nil {
api.RespondError(c, http.StatusBadRequest, err.Error())
return
}
api.RespondSuccessMessage(c, "插件上传成功", info)
}
// ServePluginFile serves a static file from an installed plugin directory,
// used by injected plugin admin pages.
func ServePluginFile(c *gin.Context) {
+2 -2
View File
@@ -302,7 +302,7 @@ func InstallPluginFromMarket(c *gin.Context) {
api.RespondError(c, http.StatusBadRequest, "This plugin does not provide an installable package")
return
}
data, err := downloadMarketURL(selected.Download, marketPackageMaxSize)
data, err := DownloadMarketURL(selected.Download, marketPackageMaxSize)
if err != nil {
api.RespondError(c, http.StatusBadRequest, "Failed to download plugin: "+err.Error())
return
@@ -349,7 +349,7 @@ func fetchPluginMarketCatalog(source PluginMarketSource, force bool) ([]PluginMa
return append([]PluginMarketPlugin(nil), cached.Plugins...), nil
}
}
data, err := downloadMarketURL(source.URL, marketCatalogMaxSize)
data, err := DownloadMarketURL(source.URL, marketCatalogMaxSize)
if err != nil {
return nil, err
}
+13 -110
View File
@@ -6,7 +6,6 @@ import (
"errors"
"fmt"
"io"
"net"
"net/http"
"net/url"
"os"
@@ -28,39 +27,6 @@ const (
maxThemeManifestSize = 1 << 20
)
// UploadTheme 上传主题
func UploadTheme(c *gin.Context) {
// 读取上传的文件内容
data, err := io.ReadAll(c.Request.Body)
if err != nil || len(data) == 0 {
api.RespondError(c, http.StatusBadRequest, "请选择要上传的主题文件")
return
}
// 临时文件名
tempFile := filepath.Join(os.TempDir(), "uploaded_theme.zip")
if err := os.WriteFile(tempFile, data, 0644); err != nil {
api.RespondError(c, http.StatusInternalServerError, "保存文件失败: "+err.Error())
return
}
defer os.Remove(tempFile)
// 检查文件扩展名(这里假定上传的就是zip)
if !strings.HasSuffix(strings.ToLower(tempFile), ".zip") {
api.RespondError(c, http.StatusBadRequest, "只支持ZIP格式的主题文件")
return
}
// 解压ZIP文件并验证
themeInfo, err := extractAndValidateTheme(tempFile)
if err != nil {
api.RespondError(c, http.StatusBadRequest, err.Error())
return
}
api.RespondSuccessMessage(c, "主题上传成功", themeInfo)
}
// ListThemes 列出所有主题
func ListThemes(c *gin.Context) {
dataDir := "./data/theme"
@@ -217,17 +183,7 @@ func extractAndValidateTheme(zipPath string) (models.Theme, error) {
return themeInfo, fmt.Errorf("主题配置格式错误: %v", err)
}
// 验证必填字段
if themeInfo.Name == "" || themeInfo.Short == "" {
return themeInfo, fmt.Errorf("主题配置缺少必填字段(name、short)")
}
// 验证Short字段格式(只允许字母、数字、下划线、连字符)
if !isValidMarketShort(themeInfo.Short) {
return themeInfo, fmt.Errorf("主题short字段格式无效,只允许字母、数字、下划线和连字符")
}
if err := themeInfo.ValidateConfiguration(); err != nil {
if err := validateThemeManifest(themeInfo); err != nil {
return themeInfo, err
}
@@ -324,6 +280,16 @@ func loadThemeConfig(configPath string) (models.Theme, error) {
return themeInfo, nil
}
func validateThemeManifest(themeInfo models.Theme) error {
if !models.IsLocalizedText(themeInfo.Name) || themeInfo.Short == "" {
return fmt.Errorf("主题配置缺少必填字段(name、short)")
}
if !isValidMarketShort(themeInfo.Short) {
return fmt.Errorf("主题short字段格式无效,只允许字母、数字、下划线和连字符")
}
return themeInfo.ValidateConfiguration()
}
// isValidMarketShort validates a market entry short name (shared by the
// theme and plugin markets).
func isValidMarketShort(short string) bool {
@@ -341,62 +307,8 @@ func isValidMarketShort(short string) bool {
return true
}
// downloadThemeFromURL 从URL下载主题文件
// isPrivateIP checks if the resolved IP addresses are private/internal
func isPrivateIP(host string) bool {
ips, err := net.LookupHost(host)
if err != nil {
return true // fail closed
}
for _, ipStr := range ips {
ip := net.ParseIP(ipStr)
if ip == nil {
continue
}
if ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() {
return true
}
}
return false
}
func downloadThemeFromURL(rawURL string) ([]byte, error) {
// SSRF protection: block requests to private/internal IPs
parsedURL, err := url.Parse(rawURL)
if err != nil {
return nil, fmt.Errorf("invalid URL: %v", err)
}
if parsedURL.Scheme != "http" && parsedURL.Scheme != "https" {
return nil, fmt.Errorf("only http and https schemes are allowed")
}
if isPrivateIP(parsedURL.Hostname()) {
return nil, fmt.Errorf("requests to private/internal addresses are not allowed")
}
// 发送HTTP GET请求
resp, err := http.Get(rawURL)
if err != nil {
return nil, fmt.Errorf("下载主题文件失败: %v", err)
}
defer resp.Body.Close()
// 检查响应状态码
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("下载主题文件失败,HTTP状态码: %d", resp.StatusCode)
}
// 读取响应内容
data, err := io.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("读取主题文件内容失败: %v", err)
}
// 检查文件大小
if len(data) == 0 {
return nil, errors.New("下载的主题文件为空")
}
return data, nil
return DownloadMarketURL(rawURL, marketDownloadMaxBytes)
}
// getGitHubReleaseDownloadURL 从GitHub API获取最新release的下载链接
@@ -726,15 +638,7 @@ func peekThemeFromZip(zipPath string) (models.Theme, error) {
return themeInfo, fmt.Errorf("主题配置格式错误: %v", err)
}
if themeInfo.Name == "" || themeInfo.Short == "" {
return themeInfo, fmt.Errorf("主题配置缺少必填字段(name、short)")
}
if !isValidMarketShort(themeInfo.Short) {
return themeInfo, fmt.Errorf("主题short字段格式无效,只允许字母、数字、下划线和连字符")
}
if err := themeInfo.ValidateConfiguration(); err != nil {
if err := validateThemeManifest(themeInfo); err != nil {
return themeInfo, err
}
@@ -840,7 +744,6 @@ func UpdateThemeSettings(c *gin.Context) {
api.RespondError(c, http.StatusBadRequest, "主题名称不能为空")
return
}
var req map[string]any
err := c.ShouldBindJSON(&req)
+4 -58
View File
@@ -7,7 +7,6 @@ import (
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
@@ -16,6 +15,7 @@ import (
"time"
"github.com/gin-gonic/gin"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/internal/config"
"github.com/komari-monitor/komari/web/api"
)
@@ -312,7 +312,7 @@ func InstallThemeFromMarket(c *gin.Context) {
api.RespondError(c, http.StatusBadRequest, "This theme does not provide an installable package")
return
}
data, err := downloadMarketURL(selected.Download, marketPackageMaxSize)
data, err := DownloadMarketURL(selected.Download, marketPackageMaxSize)
if err != nil {
api.RespondError(c, http.StatusBadRequest, "Failed to download theme: "+err.Error())
return
@@ -364,7 +364,7 @@ func fetchThemeMarketCatalog(source ThemeMarketSource, force bool) ([]ThemeMarke
return append([]ThemeMarketTheme(nil), cached.Themes...), nil
}
}
data, err := downloadMarketURL(source.URL, marketCatalogMaxSize)
data, err := DownloadMarketURL(source.URL, marketCatalogMaxSize)
if err != nil {
return nil, err
}
@@ -445,17 +445,7 @@ func validateThemeMarketTheme(theme ThemeMarketTheme) error {
// isMarketText reports whether a market entry field is usable: a non-empty
// string or an i18n object with at least one non-empty value.
func isMarketText(value any) bool {
switch text := value.(type) {
case string:
return strings.TrimSpace(text) != ""
case map[string]any:
for _, item := range text {
if itemText, ok := item.(string); ok && strings.TrimSpace(itemText) != "" {
return true
}
}
}
return false
return models.IsLocalizedText(value)
}
func validateMarketURLSyntax(rawURL string) error {
@@ -466,50 +456,6 @@ func validateMarketURLSyntax(rawURL string) error {
return nil
}
func downloadMarketURL(rawURL string, maxSize int64) ([]byte, error) {
validate := func(candidate string) error {
parsed, err := url.Parse(candidate)
if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") || parsed.Hostname() == "" || parsed.User != nil {
return errors.New("only public HTTP and HTTPS URLs are allowed")
}
if isPrivateIP(parsed.Hostname()) {
return errors.New("requests to private or internal addresses are not allowed")
}
return nil
}
if err := validate(rawURL); err != nil {
return nil, err
}
client := &http.Client{
Timeout: 45 * time.Second,
CheckRedirect: func(req *http.Request, via []*http.Request) error {
if len(via) >= 10 {
return errors.New("too many redirects")
}
return validate(req.URL.String())
},
}
resp, err := client.Get(rawURL)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("HTTP status %d", resp.StatusCode)
}
data, err := io.ReadAll(io.LimitReader(resp.Body, maxSize+1))
if err != nil {
return nil, err
}
if int64(len(data)) > maxSize {
return nil, fmt.Errorf("response exceeds the %d byte limit", maxSize)
}
if len(data) == 0 {
return nil, errors.New("empty response")
}
return data, nil
}
func invalidateThemeMarketCache(rawURL string) {
themeMarketCache.Lock()
delete(themeMarketCache.items, rawURL)
+87 -1
View File
@@ -1,6 +1,13 @@
package admin
import "testing"
import (
"archive/zip"
"os"
"path/filepath"
"testing"
"github.com/komari-monitor/komari/database/models"
)
// TestIsValidThemeShort_PathTraversal 防止 DeleteTheme/UpdateTheme/SetTheme
// 的路径穿越漏洞:req.Short 直接进入 filepath.Join("./data/theme", short),
@@ -45,3 +52,82 @@ func TestIsValidThemeShort_PathTraversal(t *testing.T) {
}
}
}
func TestValidateThemeManifestAcceptsLocalizedMetadata(t *testing.T) {
theme := models.Theme{
Name: map[string]any{
"zh-CN": "配置字段演示主题",
"en": "Managed Configuration Demo Theme",
},
Short: "managed-config-demo",
Description: map[string]any{
"zh-CN": "用于验证全部托管配置字段的测试主题。",
"en": "A test theme covering every managed configuration field type.",
},
Author: map[string]any{
"zh-CN": "Komari 团队",
"en": "Komari",
},
}
if err := validateThemeManifest(theme); err != nil {
t.Fatalf("localized theme metadata was rejected: %v", err)
}
}
func TestPeekThemeFromZipAcceptsLocalizedMetadata(t *testing.T) {
zipPath := filepath.Join(t.TempDir(), "localized-theme.zip")
archive, err := os.Create(zipPath)
if err != nil {
t.Fatalf("create zip: %v", err)
}
writer := zip.NewWriter(archive)
manifest, err := writer.Create("komari-theme.json")
if err != nil {
t.Fatalf("create manifest: %v", err)
}
_, err = manifest.Write([]byte(`{
"name": {"zh-CN": "本地化主题", "en": "Localized Theme"},
"short": "localized-theme",
"description": {"zh-CN": "描述", "en": "Description"},
"author": {"zh-CN": "作者", "en": "Author"},
"version": "1.0.0"
}`))
if err != nil {
t.Fatalf("write manifest: %v", err)
}
if err := writer.Close(); err != nil {
t.Fatalf("close zip writer: %v", err)
}
if err := archive.Close(); err != nil {
t.Fatalf("close zip file: %v", err)
}
theme, err := peekThemeFromZip(zipPath)
if err != nil {
t.Fatalf("localized theme package was rejected: %v", err)
}
if theme.Short != "localized-theme" {
t.Fatalf("short = %q, want localized-theme", theme.Short)
}
workDir, err := os.Getwd()
if err != nil {
t.Fatalf("get working directory: %v", err)
}
tempDir := t.TempDir()
if err := os.Chdir(tempDir); err != nil {
t.Fatalf("change working directory: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(workDir)
})
installed, err := extractAndValidateTheme(zipPath)
if err != nil {
t.Fatalf("localized theme upload package was rejected: %v", err)
}
if installed.Short != "localized-theme" {
t.Fatalf("installed short = %q, want localized-theme", installed.Short)
}
}
-45
View File
@@ -1,45 +0,0 @@
package admin
import (
"fmt"
"net/http"
"os"
"time"
"github.com/gin-gonic/gin"
logger "github.com/komari-monitor/komari/utils/log"
"github.com/komari-monitor/komari/web/api"
"github.com/komari-monitor/komari/web/backup"
)
// UploadBackup receives a backup and applies it after the service restarts.
func UploadBackup(c *gin.Context) {
file, header, err := c.Request.FormFile("backup")
if err != nil {
api.RespondError(c, http.StatusBadRequest, fmt.Sprintf("get uploaded backup: %v", err))
return
}
defer file.Close()
restoreLock, err := backup.AcquireRestoreLock()
if err != nil {
api.RespondError(c, http.StatusConflict, err.Error())
return
}
if err := restoreLock.SaveUploadedBackup(file, header.Filename); err != nil {
restoreLock.Release()
api.RespondError(c, http.StatusBadRequest, err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
"status": "success",
"message": "Backup uploaded successfully. The service will restart and apply the backup.",
"path": "./data/backup.zip",
})
go func() {
logger.InfoArgs("admin-api", "Backup uploaded, restarting service in 2 seconds to apply on startup...")
time.Sleep(2 * time.Second)
restoreLock.Release()
os.Exit(0)
}()
}
+6 -7
View File
@@ -5,21 +5,20 @@ import (
"time"
"github.com/komari-monitor/komari/database/clients"
"github.com/komari-monitor/komari/internal/metricstore"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/database/tasks"
v1 "github.com/komari-monitor/komari/protocol/v1"
"github.com/komari-monitor/komari/internal/metricstore"
v2 "github.com/komari-monitor/komari/protocol/v2"
agent_runtime "github.com/komari-monitor/komari/web/agent"
)
// ingest.go
// agent 上报数据的传输无关处理逻辑。v1 (REST/WS) 与 v2 (JSON-RPC) 两套上报入口
// 经过各自的协议解析后,统一调用这里的函数落库并更新运行时状态,消除重复。
// agent 上报数据的传输无关处理逻辑。v2 的 HTTP/WebSocket JSON-RPC 入口
// 经过协议解析后,统一调用这里的函数落库并更新运行时状态。
// ingestReport 保存一次负载上报并刷新运行时状态。
// protocolVersion 标记上报所用协议(1 或 2),用于运行时区分客户端能力。
// markPresence 为 true 时按 POST 上报会话刷新在线状态(WS 连接自行管理在线状态,应传 false)。
func ingestReport(uuid string, report v1.Report, protocolVersion int, markPresence bool) error {
func ingestReport(uuid string, report v2.Report, markPresence bool) error {
report.UUID = uuid
report.UpdatedAt = time.Now().UTC()
if err := clients.ReportVerify(report); err != nil {
@@ -30,7 +29,7 @@ func ingestReport(uuid string, report v1.Report, protocolVersion int, markPresen
return err
}
agent_runtime.RecordReport(savedReport)
agent_runtime.SetClientProtocolVersion(uuid, protocolVersion)
agent_runtime.MarkV2Client(uuid)
if markPresence {
refreshPostPresence(uuid)
}
+69
View File
@@ -0,0 +1,69 @@
package client
import (
"sync"
"time"
"github.com/komari-monitor/komari/utils/notifier"
agent_runtime "github.com/komari-monitor/komari/web/agent"
)
const (
// 如果超过这个时间没有收到任何消息,则认为连接已死
readWait = 11 * time.Second
postPresenceTTL = 35 * time.Second
)
type postPresenceEntry struct {
connID int64
timer *time.Timer
generation uint64
}
var (
postPresenceMu sync.Mutex
postPresenceStates = make(map[string]*postPresenceEntry)
)
// refreshPostPresence 管理 HTTP POST 上报者的在线/离线状态。
func refreshPostPresence(uuid string) {
postPresenceMu.Lock()
defer postPresenceMu.Unlock()
if entry, exists := postPresenceStates[uuid]; exists {
entry.generation++
entry.timer.Stop()
gen := entry.generation
entry.timer = time.AfterFunc(postPresenceTTL, func() {
postPresenceExpired(uuid, entry.connID, gen)
})
agent_runtime.KeepAlivePresence(uuid, entry.connID, postPresenceTTL)
return
}
connID := time.Now().UnixNano()
agent_runtime.KeepAlivePresence(uuid, connID, postPresenceTTL)
agent_runtime.MarkV2Client(uuid)
go notifier.OnlineNotification(uuid, connID)
defaultGeneration := uint64(0)
entry := &postPresenceEntry{connID: connID, generation: defaultGeneration}
entry.timer = time.AfterFunc(postPresenceTTL, func() {
postPresenceExpired(uuid, connID, defaultGeneration)
})
postPresenceStates[uuid] = entry
}
func postPresenceExpired(uuid string, connID int64, gen uint64) {
postPresenceMu.Lock()
e, ok := postPresenceStates[uuid]
if !ok || e.connID != connID || e.generation != gen {
postPresenceMu.Unlock()
return
}
delete(postPresenceStates, uuid)
postPresenceMu.Unlock()
agent_runtime.SetPresence(uuid, connID, false)
notifier.OfflineNotification(uuid, connID)
}
-263
View File
@@ -1,263 +0,0 @@
package client
import (
"bytes"
"encoding/json"
"fmt"
"io"
"sync"
"time"
logger "github.com/komari-monitor/komari/utils/log"
"net/http"
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
"github.com/komari-monitor/komari/database/clients"
v1 "github.com/komari-monitor/komari/protocol/v1"
"github.com/komari-monitor/komari/utils/notifier"
agent_runtime "github.com/komari-monitor/komari/web/agent"
"github.com/komari-monitor/komari/web/api"
"github.com/komari-monitor/komari/web/connection"
)
const (
// 如果超过这个时间没有收到任何消息,则认为连接已死
// 因为目前server没有存agent的信息上报间隔。只有写一个默认的
readWait = 11 * time.Second
postPresenceTTL = 35 * time.Second
)
// postPresenceEntry 保存单个客户端的 POST 上报会话状态
type postPresenceEntry struct {
connID int64
timer *time.Timer
generation uint64 // 每次 Reset 递增,用于回调中判断是否为过期的旧回调
}
var (
postPresenceMu sync.Mutex
postPresenceStates = make(map[string]*postPresenceEntry)
)
// refreshPostPresence 管理 HTTP POST 上报者的在线/离线状态。
// 每次 POST 刷新 TTL 定时器;定时器到期后触发离线通知。
func refreshPostPresence(uuid string) {
postPresenceMu.Lock()
defer postPresenceMu.Unlock()
if entry, exists := postPresenceStates[uuid]; exists {
// 已在线:递增 generation 使可能正在执行的旧回调失效
entry.generation++
entry.timer.Stop()
// 重新创建 AfterFunc 以在闭包中捕获新的 generation
gen := entry.generation
entry.timer = time.AfterFunc(postPresenceTTL, func() {
postPresenceExpired(uuid, entry.connID, gen)
})
agent_runtime.KeepAlivePresence(uuid, entry.connID, postPresenceTTL)
return
}
// 新 POST 会话:生成 connID,标记在线,启动超时定时器
connID := time.Now().UnixNano()
agent_runtime.KeepAlivePresence(uuid, connID, postPresenceTTL)
agent_runtime.SetClientProtocolVersion(uuid, 1)
go notifier.OnlineNotification(uuid, connID)
defaultGeneration := uint64(0)
entry := &postPresenceEntry{
connID: connID,
generation: defaultGeneration,
}
entry.timer = time.AfterFunc(postPresenceTTL, func() {
postPresenceExpired(uuid, connID, defaultGeneration)
})
postPresenceStates[uuid] = entry
}
// postPresenceExpired 是定时器到期的回调。
// 只有当 connID 和 generation 都与当前 entry 匹配时才执行离线清理,
// 避免 timer.Reset 竞态导致过期回调错误地清除仍活跃的会话。
func postPresenceExpired(uuid string, connID int64, gen uint64) {
postPresenceMu.Lock()
e, ok := postPresenceStates[uuid]
if !ok || e.connID != connID || e.generation != gen {
postPresenceMu.Unlock()
return
}
delete(postPresenceStates, uuid)
postPresenceMu.Unlock()
agent_runtime.SetPresence(uuid, connID, false)
notifier.OfflineNotification(uuid, connID)
}
func UploadReport(c *gin.Context) {
bodyBytes, err := io.ReadAll(c.Request.Body)
if err != nil {
logger.ErrorArgs("client-api", "Failed to read request body:", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
return
}
var data map[string]interface{}
err = json.Unmarshal(bodyBytes, &data)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
return
}
// Save report to database
var report v1.Report
err = json.Unmarshal(bodyBytes, &report)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
return
}
// 优先使用 body 中的 UUID,若为空则从中间件注入的上下文中获取
uuid := report.UUID
if uuid == "" {
if v, ok := c.Get("client_uuid"); ok {
uuid, _ = v.(string)
}
}
if uuid == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "UUID is required"})
return
}
// POST 上报:落库、更新运行时状态并刷新在线状态
if err := ingestReport(uuid, report, 1, true); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("%v", err)})
return
}
c.Request.Body = io.NopCloser(bytes.NewBuffer(bodyBytes)) // Restore the body for further use
c.JSON(200, gin.H{"status": "success"})
}
func WebSocketReport(c *gin.Context) {
// 升级ws
if !api.IsWebSocketUpgrade(c) {
c.JSON(http.StatusBadRequest, gin.H{"status": "error", "error": "Require WebSocket upgrade"})
return
}
// Upgrade the HTTP connection to a WebSocket connection
conn, err := api.UpgradeSafeConn(c)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"status": "error", "error": "Failed to upgrade to WebSocket." + err.Error()})
return
}
defer conn.Close()
_, message, err := conn.ReadMessage()
if err != nil {
logger.ErrorArgs("client-api", "Error reading message:", err)
return
}
// 第一次数据拿token
data := map[string]interface{}{}
err = json.Unmarshal(message, &data)
if err != nil {
conn.WriteJSON(gin.H{"status": "error", "error": "Invalid JSON"})
return
}
// it should ok,token was verfied in the middleware
token := ""
var errMsg string
// 优先检查查询参数中的 token
token = c.Query("token")
// 如果 token 为空,返回错误
if token == "" {
conn.WriteJSON(gin.H{"status": "error", "error": errMsg})
return
}
uuid, err := clients.GetClientUUIDByToken(token)
if err != nil {
conn.WriteJSON(gin.H{"status": "error", "error": errMsg})
return
}
// 接受新连接,并处理旧连接
if oldConn, exists := agent_runtime.GetConnectedClients()[uuid]; exists {
logger.Infof("client-api", "Client %s is reconnecting. Closing the old connection.", uuid)
// 强制关闭旧连接。这将导致旧连接的 ReadMessage() 循环出错退出。
go oldConn.Close()
}
agent_runtime.SetConnectedClients(uuid, conn)
logger.Infof("client-api", "Client %s is reconnect success, connID: %d", uuid, conn.ID)
go notifier.OnlineNotification(uuid, conn.ID)
defer func() {
agent_runtime.DeleteClientConditionally(uuid, conn)
notifier.OfflineNotification(uuid, conn.ID)
}()
// 首先处理第一次ws conn收到的消息
processMessage(conn, message, uuid)
for {
conn.SetReadDeadline(time.Now().Add(readWait))
_, message, err := conn.ReadMessage()
if err != nil {
if websocket.IsUnexpectedCloseError(err, websocket.CloseGoingAway, websocket.CloseAbnormalClosure) {
logger.Errorf("client-api", "Client %s connection error: %v", uuid, err)
}
break // 任何读错误(包括超时)都意味着连接已断开,退出循环
}
processMessage(conn, message, uuid)
}
}
// 将消息处理逻辑提取到一个函数中,方便复用
func processMessage(conn *connection.SafeConn, message []byte, uuid string) {
type MessageType struct {
Type string `json:"type"`
}
var msgType MessageType
err := json.Unmarshal(message, &msgType)
if err != nil {
conn.WriteJSON(gin.H{"status": "error", "error": "Invalid JSON"})
return
}
switch msgType.Type {
case "", "report":
report := v1.Report{}
err = json.Unmarshal(message, &report)
if err != nil {
conn.WriteJSON(gin.H{"status": "error", "error": "Invalid report format"})
return
}
// WS 连接自行管理在线状态,无需刷新 POST presence
if err := ingestReport(uuid, report, 1, false); err != nil {
conn.WriteJSON(gin.H{"status": "error", "error": fmt.Sprintf("%v", err)})
return
}
case "ping_result":
var reqBody struct {
PingTaskID uint `json:"task_id"`
PingResult int `json:"value"`
PingType string `json:"ping_type"`
}
err = json.Unmarshal(message, &reqBody)
if err != nil {
conn.WriteJSON(gin.H{"status": "error", "error": "Invalid ping result format"})
return
}
if err := ingestPingResult(uuid, reqBody.PingTaskID, reqBody.PingResult); err != nil {
conn.WriteJSON(gin.H{"status": "error", "error": "Failed to save ping result"})
}
default:
logger.Warnf("client-api", "Unknown message type: %s", msgType.Type)
conn.WriteJSON(gin.H{"status": "error", "error": "Unknown message type"})
}
}
+28 -3
View File
@@ -13,11 +13,13 @@ import (
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
"github.com/komari-monitor/komari/database/clients"
"github.com/komari-monitor/komari/database/tasks"
v2 "github.com/komari-monitor/komari/protocol/v2"
"github.com/komari-monitor/komari/utils/notifier"
agent_runtime "github.com/komari-monitor/komari/web/agent"
"github.com/komari-monitor/komari/web/api"
"github.com/komari-monitor/komari/web/connection"
"github.com/komari-monitor/komari/web/filemanager"
)
func readMaybeCompressedBody(r *http.Request) ([]byte, error) {
@@ -51,7 +53,7 @@ func handleV2RPC(uuid string, req v2.Request, allowWait bool) v2.Response {
if err := bindV2Params(req.Params, &params); err != nil {
return v2.Error(req.ID, -32602, "invalid report params", err.Error())
}
if err := ingestReport(uuid, params.Report, 2, true); err != nil {
if err := ingestReport(uuid, params.Report, true); err != nil {
return v2.Error(req.ID, -32000, "failed to save report", err.Error())
}
return v2.Success(req.ID, gin.H{
@@ -76,13 +78,26 @@ func handleV2RPC(uuid string, req v2.Request, allowWait bool) v2.Response {
return v2.Error(req.ID, -32000, "failed to save ping result", err.Error())
}
return v2.Success(req.ID, gin.H{"status": "success"})
case v2.MethodAgentTaskResult:
var params v2.TaskResultParams
if err := bindV2Params(req.Params, &params); err != nil {
return v2.Error(req.ID, -32602, "invalid task result params", err.Error())
}
finishedAt := params.FinishedAt
if finishedAt.IsZero() {
finishedAt = time.Now().UTC()
}
if err := tasks.SaveTaskResult(params.TaskID, uuid, params.Result, params.ExitCode, finishedAt); err != nil {
return v2.Error(req.ID, -32000, "failed to save task result", err.Error())
}
return v2.Success(req.ID, gin.H{"status": "success"})
case v2.MethodAgentPull:
var params v2.PullParams
if err := bindV2Params(req.Params, &params); err != nil {
return v2.Error(req.ID, -32602, "invalid pull params", err.Error())
}
refreshPostPresence(uuid)
agent_runtime.SetClientProtocolVersion(uuid, 2)
agent_runtime.MarkV2Client(uuid)
timeout := 0 * time.Second
if allowWait {
timeout = 25 * time.Second
@@ -90,6 +105,16 @@ func handleV2RPC(uuid string, req v2.Request, allowWait bool) v2.Response {
return v2.Success(req.ID, gin.H{
"events": agent_runtime.WaitV2Events(uuid, params.AckEventIDs, timeout),
})
case v2.MethodAgentFileResult:
var params v2.FileResult
if err := bindV2Params(req.Params, &params); err != nil {
return v2.Error(req.ID, -32602, "invalid file result params", err.Error())
}
params.UUID = uuid
if !filemanager.Resolve(params) {
return v2.Error(req.ID, -32004, "unknown or expired file operation", nil)
}
return v2.Success(req.ID, gin.H{"status": "success"})
default:
return v2.Error(req.ID, -32601, "method not found", req.Method)
}
@@ -140,7 +165,7 @@ func WebSocketV2RPC(c *gin.Context) {
go oldConn.Close()
}
agent_runtime.SetConnectedClients(uuid, conn)
agent_runtime.SetClientProtocolVersion(uuid, 2)
agent_runtime.MarkV2Client(uuid)
go notifierOnline(uuid, conn.ID)
defer func() {
agent_runtime.DeleteClientConditionally(uuid, conn)
-24
View File
@@ -6,8 +6,6 @@ import (
"github.com/komari-monitor/komari/database/clients"
"github.com/komari-monitor/komari/internal/config"
"github.com/komari-monitor/komari/utils/geoip"
"github.com/gin-gonic/gin"
)
func getClientIPType(ip net.IP) int {
@@ -80,25 +78,3 @@ func appendClientRegionFromGeoIP(info map[string]interface{}) {
return
}
}
func UploadBasicInfo(c *gin.Context) {
var cbi = map[string]interface{}{}
if err := c.ShouldBindJSON(&cbi); err != nil {
c.JSON(400, gin.H{"status": "error", "error": "Invalid or missing data"})
return
}
token := c.Query("token")
uuid, err := clients.GetClientUUIDByToken(token)
if uuid == "" || err != nil {
c.JSON(400, gin.H{"status": "error", "error": "Invalid token"})
return
}
if err := ingestBasicInfo(uuid, cbi, c.ClientIP()); err != nil {
c.JSON(500, gin.H{"status": "error", "error": err})
return
}
c.JSON(200, gin.H{"status": "success"})
}
+14 -14
View File
@@ -9,8 +9,13 @@ import (
func EstablishConnection(c *gin.Context) {
session_id := c.Query("id")
TerminalSessionsMutex.Lock()
session, exists := TerminalSessions[session_id]
if !exists || session == nil || session.Browser == nil {
TerminalSessionsMutex.Unlock()
authenticatedUUID, _ := c.Get("client_uuid")
authenticatedString, hasAuthenticatedUUID := authenticatedUUID.(string)
if !exists || session == nil || session.Browser == nil ||
(hasAuthenticatedUUID && authenticatedString != "" && authenticatedString != session.UUID) {
c.JSON(404, gin.H{"status": "error", "error": "Session not found"})
return
}
@@ -21,22 +26,17 @@ func EstablishConnection(c *gin.Context) {
}
conn, err := api.UpgradeSafeConn(c)
if err != nil {
TerminalSessionsMutex.Lock()
if session.Browser != nil {
session.Browser.Close()
}
delete(TerminalSessions, session_id)
TerminalSessionsMutex.Unlock()
closeSession(session_id)
return
}
_, ok := attachAgent(session_id, conn)
if !ok {
conn.Close()
return
}
session.Agent = conn
conn.SetCloseHandler(func(code int, text string) error {
delete(TerminalSessions, session_id)
// 通知 Browser 关闭终端连接
if session.Browser != nil {
session.Browser.Close()
}
suspendSession(session_id, nil, conn)
return nil
})
go ForwardTerminal(session_id)
maybeStartForwarding(session_id)
}
+45 -31
View File
@@ -1,39 +1,64 @@
package terminal
import (
"encoding/json"
"time"
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
"github.com/komari-monitor/komari/database/auditlog"
"github.com/komari-monitor/komari/web/connection"
)
func ForwardTerminal(id string) {
session, exists := TerminalSessions[id]
if !exists || session == nil || session.Agent == nil || session.Browser == nil {
func ForwardTerminal(id string, browser, agent *connection.SafeConn) {
if browser == nil || agent == nil {
return
}
auditlog.Log(session.RequesterIp, session.UserUUID, "established, terminal id:"+id, "terminal")
TerminalSessionsMutex.Lock()
session := TerminalSessions[id]
requesterIp, userUUID := "", ""
if session != nil && session.Browser == browser && session.Agent == agent {
requesterIp, userUUID = session.RequesterIp, session.UserUUID
}
TerminalSessionsMutex.Unlock()
if requesterIp == "" {
return
}
auditlog.Log(requesterIp, userUUID, "established, terminal id:"+id, "terminal")
established_time := time.Now()
errChan := make(chan error, 1)
go func() {
for {
messageType, data, err := session.Browser.ReadMessage()
messageType, data, err := browser.ReadMessage()
if err != nil {
errChan <- err
return
}
if messageType == websocket.TextMessage {
if session.Agent != nil && string(data[0:1]) == "{" {
err = session.Agent.WriteMessage(websocket.TextMessage, data)
} else if session.Agent != nil {
err = session.Agent.WriteMessage(websocket.BinaryMessage, data)
var control struct {
Type string `json:"type"`
}
} else if session.Agent != nil {
// 二进制消息,原样传递
err = session.Agent.WriteMessage(websocket.BinaryMessage, data)
if json.Unmarshal(data, &control) == nil {
if control.Type == "heartbeat" {
continue
}
if control.Type == "close" {
_ = agent.WriteJSON(gin.H{"type": "close"})
closeSession(id)
errChan <- nil
return
}
}
if len(data) > 0 && data[0] == '{' {
err = agent.WriteMessage(websocket.TextMessage, data)
} else {
err = agent.WriteMessage(websocket.BinaryMessage, data)
}
} else {
err = agent.WriteMessage(websocket.BinaryMessage, data)
}
if err != nil {
@@ -45,33 +70,22 @@ func ForwardTerminal(id string) {
go func() {
for {
_, data, err := session.Agent.ReadMessage()
_, data, err := agent.ReadMessage()
if err != nil {
errChan <- err
return
}
if session.Browser != nil {
err = session.Browser.WriteMessage(websocket.BinaryMessage, data)
if err != nil {
errChan <- err
return
}
err = browser.WriteMessage(websocket.BinaryMessage, data)
if err != nil {
errChan <- err
return
}
}
}()
// 等待错误或主动关闭
<-errChan
// 关闭连接
if session.Agent != nil {
session.Agent.Close()
}
if session.Browser != nil {
session.Browser.Close()
}
suspendSession(id, browser, agent)
disconnect_time := time.Now()
auditlog.Log(session.RequesterIp, session.UserUUID, "disconnected, terminal id:"+id+", duration:"+disconnect_time.Sub(established_time).String(), "terminal")
TerminalSessionsMutex.Lock()
delete(TerminalSessions, id)
TerminalSessionsMutex.Unlock()
auditlog.Log(requesterIp, userUUID, "disconnected, terminal id:"+id+", duration:"+disconnect_time.Sub(established_time).String(), "terminal")
}
+65 -44
View File
@@ -1,20 +1,27 @@
package terminal
import (
logger "github.com/komari-monitor/komari/utils/log"
"net/http"
"time"
"strings"
"github.com/gin-gonic/gin"
"github.com/komari-monitor/komari/database/clients"
v2 "github.com/komari-monitor/komari/protocol/v2"
"github.com/komari-monitor/komari/utils"
logger "github.com/komari-monitor/komari/utils/log"
agent_runtime "github.com/komari-monitor/komari/web/agent"
"github.com/komari-monitor/komari/web/api"
)
func dispatchTerminalRequest(uuid, id string) bool {
return agent_runtime.DispatchV2Event(uuid, v2.MethodAgentTerminal, v2.TerminalRequestParams{RequestID: id})
}
func RequestTerminal(c *gin.Context) {
uuid := c.Param("uuid")
user_uuid, _ := c.Get("uuid")
userUUID, _ := c.Get("uuid")
userID, _ := userUUID.(string)
_, isAPIKey := c.Get("api_key")
_, err := clients.GetClientByUUID(uuid)
if err != nil {
c.JSON(400, gin.H{
@@ -23,6 +30,26 @@ func RequestTerminal(c *gin.Context) {
})
return
}
id := strings.TrimSpace(c.Query("request_id"))
if id == "" {
// Only a new terminal request is a sensitive operation. Reattaching an
// existing session is authorized by its owner below and must not require
// a fresh (and potentially expired) TOTP code.
if err := api.VerifySensitive2FA(c); err != nil {
api.RespondError(c, http.StatusUnauthorized, err.Error())
return
}
} else {
TerminalSessionsMutex.Lock()
session := TerminalSessions[id]
allowed := session != nil && session.UUID == uuid && (isAPIKey || session.UserUUID == userID)
TerminalSessionsMutex.Unlock()
if !allowed {
c.JSON(http.StatusNotFound, gin.H{"status": "error", "message": "Terminal session not found"})
return
}
}
// 建立ws
if !api.IsWebSocketUpgrade(c) {
c.JSON(http.StatusBadRequest, gin.H{"status": "error", "message": "Require WebSocket upgrade"})
@@ -32,10 +59,36 @@ func RequestTerminal(c *gin.Context) {
if err != nil {
return
}
if id != "" {
session, ok := attachBrowser(id, userID, isAPIKey, conn)
if !ok || session.UUID != uuid {
conn.WriteMessage(1, []byte("Terminal session expired\n终端会话已过期\n"))
conn.Close()
return
}
conn.SetCloseHandler(func(code int, text string) error {
logger.InfoArgs("terminal", "Terminal browser connection closed:", code, text)
suspendSession(id, conn, nil)
return nil
})
conn.WriteJSON(gin.H{"request_id": id})
if !dispatchTerminalRequest(uuid, id) {
conn.WriteMessage(1, []byte("Client offline!\n被控端离线!\n"))
closeSession(id)
return
}
if session.Agent == nil {
conn.WriteMessage(1, []byte("等待被控端连接 waiting for agent...\n"))
}
maybeStartForwarding(id)
return
}
// 新建一个终端连接
id := utils.GenerateRandomString(32)
id = utils.GenerateRandomString(32)
session := &TerminalSession{
UserUUID: user_uuid.(string),
UserUUID: userID,
UUID: uuid,
Browser: conn,
Agent: nil,
@@ -44,51 +97,19 @@ func RequestTerminal(c *gin.Context) {
TerminalSessionsMutex.Lock()
TerminalSessions[id] = session
scheduleCleanup(id, session)
TerminalSessionsMutex.Unlock()
conn.SetCloseHandler(func(code int, text string) error {
logger.InfoArgs("terminal", "Terminal connection closed:", code, text)
TerminalSessionsMutex.Lock()
delete(TerminalSessions, id)
TerminalSessionsMutex.Unlock()
// 通知 Agent 关闭终端连接
if session.Agent != nil {
session.Agent.Close()
}
logger.InfoArgs("terminal", "Terminal browser connection closed:", code, text)
suspendSession(id, conn, nil)
return nil
})
if agent_runtime.GetConnectedClients()[uuid] == nil {
conn.WriteMessage(1, []byte("Client offline!\n被控端离线!\n"))
conn.WriteJSON(gin.H{"request_id": id})
if !dispatchTerminalRequest(uuid, id) {
conn.Close()
TerminalSessionsMutex.Lock()
delete(TerminalSessions, id)
TerminalSessionsMutex.Unlock()
return
}
err = agent_runtime.GetConnectedClients()[uuid].WriteJSON(gin.H{
"message": "terminal",
"request_id": id,
})
if err != nil {
conn.Close()
TerminalSessionsMutex.Lock()
delete(TerminalSessions, id)
TerminalSessionsMutex.Unlock()
closeSession(id)
return
}
conn.WriteMessage(1, []byte("等待被控端连接 waiting for agent...\n"))
// 如果没有连接上,则关闭连接
time.AfterFunc(30*time.Second, func() {
TerminalSessionsMutex.Lock()
if session.Agent == nil {
if session.Browser != nil {
session.Browser.WriteMessage(1, []byte("被控端连接超时 timeout\n"))
session.Browser.Close()
}
conn.Close()
delete(TerminalSessions, id)
}
TerminalSessionsMutex.Unlock()
})
//auditlog.Log(c.ClientIP(), user_uuid.(string), "request, terminal id:"+id+",client:"+session.UUID, "terminal")
//auditlog.Log(c.ClientIP(), userID, "request, terminal id:"+id+",client:"+session.UUID, "terminal")
}
+145 -5
View File
@@ -2,17 +2,157 @@ package terminal
import (
"sync"
"time"
"github.com/komari-monitor/komari/web/connection"
)
type TerminalSession struct {
UUID string
UserUUID string
Browser *connection.SafeConn
Agent *connection.SafeConn
RequesterIp string
UUID string
UserUUID string
Browser *connection.SafeConn
Agent *connection.SafeConn
RequesterIp string
Forwarding bool
CleanupTimer *time.Timer
}
var TerminalSessionsMutex = &sync.Mutex{}
var TerminalSessions = make(map[string]*TerminalSession)
// 与 v2 事件队列 TTL 对齐,被控端短暂离线后会话仍可恢复。
const terminalSessionRetention = 5 * time.Minute
func scheduleCleanup(id string, session *TerminalSession) {
if session.CleanupTimer != nil {
session.CleanupTimer.Stop()
}
session.CleanupTimer = time.AfterFunc(terminalSessionRetention, func() {
var browser, agent *connection.SafeConn
TerminalSessionsMutex.Lock()
if current, ok := TerminalSessions[id]; !ok || current != session {
TerminalSessionsMutex.Unlock()
return
}
browser, agent = session.Browser, session.Agent
delete(TerminalSessions, id)
TerminalSessionsMutex.Unlock()
if browser != nil {
_ = browser.Close()
}
if agent != nil {
_ = agent.Close()
}
})
}
func stopCleanup(session *TerminalSession) {
if session.CleanupTimer != nil {
session.CleanupTimer.Stop()
session.CleanupTimer = nil
}
}
func suspendSession(id string, browser, agent *connection.SafeConn) {
var otherBrowser, otherAgent *connection.SafeConn
TerminalSessionsMutex.Lock()
session, ok := TerminalSessions[id]
if !ok || session == nil ||
(browser != nil && session.Browser != browser) ||
(agent != nil && session.Agent != agent) {
TerminalSessionsMutex.Unlock()
return
}
otherBrowser, otherAgent = session.Browser, session.Agent
session.Browser = nil
session.Agent = nil
session.Forwarding = false
scheduleCleanup(id, session)
TerminalSessionsMutex.Unlock()
if otherBrowser != nil {
_ = otherBrowser.Close()
}
if otherAgent != nil {
_ = otherAgent.Close()
}
}
func closeSession(id string) {
var browser, agent *connection.SafeConn
TerminalSessionsMutex.Lock()
if session, ok := TerminalSessions[id]; ok && session != nil {
stopCleanup(session)
browser, agent = session.Browser, session.Agent
delete(TerminalSessions, id)
}
TerminalSessionsMutex.Unlock()
if browser != nil {
_ = browser.Close()
}
if agent != nil {
_ = agent.Close()
}
}
func attachBrowser(id, userUUID string, apiKey bool, conn *connection.SafeConn) (*TerminalSession, bool) {
TerminalSessionsMutex.Lock()
session, ok := TerminalSessions[id]
if !ok || session == nil {
TerminalSessionsMutex.Unlock()
return nil, false
}
if !apiKey && session.UserUUID != userUUID {
TerminalSessionsMutex.Unlock()
return nil, false
}
oldBrowser := session.Browser
session.Browser = conn
session.Forwarding = false
if session.Agent != nil {
stopCleanup(session)
}
TerminalSessionsMutex.Unlock()
if oldBrowser != nil && oldBrowser != conn {
_ = oldBrowser.Close()
}
return session, true
}
func attachAgent(id string, conn *connection.SafeConn) (*TerminalSession, bool) {
TerminalSessionsMutex.Lock()
session, ok := TerminalSessions[id]
if !ok || session == nil {
TerminalSessionsMutex.Unlock()
return nil, false
}
oldAgent := session.Agent
session.Agent = conn
session.Forwarding = false
if session.Browser != nil {
stopCleanup(session)
}
TerminalSessionsMutex.Unlock()
if oldAgent != nil && oldAgent != conn {
_ = oldAgent.Close()
}
return session, true
}
func maybeStartForwarding(id string) {
TerminalSessionsMutex.Lock()
session, ok := TerminalSessions[id]
if !ok || session == nil || session.Browser == nil || session.Agent == nil || session.Forwarding {
TerminalSessionsMutex.Unlock()
return
}
session.Forwarding = true
browser, agent := session.Browser, session.Agent
TerminalSessionsMutex.Unlock()
go ForwardTerminal(id, browser, agent)
}
+3 -3
View File
@@ -8,7 +8,7 @@ import (
"github.com/komari-monitor/komari/database/accounts"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/protocol/v1"
v2 "github.com/komari-monitor/komari/protocol/v2"
agent_runtime "github.com/komari-monitor/komari/web/agent"
)
@@ -53,11 +53,11 @@ func GetClients(c *gin.Context) {
for {
var resp struct {
Online []string `json:"online"` // 已建立连接的客户端uuid列表
Data map[string]v1.Report `json:"data"` // 最后上报的数据
Data map[string]v2.Report `json:"data"` // 最后上报的数据
}
resp.Online = []string{}
resp.Data = map[string]v1.Report{}
resp.Data = map[string]v2.Report{}
_, data, err := conn.ReadMessage()
if err != nil {
+121
View File
@@ -0,0 +1,121 @@
// Package filemanager correlates panel file requests with agent responses.
package filemanager
import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"sync"
"time"
v2 "github.com/komari-monitor/komari/protocol/v2"
agent_runtime "github.com/komari-monitor/komari/web/agent"
)
const defaultTimeout = 30 * time.Second
var (
ErrOffline = errors.New("agent is not connected")
ErrUnsupported = errors.New("agent does not support file operations")
ErrTimeout = errors.New("file operation timed out")
ErrUnknownToken = errors.New("unknown or expired file operation")
pendingMu sync.Mutex
pending = make(map[string]pendingCall)
)
type pendingCall struct {
uuid string
response chan v2.FileResult
}
type CallOptions struct {
Timeout time.Duration
}
// Call dispatches a metadata-only filesystem control operation. Binary file
// data must use the HTTP transfer endpoint exposed by the transfer package.
func Call(ctx context.Context, uuid, op string, args map[string]any, options ...CallOptions) (json.RawMessage, error) {
if !agent_runtime.IsV2Client(uuid) {
return nil, ErrUnsupported
}
timeout := defaultTimeout
if len(options) > 0 && options[0].Timeout > 0 {
timeout = options[0].Timeout
}
requestID, err := newRequestID()
if err != nil {
return nil, err
}
responses := make(chan v2.FileResult, 1)
pendingMu.Lock()
pending[requestID] = pendingCall{uuid: uuid, response: responses}
pendingMu.Unlock()
defer removePending(requestID)
ok := agent_runtime.DispatchV2Event(uuid, v2.MethodAgentFile, v2.FileOperation{
UUID: uuid,
RequestID: requestID,
Op: op,
Args: args,
})
if !ok {
return nil, ErrOffline
}
timer := time.NewTimer(timeout)
defer timer.Stop()
select {
case result := <-responses:
if !result.OK {
if result.Error == "" {
return nil, errors.New("file operation failed")
}
return nil, errors.New(result.Error)
}
return result.Result, nil
case <-timer.C:
return nil, ErrTimeout
case <-ctx.Done():
return nil, ctx.Err()
}
}
func Resolve(result v2.FileResult) bool {
if result.RequestID == "" || result.UUID == "" {
return false
}
pendingMu.Lock()
wait, ok := pending[result.RequestID]
if ok && wait.uuid != result.UUID {
pendingMu.Unlock()
return false
}
delete(pending, result.RequestID)
pendingMu.Unlock()
if !ok {
return false
}
wait.response <- result
return true
}
func removePending(requestID string) {
pendingMu.Lock()
if _, ok := pending[requestID]; ok {
delete(pending, requestID)
}
pendingMu.Unlock()
}
func newRequestID() (string, error) {
var id [16]byte
if _, err := rand.Read(id[:]); err != nil {
return "", fmt.Errorf("generate file request id: %w", err)
}
return hex.EncodeToString(id[:]), nil
}
+29
View File
@@ -0,0 +1,29 @@
package filemanager
import (
"testing"
v2 "github.com/komari-monitor/komari/protocol/v2"
)
func TestResolveMatchesRequestAndClient(t *testing.T) {
pendingMu.Lock()
pending = make(map[string]pendingCall)
response := make(chan v2.FileResult, 1)
pending["request"] = pendingCall{uuid: "client-a", response: response}
pendingMu.Unlock()
if Resolve(v2.FileResult{UUID: "client-b", RequestID: "request", OK: true}) {
t.Fatal("resolved a response from the wrong client")
}
if !Resolve(v2.FileResult{UUID: "client-a", RequestID: "request", OK: true}) {
t.Fatal("did not resolve the matching response")
}
result := <-response
if result.UUID != "client-a" || !result.OK {
t.Fatalf("unexpected response: %+v", result)
}
if Resolve(v2.FileResult{UUID: "client-a", RequestID: "request", OK: true}) {
t.Fatal("resolved the same request twice")
}
}
+696
View File
@@ -0,0 +1,696 @@
package filemanager
import (
"context"
"crypto/subtle"
"errors"
"fmt"
"io"
"net/http"
"strconv"
"strings"
"sync"
"time"
"github.com/gin-gonic/gin"
"github.com/google/uuid"
logger "github.com/komari-monitor/komari/utils/log"
)
const (
streamTransferTTL = 2 * time.Minute
streamCallTimeout = 30 * time.Minute
streamBufferSize = 512 * 1024
streamTokenHeader = "X-Komari-Transfer-Token"
streamChunkAttempts = 3
maxStreamTransfers = 32
)
type streamTransferDirection string
const (
streamUploadDirection streamTransferDirection = "upload"
streamDownloadDirection streamTransferDirection = "download"
)
// streamTransfer is a one-shot, bounded relay between the public browser
// request and the Agent's outbound HTTP request. io.Pipe deliberately keeps
// the relay back-pressured so neither side can queue an unbounded body.
type streamTransfer struct {
id string
token string
clientUUID string
direction streamTransferDirection
size int64
expiresAt time.Time
reader *io.PipeReader
writer *io.PipeWriter
slotHeld bool
mu sync.Mutex
connected bool
closed bool
}
var (
streamTransfersMu sync.Mutex
streamTransfers = make(map[string]*streamTransfer)
streamCleanupOnce sync.Once
streamBufferPool = sync.Pool{New: func() any { return make([]byte, streamBufferSize) }}
streamSlots = make(chan struct{}, maxStreamTransfers)
)
type streamCopyResult struct {
bytes int64
err error
}
func newStreamTransfer(ctx context.Context, clientUUID string, direction streamTransferDirection, size int64) (*streamTransfer, error) {
streamCleanupOnce.Do(startStreamTransferCleanup)
if ctx == nil {
ctx = context.Background()
}
select {
case streamSlots <- struct{}{}:
case <-ctx.Done():
return nil, ctx.Err()
}
reader, writer := io.Pipe()
transfer := &streamTransfer{
id: uuid.NewString(),
token: uuid.NewString(),
clientUUID: clientUUID,
direction: direction,
size: size,
expiresAt: time.Now().Add(streamTransferTTL),
reader: reader,
writer: writer,
slotHeld: true,
}
streamTransfersMu.Lock()
streamTransfers[transfer.id] = transfer
streamTransfersMu.Unlock()
return transfer, nil
}
func startStreamTransferCleanup() {
go func() {
ticker := time.NewTicker(time.Minute)
defer ticker.Stop()
for now := range ticker.C {
var expired []*streamTransfer
streamTransfersMu.Lock()
for id, transfer := range streamTransfers {
transfer.mu.Lock()
connected := transfer.connected
transfer.mu.Unlock()
if !connected && now.After(transfer.expiresAt) {
delete(streamTransfers, id)
expired = append(expired, transfer)
}
}
streamTransfersMu.Unlock()
for _, transfer := range expired {
transfer.close(errors.New("file transfer expired"))
}
}
}()
}
func (transfer *streamTransfer) markConnected() error {
transfer.mu.Lock()
defer transfer.mu.Unlock()
if transfer.closed {
return errors.New("file transfer is closed")
}
if transfer.connected {
return errors.New("file transfer is already connected")
}
transfer.connected = true
return nil
}
func (transfer *streamTransfer) close(err error) {
success := err == nil
if err == nil {
err = io.EOF
}
transfer.mu.Lock()
if transfer.closed {
transfer.mu.Unlock()
return
}
transfer.closed = true
slotHeld := transfer.slotHeld
transfer.slotHeld = false
transfer.mu.Unlock()
streamTransfersMu.Lock()
if current := streamTransfers[transfer.id]; current == transfer {
delete(streamTransfers, transfer.id)
}
streamTransfersMu.Unlock()
if success {
// A successful producer already closes the PipeWriter after the exact
// byte count. Do not close the reader side here: a consumer may still be
// draining the final bytes, and CloseWithError would turn a normal EOF
// into io.ErrClosedPipe. The writer's EOF is enough to finish readers.
_ = transfer.writer.CloseWithError(io.EOF)
} else {
_ = transfer.reader.CloseWithError(err)
_ = transfer.writer.CloseWithError(err)
}
if slotHeld {
<-streamSlots
}
}
func lookupStreamTransfer(c *gin.Context) (*streamTransfer, error) {
id := strings.TrimSpace(c.Param("id"))
if headerID := strings.TrimSpace(c.GetHeader("X-Komari-Transfer-ID")); headerID != "" && headerID != id {
return nil, errors.New("file transfer id does not match request")
}
token := strings.TrimSpace(c.GetHeader(streamTokenHeader))
if token == "" {
token = strings.TrimSpace(c.Query("transfer_token"))
}
if id == "" || token == "" {
return nil, errors.New("file transfer credentials are required")
}
clientValue, ok := c.Get("client_uuid")
clientID, ok := clientValue.(string)
if !ok || clientID == "" {
return nil, errors.New("agent identity is required")
}
streamTransfersMu.Lock()
transfer := streamTransfers[id]
streamTransfersMu.Unlock()
if transfer == nil {
return nil, errors.New("unknown or expired file transfer")
}
if subtle.ConstantTimeCompare([]byte(transfer.token), []byte(token)) != 1 || transfer.clientUUID != clientID {
return nil, errors.New("invalid file transfer credentials")
}
if time.Now().After(transfer.expiresAt) {
transfer.close(errors.New("file transfer expired"))
return nil, errors.New("file transfer expired")
}
return transfer, nil
}
// AgentTransfer is the data-plane endpoint used by the Agent. POST is used for
// both directions: a download carries bytes in the request body, while an
// upload receives the browser bytes in the response body. GET remains an alias
// for upload so an interrupted rollout cannot strand an already-open session.
func AgentTransfer(c *gin.Context) {
transfer, err := lookupStreamTransfer(c)
if err != nil {
logger.Warnf("file-transfer", "agent stream authentication failed method=%s path=%s remote=%s: %v", c.Request.Method, c.Request.URL.Path, c.ClientIP(), err)
_ = c.Error(err)
c.String(http.StatusUnauthorized, err.Error())
return
}
switch c.Request.Method {
case http.MethodGet:
if transfer.direction != streamUploadDirection {
logger.Warnf("file-transfer", "agent stream method rejected id=%s direction=%s method=%s", transfer.id, transfer.direction, c.Request.Method)
c.String(http.StatusMethodNotAllowed, "transfer direction does not allow GET")
return
}
serveUploadTransfer(c, transfer)
case http.MethodPost:
if transfer.direction == streamUploadDirection {
if c.Request.ContentLength > 0 {
transfer.close(errors.New("upload transfer request must not contain a body"))
logger.Warnf("file-transfer", "agent upload stream sent an unexpected request body id=%s content_length=%d", transfer.id, c.Request.ContentLength)
_ = c.Error(errors.New("invalid upload transfer request"))
c.String(http.StatusBadRequest, "invalid upload transfer request")
return
}
serveUploadTransfer(c, transfer)
return
}
if c.Request.ContentLength >= 0 && c.Request.ContentLength != transfer.size {
transfer.close(fmt.Errorf("download stream content length %d, want %d", c.Request.ContentLength, transfer.size))
logger.Warnf("file-transfer", "agent download stream length mismatch id=%s got=%d want=%d", transfer.id, c.Request.ContentLength, transfer.size)
_ = c.Error(fmt.Errorf("download stream content length %d, want %d", c.Request.ContentLength, transfer.size))
c.String(http.StatusBadRequest, "invalid transfer content length")
return
}
if err := transfer.markConnected(); err != nil {
logger.Warnf("file-transfer", "agent download stream connection rejected id=%s: %v", transfer.id, err)
_ = c.Error(err)
c.String(http.StatusConflict, err.Error())
return
}
written, copyErr := copyExactStream(transfer.writer, c.Request.Body, transfer.size, false, nil)
if copyErr == nil && written != transfer.size {
copyErr = fmt.Errorf("download stream ended after %d of %d bytes", written, transfer.size)
}
if copyErr != nil {
_ = transfer.writer.CloseWithError(copyErr)
transfer.close(copyErr)
logger.Errorf("file-transfer", "agent download stream failed id=%s received=%d want=%d: %v", transfer.id, written, transfer.size, copyErr)
_ = c.Error(copyErr)
c.String(http.StatusBadRequest, copyErr.Error())
return
}
_ = transfer.writer.Close()
c.JSON(http.StatusOK, gin.H{"received": written})
transfer.close(nil)
default:
c.String(http.StatusMethodNotAllowed, "method not allowed")
}
}
func serveUploadTransfer(c *gin.Context, transfer *streamTransfer) {
if err := transfer.markConnected(); err != nil {
logger.Warnf("file-transfer", "agent upload stream connection rejected id=%s: %v", transfer.id, err)
_ = c.Error(err)
c.String(http.StatusConflict, err.Error())
return
}
committed := false
commitHeaders := func() {
if committed {
return
}
c.Header("Content-Type", "application/octet-stream")
c.Header("Content-Length", strconv.FormatInt(transfer.size, 10))
c.Header("Content-Encoding", "identity")
c.Header("Cache-Control", "no-store, no-transform")
c.Header("X-Accel-Buffering", "no")
c.Status(http.StatusOK)
committed = true
}
written, copyErr := copyExactStream(c.Writer, transfer.reader, transfer.size, true, commitHeaders)
if copyErr == nil && written != transfer.size {
copyErr = fmt.Errorf("upload stream ended after %d of %d bytes", written, transfer.size)
}
if copyErr != nil {
transfer.close(copyErr)
logger.Errorf("file-transfer", "agent upload stream failed id=%s sent=%d want=%d: %v", transfer.id, written, transfer.size, copyErr)
_ = c.Error(copyErr)
if !committed {
c.String(http.StatusBadRequest, copyErr.Error())
}
return
}
transfer.close(nil)
}
// streamDownloadRelay opens one bounded transfer for a logical download
// chunk, then forwards its body directly to the browser response. A new
// transfer per logical chunk keeps retries and CDN request sizes bounded while
// the pipe keeps the memory footprint independent of that chunk size.
func streamDownloadRelay(c *gin.Context, clientUUID, path string, fileSize int64, modifiedAt time.Time, start, size, chunkSize int64, commitHeaders func()) error {
if size <= 0 {
rememberDownloadChunkSize(clientUUID, chunkSize)
return nil
}
if chunkSize <= 0 || chunkSize > MaxTransferChunkSize {
return fmt.Errorf("invalid download chunk size %d", chunkSize)
}
end := start + size
offset := start
chunkIndex := int64(0)
currentChunkSize := chunkSize
c.Set(downloadChunkSizeContextKey, currentChunkSize)
// Keep the upper bound from the most recent 413 response. The lower bound
// is the configured 1 MiB floor; each retry chooses the midpoint so a
// restrictive proxy is discovered in logarithmic time.
adaptiveUpperBound := chunkSize
for offset < end {
length := min(currentChunkSize, end-offset)
var chunkErr error
var lastCopyResult streamCopyResult
for {
resized := false
chunkErr = nil
for attempt := 1; attempt <= streamChunkAttempts; attempt++ {
commit := commitHeaders
if chunkIndex > 0 {
commit = nil
}
copyResult, streamErr := streamDownloadChunk(c, clientUUID, path, fileSize, modifiedAt, offset, length, chunkIndex, commit)
lastCopyResult = copyResult
if copyResult.err == nil && copyResult.bytes == length {
// The request has proven that this logical block size is
// accepted. Remember it immediately so concurrent/follow-up
// downloads do not repeat a known probe.
rememberDownloadChunkSize(clientUUID, currentChunkSize)
chunkErr = nil
break
}
// streamDownloadChunk returns the Agent/RPC error separately when a
// relay pipe was closed as a consequence of that error. Prefer it so
// the caller sees the real status/body instead of io.ErrClosedPipe.
chunkErr = streamErr
if chunkErr == nil {
chunkErr = copyResult.err
}
if chunkErr == nil {
chunkErr = fmt.Errorf("download chunk %d returned %d of %d bytes", chunkIndex, copyResult.bytes, length)
}
logger.Warnf("file-transfer", "download chunk failed client=%s path=%q chunk=%d offset=%d length=%d attempt=%d/%d bytes=%d: %v", clientUUID, path, chunkIndex, offset, length, attempt, streamChunkAttempts, copyResult.bytes, chunkErr)
// A 413 is a deterministic request-size rejection. Retry the same
// offset with a smaller logical block before considering normal
// transient retry limits. No browser bytes may have been committed
// for this chunk when this branch is taken.
if copyResult.bytes == 0 && isPayloadTooLargeError(chunkErr) {
adaptiveUpperBound = min(adaptiveUpperBound, length)
if next, ok := nextReducedTransferChunkSize(length, adaptiveUpperBound); ok && next < length {
previousLength := length
currentChunkSize = next
c.Set(downloadChunkSizeContextKey, currentChunkSize)
length = min(currentChunkSize, end-offset)
logger.Warnf("file-transfer", "reducing download chunk size after 413 client=%s path=%q from=%d to=%d", clientUUID, path, previousLength, currentChunkSize)
resized = true
break
}
}
if copyResult.bytes == 0 && isNonRetryableStreamError(chunkErr) {
return chunkErr
}
if copyResult.bytes == 0 && attempt == streamChunkAttempts && isChunkTransportError(chunkErr) {
adaptiveUpperBound = min(adaptiveUpperBound, length)
if next, ok := nextReducedTransferChunkSize(length, adaptiveUpperBound); ok && next < length {
previousLength := length
currentChunkSize = next
c.Set(downloadChunkSizeContextKey, currentChunkSize)
length = min(currentChunkSize, end-offset)
logger.Warnf("file-transfer", "reducing download chunk size after transport failure client=%s path=%q from=%d to=%d: %v", clientUUID, path, previousLength, currentChunkSize, chunkErr)
resized = true
break
}
}
// Once a byte of this logical chunk reached the browser, retrying
// would overlap an already committed HTTP response. Only retry when
// the failure happened before the first byte.
if copyResult.bytes > 0 || attempt == streamChunkAttempts || c.Request.Context().Err() != nil {
break
}
time.Sleep(time.Duration(attempt) * 150 * time.Millisecond)
}
if resized {
// Retry this offset using the newly selected block size. The
// attempt counter is intentionally reset for the resized request.
continue
}
break
}
if chunkErr != nil {
// A downstream browser/proxy can close a long response after some
// bytes were delivered. The current HTTP response cannot be resumed,
// but the next Range request can avoid repeating the same oversized
// logical block by remembering a smaller probe size.
if lastCopyResult.bytes > 0 && isClientDisconnectError(lastCopyResult.err) {
if next, ok := nextReducedTransferChunkSize(currentChunkSize, currentChunkSize); ok && next < currentChunkSize {
rememberDownloadChunkSize(clientUUID, next)
logger.Warnf("file-transfer", "reducing download chunk size after downstream disconnect client=%s from=%d to=%d", clientUUID, currentChunkSize, next)
}
}
return chunkErr
}
offset += length
chunkIndex++
}
rememberDownloadChunkSize(clientUUID, currentChunkSize)
return nil
}
// nextReducedTransferChunkSize returns the next 1 MiB-aligned midpoint below
// a rejected request size. upperBound is exclusive and is tightened after
// every 413, allowing the caller to converge without probing below 1 MiB.
func nextReducedTransferChunkSize(failedSize, upperBound int64) (int64, bool) {
if failedSize <= MinTransferChunkSize {
return 0, false
}
high := failedSize
if upperBound > 0 && upperBound < high {
high = upperBound
}
if high <= MinTransferChunkSize {
return MinTransferChunkSize, true
}
candidate := MinTransferChunkSize + (high-MinTransferChunkSize)/2
candidate = (candidate / MinTransferChunkSize) * MinTransferChunkSize
if candidate < MinTransferChunkSize {
candidate = MinTransferChunkSize
}
if candidate >= failedSize {
candidate = failedSize - 1
}
if candidate < MinTransferChunkSize {
return 0, false
}
return candidate, true
}
func streamDownloadChunk(c *gin.Context, clientUUID, path string, fileSize int64, modifiedAt time.Time, offset, length, chunkIndex int64, commitHeaders func()) (streamCopyResult, error) {
transfer, transferErr := newStreamTransfer(c.Request.Context(), clientUUID, streamDownloadDirection, length)
if transferErr != nil {
return streamCopyResult{}, transferErr
}
defer transfer.close(nil)
args := map[string]any{
"transfer_id": transfer.id,
"transfer_token": transfer.token,
"path": path,
"offset": offset,
"length": length,
"chunk_index": chunkIndex,
"file_size": fileSize,
}
if !modifiedAt.IsZero() {
args["modified_at"] = modifiedAt.UTC().Format(time.RFC3339Nano)
}
ctx, cancel := context.WithCancel(c.Request.Context())
defer cancel()
callDone := make(chan error, 1)
go func() {
_, callErr := Call(ctx, clientUUID, "download_stream", args, CallOptions{Timeout: streamCallTimeout})
callDone <- callErr
}()
copyDone := make(chan streamCopyResult, 1)
go func() {
written, copyErr := copyExactStream(c.Writer, transfer.reader, length, true, commitHeaders)
if copyErr == nil && written != length {
copyErr = fmt.Errorf("download stream ended after %d of %d bytes", written, length)
}
copyDone <- streamCopyResult{bytes: written, err: copyErr}
}()
var copyResult streamCopyResult
var callErr error
copyFinished, callFinished := false, false
contextDone := c.Request.Context().Done()
for !copyFinished || !callFinished {
select {
case copyResult = <-copyDone:
copyFinished = true
if copyResult.err != nil {
cancel()
transfer.close(copyResult.err)
}
case callErr = <-callDone:
callFinished = true
if callErr != nil {
cancel()
transfer.close(callErr)
}
case <-contextDone:
cancel()
transfer.close(c.Request.Context().Err())
contextDone = nil
}
}
// The bytes already delivered to the browser are authoritative. The Agent
// may report an error only while reading the final JSON acknowledgement
// after the complete body has arrived (for example after a proxy reset).
if copyResult.err == nil && copyResult.bytes == length {
return copyResult, nil
}
if copyResult.err != nil {
if callErr != nil && isRelayPipeError(copyResult.err) {
return copyResult, callErr
}
return copyResult, copyResult.err
}
if callErr != nil {
return copyResult, callErr
}
if copyResult.bytes != length {
return copyResult, fmt.Errorf("download stream returned %d of %d bytes", copyResult.bytes, length)
}
return copyResult, nil
}
func isRelayPipeError(err error) bool {
if err == nil {
return false
}
if errors.Is(err, io.ErrClosedPipe) || errors.Is(err, io.EOF) {
return true
}
message := strings.ToLower(err.Error())
return strings.Contains(message, "closed pipe") || strings.Contains(message, "file transfer is closed")
}
func isClientDisconnectError(err error) bool {
if err == nil {
return false
}
message := strings.ToLower(err.Error())
for _, marker := range []string{
"broken pipe",
"connection reset by peer",
"connection reset",
"write: closed network connection",
"client disconnected",
} {
if strings.Contains(message, marker) {
return true
}
}
return false
}
func isChunkTransportError(err error) bool {
if err == nil {
return false
}
message := strings.ToLower(err.Error())
for _, marker := range []string{
" 502 ",
" 503 ",
" 504 ",
"broken pipe",
"connection reset",
"unexpected eof",
"timeout",
} {
if strings.Contains(message, marker) {
return true
}
}
return false
}
func isNonRetryableStreamError(err error) bool {
if err == nil {
return false
}
if isUnsupportedAgentFileOperation(err) {
return true
}
message := strings.ToLower(err.Error())
for _, status := range []string{" 400 ", " 401 ", " 403 ", " 404 ", " 405 ", " 409 ", " 410 ", " 411 ", " 413 ", " 415 ", " 422 ", " 501 "} {
if strings.Contains(message, status) {
return true
}
}
return false
}
func isPayloadTooLargeError(err error) bool {
if err == nil {
return false
}
var maxBytesError *http.MaxBytesError
if errors.As(err, &maxBytesError) {
return true
}
message := strings.ToLower(err.Error())
for _, marker := range []string{
" 413 ",
" 413",
"413 ",
"413:",
"status 413",
"status code 413",
"request entity too large",
"payload too large",
"content too large",
} {
if strings.Contains(message, marker) {
return true
}
}
return false
}
// copyStream copies at most limit bytes using one pooled buffer.
func copyStream(dst io.Writer, src io.Reader, limit int64, flush bool, onFirstWrite func()) (int64, error) {
if limit <= 0 {
return 0, nil
}
buffer := streamBufferPool.Get().([]byte)
defer streamBufferPool.Put(buffer)
reader := io.Reader(src)
if limit > 0 {
reader = io.LimitReader(src, limit)
}
var total int64
first := true
for {
read, readErr := reader.Read(buffer)
if read > 0 {
// Response headers must be installed before the first Write; net/http
// commits them as soon as the write starts.
if first && onFirstWrite != nil {
onFirstWrite()
first = false
}
written, writeErr := dst.Write(buffer[:read])
if writeErr != nil {
return total + int64(written), writeErr
}
if written != read {
return total + int64(written), io.ErrShortWrite
}
total += int64(written)
if flush {
if flusher, ok := dst.(http.Flusher); ok {
flusher.Flush()
}
}
}
if readErr != nil {
if errors.Is(readErr, io.EOF) {
return total, nil
}
return total, readErr
}
}
}
// copyExactStream copies exactly expected bytes and probes one additional byte
// so an oversized request is rejected without forwarding the extra byte.
func copyExactStream(dst io.Writer, src io.Reader, expected int64, flush bool, onFirstWrite func()) (int64, error) {
if expected < 0 {
return 0, errors.New("negative expected stream size")
}
written, err := copyStream(dst, src, expected, flush, onFirstWrite)
if err != nil {
return written, err
}
if written != expected {
return written, fmt.Errorf("stream ended after %d of %d bytes", written, expected)
}
var extra [1]byte
read, readErr := src.Read(extra[:])
if read > 0 {
return written, fmt.Errorf("stream exceeds %d bytes", expected)
}
if readErr != nil && !errors.Is(readErr, io.EOF) {
return written, readErr
}
return written, nil
}
+208
View File
@@ -0,0 +1,208 @@
package filemanager
import (
"bytes"
"context"
"io"
"net/http"
"net/http/httptest"
"net/url"
"testing"
"time"
"github.com/gin-gonic/gin"
)
func newAgentTransferTestContext(method, transferID, token, clientUUID string, body io.Reader) (*gin.Context, *httptest.ResponseRecorder) {
gin.SetMode(gin.TestMode)
recorder := httptest.NewRecorder()
request := httptest.NewRequest(method, "/api/clients/transfer/"+url.PathEscape(transferID)+"?transfer_token="+url.QueryEscape(token), body)
context, _ := gin.CreateTestContext(recorder)
context.Request = request
context.Params = gin.Params{{Key: "id", Value: transferID}}
context.Set("client_uuid", clientUUID)
return context, recorder
}
func TestAgentTransferUploadRelaysBody(t *testing.T) {
data := []byte("streamed upload data")
transfer, err := newStreamTransfer(context.Background(), "agent-upload", streamUploadDirection, int64(len(data)))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { transfer.close(nil) })
context, recorder := newAgentTransferTestContext(http.MethodPost, transfer.id, transfer.token, transfer.clientUUID, nil)
done := make(chan struct{})
go func() {
AgentTransfer(context)
close(done)
}()
go func() {
_, _ = transfer.writer.Write(data)
_ = transfer.writer.Close()
}()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("AgentTransfer did not finish")
}
if recorder.Code != http.StatusOK {
t.Fatalf("status = %d, want %d", recorder.Code, http.StatusOK)
}
if !bytes.Equal(recorder.Body.Bytes(), data) {
t.Fatalf("body = %q, want %q", recorder.Body.Bytes(), data)
}
}
func TestAgentTransferDownloadRelaysBody(t *testing.T) {
data := []byte("streamed download data")
transfer, err := newStreamTransfer(context.Background(), "agent-download", streamDownloadDirection, int64(len(data)))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { transfer.close(nil) })
context, recorder := newAgentTransferTestContext(http.MethodPost, transfer.id, transfer.token, transfer.clientUUID, bytes.NewReader(data))
done := make(chan struct{})
go func() {
AgentTransfer(context)
close(done)
}()
got, readErr := io.ReadAll(transfer.reader)
if readErr != nil {
t.Fatalf("read relayed body: %v", readErr)
}
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("AgentTransfer did not finish")
}
if recorder.Code != http.StatusOK {
t.Fatalf("status = %d, want %d", recorder.Code, http.StatusOK)
}
if !bytes.Equal(got, data) {
t.Fatalf("body = %q, want %q", got, data)
}
}
func TestAgentTransferDownloadKeepsNormalEOFForSlowConsumer(t *testing.T) {
data := bytes.Repeat([]byte("slow-consumer-"), 4096)
transfer, err := newStreamTransfer(context.Background(), "slow-agent", streamDownloadDirection, int64(len(data)))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { transfer.close(nil) })
context, recorder := newAgentTransferTestContext(http.MethodPost, transfer.id, transfer.token, transfer.clientUUID, bytes.NewReader(data))
done := make(chan struct{})
go func() {
AgentTransfer(context)
close(done)
}()
got := make([]byte, 0, len(data))
buffer := make([]byte, 257)
for {
read, readErr := transfer.reader.Read(buffer)
if read > 0 {
got = append(got, buffer[:read]...)
time.Sleep(time.Microsecond)
}
if readErr != nil {
if readErr != io.EOF {
t.Fatalf("slow consumer read: %v", readErr)
}
break
}
}
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("AgentTransfer did not finish")
}
if recorder.Code != http.StatusOK {
t.Fatalf("status = %d, want %d", recorder.Code, http.StatusOK)
}
if !bytes.Equal(got, data) {
t.Fatalf("body length = %d, want %d", len(got), len(data))
}
}
func TestAgentTransferRelaysLargeBodyOverHTTP(t *testing.T) {
testAgentTransferOverHTTP(t, false)
}
func TestAgentTransferRelaysLargeBodyOverHTTP2(t *testing.T) {
testAgentTransferOverHTTP(t, true)
}
func testAgentTransferOverHTTP(t *testing.T, http2 bool) {
t.Helper()
data := bytes.Repeat([]byte("komari-stream-"), (6*1024*1024)/len("komari-stream-"))
transfer, err := newStreamTransfer(context.Background(), "large-agent", streamUploadDirection, int64(len(data)))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { transfer.close(nil) })
router := gin.New()
router.POST("/api/clients/transfer/:id", func(c *gin.Context) {
c.Set("client_uuid", transfer.clientUUID)
AgentTransfer(c)
})
server := httptest.NewUnstartedServer(router)
if http2 {
server.EnableHTTP2 = true
server.StartTLS()
} else {
server.Start()
}
defer server.Close()
client := server.Client()
endpoint := server.URL + "/api/clients/transfer/" + transfer.id + "?transfer_token=" + url.QueryEscape(transfer.token)
writeDone := make(chan struct{})
go func() {
_, _ = transfer.writer.Write(data)
_ = transfer.writer.Close()
close(writeDone)
}()
request, err := http.NewRequest(http.MethodPost, endpoint, nil)
if err != nil {
t.Fatal(err)
}
response, err := client.Do(request)
if err != nil {
t.Fatal(err)
}
defer response.Body.Close()
got, err := io.ReadAll(response.Body)
if err != nil {
t.Fatal(err)
}
select {
case <-writeDone:
case <-time.After(5 * time.Second):
t.Fatal("large stream writer did not finish")
}
if response.StatusCode != http.StatusOK {
t.Fatalf("status = %d, want %d", response.StatusCode, http.StatusOK)
}
if response.ContentLength != int64(len(data)) {
t.Fatalf("content length = %d, want %d", response.ContentLength, len(data))
}
if !bytes.Equal(got, data) {
t.Fatalf("body length = %d, want %d", len(got), len(data))
}
}
func TestCopyExactStreamRejectsOversizedBody(t *testing.T) {
var destination bytes.Buffer
written, err := copyExactStream(&destination, bytes.NewReader([]byte("12345")), 4, false, nil)
if err == nil {
t.Fatal("copyExactStream accepted an oversized body")
}
if written != 4 {
t.Fatalf("written = %d, want 4", written)
}
if got := destination.String(); got != "1234" {
t.Fatalf("destination = %q, want %q", got, "1234")
}
}
File diff suppressed because it is too large. Load diff
+199
View File
@@ -0,0 +1,199 @@
package filemanager
import (
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/gin-gonic/gin"
)
func TestUploadSessionLimitAndRelease(t *testing.T) {
resetUploadSessionsForTest()
t.Cleanup(resetUploadSessionsForTest)
sessions := make([]*uploadSession, 0, maxConcurrentUploads)
for index := 0; index < maxConcurrentUploads; index++ {
session, created, err := acquireUploadSession("", "client", "/tmp/file", 100, 0)
if err != nil || !created {
t.Fatalf("create session %d: created=%v err=%v", index, created, err)
}
sessions = append(sessions, session)
}
if _, _, err := acquireUploadSession("", "client", "/tmp/extra", 100, 0); !errors.Is(err, ErrTooManyUploads) {
t.Fatalf("fifth session error = %v, want %v", err, ErrTooManyUploads)
}
finishUploadSession(sessions[0])
if _, created, err := acquireUploadSession("", "client", "/tmp/reused", 100, 0); err != nil || !created {
t.Fatalf("slot was not released: created=%v err=%v", created, err)
}
}
func TestUploadSessionMustMatchRequest(t *testing.T) {
resetUploadSessionsForTest()
t.Cleanup(resetUploadSessionsForTest)
session, _, err := acquireUploadSession("", "client-a", "/tmp/file", 100, 0)
if err != nil {
t.Fatal(err)
}
if _, _, err := acquireUploadSession(session.ID, "client-b", "/tmp/file", 100, 0); err == nil {
t.Fatal("session accepted a different client")
}
if _, _, err := acquireUploadSession(session.ID, "client-a", "/tmp/other", 100, 0); err == nil {
t.Fatal("session accepted a different path")
}
if _, _, err := acquireUploadSession(session.ID, "client-a", "/tmp/file", 101, 0); err == nil {
t.Fatal("session accepted a different size")
}
if _, _, err := acquireUploadSession(session.ID, "client-a", "", 0, 0); err != nil {
t.Fatalf("continuation request was rejected: %v", err)
}
}
func TestParseSingleByteRange(t *testing.T) {
tests := []struct {
name string
header string
size int64
start, end int64
ok bool
}{
{name: "closed", header: "bytes=2-5", size: 10, start: 2, end: 5, ok: true},
{name: "open", header: "bytes=7-", size: 10, start: 7, end: 9, ok: true},
{name: "suffix", header: "bytes=-3", size: 10, start: 7, end: 9, ok: true},
{name: "clamped", header: "bytes=8-99", size: 10, start: 8, end: 9, ok: true},
{name: "multiple", header: "bytes=0-1,4-5", size: 10, ok: false},
{name: "outside", header: "bytes=10-", size: 10, ok: false},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
start, end, ok := parseSingleByteRange(test.header, test.size)
if ok != test.ok || (ok && (start != test.start || end != test.end)) {
t.Fatalf("parseSingleByteRange(%q, %d) = %d-%d, %v", test.header, test.size, start, end, ok)
}
})
}
}
func TestIfRangeAllowsRange(t *testing.T) {
modifiedAt := time.Date(2026, time.August, 28, 12, 0, 0, 123456789, time.UTC)
etag := `"100-200"`
date := modifiedAt.Format(http.TimeFormat)
for _, test := range []struct {
name string
value string
allow bool
}{
{name: "missing", value: "", allow: true},
{name: "matching etag", value: etag, allow: true},
{name: "matching date", value: date, allow: true},
{name: "stale date", value: modifiedAt.Add(-time.Minute).Format(http.TimeFormat), allow: false},
{name: "invalid", value: "not-a-validator", allow: false},
} {
t.Run(test.name, func(t *testing.T) {
if got := ifRangeAllowsRange(test.value, etag, modifiedAt); got != test.allow {
t.Fatalf("ifRangeAllowsRange(%q) = %v, want %v", test.value, got, test.allow)
}
})
}
}
func TestFormatDownloadContentDisposition(t *testing.T) {
regular := formatDownloadContentDisposition("inline", "中文 文档.docx", false)
if !strings.Contains(regular, "filename=download.docx") {
t.Fatalf("regular disposition = %q, want an ASCII fallback", regular)
}
if !strings.Contains(regular, "filename*=UTF-8''") {
t.Fatalf("regular disposition = %q, want an RFC 5987 filename", regular)
}
office := formatDownloadContentDisposition("inline", "中文 文档.docx", true)
if !strings.Contains(office, "filename=download.docx") {
t.Fatalf("Office disposition = %q, want an ASCII fallback", office)
}
if strings.Contains(office, "filename*=") {
t.Fatalf("Office disposition = %q, must not include filename*", office)
}
}
func TestRespondTransferErrorMarksUnsupportedAgentAsNonRetryable(t *testing.T) {
gin.SetMode(gin.TestMode)
recorder := httptest.NewRecorder()
context, _ := gin.CreateTestContext(recorder)
context.Request = httptest.NewRequest(http.MethodPost, "/file", nil)
respondTransferError(context, errors.New("unsupported file operation: upload_stream"))
if recorder.Code != http.StatusNotImplemented {
t.Fatalf("status = %d, want %d", recorder.Code, http.StatusNotImplemented)
}
if len(context.Errors) == 0 {
t.Fatal("transfer error was not recorded in gin context")
}
}
func TestRespondTransferErrorPreservesPayloadTooLarge(t *testing.T) {
gin.SetMode(gin.TestMode)
recorder := httptest.NewRecorder()
context, _ := gin.CreateTestContext(recorder)
context.Request = httptest.NewRequest(http.MethodPost, "/file", nil)
respondTransferError(context, errors.New("download_stream: file transfer endpoint returned 413 Request Entity Too Large"))
if recorder.Code != http.StatusRequestEntityTooLarge {
t.Fatalf("status = %d, want %d", recorder.Code, http.StatusRequestEntityTooLarge)
}
}
func TestNextReducedTransferChunkSize(t *testing.T) {
megabyte := int64(1024 * 1024)
tests := []struct {
name string
failed int64
upper int64
want int64
wantOK bool
}{
{name: "default", failed: 25 * megabyte, upper: 25 * megabyte, want: 13 * megabyte, wantOK: true},
{name: "second probe", failed: 13 * megabyte, upper: 25 * megabyte, want: 7 * megabyte, wantOK: true},
{name: "floor", failed: 2 * megabyte, upper: 2 * megabyte, want: megabyte, wantOK: true},
{name: "already at floor", failed: megabyte, upper: megabyte, wantOK: false},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
got, ok := nextReducedTransferChunkSize(test.failed, test.upper)
if ok != test.wantOK || (ok && got != test.want) {
t.Fatalf("nextReducedTransferChunkSize(%d, %d) = %d, %v; want %d, %v", test.failed, test.upper, got, ok, test.want, test.wantOK)
}
})
}
}
func TestDownloadChunkSizeRequestUsesSaferCachedValue(t *testing.T) {
downloadChunkMu.Lock()
downloadChunkSizes = map[string]int64{"client": 4 * 1024 * 1024}
downloadChunkMu.Unlock()
t.Cleanup(func() {
downloadChunkMu.Lock()
downloadChunkSizes = make(map[string]int64)
downloadChunkMu.Unlock()
})
gin.SetMode(gin.TestMode)
request := httptest.NewRequest(http.MethodGet, "/file?chunk_size=26214400", nil)
context, _ := gin.CreateTestContext(httptest.NewRecorder())
context.Request = request
if got, want := downloadChunkSizeForRequest(context, "client"), int64(4*1024*1024); got != want {
t.Fatalf("download chunk size = %d, want %d", got, want)
}
}
func resetUploadSessionsForTest() {
uploadMu.Lock()
uploadSessions = make(map[string]*uploadSession)
for len(uploadSlots) > 0 {
<-uploadSlots
}
uploadMu.Unlock()
}
+19 -13
View File
@@ -15,12 +15,13 @@ import (
"github.com/gin-gonic/gin"
"github.com/komari-monitor/komari/database/accounts"
"github.com/komari-monitor/komari/internal/metricstore"
"github.com/komari-monitor/komari/database/models"
appconfig "github.com/komari-monitor/komari/internal/config"
"github.com/komari-monitor/komari/internal/metricstore"
logger "github.com/komari-monitor/komari/utils/log"
"github.com/komari-monitor/komari/web/api"
"github.com/komari-monitor/komari/web/backup"
"github.com/komari-monitor/komari/web/upload"
"gorm.io/gorm"
)
@@ -64,7 +65,16 @@ func (c *Controller) Register(r *gin.Engine) {
g := r.Group(APIPath, c.requireActive)
g.GET("/status", c.status)
g.POST("/complete", c.complete)
g.POST("/restore", c.restore)
uploadHandler := upload.NewHandler(upload.DefaultStore, map[upload.Purpose]upload.Finalizer{
upload.PurposeBackup: c.finalizeBackupUpload,
})
uploadGroup := g.Group("/upload")
{
uploadGroup.POST("/init", uploadHandler.Init)
uploadGroup.POST("/chunk", uploadHandler.Chunk)
uploadGroup.POST("/merge", uploadHandler.Merge)
uploadGroup.POST("/cancel", uploadHandler.Cancel)
}
}
func (c *Controller) requireActive(ctx *gin.Context) {
@@ -82,25 +92,21 @@ func (c *Controller) status(ctx *gin.Context) {
api.RespondSuccess(ctx, Status{State: state, Required: state != "completed"})
}
// restore stages a verified backup before the guide creates any new data.
func (c *Controller) restore(ctx *gin.Context) {
file, header, err := ctx.Request.FormFile("backup")
func (c *Controller) finalizeBackupUpload(session upload.Session) (upload.Result, error) {
archive, err := os.Open(session.ArchivePath)
if err != nil {
api.RespondError(ctx, http.StatusBadRequest, fmt.Sprintf("get uploaded backup: %v", err))
return
return upload.Result{}, fmt.Errorf("open merged backup: %w", err)
}
defer file.Close()
if err := backup.SaveUploadedBackup(file, header.Filename); err != nil {
api.RespondError(ctx, http.StatusBadRequest, err.Error())
return
defer archive.Close()
if err := backup.SaveUploadedBackup(archive, session.Metadata.Filename); err != nil {
return upload.Result{}, err
}
api.RespondSuccessMessage(ctx, "backup uploaded; restarting to restore", gin.H{})
go func() {
logger.InfoArgs("install", "Backup uploaded, restarting service to restore it on startup...")
time.Sleep(2 * time.Second)
os.Exit(0)
}()
return upload.Result{Message: "backup uploaded; restarting to restore", Data: gin.H{}}, nil
}
func (c *Controller) complete(ctx *gin.Context) {
+5 -4
View File
@@ -30,6 +30,7 @@ const (
)
const largeDatasetThreshold int64 = 300_000
const structureProgressCeiling = 80.0
type Mode string
@@ -471,7 +472,7 @@ func (c *Controller) failTarget(err error, dsn, phase string) {
func structureProgressPercent(progress metricstore.RestructureProgress) float64 {
if progress.Phase == "reclaiming" {
return 95
return structureProgressCeiling
}
if progress.Phase == "discarding" && progress.RowsTotal == 0 {
return 1
@@ -479,12 +480,12 @@ func structureProgressPercent(progress metricstore.RestructureProgress) float64
if progress.RowsTotal <= 0 {
return 0
}
value := float64(progress.RowsDone) / float64(progress.RowsTotal) * 95
value := float64(progress.RowsDone) / float64(progress.RowsTotal) * structureProgressCeiling
if value < 0 {
return 0
}
if value > 95 {
return 95
if value > structureProgressCeiling {
return structureProgressCeiling
}
return value
}
+9 -3
View File
@@ -85,7 +85,7 @@ func TestRestrictedControllersRegisterUnifiedRoutes(t *testing.T) {
if routes["POST "+base+"/discard"] != test.discard {
t.Fatalf("POST %s/discard presence = %v, want %v", base, routes["POST "+base+"/discard"], test.discard)
}
if routes["GET /api/public"] || routes["GET /api/rpc2"] || routes["POST /api/clients/report"] {
if routes["GET /api/public"] || routes["GET /api/rpc2"] || routes["GET /api/clients/v2/rpc"] {
t.Fatalf("ordinary APIs leaked into restricted routes: %#v", routes)
}
@@ -126,7 +126,13 @@ func TestUnifiedAuthIncludesMode(t *testing.T) {
}
func TestStructureProgressPercent(t *testing.T) {
if got := structureProgressPercent(metricstore.RestructureProgress{RowsDone: 50, RowsTotal: 100}); got != 47.5 {
t.Fatalf("copying percent = %v, want 47.5", got)
if got := structureProgressPercent(metricstore.RestructureProgress{RowsDone: 50, RowsTotal: 100}); got != 40 {
t.Fatalf("copying percent = %v, want 40", got)
}
if got := structureProgressPercent(metricstore.RestructureProgress{Phase: "reclaiming"}); got != 80 {
t.Fatalf("reclaiming percent = %v, want 80", got)
}
if got := structureProgressPercent(metricstore.RestructureProgress{RowsDone: 100, RowsTotal: 100}); got != 80 {
t.Fatalf("capped percent = %v, want 80", got)
}
}
+75
View File
@@ -0,0 +1,75 @@
package public
import (
"archive/tar"
"bytes"
"fmt"
"io"
"path"
"strings"
"github.com/klauspost/compress/zstd"
)
var defaultDistFiles map[string][]byte
func loadEmbeddedDist() (map[string][]byte, error) {
return decodeEmbeddedDist(embeddedDistArchive)
}
func decodeEmbeddedDist(archive []byte) (map[string][]byte, error) {
decoder, err := zstd.NewReader(nil)
if err != nil {
return nil, fmt.Errorf("open zstd archive: %w", err)
}
defer decoder.Close()
tarBytes, err := decoder.DecodeAll(archive, nil)
if err != nil {
return nil, fmt.Errorf("decode zstd archive: %w", err)
}
files := make(map[string][]byte)
reader := tar.NewReader(bytes.NewReader(tarBytes))
for {
header, err := reader.Next()
if err == io.EOF {
break
}
if err != nil {
return nil, fmt.Errorf("read tar entry: %w", err)
}
name := path.Clean(strings.ReplaceAll(header.Name, "\\", "/"))
if header.Typeflag == tar.TypeDir && name == "." {
continue
}
if name == "." || name == ".." ||
strings.HasPrefix(name, "../") ||
strings.HasPrefix(name, "/") ||
strings.ContainsRune(name, '\x00') {
return nil, fmt.Errorf("invalid embedded tar path %q", name)
}
switch header.Typeflag {
case tar.TypeDir:
continue
case tar.TypeReg, tar.TypeRegA:
content, err := io.ReadAll(reader)
if err != nil {
return nil, fmt.Errorf("read tar entry %q: %w", name, err)
}
if _, exists := files[name]; exists {
return nil, fmt.Errorf("duplicate tar entry %q", name)
}
files[name] = content
default:
return nil, fmt.Errorf("unsupported tar entry %q type %d", name, header.Typeflag)
}
}
if _, ok := files[IndexFile]; !ok {
return nil, fmt.Errorf("tar archive does not contain %q", IndexFile)
}
return files, nil
}
+17 -5
View File
@@ -15,9 +15,12 @@ import (
"github.com/komari-monitor/komari/internal/config"
)
//go:embed defaultTheme
//go:embed defaultTheme/komari-theme.json
var PublicFS embed.FS
//go:embed defaultTheme/dist.tar.zst
var embeddedDistArchive []byte
// 常量定义
const (
DataDir = "./data"
@@ -33,6 +36,12 @@ const (
func init() {
_ = os.MkdirAll("./data/theme", 0755)
var err error
defaultDistFiles, err = loadEmbeddedDist()
if err != nil {
panic("load embedded default frontend: " + err.Error())
}
}
func normalizeHTMLLanguage(language string) string {
@@ -123,11 +132,10 @@ func StaticRestricted(r *gin.RouterGroup, noRoute func(handlers ...gin.HandlerFu
}
func static(r *gin.RouterGroup, noRoute func(handlers ...gin.HandlerFunc), forceDefaultTheme bool) {
// 初始化嵌入式文件系统,指向 defaultTheme 根目录
// 假设 defaultTheme 内部结构也是: dist/, theme.json 等
// 初始化嵌入式文件系统,指向 defaultTheme 根目录。
defaultThemeFS, err := fs.Sub(PublicFS, "defaultTheme")
if err != nil {
panic("you may forget to put dist of frontend to web/public/defaultTheme/dist")
panic("embedded default theme metadata is unavailable: " + err.Error())
}
getConfig := func() map[string]any {
@@ -179,7 +187,11 @@ func static(r *gin.RouterGroup, noRoute func(handlers ...gin.HandlerFunc), force
return nil, "", false
}
if content, err := fs.ReadFile(defaultThemeFS, embedPath); err == nil {
if strings.HasPrefix(embedPath, DistDir+"/") {
if content, ok := defaultDistFiles[strings.TrimPrefix(embedPath, DistDir+"/")]; ok {
return content, mime.TypeByExtension(filepath.Ext(embedPath)), true
}
} else if content, err := fs.ReadFile(defaultThemeFS, embedPath); err == nil {
return content, mime.TypeByExtension(filepath.Ext(embedPath)), true
}
+9
View File
@@ -76,6 +76,15 @@ func TestReplaceHTMLLanguage(t *testing.T) {
}
}
func TestEmbeddedDistDoesNotEmbedRawFiles(t *testing.T) {
if _, err := PublicFS.ReadFile("defaultTheme/dist/index.html"); err == nil {
t.Fatal("PublicFS still embeds the raw frontend files")
}
if content, ok := defaultDistFiles[IndexFile]; !ok || len(content) == 0 {
t.Fatalf("embedded dist does not contain a non-empty %q", IndexFile)
}
}
func TestStaticRestrictedDoesNotServeCustomAssetOverride(t *testing.T) {
gin.SetMode(gin.TestMode)
t.Chdir(t.TempDir())
+11 -9
View File
@@ -9,9 +9,9 @@
### Build Requirements
1. Clone the frontend repository and build the static files
2. Copy the generated `dist` files to `web/public/defaultTheme/dist` in the backend repository
2. Pack the generated `dist` directory as `tar + zstd -19` to `web/public/defaultTheme/dist.tar.zst` in the backend repository
3. Copy `komari-theme.json` to `web/public/defaultTheme` if you want the default theme metadata and managed configuration to be available
4. Ensure `web/public/defaultTheme/dist/index.html` exists before building the backend
4. Ensure `web/public/defaultTheme/dist.tar.zst` contains `index.html` before building the backend
### Important Note
@@ -28,9 +28,9 @@
### 构建要求
1. 克隆前端仓库并构建静态文件
2. 将生成的 `dist` 文件复制到后端仓库内的 `web/public/defaultTheme/dist`
2. 将生成的 `dist` 目录使用 `tar + zstd -19` 打包为后端仓库内的 `web/public/defaultTheme/dist.tar.zst`
3. 如需让后台显示默认主题元数据和可管理配置,将 `komari-theme.json` 复制到 `web/public/defaultTheme`
4. 构建后端前,确保 `web/public/defaultTheme/dist/index.html` 存在
4. 构建后端前,确保 `web/public/defaultTheme/dist.tar.zst` 包含 `index.html`
### 重要提醒
@@ -47,9 +47,9 @@
### ビルド要件
1. フロントエンドリポジトリをクローンして静的ファイルをビルドする
2. 生成された `dist` ファイルをバックエンドリポジトリ内の `web/public/defaultTheme/dist` にコピーする
2. 生成された `dist` ディレクトリを `tar + zstd -19` で圧縮し、バックエンドリポジトリ内の `web/public/defaultTheme/dist.tar.zst` に配置する
3. デフォルトテーマのメタデータと管理設定を利用する場合は、`komari-theme.json` を `web/public/defaultTheme` にコピーする
4. バックエンドをビルドする前に、`web/public/defaultTheme/dist/index.html` が存在することを確認する
4. バックエンドをビルドする前に、`web/public/defaultTheme/dist.tar.zst` に `index.html` が含まれていることを確認する
### 重要な注意事項
@@ -68,8 +68,10 @@ cd komari-web
npm install
npm run build
# Copy frontend assets into the backend embed directory / 复制到后端 embed 目录 / バックエンドの embed ディレクトリにコピー
mkdir -p /path/to/komari/web/public/defaultTheme/dist
cp -r dist/* /path/to/komari/web/public/defaultTheme/dist/
# Pack frontend assets into the backend embed archive / 打包到后端 embed 归档 / バックエンドの embed アーカイブに圧縮
mkdir -p /path/to/komari/web/public/defaultTheme
tar -cf /tmp/komari-dist.tar -C dist .
zstd -19 -T0 -f /tmp/komari-dist.tar -o /path/to/komari/web/public/defaultTheme/dist.tar.zst
rm -f /tmp/komari-dist.tar
cp komari-theme.json /path/to/komari/web/public/defaultTheme/
```
+26 -19
View File
@@ -7,6 +7,7 @@ import (
"github.com/komari-monitor/komari/web/api/client"
public_api "github.com/komari-monitor/komari/web/api/public"
"github.com/komari-monitor/komari/web/api/terminal"
"github.com/komari-monitor/komari/web/filemanager"
"github.com/komari-monitor/komari/web/public"
jsonRpc "github.com/komari-monitor/komari/web/rpc/jsonrpc"
)
@@ -38,6 +39,9 @@ func registerPublicRoutes(r *gin.Engine) {
r.GET("/api/oauth_callback", public_api.OAuthCallback)
// 插件公开页面(visibility=public 的 iframe 页面),无需鉴权。
r.GET("/api/plugin/:short/*filepath", public_api.ServePluginFile)
// 短期文件预览令牌公开下载入口,供 Office 在线预览等服务端抓取。
r.GET("/api/preview/client/:uuid/file/download", filemanager.PreviewDownload)
r.HEAD("/api/preview/client/:uuid/file/download", filemanager.PreviewDownload)
// /api/clients 是 WebSocket 端点(客户端发 "get"/"get <uuid>" 拉取在线列表与最新上报),
// 非 JSON-RPC,保留为 WS handler。
r.GET("/api/clients", api.GetClients)
@@ -64,18 +68,13 @@ func registerAgentRoutes(r *gin.Engine) {
tokenAuthorized := r.Group("/api/clients", api.RequireRole(api.RoleAdmin, api.RoleClient))
{
// 上报类(WS / 原始流 / 兼容协议)保留 REST handler。
tokenAuthorized.GET("/report", client.WebSocketReport)
tokenAuthorized.POST("/uploadBasicInfo", client.UploadBasicInfo)
tokenAuthorized.POST("/report", client.UploadReport)
// Agent 上报统一使用 v2 JSON-RPC。
tokenAuthorized.GET("/v2/rpc", client.WebSocketV2RPC)
tokenAuthorized.POST("/v2/rpc", client.UploadV2RPC)
// File data uses a short-lived, raw HTTP stream opened by a file RPC.
tokenAuthorized.GET("/transfer/:id", filemanager.AgentTransfer)
tokenAuthorized.POST("/transfer/:id", filemanager.AgentTransfer)
tokenAuthorized.GET("/terminal", terminal.EstablishConnection)
// JSON 接口 -> RPC2 (client: 命名空间)。
tokenAuthorized.POST("/task/result", jsonRpc.Bind("client:taskResult", jsonRpc.WithRaw()))
tokenAuthorized.GET("/ping/tasks", jsonRpc.Bind("client:getPingTasks", jsonRpc.WithRaw()))
tokenAuthorized.POST("/ping/result", jsonRpc.Bind("client:uploadPingResult", jsonRpc.WithRaw()))
}
}
@@ -86,11 +85,14 @@ func registerAdminRoutes(r *gin.Engine) {
// --- 二进制/流/重定向类,保留 REST handler ---
g.GET("/download/backup", admin.DownloadBackup)
g.POST("/upload/backup", admin.UploadBackup)
// chunk upload 用于大备份文件分块上传,提高稳定性
g.POST("/upload/backup/init", admin.InitChunkUpload)
g.POST("/upload/backup/chunk", admin.UploadChunk)
g.POST("/upload/backup/merge", admin.MergeChunkUpload)
uploadHandler := admin.NewArchiveUploadHandler()
uploadGroup := g.Group("/upload")
{
uploadGroup.POST("/init", uploadHandler.Init)
uploadGroup.POST("/chunk", uploadHandler.Chunk)
uploadGroup.POST("/merge", uploadHandler.Merge)
uploadGroup.POST("/cancel", uploadHandler.Cancel)
}
g.GET("/test/geoip", jsonRpc.Bind("admin:testGeoip", jsonRpc.WithQuery("ip")))
g.POST("/test/sendMessage", jsonRpc.Bind("admin:testSendMessage"))
g.POST("/update/mmdb", admin.UpdateMmdbGeoIP)
@@ -98,10 +100,9 @@ func registerAdminRoutes(r *gin.Engine) {
g.PUT("/update/favicon", admin.UploadFavicon)
g.POST("/update/favicon", admin.DeleteFavicon)
// theme 含文件上传,保留 REST handler。
// theme 的安装流程通过统一的分片上传接口;其余主题接口保留 REST handler。
theme := g.Group("/theme")
{
theme.PUT("/upload", admin.UploadTheme)
theme.GET("/list", admin.ListThemes)
theme.POST("/delete", admin.DeleteTheme)
theme.GET("/set", admin.SetTheme)
@@ -174,7 +175,14 @@ func registerAdminRoutes(r *gin.Engine) {
clientGroup.POST("/:uuid/remove", jsonRpc.Bind("admin:removeClient", jsonRpc.WithPath("uuid")))
clientGroup.GET("/:uuid/token", jsonRpc.Bind("admin:getClientToken", jsonRpc.WithPath("uuid"), jsonRpc.WithFlat()))
clientGroup.POST("/order", jsonRpc.Bind("admin:orderClients"))
clientGroup.GET("/:uuid/terminal", api.RequireSensitive2FA(), terminal.RequestTerminal)
// RequestTerminal validates 2FA only when creating a new session. Reattach
// requests are authenticated against the existing session owner so a short
// network flap does not depend on the current TOTP window.
clientGroup.GET("/:uuid/terminal", terminal.RequestTerminal)
clientGroup.POST("/:uuid/file/upload", filemanager.Upload)
clientGroup.GET("/:uuid/file/download", filemanager.Download)
clientGroup.HEAD("/:uuid/file/download", filemanager.Download)
clientGroup.GET("/:uuid/file/preview-token", filemanager.CreatePreviewToken)
}
// records
@@ -205,13 +213,12 @@ func registerAdminRoutes(r *gin.Engine) {
clipboardGroup.POST("/:id/remove", jsonRpc.Bind("admin:deleteClipboard", jsonRpc.WithPath("id")))
}
// plugins: 上传走 REST(zip 二进制),启停/列表/日志走 RPC2,市场对齐主题市场。
// plugins: 安装流程通过统一的分片上传接口,启停/列表/日志走 RPC2,市场对齐主题市场。
pluginGroup := g.Group("/plugin")
{
pluginGroup.GET("/list", jsonRpc.Bind("admin:listPlugins"))
pluginGroup.POST("/enabled", jsonRpc.Bind("admin:setPluginEnabled"))
pluginGroup.GET("/logs", jsonRpc.Bind("admin:getPluginLogs", jsonRpc.WithQuery("short")))
pluginGroup.POST("/install", admin.UploadPlugin)
pluginGroup.GET("/market/sources", admin.ListPluginMarketSources)
pluginGroup.POST("/market/sources", admin.CreatePluginMarketSource)
pluginGroup.PUT("/market/sources/:id", admin.UpdatePluginMarketSource)
+3 -3
View File
@@ -115,7 +115,7 @@ handler 退化为薄适配层:解析 gin 参数 → 调 RPC → 把响应映
| --- | --- |
| `admin` | client CRUD、ping task、session/settings/weight、notification(load/offline/traffic)、clipboard、provider(messageSender/oidc)、task 查询、system(logs/cloudflared/exec/test)、xtermjs |
| `public` | getMe、getNodesInformation、getPublicSettings、getVersion、getClientRecentRecords、getRecordsByUUID、getPingRecords、getPublicPingTasks、recordVisitorEvent |
| `client` | getPingTasks、uploadPingResult、taskResult |
| Agent v2 | `agent.report`、`agent.basicInfo`、`agent.pingResult`、`agent.taskResult`、`agent.pull`(`/api/clients/v2/rpc`) |
### 声明式路由桥 `Bind`
@@ -136,6 +136,6 @@ r.GET("/api/admin/client/:uuid", jsonRpc.Bind("admin:getClient", jsonRpc.WithPat
### 保留为 REST 的接口(不走 RPC 桥)
二进制/流/重定向/特殊鉴权类,集中在 `web/api/admin`(2fa/theme/backup/update/oauth 绑定)、
`web/api/public`(login/logout/oauth/plugin)、`web/api/client`(report WS+POST、v2 RPC、uploadBasicInfo、terminal、AutoDiscovery 注册)。
`web/api/public`(login/logout/oauth/plugin)、`web/api/client`(v2 RPC、terminal、AutoDiscovery 注册)。
agent v1/v2 上报的核心逻辑已统一到 `web/api/client/ingest.go`。
agent v2 上报的核心逻辑统一在 `web/api/client/ingest.go`。
Loaded 100 of 113 files, more files were not shown because too many files have changed in this diff. Show more