Compare commits

..
621 Commits
Author SHA1 Message Date
Akizon 5d26f1b18a Update generate-release-notes.yml 2026-07-20 09:53:15 +08:00
Akizon77 80cc39b02d Include commits in release notes 2026-07-20 09:53:15 +08:00
Akizon77 11c65ddbd0 Use stable release base for snapshot notes 2026-07-20 09:53:15 +08:00
Akizon77 ad74b57ecc Read release note workflow inputs 2026-07-20 09:53:15 +08:00
Akizon77 e13f79577d Improve release notes generation 2026-07-20 09:53:14 +08:00
Akizon77 832b80054f ci: generate AI-assisted release notes from stable tags 2026-07-20 09:53:14 +08:00
Akizon77 09d70996d3 ci: reuse frontend and toolchain actions in workflows
Extract repeated frontend build, frontend restore, and Zig setup steps into shared GitHub Actions so CI workflows use the same artifact and toolchain setup path. Update binary and Docker builds to depend on the frontend artifact, and require an explicit Docker dispatch version for controlled image tagging.
2026-07-20 09:53:14 +08:00
Akizon77 303f2bbcf5 Use UTC+8 for snapshot versions 2026-07-20 09:53:14 +08:00
airium 28ce225aa0 ci: align snapshot workflow behavior 2026-07-20 09:53:14 +08:00
Akizon77 683b7f652b fix: avoid sqlite database is locked failures for record writes 2026-07-20 09:53:05 +08:00
guocn a181b22071 fix(theme): prevent path traversal in theme operations
DeleteTheme/UpdateTheme/SetTheme used the user-supplied theme name
(short) directly in filepath.Join("./data/theme", short) before
filesystem operations (os.RemoveAll / os.Stat). Without validation,
a crafted short value like "../../" normalizes outside the theme
directory, letting an authenticated admin delete or access arbitrary
directories within the process working directory.

Reuse the existing isValidThemeShort validator (allowlist of
[a-zA-Z0-9_-]) in all three handlers before the path is built.
Add a regression test covering traversal payloads and absolute paths.
2026-07-20 09:53:05 +08:00
Torther d2f9261bc2 fix(telegram): include API error description in non-200 responses
SendTextMessage checked resp.StatusCode before decoding the JSON body, so non-200 responses returned only the status code without the error reason.

Move JSON decode before the status check. When the status is not 200, use the description field from the API response as the error message. Fall back to the original status code message if decode fails or description is empty.
2026-07-20 09:53:04 +08:00
Akizon77 5f50d1ba00 Merge branch 'fix/temp-share-private-site' 2026-06-28 21:00:52 +08:00
Akizon77 7e19909dd6 fix: allow temp-share guests to bypass private site block
Private site dispatch only checked anonymous principal against the login whitelist and ignored meta.TempShareValid, so valid temp_key share links were rejected with -32041 on the analysis methods. Add !meta.TempShareValid to the gate; CheckPrincipal still limits anonymous callers to public:* (guest).
2026-06-28 21:00:41 +08:00
Akizon77 9db4daac10 ci: limit build workflow to PRs and fix snapshot release creation 2026-06-28 20:14:17 +08:00
Akizon 0a4cd6bd8c Merge pull request #578 from komari-monitor/feat/vacuum-database
feat: add database vacuum and size API endpoints
2026-06-28 19:33:44 +08:00
Akizon77 20c865ed20 feat(install): add TUI support with whiptail/dialog and text fallback 2026-06-28 19:32:07 +08:00
Akizon77 e740bf2797 feat(install): add stable/snapshot channel selection for binary install and upgrade 2026-06-28 19:28:43 +08:00
Akizon77 074958fad7 feat: add database vacuum and size API endpoints
Add admin RPC methods getDatabaseSize and vacuumDatabase, and register /api/admin/database/size and /api/admin/database/vacuum routes. VACUUM compacts the SQLite database to reclaim disk space.
2026-06-28 18:59:16 +08:00
Akizon77 11d3ba8236 fix(rpc): allow login-page endpoints in private-site mode
Fixes #567 where enabling private site blocked the login page itself,
making it impossible to log in from external IPs.

The Dispatch private-site check was too broad, blocking all anonymous
requests including the meta endpoints needed to render the login page
(getMe, getPublicSettings, getVersion). Now exempt these from the
private-site block via whitelist.

Adds regression test to guard the required whitelist entries.
2026-06-28 00:38:29 +08:00
Akizon77 ce41745edb refactor(rpc): migrate ACL to capability-based Principal model 2026-06-28 00:19:48 +08:00
Akizon77 036121c148 refactor(rpc): unify identity into Principal model 2026-06-27 23:50:21 +08:00
Akizonandyuanhhs dd0539392b Preserve RPC node order (#566)
Co-authored-by: yuanhhs <yuanhhs@users.noreply.github.com>
2026-06-27 23:23:36 +08:00
Akizon77 50ba8310ca Merge PR #571: fix(rpc): enforce sensitive-operation 2FA at the RPC boundary 2026-06-27 23:20:35 +08:00
carbofish 50fe286528 fix(rpc): enforce sensitive-operation 2FA at the RPC boundary
Sensitive methods (admin:exec) were gated by sensitive-operation 2FA only through REST route middleware. Move the enforcement to the JSON-RPC boundary so every entrypoint applies the same check:

  - pkg/rpc: add a transport-agnostic sensitive-method registry (MarkSensitive / IsSensitive).
  - web/rpc/jsonrpc: verify sensitive-operation 2FA in dispatchWithSensitive before dispatch, covering the REST bridge, /api/rpc2 POST and WebSocket.
  - mark admin:exec sensitive.

API-key callers and the existing no-2FA-configured semantics are preserved; non-sensitive methods are unaffected.
2026-06-27 05:18:45 +08:00
yuanhhs 144efdc8ad Preserve RPC node order 2026-06-22 07:42:49 +08:00
Akizon77 e446bee900 ci: expand release notes 2026-06-21 20:53:30 +08:00
Akizon77 a980befd60 fix(metric): isolate compaction scan and delete to prevent data loss
Add compactTxOptions to the dialect interface so the raw scan and raw
deletion in CompactMetric observe a consistent view. PostgreSQL and
MySQL escalate to SERIALIZABLE, ensuring a point inserted between the
scan and the deletion cannot be removed before it is rolled up; on
PostgreSQL this surfaces as a serialization failure that is retried,
while InnoDB blocks the conflicting insert via locking reads. SQLite
returns nil since its single write connection already serializes writes.

Add regression tests covering partial fine-row overwrites of coarse
rollups during compaction.
2026-06-21 17:00:01 +08:00
Akizon77 2e0c598dae feat(metric): support arbitrary percentile aggregations 2026-06-21 01:11:07 +08:00
Akizon77 771a192a3d chore(deps): upgrade Go and add SQL database drivers
Bump the module target to Go 1.25 and refresh dependency metadata.

Add MySQL, PostgreSQL, and SQLite drivers as direct dependencies to support SQL backends, while updating related transitive modules and checksums.
2026-06-20 20:11:16 +08:00
Akizon77 95cc65e28d docs: add abuse warning and remote control disclosure 2026-06-17 19:53:46 +08:00
airium b6e36ee0c4 Merge pull request #555 from airium/fix-traffic-report
fix: 修复流量报告为 0 B 问题
2026-06-17 19:21:30 +08:00
Akizon 25f3e1d2bd Merge pull request #551 from komari-monitor/fix/rpc2-route-contract-regressions
fix(router): 修复 RPC2 迁移引入的接口契约回归
2026-06-11 17:06:51 +08:00
Akizon77andClaude Opus 4.8 091eafa743 fix(router): 修复 RPC2 迁移引入的接口契约回归
对照 1.1.7 与重构前基线逐路由核对,修复 3 处契约破坏:

- /api/clients 是 WebSocket 端点(api.GetClients,客户端发 "get"/"get <uuid>"
  拉取在线列表与最新上报),迁移时误绑为 REST public:getNodesInformation,
  导致 wss:// 连接失败。还原为 api.GetClients。
- POST /api/admin/settings/oidc 与 /message-sender:原响应为
  {status,message:"",data:{message:"..."}},误用 WithFlat() 变成
  {status,message:"..."}(丢失 data 层)。移除 WithFlat,恢复 data.message 契约。

其余路由(WS/流端点、响应渲染器、完整路径)经与 1.1.7 全量比对确认一致。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 17:03:24 +08:00
Akizon cbe665591f Merge pull request #550 from komari-monitor/refactor/rpc2-unified-pipeline
refactor(rpc): 统一 RPC2 管道并全量迁移 REST 接口
2026-06-11 16:36:57 +08:00
Akizon77andClaude Opus 4.8 0abd9ff9bc refactor(rpc): 统一 RPC2 管道并全量迁移 REST 接口
将混乱的「Web 接口 → 业务逻辑」收敛为统一管道:
接口 → 生成 RPC 请求 → Dispatch(权限+命名空间) → RPC handler(业务),
为后续 JS 插件接入打地基(插件本身未实现)。

核心:
- pkg/rpc/permission.go:声明式 ACL + 通配符权限,按特异性裁决
  (精确 > 字面前缀更长的通配 > 全局 *)。Allow/RegisterNamespace 供插件扩展。
- pkg/rpc:新增 Unregister(动态注销,供插件卸载);ContextMeta 增 SessionToken/TempShareValid。
- web/rpc/jsonrpc 分层:dispatch(统一 Dispatch+OnInternalRequest)、
  transport(HTTP/WS+CallFromGin)、register、bridge(声明式路由桥)。
  WS 改同步写,删除旧 JsonRpc.go 的死代码 registry。

全量迁移:
- 所有 JSON 接口迁为 RPC2 方法(admin/public/client 三命名空间),
  router 经 bridge.Bind 直接绑定,删除整个 per-resource gin handler 层。
- 渲染器覆盖全部响应形状:standard/flat/raw/message。
- agent v1/v2 上报逻辑统一到 web/api/client/ingest.go。

文件收敛:
- web/api/admin 从 ~20 文件+5 子目录降到 6 个 REST-binary 文件
  (2fa/theme/download/upload/update/oauth),update 子包合并。
- 二进制/流/重定向/特殊鉴权类保留为 REST。

开发者文档见 web/rpc/jsonrpc/README.md。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 15:05:49 +08:00
airium 618ced3b8f test(auth): remove obsolete dev cookie bypass coverage (#547) 2026-06-11 12:22:27 +08:00
Akizon77 f7fd459723 fix(auth): only set secure session cookie over HTTPS 2026-06-10 20:48:43 +08:00
Akizon77 bb22a818bf fix(client): fill GeoIP region for v2 basic info 2026-06-10 01:12:02 +08:00
Akizon77 b66fe47487 fix(config): rename CORS toggle to origin check setting
Replace the legacy allow_cors setting with cors_origin_check_enabled
throughout server setup, public info output, and configuration constants.
This clarifies that the option controls API CORS origin validation rather
than broadly enabling CORS.
2026-06-09 17:27:06 +08:00
Akizon77 98122fa4d1 ci: generate release notes from GitHub 2026-06-09 16:46:33 +08:00
Akizon77 c3f65b1a32 fix go tests 2026-06-08 21:14:27 +08:00
Akizon77 7da616ce4c feat(security): add configurable CORS and origin checks
Add settings for allowed CORS origins and WebSocket origin validation so
deployments can restrict cross-origin access instead of using a global
wildcard policy.

Move CORS handling into the security middleware and update report cache
usage to use the dedicated report package.
2026-06-08 20:45:00 +08:00
airium 9d914e70b9 fix: harden traffic report accounting and validation (#542)
- Store per-record traffic deltas and sum them for scheduled reports
- Preserve traffic delta totals during long-term record compaction
- Compact only complete 15-minute buckets to avoid partial-bucket undercounts
- Serialize report saves per client so concurrent uploads cannot double-count deltas
- Enforce enabled traffic report cadence validation across edit and enable paths
- Add regression coverage for compaction boundaries, cadence validation, and client save locking
2026-06-08 11:26:13 +08:00
Akizon77 6f0624e9b4 Merge branch 'main' into codex/theme-configuration-types 2026-06-08 01:58:52 +08:00
Akizon77 3cc67e11cd fix: align theme redirect and html language handling 2026-06-08 01:53:28 +08:00
Akizon77 742a5e953e feat: support raw and redirect themes 2026-06-08 01:53:28 +08:00
Akizon77 3b82cee4b1 fix: align theme redirect and html language handling 2026-06-08 01:53:20 +08:00
Akizon77 2b010954a3 feat: support raw and redirect themes 2026-06-08 00:48:37 +08:00
Akizon77 6f23565517 refactor: isolate startup migrations 2026-06-08 00:01:32 +08:00
airium 7c2896c3fe fix(remote-exec): preserve command text during validation (#543)
Reject whitespace-only remote exec commands without trimming the submitted
command body, so multiline scripts and meaningful leading/trailing content are
preserved through dispatch.
2026-06-07 23:54:43 +08:00
Akizon77 721d61bc4e docs: update default theme build path 2026-06-06 22:59:38 +08:00
kylvenp 014b236745 feat: add network test rpc protocol structs 2026-06-06 21:34:48 +08:00
Akizon77 80c8d7e5ee Merge pull request #537 from airium/fix-http-development 2026-06-06 20:05:30 +08:00
Akizon77 8423881cc4 Merge pull request #541 from airium/support-phy-cpu-count 2026-06-06 19:59:16 +08:00
Akizon77 a1d644977c Merge pull request #538 from airium/fix-theme-path 2026-06-06 19:59:05 +08:00
Akizon77 301fb536f2 Merge pull request #539 from aijwfsibari/main 2026-06-06 19:53:17 +08:00
airium 49a8d772c8 feat: support physical CPU core count 2026-06-06 19:15:03 +08:00
aijwfsibari a9adae29d2 add Traffic report patch 2026-06-05 15:04:21 +08:00
airium d6e23baa19 fix: correct defaultTheme path across CI and docs
The Go embed directive in web/public/public.go resolves defaultTheme
relative to the package directory (web/public/defaultTheme/),
but all CI workflows and documentation referenced public/defaultTheme/
which would have caused builds to fail or embed an empty directory.
2026-06-02 13:24:23 +08:00
airium 658e7603f8 fix: allow insecure session cookies in development build 2026-06-02 02:31:01 +08:00
Akizon77 fb0b9b9910 feat(agent): add v2 pull events for queued agent tasks
Introduce agent.pull protocol types with event metadata and ACK fields.
Route ping dispatch through the agent API so tasks can be queued for
pull-based clients instead of only sent over active WebSocket sessions.
2026-05-31 19:01:02 +08:00
Akizon77 e149e8b8b7 refactor: remove legacy client APIs and support v2 pings
Remove deprecated client database helpers tied to common protocol types and
update report handling to use protocol/v1 models. Send ping tasks with the
v2 JSON-RPC payload for v2 clients while preserving existing behavior for
legacy websocket clients.
2026-05-31 16:50:04 +08:00
Akizon77 d2cc1503de refactor: centralize scheduled jobs with corn package
Replace ad-hoc ticker loops with named cron-based scheduled tasks for cleanup,
minute reporting, and expiration checks. Add scheduler shutdown handling and
log errors when schedules fail to reload or register.

Update imports to use the relocated pkg/config package across affected code.
2026-05-31 15:54:23 +08:00
Akizon77 f86803feff refactor(web): move HTTP route setup out of server
Relocate API routing, OAuth, and Nezha compatibility wiring into the web package to keep server startup focused on initialization concerns and improve code organization.
2026-05-31 15:35:47 +08:00
flyflas a2e3c7218d Merge pull request #525 from flyflas/feat/addXtermjsSetting
Feat/add xtermjs setting
2026-05-31 14:14:16 +08:00
Akizon77 567d2b66fb fix(sessions): 添加移除过期会话功能并更新定时任务 2026-05-08 12:44:37 +08:00
Akizon 473dbb9057 Merge pull request #519 from iamsxm/feature/choose-all
feat(ping): 延迟监测增加默认开启选项
2026-05-07 20:57:29 +08:00
Akizon 89bd47d5df Merge pull request #510 from KanameMadoka520/feature/cloudflared-followup
feat: improve built-in cloudflared tunnel management
2026-05-07 20:52:04 +08:00
Akizon 10c4a9dafc Merge pull request #511 from Lixsp11/db-opt
fix: Optimize database indexes and improve performance on I/O-constrained hosts
2026-05-07 20:51:49 +08:00
Akizon f4d73bf8c9 Merge pull request #504 from paulzzh/patch-1
fix: mjpeg 价格周期与剩余时间显示错误
2026-05-07 20:48:16 +08:00
xiaomi 441e2e9194 feat(ping): 延迟监测增加默认开启选项 2026-05-01 23:21:40 +08:00
lixsp11 c15881c463 fix: Avoid rewriting the database at startup, and create composite indexes to boost query. 2026-04-16 23:04:05 +08:00
KanameMadoka520 f5acbd2efe refactor: narrow cloudflared backend PR scope 2026-04-15 21:49:59 +08:00
KanameMadoka520 e07378c92b docs: document built-in cloudflared support 2026-04-15 07:28:21 +08:00
KanameMadoka520 cd38d32a7d build: bundle cloudflared in the Docker image 2026-04-15 07:18:20 +08:00
KanameMadoka520 53f326390e feat: add managed cloudflared backend APIs 2026-04-15 07:18:10 +08:00
Akizon 651d24128c Merge pull request #509 from airium/fix-timeout-if-empty-exe-result
fix: 防止无返回文本的远程执行结果被后端拒绝而变成超时
2026-04-14 01:32:22 +08:00
airium 2660026917 fix: make task result field optional in request 2026-04-13 21:40:59 +08:00
paulzzh ac7ace27cb refactor: 更名翻译键 2026-04-12 04:43:30 +08:00
paulzzh 1de008269b fix: mjpeg 剩余时间显示错误 2026-04-12 04:32:53 +08:00
paulzzh 4bdbff0377 fix: mjpeg 价格周期显示错误 2026-04-12 04:21:10 +08:00
Akizon 3415c06256 Merge pull request #485 from airium/main
feat: 支持自定义排序延迟检测任务
2026-03-30 16:38:14 +08:00
Akizon 9f3da4c17a Merge pull request #495 from tranquac/fix/ssrf-theme-download
fix: prevent SSRF in theme download by blocking private/internal IPs
2026-03-30 16:37:43 +08:00
quactv 3d0c83557e fix: prevent SSRF in theme download by blocking private/internal IPs
Signed-off-by: tranquac <tranquac@users.noreply.github.com>
2026-03-27 21:50:41 +07:00
airium e319eb7f20 fix: harden ping task ordering update semantics 2026-03-14 23:31:27 +08:00
airium 7f007b7eb9 feat: Add reorder API to ping tasks 2026-03-13 18:18:57 +08:00
Akizon 9e665005e6 Merge pull request #484 from komari-monitor/fix/流量限制溢出
fix: #393 验证流量限制字段为有效的非负整数值
2026-03-10 15:41:33 +08:00
Akizon77 b5b32b5f64 fix: #393 验证流量限制字段为有效的非负整数值 2026-03-10 15:40:53 +08:00
Akizon 89a699b6cb Merge pull request #483 from komari-monitor/fix/IP地址乱码
fix: #363 IP地址出现乱码
2026-03-10 15:29:22 +08:00
Akizon77 79cbf2d2d1 fix: #363 IP地址出现乱码 2026-03-10 15:28:38 +08:00
Akizon ab00c9529e refactor: 按照业务整理api接口 (#482)
* refactor(wip): 按照业务整理API接口

* refactor: 整合认证中间件
fix: #481 Api 鉴权获取UUID为空
2026-03-10 15:23:54 +08:00
Akizon77 21500a6727 refactor: 整合认证中间件
fix: #481 Api 鉴权获取UUID为空
2026-03-10 15:21:02 +08:00
Akizon77 066b944789 refactor(wip): 按照业务整理API接口 2026-03-10 14:37:22 +08:00
Akizon d7dd816a26 Merge pull request #478 from Real-Wanger/main
fix: reset traffic cache when traffic drops
2026-03-04 14:17:34 +08:00
Wanger c5b464c000 fix: reset traffic cache when traffic drops
修复当节点流量被清零重置时,内存中的告警进度缓存未同步清理,导致无法触发后续阈值通知的 Bug。
2026-03-02 14:27:42 +08:00
Akizon 645c733393 Merge pull request #468 from wayjam/main
feat: import theme api #467
2026-02-26 19:46:09 +08:00
Akizon 85088e64ca Merge pull request #471 from vhvy/feat/keepalive-via-post
feat: 支持 HTTP POST 上报实时信息和在线状态管理,添加 HTTP 接口获取 Ping 任务和提交 Ping 结果
2026-02-26 19:40:05 +08:00
Akizon77 896b3e45fc fix: 添加 base_scripts_url 字段到 Legacy 结构体 #472 2026-02-26 19:04:18 +08:00
vhvy 490f951b59 feat: 添加 HTTP 接口获取 Ping 任务和提交 Ping 结果 2026-02-23 21:27:03 +08:00
vhvy 69c96255d7 feat: 支持 HTTP POST 上报实时信息保活及在线状态管理 2026-02-23 21:24:14 +08:00
WayJam So 487542e55d feat: import theme api #467 2026-02-21 16:55:50 +08:00
Akizon ca8fe76c06 Merge pull request #463 from komari-monitor/refactor/增强临时分享的兼容性
refactor: 增强临时分享的兼容性
2026-02-19 00:15:29 +08:00
Akizon77 e6e3619cd2 refactor: 增强临时分享的兼容性 2026-02-19 00:14:23 +08:00
Akizon77 92ab67d5f3 feat: 添加临时访问许可功能,允许使用临时密钥访问私有站点 2026-02-16 00:34:43 +08:00
Akizon77 77f5bef1a5 fix: 配置默认值 2026-02-15 14:17:03 +08:00
Akizon77 58e43a0862 fix: 增强 WebSocket 升级错误消息,包含详细错误信息 2026-02-12 13:41:01 +08:00
Akizon 80c1f2747a Merge pull request #455 from komari-monitor/fix/expire-notification-panic
fix: 修复过期通知的通知提前天数类型转换错误
2026-02-12 13:13:12 +08:00
Akizon77 b7b2d8b050 fix: 修复过期通知的通知提前天数类型转换错误 2026-02-12 13:12:16 +08:00
Akizon77 b9eabd7cdc docs: 添加 1Panel 应用商店的安装说明 2026-02-10 22:54:48 +08:00
Akizon77 0a19465140 docs: 添加破碎工坊云的介绍和链接 2026-02-09 23:08:57 +08:00
Akizon e874b5db18 Merge pull request #451 from komari-monitor/fix/config-panic
Fix/config panic
2026-02-08 15:49:00 +08:00
Akizon77 23e7d97e3c fix: 流量提醒 2026-02-08 15:48:02 +08:00
Akizon77 a222de2c55 fix: 修复 GetAs 函数中的默认值类型不匹配错误 2026-02-08 13:26:43 +08:00
Akizon bb1eed1ccb Merge pull request #448 from komari-monitor/fix-path-traversal-vulnerability
fix: 修复路径穿透安全漏洞
2026-02-07 23:39:57 +08:00
Akizon77 c34bc8da69 fix: 修复路径穿透安全漏洞
- 添加 isSafePath 函数验证路径是否在允许的目录内
- 在 getFileContent 中验证主题 ID 和文件路径
- 清理所有用户输入的路径,防止 ../ 等路径穿透攻击
- 确保嵌入式文件系统路径也经过验证

Security: 防止攻击者通过构造特殊路径访问系统任意文件
2026-02-07 23:39:08 +08:00
Akizon77 249c4376f1 fix: 异步发送登录信息 2026-02-07 23:02:45 +08:00
Akizon77 d36bde7d94 fix: 默认主题不显示 2026-02-07 22:59:50 +08:00
Akizon 493d2a2707 Merge pull request #447 from komari-monitor/feat/default-theme
feat: default theme
2026-02-07 20:39:53 +08:00
Akizon77 93a9fbe5cc feat: default theme 2026-02-04 22:20:45 +08:00
Akizon b070f33bf9 Merge pull request #445 from komari-monitor/refactor/config-structure
refactor: 调整配置结构
2026-02-04 19:46:01 +08:00
Akizon ee5480dba7 Merge pull request #442 from flyflas/add-email-html-support
feat:add_email_html_support
2026-02-04 19:43:09 +08:00
Akizon77 2e9ef3d5ba refactor: 调整配置结构 2026-02-04 19:39:48 +08:00
xiaobai 1061b84087 feat:add_email_html_support 2026-01-29 14:44:58 +08:00
Akizon77 0dc80f7696 更新依赖项 2026-01-29 01:17:36 +08:00
Akizon77 212e3c689b feat: 添加网络流量指标支持 2026-01-29 01:05:49 +08:00
Akizon77 488ca451ae feat: RPC 内部请求,MJPEG流 2026-01-29 00:44:19 +08:00
Akizon77 f50c167b37 fix: 当您修改历史数据保存时间相关设置时,需要重新启动服务端才会生效 2026-01-29 00:33:29 +08:00
Akizon77 4e91d62d48 feat: 添加 WAL 检查点优化数据库性能 2025-11-14 16:44:48 +08:00
Akizon a1b0b1e177 Merge pull request #355 from komari-monitor/dev
docs: 添加镜像站链接 https://github.com/komari-monitor/komari-document/issue…
2025-11-09 11:52:36 +08:00
Akizon77 147dd2011a docs: 添加镜像站链接 https://github.com/komari-monitor/komari-document/issues/10 2025-11-09 11:43:05 +08:00
Akizon77 d24b71d4af docs: 移除 ClawRun - 云平台未更新维护且开发者无法更改内容 2025-11-04 23:16:14 +08:00
Akizon 483e3a94e9 remove Claw Run 2025-11-04 22:10:48 +08:00
Akizon e612f5ed8f Add badges for GitHub and DeepWiki to README 2025-11-04 21:38:45 +08:00
github-actions[bot] 78f3c8fd4b Merge pull request #350 from komari-monitor/dev
fix: #344 引入更宽松的认证方式
2025-11-04 06:50:01 +00:00
Akizon77 5a4e073d63 fix: #344 引入更宽松的认证方式 2025-11-04 14:49:34 +08:00
Akizon77 22cbfecfa9 feat: 整合日志功能 2025-11-04 14:36:40 +08:00
github-actions[bot] d1b059fc2f Merge pull request #349 from komari-monitor/dev
feat: 添加对 Server酱Turbo & Server酱³ 推送通道支持(SendKey)
2025-11-04 05:02:09 +00:00
Akizon cf5c410aa6 Merge branch 'main' into dev 2025-11-04 13:01:53 +08:00
Akizon 9c46673a1a Merge pull request #348 from RSJWY/dev
feat: 添加对 Server酱Turbo & Server酱³ 推送通道支持(SendKey)
2025-11-04 00:50:10 +08:00
RSJWY 45c27ae981 feat: 添加对 Server酱Turbo & Server酱³ 推送通道支持(SendKey) 2025-11-03 22:29:55 +08:00
akizon77 b62fb1f708 feat: 添加 JavaScript 消息发送器,支持 fetch 和 XMLHttpRequest API 2025-10-25 20:43:14 +08:00
github-actions[bot] 579c196643 Merge pull request #335 from komari-monitor/dev
feat: 添加 NOTICE 文件,包含版权和许可证信息
2025-10-18 09:16:09 +00:00
Akizon77 e8da6e213a feat: 添加 NOTICE 文件,包含版权和许可证信息 2025-10-18 17:15:20 +08:00
github-actions[bot] d9426e43a9 Merge pull request #333 from komari-monitor/dev
feat: 添加 EULA 接受字段到配置模型
2025-10-17 17:15:38 +00:00
Akizon77 aad0274d56 feat: 添加 EULA 接受字段到配置模型 2025-10-18 00:15:05 +08:00
github-actions[bot] d65f7ca924 Merge pull request #331 from komari-monitor/dev
fix: 重复注册Bark
2025-10-15 05:14:01 +00:00
Akizon77 a4d95640a5 fix: 重复注册Bark 2025-10-15 13:13:31 +08:00
Akizon77 adda3a2488 docs: 移除Crowdin 2025-10-15 13:09:18 +08:00
github-actions[bot] 34654c0969 Merge pull request #320 from komari-monitor/dev
fix: #319 伪随机 token 可预测
2025-10-09 11:09:44 +00:00
Akizon77 b708f9817d fix: 伪随机 token 可预测 2025-10-09 19:09:11 +08:00
github-actions[bot] 1210a1cbd9 Merge pull request #314 from komari-monitor/dev
refactor: 重写备份上传功能
2025-10-06 08:10:07 +00:00
Akizon77 24f1a673b9 refactor: 重写备份上传功能 2025-10-06 16:07:36 +08:00
github-actions[bot] a3162e1992 Merge pull request #313 from komari-monitor/dev
feat: 添加配置项以控制是否向访客发送 IP 地址
2025-10-06 07:31:40 +00:00
Akizon77 17f904e4ab feat: 添加配置项以控制是否向访客发送 IP 地址 2025-10-06 13:52:12 +08:00
github-actions[bot] 1324f4fa21 Merge pull request #312 from komari-monitor/dev
refactor: 重写下载备份功能
2025-10-06 05:06:40 +00:00
Akizon77 cf15144bb9 refactor: 重写下载备份功能 2025-10-06 13:06:01 +08:00
github-actions[bot] 946b27b3b8 Merge pull request #309 from komari-monitor/dev
feat: 启用SQLite的WAL模式
2025-10-03 06:12:48 +00:00
Akizon77 aed5254cf0 feat: 启用SQLite的WAL模式 2025-10-03 14:11:47 +08:00
github-actions[bot] 7b8922b7ac Merge pull request #296 from komari-monitor/dev
feat: 添加自定义脚本连接域名配置
2025-09-26 13:58:45 +00:00
Akizon77 733ad4c6c0 feat: 添加自定义脚本连接域名配置 2025-09-26 21:58:01 +08:00
github-actions[bot] ef98ddd1d3 Merge pull request #295 from komari-monitor/dev
fix: 优化默认通知模板
2025-09-26 13:55:08 +00:00
Akizon77 ca28c40aca fix: 优化默认通知模板 2025-09-26 21:54:34 +08:00
github-actions[bot] 0af5903bfe Merge pull request #283 from komari-monitor/dev
fix: 增强模板替换功能,支持 JSON 内容类型的占位符转义
2025-09-21 14:09:52 +00:00
akizon77 717b7b4a88 fix: 增强模板替换功能,支持 JSON 内容类型的占位符转义 2025-09-21 22:08:54 +08:00
github-actions[bot] 05eec9515b Merge pull request #282 from komari-monitor/dev
fix: 处理QQ发送消息时的短响应错误
2025-09-21 13:42:07 +00:00
akizon77 877852fc29 fix: 处理QQ发送消息时的短响应错误 2025-09-21 21:38:00 +08:00
github-actions[bot] dbb9075a52 Merge pull request #281 from komari-monitor/dev
fix: 实现按任务ID分组的记录比例下采样
2025-09-20 16:43:49 +00:00
akizon77 eafcf7fe76 fix: 实现按任务ID分组的记录比例下采样 2025-09-21 00:43:00 +08:00
github-actions[bot] 094ce6c1ba Merge pull request #280 from komari-monitor/dev
feat: 更新通知机制,使用事件消息替代文本消息
2025-09-20 11:41:25 +00:00
akizon77 2eed2cf7ba fix: 测试消息使用事件 2025-09-20 17:37:22 +08:00
akizon77 88b9d3bdac Merge branch 'main' into dev 2025-09-20 17:22:29 +08:00
Akizon 9c87227613 Merge pull request #269 from kdwycz/dev
新增了GPU监控数据接收存储
2025-09-17 16:53:46 +08:00
Akizon77 bc737b1ef9 feat: 更新通知模板,添加表情和消息内容 2025-09-17 13:50:19 +08:00
Akizon77 911d990690 feat: 更新通知机制,使用事件消息替代文本消息 2025-09-17 13:41:17 +08:00
kdwycz 0f7cd5f438 新增了GPU监控数据接收存储
接收agent发送的gpu监控数据.
参考Record实现, 新增GPURecord表存储监控数据和压缩监控数据.
2025-09-15 20:23:16 +08:00
github-actions[bot] 354903b97b Merge pull request #264 from komari-monitor/dev
fix: 修正 WebSocket 升级逻辑
2025-09-14 05:06:52 +00:00
Akizon77 b432396111 fix: 修正 WebSocket 升级逻辑 2025-09-14 13:06:02 +08:00
github-actions[bot] 491c0ad531 Merge pull request #263 from komari-monitor/dev
feat: 延迟显示更多信息
2025-09-14 02:41:08 +00:00
Akizon77 1540b78487 feat: 延迟显示更多信息 2025-09-14 01:17:51 +00:00
Akizon77 11f43b116d fix: 修正Uptime 2025-09-13 20:05:46 +08:00
github-actions[bot] a0760c8e93 Merge pull request #261 from komari-monitor/dev
feat: rpc2最近信息补全延迟、uptime
2025-09-13 09:30:40 +00:00
github-actions[bot] 85ecfc3500 Merge pull request #260 from komari-monitor/dev
feat: 添加 Nezha 兼容 gRPC 服务器的启动和停止功能
2025-09-13 09:29:10 +00:00
Akizon77 4c2308622a feat: rpc2最近信息补全延迟、uptime 2025-09-13 17:28:04 +08:00
Akizon77 25b6a25f9a fix: 更新 OAuthCallback 重定向路径,从 /manage 改为 /admin 2025-09-13 16:19:51 +08:00
Akizon77 7138c22d02 refactor: 移除不必要的 nezhaBoot 变量,更新 StopNezhaCompat 以强制停止服务器 2025-09-13 15:52:57 +08:00
Akizon77 a06a5c9e72 feat: 替换 StartNezhaCompatServer 为 StartNezhaCompat,标记为已弃用 2025-09-13 15:45:20 +08:00
Akizon77 668eb63668 feat: 添加 Nezha 兼容 gRPC 服务器的启动和停止功能 2025-09-13 15:09:34 +08:00
Akizon77 7d75f1ba37 feat: 在客户端自动创建时记录审计日志事件 2025-09-13 15:01:20 +08:00
github-actions[bot] 2c7b59985e Merge pull request #259 from komari-monitor/dev
feat: 添加数据掩码功能以保护敏感信息
2025-09-13 06:36:17 +00:00
Akizon77 b616eeb2d4 feat: 添加数据掩码功能以保护敏感信息 2025-09-13 14:35:42 +08:00
Akizon77 0f0e359c63 wip: 通知 2025-09-12 03:01:42 +00:00
Akizon77 1b83644788 docs: 更新 Sharon 图片 2025-09-10 04:40:13 +00:00
github-actions[bot] 9a9200744e Merge pull request #250 from komari-monitor/dev
feat: 兼容哪吒Agent
2025-09-09 14:41:47 +00:00
Akizon77 e763bc12c4 docs: 添加 SharonNetworks 的赞助信息 2025-09-09 14:39:39 +00:00
Akizon77 8d7b47a7de feat: 兼容哪吒Agent 2025-09-08 13:38:47 +00:00
github-actions[bot] d7b3c3443d Merge pull request #240 from komari-monitor/dev
feat: 未绑定第三方账户时不能关闭密码登录
2025-09-06 04:03:58 +00:00
Akizon77 8559083d66 feat: 未绑定第三方账户时不能关闭密码登录 2025-09-06 04:02:22 +00:00
github-actions[bot] 96ef3c9fd5 Merge pull request #239 from komari-monitor/dev
feat: 添加JSON-RPC2接口
2025-09-06 03:21:32 +00:00
Akizon77 3cace1fce0 feat: 添加 maxCount 参数以限制记录数量并实现分组下采样 2025-09-06 03:16:18 +00:00
Akizon77 f2adfcc052 feat: 添加获取记录的 RPC 方法及相关逻辑 2025-09-06 03:02:11 +00:00
Akizon77 b31bfa8ba9 feat: 添加获取版本和节点最近状态的方法 2025-09-06 02:44:27 +00:00
Akizon77 b098509df4 feat: 添加 getMe 方法 2025-09-02 14:12:00 +00:00
Akizon77 2df23b6ddb feat: 更新 getNodesLatestStatus 支持通过多个 UUID 获取节点最新状态 2025-08-31 20:44:32 +08:00
Akizon77 6dbfdf7f89 feat: 添加获取节点最新状态的 RPC 方法及相关逻辑 2025-08-31 20:37:46 +08:00
Akizon77 5d8b0d9c91 feat: rpc 添加公开信息方法 2025-08-31 13:15:39 +08:00
Akizon77 fa00885d8b feat: 增强 JSON-RPC 支持,添加批量请求处理及权限管理 2025-08-31 12:47:36 +08:00
Akizon77 f3b52b3282 feat: 添加 JSON-RPC2 支持及相关方法注册 2025-08-30 18:43:36 +08:00
svnmoe 10ddde6592 feat:update /api/records/ping API (#232)
* feat:update /api/records/ping API
Request method:GET
parameters:
  uuid | string | client uuid
  task_id | int | ping_task_id
  hours   | int | ping_record_hours
*At least one of uuid or task_id must be provided
examples:
  GET /api/records/ping?uuid= &hours=
  GET /api/records/ping?task_id= &hours=
  GET /api/records/ping?uuid= &task_id= &hours=\

* fix:remove target in /api/records/ping

* feat:add load_type parameters for api/records
load_type parameters can be:
cpu
gpu
ram
swap
load
temp
disk
network
process
connections
all (or not passing a parameter)
example:
GET /api/records/load?uuid= &load_type=cpu&hours=
GET /api/records/load?uuid= &hours=4
2025-08-30 13:05:12 +08:00
github-actions[bot] 9dff938db3 Merge pull request #229 from komari-monitor/dev
refactor: 简化 CFACCESS OAuth 逻辑
2025-08-28 03:18:56 +00:00
Akizon77 44ae7b218f refactor: 简化 CFACCESS OAuth 逻辑 2025-08-28 11:14:30 +08:00
Akizon77 f6fa61fe56 Merge branch 'main' of https://github.com/komari-monitor/komari 2025-08-28 10:39:48 +08:00
xxddff 207c3fa75d Merge pull request #228 from xxddff/cloudflare-access-simply
feat: 添加对 Cloudflare Access 的支持
2025-08-28 10:36:23 +08:00
MoGuQAQ 027c6c8c14 Merge pull request #221 from mogumc/dev
fix: some feature
2025-08-25 12:40:24 +08:00
Akizon 709fafd64a Merge pull request #219 from svnmoe/dev
feat:add GET /api/records/ping?task_id= &hours=
2025-08-24 14:13:17 +08:00
svnmoe 772e43d5b4 feat:add GET /api/records/ping?task_id= &hours= 2025-08-24 14:10:09 +08:00
Akizon 2c0f01be1a Merge pull request #218 from svnmoe/dev
fix:remove target in /api/task/ping
2025-08-24 12:50:55 +08:00
svnmoe f7bec7189c fix:remove target in /api/task/ping 2025-08-24 12:49:25 +08:00
Akizon 1730f849ed Merge pull request #217 from svnmoe/dev
feat:add /api/task/ping to get all PingTask info
2025-08-24 12:43:10 +08:00
svnmoe 1d46194f99 feat:add /api/task/ping to get all PingTask info 2025-08-24 12:40:13 +08:00
Akizon 8d652049af Merge pull request #210 from komari-monitor/copilot/fix-b576a6c5-0ea2-4f16-8483-2eb7767cd8ad
Add PR and manual triggers to binary build workflow
2025-08-22 16:42:44 +08:00
copilot-swe-agent[bot]andAkizon77 7af2a956d2 Add PR and manual triggers to binary build workflow
Co-authored-by: Akizon77 <64686663+Akizon77@users.noreply.github.com>
2025-08-20 14:31:50 +00:00
copilot-swe-agent[bot] 45b23a04c9 Initial plan 2025-08-20 14:26:30 +00:00
github-actions[bot] 1a7c5990ae Merge pull request #209 from komari-monitor/dev
feat: #159 流量提醒
2025-08-20 14:23:53 +00:00
Akizon77 9133dac3d0 feat: #159 流量提醒 2025-08-20 22:09:57 +08:00
Akizon77 2709166a5b feat: 添加流量监控功能,每分钟检查并提醒流量使用情况 2025-08-20 22:01:47 +08:00
github-actions[bot] fcb5284d57 Merge pull request #208 from komari-monitor/dev
feat: 添加登录通知
2025-08-20 11:08:27 +00:00
Akizon77 92d511253f fix: 调整登录通知消息格式 2025-08-20 18:53:55 +08:00
Akizon77 9874c4d339 feat: 添加登录通知 2025-08-20 17:34:28 +08:00
MoGuQAQ 86e97c66b9 Merge pull request #206 from mogumc/main
feat: ws支持请求特定服务器
2025-08-20 16:47:34 +08:00
Akizon f2afed643e ci: Disable auto run 2025-08-18 16:44:06 +08:00
Akizon f502e3df76 Update cleanup-packages.yml 2025-08-18 16:43:10 +08:00
Akizon 7491da96e6 Merge pull request #204 from guodonn01/main
Feat: ws支持快速重连,fix: 精确时间段下发ping task
2025-08-17 20:38:41 +08:00
root 0b887689f0 Feat: ws支持快速重连,fix: 精确时间段下发ping task 2025-08-17 19:59:36 +08:00
Akizon 9cce96c80c Merge pull request #200 from guodonn01/main
Fix: 下发ping task 应该时间更精确得以interval为间隔
2025-08-16 14:49:11 +08:00
gordon 2711d27c03 Fix: 下发ping task 应该时间更精确得以interval为间隔 2025-08-16 13:27:16 +08:00
github-actions[bot] db3f31aba5 Merge pull request #198 from komari-monitor/dev
fix: 确保主题声明回传有值
2025-08-15 15:20:43 +00:00
Akizon77 29a7e3607f fix: 确保主题声明回传有值 2025-08-15 13:46:37 +00:00
Akizon d7f6c2920d Merge pull request #196 from mogumc/main
feat: Agent无ip返回时使用ClientIP
2025-08-14 18:56:09 +08:00
mogumc 6bff7e86f6 feat: Agent无ip返回时使用ClientIP 2025-08-14 18:23:56 +08:00
Akizon a7a78a316e Merge pull request #192 from mogumc/main
fix: 安全数据校验
2025-08-14 15:31:45 +08:00
github-actions[bot] 17ff74854a Merge pull request #195 from komari-monitor/dev
feat: 更新WebSocket连接的跨域检查逻辑
2025-08-14 07:26:38 +00:00
Akizon77 a101601cf4 feat: 更新WebSocket连接的跨域检查逻辑 2025-08-14 07:25:14 +00:00
MoGuQAQ 1904192f55 Merge branch 'komari-monitor:main' into main 2025-08-14 00:57:51 +08:00
github-actions[bot] 7fd49ab55c Merge pull request #193 from komari-monitor/dev
feat: 新增主题配置项、流量限制
2025-08-13 15:10:30 +00:00
Akizon77 3d7138ea30 feat: 管理主题配置
fix: recent鉴权
2025-08-13 15:08:33 +00:00
mogumc 835f8f17ec chore: 删除调试信息 2025-08-13 16:57:31 +08:00
mogumc 488c1afcab fix: 安全数据校验 2025-08-13 16:51:18 +08:00
mogumc 1fd59efe6c fix: 修复未登录可以通过uuid获取流量与ping记录 2025-08-13 15:47:20 +08:00
Akizon 9cf0366aca Merge pull request #191 from mogumc/main
fix: 修复未登录状态可以通过uuid获取隐藏信息
2025-08-13 13:43:01 +08:00
mogumc 15d7480547 fix: 修复未登录状态可以通过uuid获取隐藏信息 2025-08-13 13:37:00 +08:00
Akizon77 819c55d691 feat: 新增流量限制类型字段 2025-08-12 15:02:27 +00:00
Akizon77 67b4c5ebd8 feat: 新增流量限制字段 2025-08-12 13:31:38 +00:00
github-actions[bot] 4bd9fda728 Merge pull request #190 from komari-monitor/dev
feat: 支持隐藏特定服务器
2025-08-12 09:11:10 +00:00
Akizon77 33b8308b3f chore: 更正注释 2025-08-12 09:09:48 +00:00
Akizon77 57cf9c243f feat: 支持隐藏特定服务器
chore: 迁移ws的API接口到api包下
2025-08-12 08:59:01 +00:00
github-actions[bot] f497a1e48b Merge pull request #189 from komari-monitor/dev
fix: 修正createGETRequest方法中的URL替换逻辑
2025-08-12 04:46:17 +00:00
Akizon77 f23fa95a56 fix: 修正createGETRequest方法中的URL替换逻辑 2025-08-12 04:45:20 +00:00
github-actions[bot] 8e8033bb35 Merge pull request #187 from komari-monitor/dev
feat: 支持通过环境变量显式关闭Origin校验
2025-08-10 08:50:53 +00:00
Akizon77 71b0903b0e feat: 支持通过环境变量显式关闭Origin校验 2025-08-10 08:49:23 +00:00
Gil Schneider d0eefe2dd2 feat: 新增聚合登录(QQ)功能 (#185)
* feat:新增单点登录提供商“聚合登录”;
feat:新增集合登录的登录参数设置;

* feat:移除nsmao接口支持。

* 移除调试环境生成的二进制文件
2025-08-10 15:04:49 +08:00
github-actions[bot] 2c547bab68 Merge pull request #186 from komari-monitor/dev
ci: 禁用ghcr缓存
2025-08-10 07:03:45 +00:00
Akizon77 2dc04b5ecf ci: 禁用ghcr缓存 2025-08-10 07:02:48 +00:00
github-actions[bot] e2dae1ec7b Merge pull request #182 from komari-monitor/dev
fix: 修复2FA验证逻辑
2025-08-09 08:36:41 +00:00
Akizon77 59d923e168 fix: 修复2FA验证逻辑 2025-08-09 08:35:46 +00:00
github-actions[bot] 51844c27f1 Merge pull request #181 from komari-monitor/dev
feat: 添加CheckOrigin函数并更新WebSocket连接的源检查逻辑
2025-08-09 07:31:50 +00:00
Akizon77 53171affca feat: 添加CheckOrigin函数并更新WebSocket连接的源检查逻辑 2025-08-09 07:30:44 +00:00
Akizon d31d12e59f Merge commit from fork
fix: WS跨域检查
2025-08-09 14:56:45 +08:00
Akizon77 3715f2b3f9 fix: WS跨域检查 2025-08-09 06:52:39 +00:00
github-actions[bot] cc3d54bff4 Merge pull request #180 from komari-monitor/dev
fix: 修复2FA验证逻辑中的错误处理
2025-08-09 06:40:56 +00:00
Akizon77 ef376a88b8 fix: 修复2FA验证逻辑中的错误处理 2025-08-09 06:39:48 +00:00
github-actions[bot] e639238f0e Merge pull request #179 from komari-monitor/dev
feat: 在删除客户端时同时删除连接的客户端和最新报告
2025-08-09 05:54:31 +00:00
Akizon77 cf8241a77a feat: 在删除客户端时同时删除连接的客户端和最新报告 2025-08-09 05:53:09 +00:00
Akizon77 eb87a4fc33 feat: 计算任务的丢包率 2025-08-09 04:21:34 +00:00
github-actions[bot] 0727fb4343 Merge pull request #178 from komari-monitor/dev
feat: 即使没有ping数据,也返回任务信息
2025-08-09 04:13:08 +00:00
Akizon77 d55d759fec feat: 即使没有ping数据,也返回任务信息 2025-08-09 04:12:10 +00:00
github-actions[bot] aad279ce55 Merge pull request #177 from komari-monitor/dev
feat: 增强数据库备份功能
2025-08-09 04:05:07 +00:00
Akizon77 e9b1a6b2cd feat: 增强数据库备份功能 2025-08-09 04:04:04 +00:00
github-actions[bot] bd5a6934e1 Merge pull request #174 from komari-monitor/dev
feat: 添加自动发现
2025-08-07 05:42:15 +00:00
Akizon77 2f04008453 feat: 添加自动发现 2025-08-07 05:15:21 +00:00
Akizon77 fbaa410fd4 fix: 修改任务结果处理中的时间类型 2025-08-06 16:05:08 +00:00
Akizon77 1d4f6f2814 fix: 移除无用的日志记录以简化API密钥验证 2025-08-06 16:02:53 +00:00
github-actions[bot] ad2f5160fb Merge pull request #173 from komari-monitor/dev
feat: 添加API密钥认证
2025-08-06 15:07:02 +00:00
Akizon77 d284405310 忽略debug文件 2025-08-06 15:05:41 +00:00
Akizon77 ff172ccbf4 feat: 添加API密钥认证 2025-08-06 15:05:27 +00:00
github-actions[bot] 14635948b6 Merge pull request #172 from komari-monitor/dev
fix: GET请求添加占位符
2025-08-06 07:47:48 +00:00
Akizon77 012fd1dcc2 fix: GET请求添加占位符 2025-08-06 07:46:52 +00:00
Akizon77 94fa5862b1 fix: 更新自动合并工作流,移除不必要的注释和步骤 2025-08-06 07:43:30 +00:00
Akizon77 e7ec6c71f7 Merge branch 'main' into dev 2025-08-06 07:43:11 +00:00
Akizon 59b187b2a6 fix: 移除主题更新中的下载链接保存逻辑 (#171) 2025-08-06 07:15:52 +00:00
Akizon77 5e7cb0f9f1 fix: 移除主题更新中的下载链接保存逻辑 2025-08-06 07:13:29 +00:00
Akizon77 dfe51b0dc5 Merge branch 'main' into dev 2025-08-06 07:05:37 +00:00
Akizon 7a4dd4bf66 fix: 安装时多一个嵌套的/data (#170)
* fix: 安装时多一个嵌套的/data

* ci: 添加自动合并

* ci: 申请auto-merge权限
2025-08-06 06:36:49 +00:00
Akizon77 0b34beea59 ci: 申请auto-merge权限 2025-08-06 06:35:50 +00:00
Akizon77 90ee9bc731 ci: 添加自动合并 2025-08-06 06:25:39 +00:00
Akizon77 046f38e7c4 fix: 安装时多一个嵌套的/data 2025-08-06 06:07:23 +00:00
Akizon eb733cea45 Merge pull request #168 from dbedu/main
feat: support theme online update.
2025-08-06 12:39:09 +08:00
Akizon eb68f0c629 Merge pull request #167 from komari-monitor/dev
feat: #162 添加Webhook消息发送功能
2025-08-05 15:36:41 +08:00
Akizon77 1cbcac4a00 fix: 修正拼写错误 2025-08-05 07:35:28 +00:00
Akizon77 2d6d58edab feat: #162 添加Webhook消息发送功能 2025-08-05 07:34:05 +00:00
Akizon77 c01fadbb48 feat: 添加option选项 2025-08-05 07:26:11 +00:00
Akizon77 41ef2e239d fix: 更新任务结果保存逻辑以使用模型中的时间类型 2025-08-05 06:44:24 +00:00
Akizon77 7176180343 fix: 修复任务创建时未包含离线客户端 2025-08-05 06:32:45 +00:00
Akizon77 0b9a01c8ac fix: 更新Exec函数以处理离线客户端并保存任务结果 2025-08-05 05:56:00 +00:00
Akizon 50ef62a64a Merge pull request #163 from imlonghao/fix/oidc
fix(oidc): 修正请求 token 阶段的请求包
2025-08-05 13:17:30 +08:00
Akizon 384fa4bbe1 Merge pull request #165 from JohnsonRan/bark
feat: add Bark notification provider
2025-08-04 22:36:42 +08:00
JohnsonRan 10f84a8052 feat: add Bark notification provider 2025-08-04 22:09:49 +08:00
imlonghao 53bd83af8b fix(oidc): 修正请求 token 阶段的请求包 2025-08-04 21:22:20 +08:00
dbedu 53577e21e7 feat: support update theme from github release 2025-08-04 21:01:27 +08:00
dbedu 46c0cb9bfb feat: support update theme from github release 2025-08-04 20:48:10 +08:00
dbedu ac8ee8f9fa feat: add theme online update 2025-08-04 19:56:43 +08:00
Akizon 787e8bdd80 Merge pull request #160 from imlonghao/fix/oidc-redirect-uri
fix(oidc): 增加 redirect_uri
2025-08-03 21:48:47 +08:00
Akizon 83a39e3143 Merge pull request #161 from imlonghao/fix/oidc-reload-config
fix(oidc): 修复配置无法更新的问题
2025-08-03 21:43:21 +08:00
imlonghao 5e2ed79929 fix(oidc): 修复配置无法更新的问题 2025-08-03 21:40:54 +08:00
imlonghao c8f0ba926d fix(oidc): 增加 redirect_uri 2025-08-03 21:22:37 +08:00
Akizon77 17e0e14aef fix: 初次启动时无法加载oauth 2025-08-03 09:54:07 +00:00
Akizon77 731602a80d Merge branch 'main' of https://github.com/komari-monitor/komari 2025-08-03 09:42:55 +00:00
Akizon77 37f0e449de fix: 优化OIDC提供者加载逻辑,添加空配置处理 2025-08-03 09:42:52 +00:00
Akizon 56dd512d3c Merge pull request #157 from imlonghao/feat/5years
feat: 增加五年期的账单周期
2025-08-02 23:40:31 +08:00
Akizon 16f11b052a Merge pull request #158 from JohnsonRan/kernel-version
feat: add kernel_version field to Client model
2025-08-02 23:39:56 +08:00
JohnsonRan 06bae7efcb feat: add kernel_version field to Client model 2025-08-02 22:52:17 +08:00
imlonghao 39b017620f feat: 增加五年期的账单周期 2025-08-02 22:21:10 +08:00
Akizon77 45e8a3a530 Merge branch 'main' of https://github.com/komari-monitor/komari 2025-08-01 14:28:27 +00:00
Akizon77 8dd64a9fa3 fix: 更新OAuth回调中文案消息 2025-08-01 14:04:19 +00:00
Akizon c1e8b0779f Merge pull request #155 from komari-monitor/dev
feat: #149 支持超级群组话题
2025-08-01 21:53:47 +08:00
Akizon77 d16daf1862 feat: #149 支持超级群组话题 2025-08-01 13:45:06 +00:00
Akizon77 491b092ac1 feat: 添加Help字段到Item结构 2025-08-01 13:17:26 +00:00
Akizon77 e2304c6b1b refactor: 重构messageSender接口 2025-08-01 09:05:43 +00:00
Akizon77 ea52f61a2c Merge branch 'dev' 2025-08-01 06:53:12 +00:00
Akizon77 3db6edcc3d fix: 保存第三方登陆后会自动重载 2025-08-01 06:52:39 +00:00
Akizon77 2045dd3f30 Merge branch 'main' of https://github.com/komari-monitor/komari 2025-08-01 06:33:01 +00:00
Akizon77 7a37c93a02 fix: 修复私有站点中对非/api路径的放行逻辑 2025-08-01 06:32:26 +00:00
Akizon 1d49300f8c Merge pull request #154 from komari-monitor/dev
fix: OAuth校验state失效、打开私有化后客户点连接401的问题
2025-08-01 14:24:10 +08:00
Akizon77 f5641df0d9 fix: OAuth校验state失效 2025-08-01 06:22:10 +00:00
Akizon77 b9c1e58703 fix: /api/clients 绕过 2025-08-01 06:18:18 +00:00
Akizon77 22b9fada55 fix: 打开私有化后客户点连接401的问题 2025-07-31 16:07:00 +00:00
Akizon d34b286c4f Merge pull request #152 from komari-monitor/dev
refactor: 重构OAuth
2025-07-31 16:42:02 +08:00
Akizon77 6dab3c8450 refactor: 重写oauth逻辑 2025-07-31 08:23:34 +00:00
Akizon77 32dd5c9414 feat: OIDC接口 2025-07-31 07:24:00 +00:00
Akizon77 2f09d28b94 feat: 添加OIDC接口 2025-07-31 06:08:09 +00:00
Akizon77 77ce900bbf wip: 添加通用OIDC提供者实现及相关注册逻辑 2025-07-31 04:51:27 +00:00
Akizon77 0675634b27 wip: OIDC接口 2025-07-30 09:45:14 +00:00
Akizon77 74db186ab3 feat: 更新备份恢复成功消息并添加服务重启逻辑 2025-07-30 07:10:56 +00:00
Akizon 1a5bc31b9b Merge pull request #151 from komari-monitor/dev
feat: #143 添加全站私有化
2025-07-30 13:37:54 +08:00
Akizon77 f6a94db702 Merge branch 'main' into dev 2025-07-29 08:11:39 +00:00
Akizon77 6213ce7556 feat: #143 添加全站私有化 2025-07-29 08:11:27 +00:00
Akizon77 ff91e548b8 docs: 添加英语、日文和繁体中文的 README 文件 2025-07-29 07:28:26 +00:00
Akizon77 6d59070184 Merge branch 'main' into dev 2025-07-28 08:21:22 +00:00
Akizon77 c8126e551f refactor: 将 UTCTime 类型替换为 LocalTime 2025-07-28 08:20:42 +00:00
Akizon77 589eae0d3f ci: 更新清理旧包的工作流配置 2025-07-28 04:20:26 +00:00
Akizon77 8e25bc9ce3 ci: 删除没有标签的版本 2025-07-28 04:15:24 +00:00
Akizon77 9aa4dcc206 ci: 更新 Docker 发布工作流 2025-07-28 04:12:23 +00:00
Akizon77 b2b5966699 fix: 修改 Ping 记录时间格式为 RFC3339 2025-07-26 07:20:34 +00:00
Akizon77 e9c7e0031c fix: #146 优化 Ping 记录的输出结构 2025-07-26 07:19:31 +00:00
Akizon77 1e3d29ffdb Merge branch 'main' of https://github.com/komari-monitor/komari 2025-07-26 07:10:15 +00:00
Akizon77 84670ac231 feat: 恢复备份添加锁、添加备份标记文件并在上传时验证 2025-07-26 06:56:02 +00:00
Akizon 6b17c5c498 Update README.md 2025-07-26 14:11:21 +08:00
Akizon eed9820ce6 Merge pull request #148 from dbedu/main
feat: 备份文件恢复
2025-07-26 10:06:35 +08:00
dbedu 858d914ae3 新功能: 新增备份文件恢复 2025-07-26 00:46:07 +08:00
Akizon be7c3812db docs: 更新文档链接 2025-07-25 01:00:17 +08:00
Akizon 44c65f1eb8 docs: 容器云标注价格 2025-07-25 00:36:50 +08:00
Akizon dd742c1038 docs: 添加雨云一键部署 2025-07-25 00:16:53 +08:00
Akizon d19b64ec1d Merge pull request #144 from komari-monitor/dev
feat: 优化静态资源服务
2025-07-24 16:25:33 +08:00
Akizon77 6ea22e3241 feat: 优化静态资源服务 2025-07-24 08:10:09 +00:00
Akizon77 ea99966e7a feat: 配置获取添加并发控制 2025-07-24 08:09:57 +00:00
Akizon77 fa13d4d1f9 feat: 自定义端口 2025-07-23 14:31:45 +00:00
Akizon77 15e4f583e0 chore: 删除待办清单文件 2025-07-23 14:25:40 +00:00
Akizon77 3fbbc7bdce docs: 添加前端构建说明文档 2025-07-23 14:23:54 +00:00
Akizon77 5fcd1d313c fix: #138 Docker容器添加时区数据 2025-07-23 04:00:24 +00:00
Akizon 967b7ab8a7 docs: 更新文档链接 2025-07-22 20:52:24 +08:00
Akizon77 d113dde3cc docs: 添加容器云一键部署 2025-07-21 07:52:12 +00:00
Akizon77 60c24a5ee1 docs: 添加一键脚本的指南 2025-07-21 06:48:51 +00:00
Akizon77 d5b8982763 docs: 添加Telegram群组链接 2025-07-21 06:40:45 +00:00
Akizon77 88a441da23 feat: 添加manifest.json路由 2025-07-20 08:28:15 +00:00
akizon77 03f27e7eb1 docs: 更新鸣谢 2025-07-19 01:01:56 +08:00
Akizon 6bba4526f0 Merge pull request #129 from komari-monitor/dev
feat: 定时自动续费
2025-07-16 11:26:14 +08:00
Akizon77 ebb84862a2 feat: 定时自动续费 2025-07-16 11:16:05 +08:00
Akizon77 28480a93d4 feat: 自动续费 2025-07-16 11:07:27 +08:00
Akizon 5c7c5a4e99 Merge pull request #128 from komari-monitor/main
Sync to dev branch
2025-07-16 10:21:57 +08:00
Akizon 90ee1da9c7 Merge pull request #127 from ours1505/dev
feat: 自动续费
2025-07-16 10:18:56 +08:00
Nya 596dbd0c2d feat: 自动续费 2025-07-16 10:02:26 +08:00
Akizon de5db97505 Merge pull request #124 from komari-monitor/dev
docs: 更新前端开发指南
2025-07-15 16:33:32 +08:00
Akizon77 0db40294ba docs: 更新前端开发指南 2025-07-15 16:32:17 +08:00
Akizon 74cd074c98 Merge pull request #123 from komari-monitor/dev
feat: 终端页面不使用主题
2025-07-15 16:05:01 +08:00
Akizon77 bb43b25e37 fix: 主题short不可以为default 2025-07-15 14:44:14 +08:00
Akizon77 b28431e4bd feat: 终端页面不使用主题 2025-07-15 11:37:42 +08:00
Akizon77 ee999810af feat: 在删除客户端时同步删除ws报告 2025-07-15 11:32:36 +08:00
Akizon 860f498e21 Merge pull request #122 from komari-monitor/dev
docs: 更新readme
2025-07-14 23:55:07 +08:00
Akizon77 798e900a9a docs: 更新readme 2025-07-14 23:54:38 +08:00
Akizon 9617cf575f Merge pull request #121 from komari-monitor/dev
feat: 添加主题管理
2025-07-14 23:48:29 +08:00
Akizon77 0ebba0f591 feat: 添加主题管理功能,包括上传、列出、删除和设置主题 2025-07-14 23:23:47 +08:00
Akizon77 01fd429ec3 refactor: 提取RunServer 2025-07-14 19:18:32 +08:00
Akizon 701aacf45c Merge pull request #119 from komari-monitor/dev
feat: #107 支持自定义Tags
2025-07-14 19:13:13 +08:00
Akizon77 2120168d1f feat: #107 支持自定义Tags 2025-07-14 18:53:19 +08:00
Akizon 171e4ceece Merge pull request #117 from komari-monitor/dev
refactor: 替换logOperation为auditlog、将notification模块重构为notifier
2025-07-13 15:47:09 +08:00
Akizon77 1d4c394ad8 refactor: 将notification模块重构为notifier 2025-07-13 15:39:21 +08:00
Akizon77 837313e146 refactor: 替换logOperation为auditlog 2025-07-13 15:36:56 +08:00
Akizon 3e415da8d4 Merge pull request #114 from komari-monitor/dev
feat: admin页面不使用自定义样式
2025-07-12 18:42:05 +08:00
Akizon77 a125466b75 feat: admin页面不使用自定义样式 2025-07-12 18:39:04 +08:00
Akizon a9306cd651 Merge pull request #110 from komari-monitor/dev
feat: 添加客户端分组字段
2025-07-09 14:47:21 +08:00
Akizon77 6af72f41c0 feat: 添加客户端分组字段 2025-07-09 14:19:56 +08:00
Akizon77 e9e04593c7 feat: 负载通知 2025-07-08 23:31:20 +08:00
Akizon 43b39c9f68 Merge pull request #109 from komari-monitor/dev
refactor: 重建数据库关系
2025-07-08 16:58:28 +08:00
Akizon77 8d5ce55301 refactor: 离线通知不回传ClientInfo 2025-07-08 16:57:23 +08:00
Akizon77 e72d461869 refactor: 重建数据库关系 2025-07-08 16:52:21 +08:00
Akizon 5cc4b518ba Merge pull request #106 from komari-monitor/dev
feat: 管理脚本
2025-07-06 00:49:32 +08:00
Akizon77 8aed055cc4 feat: 精简安装脚本 2025-07-06 00:48:44 +08:00
Akizon77 a306b6911a feat: 显示默认登录信息 2025-07-05 23:29:59 +08:00
Akizon77 776dfee23b feat: 移除日志配置前缀颜色 2025-07-05 23:26:44 +08:00
Akizon77 02e01a1d7c feat: 修正二进制文件命名格式 2025-07-05 23:22:26 +08:00
Akizon77 6b0b5b4855 feat: 添加安装脚本 2025-07-05 23:17:46 +08:00
Akizon 9dd09731bd Merge pull request #100 from komari-monitor/dev
feat: 补全相关日志记录
2025-07-04 00:47:00 +08:00
Akizon77 963c829164 feat: 添加重试机制以增强消息发送的可靠性 2025-07-04 00:45:43 +08:00
Akizon77 d01fafbf0f feat: 更新终端相关日志记录 2025-07-04 00:42:50 +08:00
Akizon77 729db05346 feat: 增强cloudflared运行时日志记录,添加错误处理和成功退出日志 2025-07-04 00:35:25 +08:00
Akizon 70dc06a761 Merge pull request #90 from komari-monitor/dev
docs: 更新 Agent 开发指南
2025-07-01 22:40:38 +08:00
Akizon77 be059b7aa3 feat: 更新 Agent 开发指南 2025-07-01 22:39:36 +08:00
Akizon b80c917f25 Merge pull request #89 from komari-monitor/dev
fix: 数据库可能两次打包在压缩包内
2025-07-01 17:09:42 +08:00
Akizon77 6b74bf90ce fix: 数据库可能两次打包在压缩包内 2025-07-01 17:09:11 +08:00
Akizon a2e4665ff5 Merge pull request #88 from komari-monitor/dev
feat: 优化报告平均值计算
2025-07-01 16:57:04 +08:00
Akizon77 2c9bd9c847 feat: 修改AverageReport函数以支持topPercentage参数,优化报告平均值计算 2025-07-01 16:56:31 +08:00
Akizon77 54eb957618 feat: 为GeoIP服务提供者添加名称方法 2025-07-01 16:44:56 +08:00
Akizon 370f0ff815 Merge pull request #87 from komari-monitor/dev
feat: #68 集成cloudflared
2025-07-01 15:45:15 +08:00
Akizon77 9bd9a280e5 feat: #68 集成cloudflared 2025-07-01 15:34:00 +08:00
Akizon daf6ae291f Merge pull request #86 from komari-monitor/dev
feat: 添加IP查询缓存机制
2025-07-01 15:03:06 +08:00
Akizon77 3d774eec81 feat: 添加IP查询缓存机制 2025-07-01 14:58:56 +08:00
Akizon77 129e0be151 refactor: 简化GeoIP相关调用,可以插中间件了 2025-07-01 14:51:17 +08:00
Akizon 862657ea86 Merge pull request #83 from komari-monitor/dev
feat: 在获取的任务中添加间隔时间字段
2025-06-30 18:38:06 +08:00
Akizon77 70b519a345 feat: 在获取的任务中添加间隔时间字段 2025-06-30 18:09:31 +08:00
Akizon 6600e4014d Merge pull request #82 from komari-monitor/dev
feat: #53 通知方式兼容SMTP邮件、支持下载备份
2025-06-29 23:50:00 +08:00
Akizon77 b8a6f18a77 feat: 支持下载备份 2025-06-29 23:29:19 +08:00
Akizon77 03e08cdfbb fix: 只有在通知方式变动时重载MessageSenderProvider 2025-06-29 22:30:28 +08:00
Akizon77 b07f7345ee feat: 通知方式兼容SMTP邮件 2025-06-29 22:28:04 +08:00
Akizon77 e85996178a refactor: 抽象消息发送接口 2025-06-29 22:17:35 +08:00
Akizon77 5547fcd067 ci: 发布main分支的docker镜像,修正release 2025-06-29 21:54:41 +08:00
Akizon aef4344997 Merge pull request #81 from komari-monitor/dev
fix: #79 GeoIP无法更新
feat: IP查询兼容 ipinfo.io
2025-06-29 21:44:43 +08:00
Akizon77 061197cd77 ci: 发布到测试环境不显示部分日志 2025-06-29 21:40:14 +08:00
Akizon77 c00d1d0cb2 ci: 发布docker镜像时预编译二进制 2025-06-29 21:36:30 +08:00
Akizon77 b6a7511e33 fix: #79 GeoIP无法更新 2025-06-29 01:31:54 +08:00
Akizon77 13450a50eb feat: IP查询兼容 ipinfo.io 2025-06-28 21:04:37 +08:00
Akizon eb883ccc6f Merge pull request #78 from komari-monitor/dev
feat: 到期提醒显示具体还剩多少天
2025-06-28 20:20:43 +08:00
Akizon77 2ae9b3516c fix: MaxMind 数据库 initialize 方法需要在解锁后调用,以避免死锁 2025-06-28 20:15:31 +08:00
Akizon77 7dc2a0294e feat: 到期提醒显示具体还剩多少天 2025-06-28 20:02:32 +08:00
Akizon f97b91c19c Merge pull request #77 from komari-monitor/dev
feat: 添加过期通知功能,支持配置过期前通知天数
2025-06-27 23:10:54 +08:00
Akizon77 bfac98dc67 feat: 添加过期通知功能,支持配置过期前通知天数 2025-06-27 22:28:42 +08:00
Akizon77 0f0f9f5985 feat: 添加记录删除功能,支持在记录禁用时清除所有记录 2025-06-27 22:13:24 +08:00
Akizon efebe66518 Merge pull request #76 from komari-monitor/dev
feat: 添加对ip-api和geojs服务的支持
2025-06-27 21:38:10 +08:00
Akizon77 37a05e8f3d fix: 更新时间条件,调整为四小时之前的记录查询 2025-06-27 21:36:04 +08:00
Akizon77 7473c438c2 feat: 订阅配置变化更新geoip配置 2025-06-27 21:22:50 +08:00
Akizon77 6e4e6d0653 feat: 添加对ip-api和geojs服务的支持,更新GeoIP配置 2025-06-27 21:20:31 +08:00
Akizon77 c09bec5c20 refactor: 更新GeoIP服务实现 2025-06-27 21:05:59 +08:00
Akizon77 14ce2b8f53 fix: 调整高位百分比和删除旧数据的时间条件 2025-06-27 20:26:49 +08:00
Akizon 8ecb6aa7aa Merge pull request #75 from komari-monitor/dev
ci: docker发布
2025-06-27 17:36:26 +08:00
Akizon77 cedc144173 ci: docker发布 2025-06-27 17:34:26 +08:00
Akizon 434e09ae95 Merge pull request #74 from komari-monitor/dev
refactor: 将time.Time 迁移到UTCTime
2025-06-27 13:55:34 +08:00
Akizon77 06ffc44530 refactor: 将time.Time 迁移到UTCTime 2025-06-27 13:53:03 +08:00
Akizon a7c24e2323 Merge pull request #72 from komari-monitor/dev
ci: 限定Matrix
2025-06-25 23:15:26 +08:00
Akizon77 726cfa513e ci: 限定Matrix 2025-06-25 23:14:33 +08:00
Akizon 62a1317d88 Merge pull request #71 from komari-monitor/dev
ci: 使用Zig作为C编译器
2025-06-25 23:12:20 +08:00
Akizon77 1221feae45 ci: 使用Zig作为C编译器 2025-06-25 23:11:53 +08:00
Akizon 884c5686bd Merge pull request #70 from komari-monitor/dev
fix: #69  (概率)离线通知未按预期工作
2025-06-25 19:09:46 +08:00
Akizon77 9b82a19528 fix: #69 (概率)离线通知未按预期工作 2025-06-25 19:09:04 +08:00
Akizon 2db4dfb942 Merge pull request #67 from komari-monitor/dev
fix: 修复了每次建立连接都会通知客户端上线的问题
2025-06-24 14:52:07 +08:00
Akizon77 09df09c29a fix: 修复了每次建立连接都会通知客户端上线的问题 2025-06-24 14:51:27 +08:00
Akizon77 215fdc822c feat: 使用go-cache替代安全映射
ci: main自动发布附件
2025-06-24 14:36:30 +08:00
Akizon 1572b92751 Merge pull request #65 from komari-monitor/dev
fix: #57 api.Records引入线程安全的泛型映射
2025-06-23 23:39:40 +08:00
Akizon77 8da28d6b95 fix: #57 api.Records引入线程安全的泛型映射 2025-06-23 23:09:21 +08:00
Akizon 6f24268683 Merge pull request #63 from komari-monitor/dev
feat: 不再自动清除ping记录,重命名Telegram测试为发送消息
2025-06-22 23:17:46 +08:00
Akizon77 abe9de2bd1 feat: 在获取Ping记录特定任务过滤 2025-06-22 22:53:13 +08:00
Akizon77 342aacd060 fix: 只有订阅了ping task的节点能获取相应的task信息 2025-06-22 22:06:18 +08:00
Akizon77 a95d621bdb feat: 不再自动清除ping记录,重命名Telegram测试为发送消息 2025-06-22 21:53:19 +08:00
Akizon77 68a8f01b32 fix: 修正获取记录时的时间范围和分组逻辑 2025-06-22 19:33:43 +08:00
Akizon ca07f1b904 Merge pull request #61 from komari-monitor/dev
feat: 增加Ping任务管理功能
2025-06-22 15:05:43 +08:00
Akizon77 1aa76be031 feat: 在获取Ping记录时添加任务列表响应 2025-06-22 15:04:22 +08:00
Akizon77 38be68455c feat: 更新获取Ping记录的API路由 2025-06-22 14:56:58 +08:00
Akizon77 2f3cfd5387 fix: 修正获取4小时内记录的时间范围,增加10分钟偏移 2025-06-22 14:54:57 +08:00
Akizon77 d3a99c0ff4 feat: 获取长期记录时,4小时内的数据按15分钟分组获取最近记录 2025-06-22 14:50:08 +08:00
Akizon77 b4d5a40ca4 feat: 更新AddPingTask和EditPingTask函数以支持名称字段 2025-06-21 21:43:23 +08:00
Akizon77 356e71ad22 feat: 添加PingTask的名称字段 2025-06-21 19:27:19 +08:00
Akizon77 d6b58f6eed feat: 增加Ping任务管理功能 2025-06-21 16:47:39 +08:00
Akizon77 785f448baf wip: ping task
fix: Cannot add a column with non-constant default
2025-06-20 22:40:03 +08:00
Akizon77 194987f66a fix: 移除冷却时间检查 2025-06-20 22:28:30 +08:00
Akizon77 bbff63bd7d fix: #57 自动创建data文件夹 2025-06-20 22:19:33 +08:00
Akizon f654b71b31 Merge pull request #58 from komari-monitor/dev
feat: #57 增加connectedClients、latestReport线程安全
2025-06-20 22:07:15 +08:00
Akizon77 b85d1d1a48 feat: #57 增加connectedClients、latestReport线程安全 2025-06-20 21:57:30 +08:00
Akizon d669d2862e Update README.md 2025-06-19 21:48:33 +08:00
Akizon e6e71716c8 Merge pull request #55 from komari-monitor/dev
feat: 可配置的历史记录保存时长
2025-06-19 15:08:27 +08:00
Akizon77 5022148fd9 feat: 可配置的历史记录保存时长
wip: PING 检测
2025-06-19 15:07:17 +08:00
Akizon77 f069edc026 Merge branch 'dev' of https://github.com/komari-monitor/komari into dev 2025-06-19 14:31:06 +08:00
Akizon77 65cf7b30c3 wip: Ping、TCPing、HTTPing 2025-06-19 14:31:03 +08:00
Akizon 5110b42f0a Update todo.md 2025-06-19 12:54:31 +08:00
Akizon 92475f7331 Merge pull request #54 from komari-monitor/dev
feat: 补上站点名称和描述的坑
2025-06-18 23:54:57 +08:00
Akizon77 12ad38fc78 feat: 补上站点名称和描述的坑 2025-06-18 23:54:01 +08:00
Akizon77 dbf6e24d68 fix: 修复在保存客户端报告时的空UUID处理逻辑 2025-06-18 15:14:37 +08:00
Akizon 19d42a41dd Merge pull request #52 from komari-monitor/dev
refactor: 重构压缩数据库的方法
2025-06-17 17:53:29 +08:00
Akizon77 573fbdb6d8 feat: 下放获取历史记录的权限 2025-06-17 17:49:24 +08:00
Akizon77 52a7024e82 feat: 获取历史负载记录 2025-06-17 17:38:42 +08:00
Akizon77 600ae2e36f refactor: 重构压缩数据库的方法 2025-06-17 17:28:58 +08:00
Akizon d7b744a01a Merge pull request #50 from komari-monitor/main
同步
2025-06-16 00:34:57 +08:00
Akizon a3d4f8ca6a Merge pull request #49 from komari-monitor/dev
feat: 在启动服务器时根据版本哈希设置Gin模式
2025-06-16 00:34:17 +08:00
Akizon77 18ad4fe9de feat: 在启动服务器时根据版本哈希设置Gin模式 2025-06-16 00:33:38 +08:00
Akizon77 efb4cc6a4d ci/cd: 添加 GIN_MODE 环境变量以支持构建 2025-06-16 00:20:16 +08:00
Akizon77 09dd2ad329 ci/cd: 排除 Windows 平台的 arm64 架构 2025-06-16 00:03:38 +08:00
Akizon77 5f079c46d6 ci/cd: 更新交叉编译配置以支持 arm64 和 amd64 2025-06-15 23:59:53 +08:00
Akizon77 ec899e3ee4 ci/cd: arm64 2025-06-15 23:10:48 +08:00
Akizon f7813e4b1b Merge pull request #48 from komari-monitor/dev
ci/cd: 交叉编译
2025-06-15 22:58:47 +08:00
Akizon f5d00c0a2a Merge branch 'main' into dev 2025-06-15 22:58:41 +08:00
Akizon77 9b9348b341 ci/cd: 交叉编译 2025-06-15 22:56:51 +08:00
Akizon b5266cda66 Merge pull request #47 from komari-monitor/dev
feat: 添加离线通知功能,每个服务器可以单独设置
feat: 添加强制禁用2FA功能
2025-06-15 22:29:49 +08:00
Akizon77 d6d716c5ab fix: 修正通知开关字段的JSON标签拼写错误 2025-06-15 18:36:43 +08:00
Akizon77 de5a611718 feat: 添加强制禁用2FA功能 2025-06-15 17:41:09 +08:00
Akizon77 d595acbde3 feat: 添加离线通知功能,每个服务器可以单独设置 2025-06-15 17:32:27 +08:00
Akizon77 638ce9ae6c wip: 泛型查询 2025-06-15 15:26:14 +08:00
Akizon77 bd2c67d725 wip: 添加负载和离线通知模型 2025-06-15 15:18:03 +08:00
Akizon f8ae5418c9 fix: #45 当前编译禁用了 CGO 2025-06-15 00:55:16 +08:00
Akizon 5e1959dec3 Merge pull request #46 from komari-monitor/dev
feat: 添加在线和离线通知功能,支持冷却和宽限期设置
2025-06-15 00:18:52 +08:00
Akizon77 b7a276dd05 feat: 添加在线和离线通知功能,支持冷却和宽限期设置 2025-06-15 00:03:25 +08:00
Akizon b2b3db616a Update README.md 2025-06-14 21:25:59 +08:00
Akizon 17e1a8828c Merge pull request #43 from komari-monitor/dev
feat: 添加命令剪切板功能
2025-06-14 17:19:08 +08:00
Akizon77 1674220340 feat: 在Clipboard模型中添加Weight字段 2025-06-14 13:01:53 +08:00
Akizon77 ab76d51b84 feat: 添加命令剪切板功能 2025-06-13 22:57:19 +08:00
Akizon77 4863e623ae wip: 命令剪切板 2025-06-13 22:14:58 +08:00
Akizon77 ac88159d56 feat: 为Client模型添加Currency,默认值为'$' 2025-06-13 21:28:50 +08:00
Akizon 433bc52788 Merge pull request #42 from komari-monitor/dev
feat: 添加2FA支持
2025-06-11 19:35:59 +08:00
Akizon77 6aeb444b2b feat: 在更新用户信息时添加用户存在性检查 2025-06-07 17:06:24 +08:00
Akizon77 bbf631714d feat: 添加双因素认证支持 2025-06-07 16:30:16 +08:00
Akizon c204094173 Update todo.md 2025-06-07 15:38:52 +08:00
Akizon 417cec5d44 Update README.md 2025-06-07 15:34:03 +08:00
Akizon 5401b4c10e Update README.md 2025-06-07 15:32:54 +08:00
Akizon f1cfbd9391 Merge pull request #41 from komari-monitor/dev
feat: 添加日志功能
2025-06-06 18:02:32 +08:00
Akizon77 ae6f9a7840 refactor: 移除冗余的版本文件 2025-06-06 18:00:56 +08:00
Akizon77 a75ac3e97d feat: session管理更加细致,识别UA、IP 2025-06-06 17:34:53 +08:00
Akizon77 00fa79e31c fix: 登陆后的重定向从/admin更改为/manage 2025-06-06 17:14:54 +08:00
Akizon77 392b05be94 feat: 更新日志获取接口,返回日志总数 2025-06-06 16:54:16 +08:00
Akizon77 26e120cc89 fix: 查询日志传入小于等于0的页数报500 2025-06-06 16:29:40 +08:00
Akizon77 413dd780f2 feat: 优化设置更新日志,忽略特定字段 2025-06-06 16:21:30 +08:00
Akizon77 3d26743492 feat: 添加服务器启动和关闭的信号处理,记录日志信息 2025-06-06 16:17:12 +08:00
Akizon77 0a6927805e feat: 添加分页功能,支持限制日志数量和选择页码 2025-06-06 16:06:29 +08:00
Akizon77 d16889c103 feat: 更新日志记录,添加客户端UUID信息 2025-06-06 16:01:19 +08:00
Akizon77 7894990d91 feat: 添加终端连接建立和断开的时间记录 2025-06-06 15:55:58 +08:00
Akizon77 fe3afceca9 feat: 终端添加日志 2025-06-06 15:54:42 +08:00
Akizon77 014ee67d69 feat:记录用户通过密码和OAuth登录的操作 2025-06-06 15:29:32 +08:00
Akizon77 ec4dee765a feat: 支持获取和删除旧日志 2025-06-06 15:05:39 +08:00
Akizon77 00d8936455 feat: 添加日志记录功能,记录用户操作及相关信息 2025-06-06 15:00:35 +08:00
Akizon77 4b97ae92da feat: 添加Telegram消息发送功能及GeoIP查询接口 2025-06-06 14:41:44 +08:00
Akizon77 a33885255c fix: 统一未授权响应处理,使用统一的错误响应函数 2025-06-05 19:01:15 +08:00
Akizon77 fd8a4c4895 feat: 统一API响应处理,简化错误和成功响应逻辑 2025-06-05 18:59:17 +08:00
Akizon77 039fd6b8d9 feat: 统一(大部分)API响应格式 2025-06-05 18:48:50 +08:00
Akizon77 09c346180e feat: 允许禁止密码登录
refactor: 大致上应该错误都返回message
2025-06-05 17:54:40 +08:00
Akizon77 179606ea6e fix: 自定义head不生效的bug 2025-06-04 23:00:28 +08:00
Akizon 2118648a58 Merge pull request #40 from komari-monitor/dev
feat: 添加上传和删除 favicon 的功能
2025-06-04 22:21:20 +08:00
Akizon77 db26327140 feat: 添加上传和删除 favicon 的功能 2025-06-04 22:12:39 +08:00
Akizon 5132b60f18 Merge pull request #39 from komari-monitor/dev
feat: 支持更新用户信息 & 手动更新geoip数据库
2025-06-04 20:48:44 +08:00
Akizon77 43cfbca62f feat: 更新用户信息 & 手动更新geoip数据库 2025-06-04 20:44:16 +08:00
Akizon77 1cc6f22a5a feat: 增加动态 CORS 支持和配置事件发布机制 2025-06-04 20:08:56 +08:00
Akizon dfc242dc0e Merge pull request #38 from komari-monitor/dev
fix: 修复 CORS
2025-06-04 19:21:08 +08:00
Akizon77 f1d17d9bcd feat: 清理过期任务结果 2025-06-04 19:17:41 +08:00
Akizon77 403879e705 fix: 修复配置中的 CORS 列名错误 2025-06-04 19:15:21 +08:00
Akizon77 cbc91cfb0c fix: 修复 CORS 2025-06-04 18:19:22 +08:00
Akizon b709ac9e98 Merge pull request #37 from komari-monitor/dev
feat: 添加批量执行任务
2025-06-04 17:46:27 +08:00
Akizon77 6d29b165ca feat: 增加任务结果的详细信息 2025-06-04 17:46:06 +08:00
Akizon77 44b1106dec feat: task 存储数组 2025-06-04 17:31:39 +08:00
Akizon77 2bb96b7272 Merge branch 'dev' of https://github.com/komari-monitor/komari into dev 2025-06-04 16:19:27 +08:00
Akizon77 2cd1159f94 feat: 添加批量执行任务 2025-06-04 16:19:22 +08:00
Akizon 5278803fb1 Merge pull request #36 from komari-monitor/dev
移除mmdb下载地址代理,增加版本号信息
2025-06-04 13:26:51 +08:00
Akizon 662a3033f7 Merge branch 'main' into dev 2025-06-04 13:26:43 +08:00
Akizon77 4c55e52374 feat: 添加版本信息API接口,支持获取当前版本和哈希 2025-06-03 23:25:50 +08:00
Akizon77 864b91fad1 ci/cd: 版本信息构建 2025-06-03 23:21:57 +08:00
Akizon77 8ed2e5a774 cd: 使用完整前端 2025-06-03 23:07:26 +08:00
Akizon77 907419ec73 cd: 工作流处理文件已存在 2025-05-30 16:53:12 +08:00
Akizon77 dc05937331 fix: 移除geoip下载地址代理 2025-05-30 16:44:21 +08:00
Akizon77 3169ec687b feat: 添加开发环境工作流,支持多平台构建和生产部署 2025-05-30 15:47:17 +08:00
Akizon 9160c1eef7 Merge pull request #35 from komari-monitor/dev
refactor: Client优化模型
2025-05-29 18:08:36 +08:00
Akizon77 7b95e270b5 fix: 修复EditClient函数中UUID的处理逻辑 2025-05-29 18:07:03 +08:00
Akizon77 95cf1d78d0 feat: 添加SSO相关信息返回 2025-05-29 17:51:04 +08:00
Akizon77 fefdcee60d refactor: 更新Client相关逻辑,迁移旧版ClientInfo数据并优化模型 2025-05-29 17:29:02 +08:00
Akizon 1f7bdf60c2 Merge pull request #34 from komari-monitor/dev
fix: 数据库charset=utf8mb4
2025-05-27 23:00:16 +08:00
Akizon77 554caf7a37 fix: 数据库charset=utf8mb4 2025-05-27 22:12:52 +08:00
Akizon 61277a8fc8 Merge pull request #33 from komari-monitor/dev
refactor: 默认服务端配置的创建
2025-05-27 17:19:29 +08:00
Akizon b178ceb62f Update docker-publish.yml 2025-05-27 17:18:28 +08:00
Akizon77 793852f7dc refactor: 优化Get函数,移除重复的默认配置创建逻辑 2025-05-27 17:17:35 +08:00
Akizon77 276feae7f6 feat: 在Update函数中添加默认配置创建逻辑 2025-05-27 17:13:43 +08:00
Akizon 3f6073cab4 Merge pull request #32 from komari-monitor/dev
fix: 修复Session表的索引冲突并优化数据库清理逻辑
2025-05-27 16:23:25 +08:00
Akizon77 a9f2c7658d fix: 修复Session表的索引冲突并优化数据库清理逻辑 2025-05-27 16:21:56 +08:00
Akizon da6eaddf3f Merge pull request #31 from komari-monitor/dev
refactor: 更新UploadBasicInfo函数以使用SaveClientInfo,移除UpdateOrInsertBasicI…
2025-05-27 15:50:07 +08:00
Akizon77 e0175e3b6c refactor: 更新UploadBasicInfo函数以使用SaveClientInfo,移除UpdateOrInsertBasicInfo的依赖 2025-05-27 15:46:43 +08:00
Akizon bd2d9117d9 Merge pull request #30 from komari-monitor/dev
feat: 添加mysql支持,新增部分公开字段
2025-05-26 20:26:29 +08:00
Akizon77 cde3a55451 feat: 添加billing_cycle 2025-05-26 19:32:17 +08:00
Akizon77 d3ad29439a feat: 添加虚拟化 2025-05-26 19:29:37 +08:00
Akizon77 fb80e48010 feat: 添加数据库配置,支持MySQL,修改uuid类型为varchar(36),移除默认时间字段 2025-05-25 18:50:33 +08:00
Akizon77 513e59dbff feat: 添加终端连接功能和会话管理 2025-05-16 22:04:44 +08:00
Akizon 7d79825345 Update README.md 2025-05-16 18:24:58 +08:00
Akizon77 735a08ad75 fix: 添加public_remark字段 2025-05-15 21:07:59 +08:00
Akizon77 874b272f55 fix: 确保发给前端的数据中有cpu usage 2025-05-15 20:59:25 +08:00
Akizon 4d86cb3632 Merge pull request #29 from komari-monitor/dev
fix: #28 修改Settings时如果不包含全部信息会导致丢失数据
fix: 修改Config结构体中的字段名称以提高一致性
feat: 添加中间件以禁用API请求的缓存
fix: 统一Record结构体字段名称以提高一致性
fix: 修复AddClient和EditClient函数中的JSON绑定和字段更新逻辑
feat: 添加获取客户端最近记录的API接口
2025-05-15 20:54:02 +08:00
Akizon77 5316a4d3b2 feat: 添加获取客户端最近记录的API接口 2025-05-15 20:49:12 +08:00
Akizon77 5d386a18e4 fix: 修复AddClient和EditClient函数中的JSON绑定和字段更新逻辑 2025-05-15 20:43:13 +08:00
Akizon77 6e65706a23 fix: 统一Record结构体字段名称以提高一致性 2025-05-15 20:33:11 +08:00
Akizon 55e48ee8f1 Update README.md 2025-05-13 00:35:33 +08:00
Akizon caf330265a Update LICENSE 2025-05-12 20:18:02 +08:00
Akizon 9e86118683 Update LICENSE 2025-05-12 20:15:44 +08:00
Akizon e5d7c2f89c Update README.md 2025-05-12 20:12:08 +08:00
Akizon77 803251fbc1 wip: 添加价格、过期时间 2025-05-12 19:33:15 +08:00
Akizon77 b6b75c9aa2 feat: 添加中间件以禁用API请求的缓存 2025-05-12 17:06:14 +08:00
Akizon77 07ff3c1cfe fix: 修改Config结构体中的字段名称以提高一致性 2025-05-12 16:57:47 +08:00
Akizon77 544aada3b9 fix: #28 修改Settings时如果不包含全部信息会导致丢失数据 2025-05-12 16:41:34 +08:00
Akizon 64683bc50a Merge pull request #27 from komari-monitor/dev
feat: #25 添加批量修改weight权重的接口
feat: #24 添加logout登出
feat: 添加OAuth功能及相关API接口
2025-05-12 16:23:11 +08:00
Akizon77 62b0dff1f4 feat: #25 添加批量修改weight权重的接口
fix: #24 logout 接口
2025-05-12 16:21:31 +08:00
Akizon77 5accc9d258 feat: #24 添加logout登出 2025-05-12 16:12:20 +08:00
Akizon77 8535b4ba7a fix: # 26add client 可以指定名称
fix: 修改日志记录方式,避免在读取index.html失败时导致程序崩溃
2025-05-12 16:09:47 +08:00
Akizon77 cc3cda12cf 改回来了www 2025-05-08 23:02:19 +08:00
Akizon77 4ecf1fccd7 npm -> pnpm 2025-05-08 23:00:54 +08:00
Akizon77 74b9b8f9c8 fix: 修复OAuth回调中cookie设置逻辑,确保状态验证后正确清除cookie 2025-05-08 20:41:42 +08:00
Akizon77 9b2a33824a fix: 修复OAuth未启用时的HTTP状态码返回 2025-05-08 20:28:53 +08:00
Akizon77 2c3b5f7a47 feat: 添加OAuth功能及相关API接口
修改API路径
2025-05-08 20:13:31 +08:00
Akizon d81ec0eee6 非常抱歉!!漏了一个 '\' 2025-05-08 16:56:57 +08:00
Akizon 0ccda568ed Update README.md 2025-05-08 16:52:08 +08:00
Akizon 2cf38a729a Merge pull request #23 from komari-monitor/dev
feat: 添加获取公共设置的API接口
2025-05-08 16:21:16 +08:00
Akizon77 77aa490696 feat: 添加获取公共设置的API接口 2025-05-08 16:20:44 +08:00
Akizon77 2e524fda42 fix: 修复配置保存时的时间戳和ID设置逻辑 2025-05-08 16:07:26 +08:00
Akizon77 9a6b2b8a35 feat: #19 1分钟的历史记录 2025-05-07 23:19:08 +08:00
259 changed files with 38438 additions and 1793 deletions

No files matched your search

+55
View File
@@ -0,0 +1,55 @@
name: Build frontend theme
description: Clone komari-web, build the default theme, and optionally upload it as an artifact.
inputs:
node-version:
description: Node.js version used to build komari-web.
default: "23"
repository:
description: komari-web repository URL.
default: https://github.com/komari-monitor/komari-web
artifact-name:
description: Optional artifact name for the packed frontend theme.
default: ""
runs:
using: composite
steps:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ inputs.node-version }}
- name: Clone and build frontend
shell: bash
run: |
set -euxo pipefail
rm -rf komari-web
git clone --depth=1 "${{ inputs.repository }}" komari-web
cd komari-web
npm install
npm run build
cd ..
mkdir -p web/public/defaultTheme/dist
rm -rf web/public/defaultTheme/dist/*
cp -r komari-web/dist/* web/public/defaultTheme/dist/
cp -f komari-web/komari-theme.json web/public/defaultTheme/
if [ -f komari-web/preview.png ]; then cp -f komari-web/preview.png web/public/defaultTheme/; fi
if [ -f komari-web/perview.png ]; then cp -f komari-web/perview.png web/public/defaultTheme/; fi
if [ -f web/public/defaultTheme/preview.png ] && [ ! -f web/public/defaultTheme/perview.png ]; then cp -f web/public/defaultTheme/preview.png web/public/defaultTheme/perview.png; fi
if [ -f web/public/defaultTheme/perview.png ] && [ ! -f web/public/defaultTheme/preview.png ]; then cp -f web/public/defaultTheme/perview.png web/public/defaultTheme/preview.png; fi
- name: Pack frontend artifact
if: ${{ inputs.artifact-name != '' }}
shell: bash
run: |
set -euxo pipefail
tar -czf "$RUNNER_TEMP/${{ inputs.artifact-name }}.tar.gz" -C web/public/defaultTheme .
- name: Upload frontend artifact
if: ${{ inputs.artifact-name != '' }}
uses: actions/upload-artifact@v4
with:
name: ${{ inputs.artifact-name }}
path: ${{ runner.temp }}/${{ inputs.artifact-name }}.tar.gz
@@ -0,0 +1,24 @@
name: Restore frontend theme
description: Download and unpack a previously built frontend theme artifact.
inputs:
artifact-name:
description: Frontend theme artifact name.
default: komari-frontend
runs:
using: composite
steps:
- name: Download frontend artifact
uses: actions/download-artifact@v4
with:
name: ${{ inputs.artifact-name }}
path: ${{ runner.temp }}/frontend-artifact
- name: Extract frontend artifact
shell: bash
run: |
set -euxo pipefail
mkdir -p web/public/defaultTheme
rm -rf web/public/defaultTheme/*
tar -xzf "$RUNNER_TEMP/frontend-artifact/${{ inputs.artifact-name }}.tar.gz" -C web/public/defaultTheme
+31
View File
@@ -0,0 +1,31 @@
name: Setup Zig
description: Restore or install the Zig toolchain used for CGO cross-compilation.
inputs:
zig-version:
description: Zig version to install.
default: 0.14.1
runs:
using: composite
steps:
- name: Restore Zig cache
id: zig-cache
uses: actions/cache@v4
with:
path: ~/.cache/komari-zig/zig-x86_64-linux-${{ inputs.zig-version }}
key: zig-${{ runner.os }}-x86_64-${{ inputs.zig-version }}
- name: Download Zig
if: ${{ steps.zig-cache.outputs.cache-hit != 'true' }}
shell: bash
run: |
set -euxo pipefail
mkdir -p "$HOME/.cache/komari-zig"
curl -fsSL "https://ziglang.org/download/${{ inputs.zig-version }}/zig-x86_64-linux-${{ inputs.zig-version }}.tar.xz" -o "$RUNNER_TEMP/zig.tar.xz"
tar -C "$HOME/.cache/komari-zig" -Jxf "$RUNNER_TEMP/zig.tar.xz"
- name: Add Zig to PATH
shell: bash
run: |
echo "$HOME/.cache/komari-zig/zig-x86_64-linux-${{ inputs.zig-version }}" >> "$GITHUB_PATH"
@@ -0,0 +1,109 @@
name: Auto Merge Dev to Main
on:
pull_request:
types: [opened, synchronize] # No longer need 'closed' trigger
branches:
- main
paths-ignore:
- '**.md'
- 'docs/**'
jobs:
# Job to enable Auto-Merge for the PR
auto-merge:
runs-on: ubuntu-latest
# Only run for open PRs from dev to main
if: github.head_ref == 'dev' && github.base_ref == 'main'
permissions:
contents: write
pull-requests: write
checks: read
statuses: read
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
token: ${{ secrets.GITHUB_TOKEN }}
fetch-depth: 0 # Required for git history
- name: Configure Git
run: |
git config --local user.email "action@github.com"
git config --local user.name "GitHub Action"
- name: Get PR number
id: get-pr
run: |
echo "pr_number=${{ github.event.number }}" >> $GITHUB_OUTPUT
- name: Check for conflicts
id: check-conflicts
run: |
# Ensure local branches are up-to-date
git fetch origin main
git fetch origin dev
# Use merge-base to find the common ancestor and check for conflicts
if git merge-tree $(git merge-base origin/main origin/dev) origin/main origin/dev | grep -q '<<<<<<<'; then
echo "has_conflicts=true" >> $GITHUB_OUTPUT
echo "❌ Merge conflicts detected"
else
echo "has_conflicts=false" >> $GITHUB_OUTPUT
echo "✅ No merge conflicts"
fi
- name: Add conflict comment
if: steps.check-conflicts.outputs.has_conflicts == 'true'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
gh pr comment ${{ steps.get-pr.outputs.pr_number }} \
--body "❌ **Merge Conflict Detection**
This PR has merge conflicts and cannot be automatically merged. Please resolve conflicts manually and push to the dev branch.
Steps to resolve conflicts:
1. \`git checkout dev\`
2. \`git merge main\`
3. Resolve conflicts
4. \`git commit\`
5. \`git push origin dev\`"
exit 1 # Exit to prevent further steps
- name: Enable auto-merge
if: steps.check-conflicts.outputs.has_conflicts == 'false'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
# Enable auto-merge using a merge commit
gh pr merge ${{ steps.get-pr.outputs.pr_number }} \
--auto \
--merge \
--delete-branch=false
# Add success comment
gh pr comment ${{ steps.get-pr.outputs.pr_number }} \
--body "✅ **Auto-merge Enabled**
This PR has no merge conflicts and auto-merge has been enabled. It will be automatically merged to the main branch once all checks pass.
Merge method: Merge commit
Branch retention: dev branch will be preserved"
- name: Summary
run: |
echo "### 🤖 Auto-merge Summary" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "- 📝 Processing PR #${{ steps.get-pr.outputs.pr_number }} (dev → main)" >> $GITHUB_STEP_SUMMARY
if [ "${{ steps.check-conflicts.outputs.has_conflicts }}" == "true" ]; then
echo "- ❌ Merge conflicts detected, manual resolution required" >> $GITHUB_STEP_SUMMARY
else
echo "- ✅ No merge conflicts, auto-merge enabled" >> $GITHUB_STEP_SUMMARY
fi
echo "" >> $GITHUB_STEP_SUMMARY
echo "**PR Link**: https://github.com/${{ github.repository }}/pull/${{ steps.get-pr.outputs.pr_number }}" >> $GITHUB_STEP_SUMMARY
+103
View File
@@ -0,0 +1,103 @@
name: Build Binaries on PR
on:
pull_request:
branches:
- main
workflow_dispatch:
jobs:
build-frontend:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Build frontend artifact
uses: ./.github/actions/build-frontend
with:
artifact-name: komari-frontend
build-binaries:
needs: build-frontend
runs-on: ubuntu-latest
strategy:
matrix:
goos: [windows, linux]
goarch: [amd64, arm64, 386, riscv64]
include:
# Windows
- goos: windows
goarch: amd64
zig_target_triple: x86_64-windows-gnu
binary_extension: .exe
- goos: windows
goarch: arm64
zig_target_triple: aarch64-windows-gnu
binary_extension: .exe
- goos: windows
goarch: 386
zig_target_triple: x86-windows-gnu
binary_extension: .exe
# Linux
- goos: linux
goarch: amd64
zig_target_triple: x86_64-linux-musl
binary_extension: ""
- goos: linux
goarch: arm64
zig_target_triple: aarch64-linux-musl
binary_extension: ""
- goos: linux
goarch: 386
zig_target_triple: x86-linux-musl
binary_extension: ""
- goos: linux
goarch: riscv64
zig_target_triple: riscv64-linux-musl
binary_extension: ""
exclude:
- goos: windows
goarch: riscv64
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Restore frontend artifact
uses: ./.github/actions/restore-frontend
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: "1.23"
- name: Install Zig cross-compiler toolchain
uses: ./.github/actions/setup-zig
- name: Build binary
env:
GOOS: ${{ matrix.goos }}
GOARCH: ${{ matrix.goarch }}
CGO_ENABLED: 1
CC: zig cc -target ${{ matrix.zig_target_triple }}
GIN_MODE: release
run: |
BINARY_NAME=komari-${{ matrix.goos }}-${{ matrix.goarch }}
# Append the appropriate executable extension based on the OS
if [ "${{ matrix.binary_extension }}" != "" ]; then
BINARY_NAME=${BINARY_NAME}${{ matrix.binary_extension }}
fi
VERSION="${{ github.sha }}"
VERSION_HASH="${{ github.sha }}"
go build -trimpath -ldflags="-s -w -X github.com/komari-monitor/komari/utils.CurrentVersion=${VERSION} -X github.com/komari-monitor/komari/utils.VersionHash=${VERSION_HASH}" -o $BINARY_NAME
- name: Upload binary as artifact
uses: actions/upload-artifact@v4
with:
name: komari-${{ matrix.goos }}-${{ matrix.goarch }}
path: komari-${{ matrix.goos }}-${{ matrix.goarch }}*
+34 -32
View File
@@ -1,8 +1,9 @@
name: Cleanup Old Packages
on:
schedule:
- cron: '0 0 * * 0' # Run every Sunday at midnight
#schedule:
# - cron: '0 0 * * 0' # Run every Sunday at midnight
# 会清理掉正常的镜像
workflow_dispatch: # Allow manual trigger
jobs:
@@ -20,40 +21,41 @@ jobs:
uses: actions/github-script@v7
with:
script: |
const { data: packages } = await github.rest.packages.getAllPackageVersionsForPackageOwnedByAuthenticatedUser({
package_type: 'container',
package_name: process.env.GITHUB_REPOSITORY.split('/')[1]
// 分页获取所有包版本
const allVersions = [];
let page = 1;
let fetched;
const packageName = process.env.GITHUB_REPOSITORY.split('/')[1];
do {
const { data: packages } = await github.rest.packages.getAllPackageVersionsForPackageOwnedByAuthenticatedUser({
package_type: 'container',
package_name: packageName,
page,
per_page: 100
});
fetched = packages.length;
allVersions.push(...packages);
page++;
} while (fetched === 100);
// 筛选仅有 SHA/sha256 标签或无标签的版本
const shaOrNoTagVersions = allVersions.filter(pkg => {
const tags = pkg.metadata.container.tags;
return (
(tags.length === 1 && (
tags[0].match(/^sha-[0-9a-f]{7}$/) ||
tags[0].match(/^sha256:[0-9a-f]{64}$/)
)) ||
tags.length === 0
);
});
// Keep packages with 'dev' or 'latest' tags, plus the 10 most recent non-dev, non-latest versions
const keepVersions = [
// Keep packages with 'dev' tag
...packages.filter(pkg => pkg.metadata.container.tags.includes('dev')),
// Keep packages with 'latest' tag, excluding those already included with 'dev'
...packages.filter(
pkg =>
pkg.metadata.container.tags.includes('latest') &&
!pkg.metadata.container.tags.includes('dev')
),
// Keep the 10 most recent non-dev, non-latest packages
...packages
.filter(
pkg =>
!pkg.metadata.container.tags.includes('dev') &&
!pkg.metadata.container.tags.includes('latest')
)
.slice(0, 10),
];
// Identify versions to delete (not in keepVersions)
const deleteVersions = packages.filter(pkg => !keepVersions.includes(pkg));
// Delete the versions
for (const version of deleteVersions) {
// 删除这些版本
for (const version of shaOrNoTagVersions) {
await github.rest.packages.deletePackageVersionForAuthenticatedUser({
package_type: 'container',
package_name: process.env.GITHUB_REPOSITORY.split('/')[1],
package_name: packageName,
package_version_id: version.id
});
console.log(`Deleted package version ${version.id}`);
console.log(`Deleted package version ${version.id} with tags: ${version.metadata.container.tags.join(', ') || 'none'}`);
}
+75
View File
@@ -0,0 +1,75 @@
name: Development Environment
on:
push:
branches:
- dev
workflow_dispatch:
jobs:
build-binaries:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
#- name: Create dummy index.html
# run: |
# mkdir -p web/public/defaultTheme/dist
# echo "<html><body><h1>This is komari development server.</h1></body></html>" > web/public/defaultTheme/dist/index.html
- name: Clone and build frontend
uses: ./.github/actions/build-frontend
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: "1.23"
- name: Build application
env:
GOOS: linux
GOARCH: amd64
run: |
VERSION_SHA="${{ github.sha }}"
go build -ldflags "-X github.com/komari-monitor/komari/utils.CurrentVersion=dev -X github.com/komari-monitor/komari/utils.VersionHash=${VERSION_SHA}" -o komari .
- name: Upload binary as artifact
uses: actions/upload-artifact@v4
with:
name: komari
path: komari
deploy-to-production:
needs: build-binaries
runs-on: ubuntu-latest
env:
PRODUCTION_SERVER: ${{ vars.PRODUCTION_SERVER }}
if: github.ref == 'refs/heads/dev'
steps:
- name: Download binary
uses: actions/download-artifact@v4
with:
name: komari
- name: Setup SSH Key
uses: webfactory/ssh-agent@v0.8.0
with:
ssh-private-key: ${{ secrets.PRODUCTION_SSH_KEY }}
- name: Add host to known hosts
run: |
mkdir -p ~/.ssh
ssh-keyscan -H ${{ env.PRODUCTION_SERVER }} >> ~/.ssh/known_hosts 2>/dev/null
- name: Deploy to production
run: |
chmod +x komari
ssh root@${{ env.PRODUCTION_SERVER }} 'mkdir -p /opt/komari/'
ssh root@${{ env.PRODUCTION_SERVER }} 'systemctl stop komari.service || true'
ssh root@${{ env.PRODUCTION_SERVER }} 'rm -f /opt/komari/komari.old'
ssh root@${{ env.PRODUCTION_SERVER }} 'mv /opt/komari/komari /opt/komari/komari.old || true'
scp komari root@${{ env.PRODUCTION_SERVER }}:/opt/komari/komari
ssh root@${{ env.PRODUCTION_SERVER }} 'systemctl start komari.service'
+72 -6
View File
@@ -1,10 +1,9 @@
name: Docker Build and Publish
name: Publish Docker Image on Main
on:
push:
branches:
- dev
- main
#push:
# branches:
# - main
workflow_dispatch:
env:
@@ -12,7 +11,61 @@ env:
IMAGE_NAME: ${{ github.repository }}
jobs:
build-and-push:
build-frontend:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Build frontend artifact
uses: ./.github/actions/build-frontend
with:
artifact-name: komari-frontend
build-docker-binaries:
needs: build-frontend
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Restore frontend artifact
uses: ./.github/actions/restore-frontend
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: "1.23"
- name: Install Zig
uses: ./.github/actions/setup-zig
- name: Build static binaries for multi-arch
env:
GOOS: linux
CGO_ENABLED: 1
GIN_MODE: release
run: |
VERSION="${{ github.ref_name }}"
VERSION_HASH="${{ github.sha }}"
LDFLAGS="-s -w -X github.com/komari-monitor/komari/utils.CurrentVersion=${VERSION} -X github.com/komari-monitor/komari/utils.VersionHash=${VERSION_HASH}"
echo "Building for linux/amd64..."
GOARCH=amd64 CC="zig cc -target x86_64-linux-musl" go build -trimpath -ldflags="$LDFLAGS" -o komari-linux-amd64
echo "Building for linux/arm64..."
GOARCH=arm64 CC="zig cc -target aarch64-linux-musl" go build -trimpath -ldflags="$LDFLAGS" -o komari-linux-arm64
- name: Upload Docker binaries
uses: actions/upload-artifact@v4
with:
name: komari-docker-binaries
path: komari-linux-*
build-and-push-multi-arch:
needs: build-docker-binaries
runs-on: ubuntu-latest
permissions:
contents: read
@@ -22,6 +75,18 @@ jobs:
- name: Checkout repository
uses: actions/checkout@v4
- name: Download Docker binaries
uses: actions/download-artifact@v4
with:
name: komari-docker-binaries
path: .
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to the Container registry
uses: docker/login-action@v3
with:
@@ -43,5 +108,6 @@ jobs:
with:
context: .
push: true
platforms: linux/amd64,linux/arm64
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
@@ -0,0 +1,608 @@
name: Generate Release Notes
on:
release:
types: [published]
workflow_dispatch:
inputs:
tag:
description: "Release tag to regenerate notes for. Leave empty to use the latest release."
required: false
type: string
base_ref:
description: "Base ref or SHA for the release comparison. Snapshot builds pass the previous main SHA."
required: false
type: string
permissions:
contents: write
jobs:
update-release-notes:
runs-on: ubuntu-latest
if: ${{ github.event_name != 'release' || !startsWith(github.event.release.tag_name, 'Snapshot-') }}
steps:
- name: Generate and apply release notes
uses: actions/github-script@v9
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_BASE_URL: ${{ secrets.OPENAI_BASE_URL }}
OPENAI_MODEL: ${{ vars.OPENAI_MODEL }}
with:
script: |
const { owner, repo } = context.repo;
const webRepo = { owner: "komari-monitor", repo: "komari-web" };
const AI_CONTEXT_CHAR_LIMIT = 120000;
const AI_MAX_COMMITS = 160;
const AI_MAX_CHANGED_FILES = 120;
const AI_MAX_PULL_REQUESTS = 80;
const AI_MAX_PATCH_CHARS_PER_FILE = 2200;
const OPENAI_TIMEOUT_MS = 120000;
const clean = (value) => (typeof value === "string" ? value.trim() : "");
const isSnapshotTag = (tag) => /^Snapshot-/i.test(clean(tag));
const releaseTimestamp = (release) => Date.parse(release.published_at || release.created_at || "");
const isStableRelease = (release) => !release.draft && !release.prerelease && !isSnapshotTag(release.tag_name);
function limitText(value, maxLength) {
const text = clean(value);
if (text.length <= maxLength) {
return text;
}
return `${text.slice(0, maxLength)}\n...[truncated ${text.length - maxLength} chars]`;
}
function appendWithinLimit(parts, value, maxLength) {
const text = clean(value);
if (!text) {
return false;
}
const currentLength = parts.reduce((total, part) => total + part.length + 2, 0);
if (currentLength + text.length + 2 <= maxLength) {
parts.push(text);
return true;
}
const remaining = maxLength - currentLength - 64;
if (remaining > 500) {
parts.push(`${text.slice(0, remaining)}\n...[context truncated]`);
}
return false;
}
async function resolveRelease() {
const requestedTag = clean(context.payload.release?.tag_name)
|| clean(context.payload.client_payload?.tag)
|| clean(context.payload.inputs?.tag);
if (requestedTag) {
const release = context.payload.release?.tag_name === requestedTag
? context.payload.release
: (await github.rest.repos.getReleaseByTag({ owner, repo, tag: requestedTag })).data;
return { tag: requestedTag, release };
}
const release = (await github.rest.repos.getLatestRelease({ owner, repo })).data;
return { tag: release.tag_name, release };
}
async function getPreviousReleaseTag(tag, release) {
const releases = await github.paginate(github.rest.repos.listReleases, {
owner,
repo,
per_page: 100,
});
const currentTime = releaseTimestamp(release);
const currentIsSnapshot = isSnapshotTag(tag);
const currentIsStable = isStableRelease(release);
const candidates = releases
.filter((item) => !item.draft && item.tag_name !== tag)
.filter((item) => {
if (!Number.isFinite(currentTime)) {
return true;
}
const itemTime = releaseTimestamp(item);
return Number.isFinite(itemTime) && itemTime < currentTime;
})
.filter((item) => {
if (currentIsSnapshot || currentIsStable) {
return isStableRelease(item);
}
return item.prerelease && !isSnapshotTag(item.tag_name);
})
.sort((left, right) => releaseTimestamp(right) - releaseTimestamp(left));
return candidates[0]?.tag_name || "";
}
async function resolveComparisonBase(tag, release) {
const requestedBaseRef = clean(context.payload.client_payload?.base_ref)
|| clean(context.payload.inputs?.base_ref);
if (requestedBaseRef) {
return {
ref: requestedBaseRef,
previousTag: "",
source: "requested base ref",
};
}
const previousTag = await getPreviousReleaseTag(tag, release);
if (previousTag) {
return {
ref: previousTag,
previousTag,
source: isSnapshotTag(tag) ? "previous stable release" : "previous release in the same channel",
};
}
if (isSnapshotTag(tag)) {
try {
const commit = (await github.rest.repos.getCommit({
owner,
repo,
ref: release.target_commitish || tag,
})).data;
const parentSha = commit.parents?.[0]?.sha || "";
if (parentSha) {
return {
ref: parentSha,
previousTag: "",
source: "snapshot target parent commit",
};
}
} catch (error) {
core.warning(`Failed to resolve parent commit for ${tag}: ${error.message}`);
}
}
return {
ref: "",
previousTag: "",
source: "none",
};
}
async function getGeneratedNotes(tag, release, previousTag) {
const params = {
owner,
repo,
tag_name: tag,
target_commitish: release.target_commitish || undefined,
};
if (previousTag) {
params.previous_tag_name = previousTag;
}
const generated = await github.request(
"POST /repos/{owner}/{repo}/releases/generate-notes",
params,
);
return clean(generated.data.body);
}
async function getComparison(baseRef, tag) {
if (!baseRef) {
return { commits: [], files: [], htmlUrl: "" };
}
const commits = [];
let files = [];
let htmlUrl = "";
for (let page = 1; page <= 20; page += 1) {
const response = await github.rest.repos.compareCommitsWithBasehead({
owner,
repo,
basehead: `${baseRef}...${tag}`,
per_page: 100,
page,
});
if (page === 1) {
files = response.data.files || [];
htmlUrl = response.data.html_url || "";
}
commits.push(...response.data.commits);
if (!response.headers.link?.includes('rel="next"')) {
break;
}
}
return { commits, files, htmlUrl };
}
function renderCommitLine(commit) {
const subject = clean(commit.commit.message.split("\n")[0]) || "(no subject)";
const shortSha = commit.sha.slice(0, 7);
const author = commit.author?.login
? ` @${commit.author.login}`
: (commit.commit.author?.name ? ` ${commit.commit.author.name}` : "");
return `- ${subject} ([${shortSha}](${commit.html_url}))${author}`;
}
function buildWhatsChangedSection(baseRef, tag, comparison) {
if (!baseRef) {
return "## What's Changed\n\nNo comparison base was found for this release.";
}
if (comparison.commits.length === 0) {
return `## What's Changed\n\nNo commits found between \`${baseRef}\` and \`${tag}\`.`;
}
const lines = ["## What's Changed", "", ...comparison.commits.map(renderCommitLine)];
if (comparison.htmlUrl) {
lines.push("", `**Full Changelog**: ${comparison.htmlUrl}`);
}
return lines.join("\n");
}
function appendCommitsToGeneratedNotes(generatedNotes, baseRef, tag, comparison) {
const notes = clean(generatedNotes);
if (!notes || comparison.commits.length === 0) {
return notes || buildWhatsChangedSection(baseRef, tag, comparison);
}
const lines = notes.split("\n");
const fullChangelogIndex = lines.findIndex((line) => /^\*\*Full Changelog\*\*:/i.test(line.trim()));
const nextTopLevelSectionIndex = lines.findIndex((line, index) => (
index > 0
&& /^##\s+/.test(line)
&& !/^##\s+What's Changed\b/i.test(line)
));
const insertionIndex = nextTopLevelSectionIndex >= 0
? nextTopLevelSectionIndex
: (fullChangelogIndex >= 0 ? fullChangelogIndex : lines.length);
const commitSection = [
"### Commits",
"",
...comparison.commits.map(renderCommitLine),
"",
];
lines.splice(insertionIndex, 0, ...commitSection);
return clean(lines.join("\n"));
}
async function getPullRequestContext(commits) {
const pullRequests = new Map();
for (const commit of commits.slice(0, AI_MAX_COMMITS)) {
try {
const response = await github.rest.repos.listPullRequestsAssociatedWithCommit({
owner,
repo,
commit_sha: commit.sha,
});
for (const pullRequest of response.data || []) {
pullRequests.set(pullRequest.html_url, pullRequest);
}
} catch (error) {
core.warning(`Failed to load pull requests for ${commit.sha.slice(0, 7)}: ${error.message}`);
}
if (pullRequests.size >= AI_MAX_PULL_REQUESTS) {
break;
}
}
return [...pullRequests.values()].slice(0, AI_MAX_PULL_REQUESTS);
}
function buildChangedFilesContext(files) {
if (!files.length) {
return "No changed file metadata was returned by GitHub.";
}
return files.slice(0, AI_MAX_CHANGED_FILES).map((file) => {
const lines = [
`File: ${file.filename}`,
`Status: ${file.status}; additions: ${file.additions}; deletions: ${file.deletions}; changes: ${file.changes}`,
];
if (file.patch) {
lines.push("Patch excerpt:");
lines.push(limitText(file.patch, AI_MAX_PATCH_CHARS_PER_FILE));
}
return lines.join("\n");
}).join("\n\n");
}
function buildCommitContext(commits) {
if (!commits.length) {
return "No commits were returned by GitHub.";
}
return commits.slice(0, AI_MAX_COMMITS).map((commit) => {
const shortSha = commit.sha.slice(0, 7);
const author = commit.author?.login || commit.commit.author?.name || "unknown";
const message = limitText(commit.commit.message, 1800);
return [
`Commit: ${shortSha}`,
`Author: ${author}`,
`URL: ${commit.html_url}`,
"Message:",
message,
].join("\n");
}).join("\n\n");
}
function buildPullRequestContext(pullRequests) {
if (!pullRequests.length) {
return "No associated pull requests were found for the compared commits.";
}
return pullRequests.map((pullRequest) => {
const labels = (pullRequest.labels || []).map((label) => label.name).filter(Boolean).join(", ") || "none";
return [
`PR: #${pullRequest.number} ${pullRequest.title}`,
`URL: ${pullRequest.html_url}`,
`Author: ${pullRequest.user?.login || "unknown"}`,
`Labels: ${labels}`,
"Body:",
limitText(pullRequest.body, 1800) || "(empty)",
].join("\n");
}).join("\n\n");
}
function buildAiContext({ tag, release, comparisonBase, whatsChangedSection, comparison, pullRequests, webReleaseSection }) {
const sections = [];
appendWithinLimit(sections, [
`Repository: ${owner}/${repo}`,
`Release tag: ${tag}`,
`Release title: ${release.name || tag}`,
`Prerelease: ${release.prerelease ? "yes" : "no"}`,
`Snapshot tag: ${isSnapshotTag(tag) ? "yes" : "no"}`,
`Target commitish: ${release.target_commitish || "(unknown)"}`,
`Comparison base: ${comparisonBase.ref || "(none)"}`,
`Comparison base source: ${comparisonBase.source}`,
`Compare URL: ${comparison.htmlUrl || "(unavailable)"}`,
].join("\n"), AI_CONTEXT_CHAR_LIMIT);
appendWithinLimit(sections, `What's Changed section:\n${whatsChangedSection}`, AI_CONTEXT_CHAR_LIMIT);
appendWithinLimit(sections, `Changed files and patch excerpts:\n${buildChangedFilesContext(comparison.files)}`, AI_CONTEXT_CHAR_LIMIT);
appendWithinLimit(sections, `Associated pull requests:\n${buildPullRequestContext(pullRequests)}`, AI_CONTEXT_CHAR_LIMIT);
appendWithinLimit(sections, `Commit details:\n${buildCommitContext(comparison.commits)}`, AI_CONTEXT_CHAR_LIMIT);
if (webReleaseSection) {
appendWithinLimit(sections, `Komari Web release notes for the same tag:\n${webReleaseSection}`, AI_CONTEXT_CHAR_LIMIT);
}
return sections.join("\n\n---\n\n");
}
function buildChatCompletionsUrl(baseUrl) {
const trimmed = clean(baseUrl).replace(/\/+$/, "");
if (!trimmed) {
return "";
}
if (trimmed.endsWith("/chat/completions")) {
return trimmed;
}
if (trimmed.endsWith("/v1")) {
return `${trimmed}/chat/completions`;
}
return `${trimmed}/v1/chat/completions`;
}
function normalizeReleaseSummary(markdown) {
const summary = clean(markdown);
if (!summary) {
return "";
}
const withoutAiHeading = clean(summary
.replace(/^#{1,6}\s+AI Summary\b\s*/im, "")
.replace(/^\*\*AI Summary\*\*\s*/im, ""));
const whatsChangedIndex = withoutAiHeading.search(/^##\s+What's Changed\b/im);
if (whatsChangedIndex >= 0) {
return clean(withoutAiHeading.slice(0, whatsChangedIndex));
}
return withoutAiHeading;
}
async function callOpenAiChatCompletion(url, apiKey, model, messages) {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), OPENAI_TIMEOUT_MS);
try {
const response = await fetch(url, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ model, messages, temperature: 0 }),
signal: controller.signal,
});
const responseText = await response.text();
let responseJson = {};
try {
responseJson = responseText ? JSON.parse(responseText) : {};
} catch {
responseJson = {};
}
if (!response.ok) {
const message = clean(responseJson.error?.message) || limitText(responseText, 500);
throw new Error(`OpenAI request failed with ${response.status}: ${message}`);
}
return clean(responseJson.choices?.[0]?.message?.content);
} finally {
clearTimeout(timeout);
}
}
async function buildReleaseSummarySection({ tag, release, comparisonBase, whatsChangedSection, comparison, webReleaseSection }) {
const apiKey = clean(process.env.OPENAI_API_KEY);
const baseUrl = clean(process.env.OPENAI_BASE_URL);
const model = clean(process.env.OPENAI_MODEL);
if (!apiKey || !baseUrl || !model) {
core.warning("OPENAI_API_KEY, OPENAI_BASE_URL, or OPENAI_MODEL is missing. Skipping generated release summary.");
return "";
}
const pullRequests = await getPullRequestContext(comparison.commits);
const aiContext = buildAiContext({
tag,
release,
comparisonBase,
whatsChangedSection,
comparison,
pullRequests,
webReleaseSection,
});
const messages = [
{
role: "system",
content: [
"You write accurate release notes for the Komari server project.",
"Use only the supplied repository context.",
"Include both Komari server changes and Komari Web frontend changes when frontend release notes are supplied.",
"Prioritize user-visible behavior, compatibility or migration notes, operations/deployment changes, security fixes, and important bug fixes.",
"Do not invent features or implementation details that are not supported by the context.",
"Write concise Markdown in Simplified Chinese.",
].join(" "),
},
{
role: "user",
content: [
`Create the release summary section for release ${tag}.`,
comparisonBase.ref
? `Summarize only changes in the comparison ${comparisonBase.ref}...${tag}.`
: "No comparison base was found, so summarize only the supplied release context.",
"Return only Markdown for the summary section that appears before '## What's Changed'.",
"Do not use the heading 'AI Summary'. Do not add a 'What's Changed' section.",
"Use these top-level headings, in this order: '## 主要更新内容', '## Bug修复', and '## 其他'.",
"Omit '## Bug修复' when there are no concrete bug fixes. Keep bullets short and specific.",
"",
aiContext,
].join("\n"),
},
];
try {
const url = buildChatCompletionsUrl(baseUrl);
const summary = await callOpenAiChatCompletion(url, apiKey, model, messages);
return normalizeReleaseSummary(summary);
} catch (error) {
core.warning(`Generated release summary failed: ${error.message}`);
return "";
}
}
async function buildWebReleaseSection(tag) {
const demoteMarkdownHeadings = (markdown) => markdown.replace(/^(#{1,5})(\s+)/gm, "#$1$2");
try {
const webRelease = (await github.rest.repos.getReleaseByTag({
owner: webRepo.owner,
repo: webRepo.repo,
tag,
})).data;
const webBody = clean(webRelease.body);
if (!webBody) {
return `## Komari Web Release Notes\n\nKomari Web release [${tag}](${webRelease.html_url}) exists, but its release notes are empty.`;
}
return [
"## Komari Web Release Notes",
"",
`Source: [komari-web ${tag}](${webRelease.html_url})`,
"",
demoteMarkdownHeadings(webBody),
].join("\n");
} catch (error) {
if (error.status === 404) {
core.info(`Komari Web release ${tag} does not exist yet. Skipping web release notes for now.`);
return "";
}
throw error;
}
}
const { tag, release } = await resolveRelease();
if (!tag) {
core.setFailed("No release tag was provided or found.");
return;
}
const comparisonBase = await resolveComparisonBase(tag, release);
core.info(comparisonBase.ref
? `Comparison base for ${tag}: ${comparisonBase.ref} (${comparisonBase.source}).`
: `No comparison base found for ${tag}.`);
const comparison = await getComparison(comparisonBase.ref, tag);
const whatsChangedSection = comparisonBase.previousTag
? appendCommitsToGeneratedNotes(
await getGeneratedNotes(tag, release, comparisonBase.previousTag),
comparisonBase.ref,
tag,
comparison,
)
: (!comparisonBase.ref && !isSnapshotTag(tag)
? await getGeneratedNotes(tag, release, "")
: buildWhatsChangedSection(comparisonBase.ref, tag, comparison));
if (!whatsChangedSection) {
core.setFailed(`Failed to build What's Changed section for ${tag}.`);
return;
}
const webReleaseSection = await buildWebReleaseSection(tag);
const releaseSummarySection = await buildReleaseSummarySection({
tag,
release,
comparisonBase,
whatsChangedSection,
comparison,
webReleaseSection,
});
const body = [
releaseSummarySection,
whatsChangedSection,
].filter(Boolean).join("\n\n");
if (clean(release.body) === clean(body)) {
core.info(`Release notes for ${tag} are already up to date.`);
} else {
await github.rest.repos.updateRelease({
owner,
repo,
release_id: release.id,
body,
});
}
await core.summary
.addHeading("Release Notes")
.addRaw(`Updated release notes for ${tag}.`)
.write();
+121
View File
@@ -0,0 +1,121 @@
name: Publish Docker Image Manually
on:
workflow_dispatch:
inputs:
version:
description: Docker image tag and embedded version.
required: true
type: string
permissions:
contents: read
packages: write
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
build-frontend:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Build frontend artifact
uses: ./.github/actions/build-frontend
with:
artifact-name: komari-frontend
build-docker-binaries:
needs: build-frontend
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Restore frontend artifact
uses: ./.github/actions/restore-frontend
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: "1.23"
- name: Install Zig
uses: ./.github/actions/setup-zig
- name: Build static binaries for multi-arch
env:
GOOS: linux
CGO_ENABLED: 1
GIN_MODE: release
VERSION: ${{ github.event.inputs.version }}
run: |
VERSION_HASH="${{ github.sha }}"
LDFLAGS="-s -w -X github.com/komari-monitor/komari/utils.CurrentVersion=${VERSION} -X github.com/komari-monitor/komari/utils.VersionHash=${VERSION_HASH}"
echo "Building for linux/amd64..."
GOARCH=amd64 CC="zig cc -target x86_64-linux-musl" go build -trimpath -ldflags="$LDFLAGS" -o komari-linux-amd64
echo "Building for linux/arm64..."
GOARCH=arm64 CC="zig cc -target aarch64-linux-musl" go build -trimpath -ldflags="$LDFLAGS" -o komari-linux-arm64
- name: Upload Docker binaries
uses: actions/upload-artifact@v4
with:
name: komari-docker-binaries
path: komari-linux-*
build-and-push-multi-arch:
needs: build-docker-binaries
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Download Docker binaries
uses: actions/download-artifact@v4
with:
name: komari-docker-binaries
path: .
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to the Container registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=raw,value=${{ github.event.inputs.version }}
type=raw,value=latest
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
push: true
platforms: linux/amd64,linux/arm64
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
no-cache: true
provenance: false
+98 -26
View File
@@ -1,4 +1,4 @@
name: Build and Attach Binaries to Release
name: Attach Release Binaries
on:
release:
@@ -13,68 +13,137 @@ env:
IMAGE_NAME: ${{ github.repository }}
jobs:
build-and-attach-binaries:
build-frontend:
runs-on: ubuntu-latest
strategy:
matrix:
goos: [windows, linux]
goarch: [amd64, arm64]
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
- name: Build frontend artifact
uses: ./.github/actions/build-frontend
with:
node-version: '23'
artifact-name: komari-frontend
- name: Clone and build frontend
run: |
git clone https://github.com/komari-monitor/komari-web web
cd web
npm install
npm run build
cd ..
mkdir -p public/dist
cp -r web/dist/* public/dist/
build-and-attach-binaries:
needs: build-frontend
runs-on: ubuntu-latest
strategy:
matrix:
goos: [windows, linux]
goarch: [amd64, arm64, 386, riscv64]
include:
# Windows
- goos: windows
goarch: amd64
zig_target_triple: x86_64-windows-gnu
binary_extension: .exe
- goos: windows
goarch: arm64
zig_target_triple: aarch64-windows-gnu
binary_extension: .exe
- goos: windows
goarch: 386
zig_target_triple: x86-windows-gnu
binary_extension: .exe
# Linux
- goos: linux
goarch: amd64
zig_target_triple: x86_64-linux-musl
binary_extension: ""
- goos: linux
goarch: arm64
zig_target_triple: aarch64-linux-musl
binary_extension: ""
- goos: linux
goarch: 386
zig_target_triple: x86-linux-musl
binary_extension: ""
- goos: linux
goarch: riscv64
zig_target_triple: riscv64-linux-musl
binary_extension: ""
exclude:
- goos: windows
goarch: riscv64
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Restore frontend artifact
uses: ./.github/actions/restore-frontend
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: '1.23'
go-version: "1.23"
- name: Install Zig cross-compiler toolchain
uses: ./.github/actions/setup-zig
- name: Build binary
env:
GOOS: ${{ matrix.goos }}
GOARCH: ${{ matrix.goarch }}
CGO_ENABLED: 0
CGO_ENABLED: 1
CC: zig cc -target ${{ matrix.zig_target_triple }}
GIN_MODE: release
run: |
BINARY_NAME=komari-${{ matrix.goos }}-${{ matrix.goarch }}
if [ "${{ matrix.goos }}" = "windows" ]; then
BINARY_NAME=${BINARY_NAME}.exe
if [ "${{ matrix.binary_extension }}" != "" ]; then
BINARY_NAME=${BINARY_NAME}${{ matrix.binary_extension }}
fi
go build -trimpath -ldflags="-s -w" -o $BINARY_NAME
VERSION="${{ github.event.release.tag_name }}"
VERSION_HASH="${{ github.sha }}" # Use commit SHA for hash
go build -trimpath -ldflags="-s -w -X github.com/komari-monitor/komari/utils.CurrentVersion=${VERSION} -X github.com/komari-monitor/komari/utils.VersionHash=${VERSION_HASH}" -o $BINARY_NAME
- name: Upload binary as artifact
uses: actions/upload-artifact@v4
with:
name: komari-${{ matrix.goos }}-${{ matrix.goarch }}
path: komari-${{ matrix.goos }}-${{ matrix.goarch }}*
- name: Upload binary to release
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
# Define the binary name for upload
BINARY_NAME=komari-${{ matrix.goos }}-${{ matrix.goarch }}
if [ "${{ matrix.goos }}" = "windows" ]; then
BINARY_NAME=${BINARY_NAME}.exe
fi
# Upload the binary as a release asset using GitHub CLI
gh release upload ${{ github.event.release.tag_name }} $BINARY_NAME --repo ${{ github.repository }}
shell: bash
build-and-push-docker:
needs: build-and-attach-binaries
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Download Docker binaries
uses: actions/download-artifact@v4
with:
pattern: komari-linux-*
path: .
merge-multiple: true
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to the Container registry
uses: docker/login-action@v3
with:
@@ -82,7 +151,7 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels) for Docker
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
@@ -96,5 +165,8 @@ jobs:
with:
context: .
push: true
platforms: linux/amd64,linux/arm64
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
labels: ${{ steps.meta.outputs.labels }}
no-cache: true
provenance: false
+289
View File
@@ -0,0 +1,289 @@
name: Publish Snapshot
on:
workflow_dispatch:
push:
branches:
- main
permissions:
contents: write
packages: write
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
concurrency:
group: snapshot-${{ github.ref }}
cancel-in-progress: true
jobs:
validate-ref:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Ensure snapshot target is main
run: |
if [ "${GITHUB_REF}" != "refs/heads/main" ]; then
echo "Snapshot builds must run from refs/heads/main. Current ref: ${GITHUB_REF}"
exit 1
fi
git fetch --no-tags --depth=1 origin main
MAIN_SHA="$(git rev-parse FETCH_HEAD)"
if [ "${GITHUB_SHA}" != "${MAIN_SHA}" ]; then
echo "Snapshot builds must use the current main commit."
echo "Workflow SHA: ${GITHUB_SHA}"
echo "Current main: ${MAIN_SHA}"
exit 1
fi
prepare:
needs: validate-ref
runs-on: ubuntu-latest
outputs:
version: ${{ steps.version.outputs.version }}
steps:
- name: Generate snapshot version
id: version
run: |
# Snapshot 版本号格式: Snapshot-yymmddhhMM (UTC+8)
VERSION="Snapshot-$(date -u -d '+8 hours' +%y%m%d%H%M)"
echo "version=${VERSION}" >> "$GITHUB_OUTPUT"
echo "Snapshot version: ${VERSION}"
build-frontend:
needs: validate-ref
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Build frontend artifact
uses: ./.github/actions/build-frontend
with:
artifact-name: komari-frontend
build-and-attach-binaries:
needs: [prepare, build-frontend]
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
goos: [windows, linux]
goarch: [amd64, arm64, 386, riscv64]
include:
# Windows
- goos: windows
goarch: amd64
zig_target_triple: x86_64-windows-gnu
binary_extension: .exe
- goos: windows
goarch: arm64
zig_target_triple: aarch64-windows-gnu
binary_extension: .exe
- goos: windows
goarch: 386
zig_target_triple: x86-windows-gnu
binary_extension: .exe
# Linux
- goos: linux
goarch: amd64
zig_target_triple: x86_64-linux-musl
binary_extension: ""
- goos: linux
goarch: arm64
zig_target_triple: aarch64-linux-musl
binary_extension: ""
- goos: linux
goarch: 386
zig_target_triple: x86-linux-musl
binary_extension: ""
- goos: linux
goarch: riscv64
zig_target_triple: riscv64-linux-musl
binary_extension: ""
exclude:
- goos: windows
goarch: riscv64
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Restore frontend artifact
uses: ./.github/actions/restore-frontend
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: "1.23"
- name: Install Zig cross-compiler toolchain
uses: ./.github/actions/setup-zig
- name: Build binary
env:
GOOS: ${{ matrix.goos }}
GOARCH: ${{ matrix.goarch }}
CGO_ENABLED: 1
CC: zig cc -target ${{ matrix.zig_target_triple }}
GIN_MODE: release
run: |
BINARY_NAME=komari-${{ matrix.goos }}-${{ matrix.goarch }}
if [ "${{ matrix.binary_extension }}" != "" ]; then
BINARY_NAME=${BINARY_NAME}${{ matrix.binary_extension }}
fi
VERSION="${{ needs.prepare.outputs.version }}"
VERSION_HASH="${{ github.sha }}" # Use commit SHA for hash
go build -trimpath -ldflags="-s -w -X github.com/komari-monitor/komari/utils.CurrentVersion=${VERSION} -X github.com/komari-monitor/komari/utils.VersionHash=${VERSION_HASH}" -o $BINARY_NAME
- name: Upload binary as artifact
uses: actions/upload-artifact@v4
with:
name: komari-${{ matrix.goos }}-${{ matrix.goarch }}
path: komari-${{ matrix.goos }}-${{ matrix.goarch }}*
publish-snapshot-release:
needs: [prepare, build-and-attach-binaries]
runs-on: ubuntu-latest
permissions:
actions: write
contents: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Ensure commit is still current main
run: |
git fetch --no-tags --depth=1 origin main
MAIN_SHA="$(git rev-parse FETCH_HEAD)"
if [ "${GITHUB_SHA}" != "${MAIN_SHA}" ]; then
echo "Snapshot publish skipped because a newer main commit exists."
echo "Workflow SHA: ${GITHUB_SHA}"
echo "Current main: ${MAIN_SHA}"
exit 1
fi
- name: Download all binaries
uses: actions/download-artifact@v4
with:
path: artifacts
merge-multiple: true
- name: Create snapshot prerelease and upload binaries
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
VERSION: ${{ needs.prepare.outputs.version }}
run: |
set -euxo pipefail
echo "Snapshot version: ${VERSION}"
ls -alR artifacts
# 如果同名 release/tag 已存在(例如重跑),先删除避免创建冲突
if gh release view "${VERSION}" --repo "${{ github.repository }}" >/dev/null 2>&1; then
echo "Release ${VERSION} 已存在,正在删除以重新创建..."
gh release delete "${VERSION}" --repo "${{ github.repository }}" --cleanup-tag --yes
fi
# 创建预发布并上传所有二进制文件
gh release create "${VERSION}" \
--repo "${{ github.repository }}" \
--target "${{ github.sha }}" \
--title "${VERSION}" \
--notes "Automated snapshot build (${VERSION}) from commit ${{ github.sha }}." \
--prerelease \
artifacts/komari-*
gh workflow run generate-release-notes.yml \
--repo "${{ github.repository }}" \
--ref main \
-f tag="${VERSION}"
gh release list \
--repo "${{ github.repository }}" \
--limit 1000 \
--json tagName,isPrerelease \
--jq '.[] | select(.isPrerelease and (.tagName | startswith("Snapshot-"))) | .tagName' |
while IFS= read -r tag; do
if [ -z "${tag}" ] || [ "${tag}" = "${VERSION}" ]; then
continue
fi
echo "Deleting old snapshot release ${tag}..."
gh release delete "${tag}" --repo "${{ github.repository }}" --cleanup-tag --yes
done
shell: bash
build-and-push-docker:
needs: [prepare, build-and-attach-binaries, publish-snapshot-release]
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Ensure commit is still current main
run: |
git fetch --no-tags --depth=1 origin main
MAIN_SHA="$(git rev-parse FETCH_HEAD)"
if [ "${GITHUB_SHA}" != "${MAIN_SHA}" ]; then
echo "Snapshot Docker publish skipped because a newer main commit exists."
echo "Workflow SHA: ${GITHUB_SHA}"
echo "Current main: ${MAIN_SHA}"
exit 1
fi
- name: Download Docker binaries
uses: actions/download-artifact@v4
with:
pattern: komari-linux-*
path: .
merge-multiple: true
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to the Container registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=raw,value=snapshot
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
push: true
platforms: linux/amd64,linux/arm64
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
no-cache: true
provenance: false
+6
View File
@@ -7,12 +7,14 @@
*.dll
*.so
*.dylib
__debug*
# local development data
komari.db
/data
/utils/geoip/data
.vscode/
/backup
# Test binary, built with `go test -c`
*.test
@@ -29,3 +31,7 @@ go.work.sum
# env file
.env
# build
komari
.codex
+31 -24
View File
@@ -1,31 +1,38 @@
FROM node:23-alpine AS frontend
WORKDIR /web
RUN apk add --no-cache git
RUN git clone https://github.com/komari-monitor/komari-web .
RUN npm install
RUN npm run build
FROM golang:1.24 AS builder
WORKDIR /app
# RUN apt-get update && apt-get install -y musl-tools
COPY go.mod go.sum ./
RUN go mod download
COPY . .
COPY --from=frontend /web/dist ./public/dist
ENV CGO_ENABLED=1
# RUN CC=musl-gcc go build -trimpath -ldflags="-s -w -linkmode external -extldflags -static" -o komari .
RUN go build -trimpath -ldflags="-s -w -linkmode external -extldflags -static" -o komari .
# FROM scratch
FROM alpine:3.21
WORKDIR /app
COPY --from=builder /app/komari .
# Docker buildx 会在构建时自动填充这些变量
ARG TARGETOS
ARG TARGETARCH
RUN apk add --no-cache ca-certificates curl tzdata
RUN set -eux; \
case "${TARGETARCH}" in \
amd64) cloudflared_arch="amd64" ;; \
386) cloudflared_arch="386" ;; \
arm64) cloudflared_arch="arm64" ;; \
arm) cloudflared_arch="arm" ;; \
*) echo "Unsupported TARGETARCH: ${TARGETARCH}" >&2; exit 1 ;; \
esac; \
curl -fsSL "https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-${cloudflared_arch}" -o /usr/local/bin/cloudflared; \
chmod +x /usr/local/bin/cloudflared
COPY komari-${TARGETOS}-${TARGETARCH} /app/komari
RUN chmod +x /app/komari
ENV GIN_MODE=release
ENV KOMARI_DB_TYPE=sqlite
ENV KOMARI_DB_FILE=/app/data/komari.db
ENV KOMARI_DB_HOST=localhost
ENV KOMARI_DB_PORT=3306
ENV KOMARI_DB_USER=root
ENV KOMARI_DB_PASS=
ENV KOMARI_DB_NAME=komari
ENV KOMARI_LISTEN=0.0.0.0:25774
EXPOSE 25774
CMD ["/app/komari","server", "-l", "0.0.0.0:25774","-d", "/app/data/komari.db"]
CMD ["/app/komari", "server"]
+27
View File
@@ -0,0 +1,27 @@
CoreOS Project
Copyright 2014 CoreOS, Inc
This product includes software developed at CoreOS, Inc.
(http://www.coreos.com/).
otp
Copyright (c) 2014, Paul Querna
This product includes software developed by
Paul Querna (http://paul.querna.org/).
Copyright 2014 gRPC authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+147 -1
View File
@@ -1 +1,147 @@
# komari
# Komari
![Badge](https://hitscounter.dev/api/hit?url=https%3A%2F%2Fgithub.com%2Fkomari-monitor%2Fkomari&label=&icon=github&color=%23a370f7&message=&style=flat&tz=UTC)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/komari-monitor/komari)
![komari](https://socialify.git.ci/komari-monitor/komari/image?description=1&font=Inter&forks=1&issues=1&language=1&logo=https%3A%2F%2Fraw.githubusercontent.com%2Fkomari-monitor%2Fkomari-web%2Fd54ce1288df41ead08aa19f8700186e68028a889%2Fpublic%2Ffavicon.png&name=1&owner=1&pattern=Plus&pulls=1&stargazers=1&theme=Auto)
[简体中文](./docs/README_zh.md) | [繁體中文](./docs/README_zh-TW.md) | [日本語](./docs/README_ja.md)
Komari is a lightweight, self-hosted server monitoring tool designed to provide a simple and efficient solution for monitoring server performance. It supports viewing server status through a web interface and collects data through a lightweight agent.
> [!WARNING]
> Komari is a self-hosted monitoring and control program intended only for systems you own or are authorized to administer. Do not weaponize Komari, deploy it without consent, or use it for unauthorized access, persistence, command execution, or other abusive activity. For context on real-world abuse risks, see Huntress' analysis: [Komari C2 agent abuse](https://www.huntress.com/blog/komari-c2-agent-abuse).
> Users are solely responsible for how they deploy and operate Komari. The developers do not accept responsibility for unauthorized or abusive use, or for any resulting consequences.
> On Windows, when remote control is enabled, the client displays a Windows notification at each user login to remind the user that Komari is remote control software.
[Documentation](https://komari-document.pages.dev/) | [文档(镜像站 By Geekertao)](https://www.komari.wiki) | [Telegram Group](https://t.me/komari_monitor)
## Features
- **Lightweight and Efficient**: Low resource consumption, suitable for servers of all sizes.
- **Self-hosted**: Complete control over data privacy, easy to deploy.
- **Web Interface**: Intuitive monitoring dashboard, easy to use.
## Quick Start
### 0. One-click Deployment with Cloud Hosting
- Rainyun - CNY 4.5/month
[![](https://rainyun-apps.cn-nb1.rains3.com/materials/deploy-on-rainyun-cn.svg)](https://app.rainyun.com/apps/rca/store/6780/NzYxNzAz_)
- 1Panel App Store
Available on 1Panel App Store. Install via **App Store > Utilities > Komari**.
### 1. Use the One-click Install Script
Suitable for distributions using systemd (Ubuntu, Debian...).
```bash
curl -fsSL https://raw.githubusercontent.com/komari-monitor/komari/main/install-komari.sh -o install-komari.sh
chmod +x install-komari.sh
sudo ./install-komari.sh
```
### 2. Docker Deployment
1. Create a data directory:
```bash
mkdir -p ./data
```
2. Run the Docker container:
```bash
docker run -d \
-p 25774:25774 \
-v $(pwd)/data:/app/data \
--name komari \
ghcr.io/komari-monitor/komari:latest
```
3. View the default username and password:
```bash
docker logs komari
```
4. Access `http://<your_server_ip>:25774` in your browser.
> [!NOTE]
> You can also customize the initial username and password through the environment variables `ADMIN_USERNAME` and `ADMIN_PASSWORD`.
### 3. Binary File Deployment
1. Visit Komari's [GitHub Release page](https://github.com/komari-monitor/komari/releases) to download the latest binary for your operating system.
2. Run Komari:
```bash
./komari server -l 0.0.0.0:25774
```
3. Access `http://<your_server_ip>:25774` in your browser. The default port is `25774`.
4. The default username and password can be found in the startup logs or set via the environment variables `ADMIN_USERNAME` and `ADMIN_PASSWORD`.
> [!NOTE]
> Ensure the binary has execute permissions (`chmod +x komari`). Data will be saved in the `data` folder in the running directory.
### Manual Build
#### Dependencies
- Go 1.18+ and Node.js 20+ (for manual build)
1. Build the frontend static files:
```bash
git clone https://github.com/komari-monitor/komari-web
cd komari-web
npm install
npm run build
```
2. Build the backend:
```bash
git clone https://github.com/komari-monitor/komari
cd komari
```
Copy the static files generated in step 1 to the `/web/public/defaultTheme/dist` folder in the root of the `komari` project, and copy `komari-theme.json` + `preview.png`/`perview.png` to `/web/public/defaultTheme`.
```bash
go build -o komari
```
3. Run:
```bash
./komari server -l 0.0.0.0:25774
```
The default listening port is `25774`. Access `http://localhost:25774`.
## Frontend Development Guide
[Komari Theme Development Guide | Komari](https://komari-document.pages.dev/dev/theme.html)
## Client Agent Development Guide
[Komari Agent Information Reporting and Event Handling Documentation](https://komari-document.pages.dev/dev/agent.html)
## Contributing
Issues and Pull Requests are welcome!
## Acknowledgements
### 破碎工坊云
[破碎工坊云 - 专业云计算服务平台,提供高效、稳定、安全的高防服务器与CDN解决方案](https://www.crash.work/)
### DreamCloud
[DreamCloud - 极高性价比解锁直连亚太高防](https://as211392.com/)
### 🚀 Sponsored by SharonNetworks
[![Sharon Networks](https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/sharon-networks.webp)](https://sharon.io)
SharonNetworks 为您的业务起飞保驾护航!
亚太数据中心提供顶级的中国优化网络接入 · 低延时&高带宽&提供Tbps级本地清洗高防服务, 为您的业务保驾护航, 为您的客户提供极致体验. 加入社区 [Telegram群组](https://t.me/SharonNetwork) 可参与公益募捐或群内抽奖免费使用
### The open source software community
All the developers who submitted PRs and created themes
## Star History
[![Star History Chart](https://api.star-history.com/svg?repos=komari-monitor/komari&type=Date)](https://www.star-history.com/#komari-monitor/komari&Date)
-31
View File
@@ -1,31 +0,0 @@
package api
import (
"net/http"
"github.com/komari-monitor/komari/database/accounts"
"github.com/gin-gonic/gin"
)
func AdminAuthMiddleware() gin.HandlerFunc {
return func(c *gin.Context) {
session, err := c.Cookie("session_token")
if err != nil {
c.JSON(http.StatusUnauthorized, gin.H{"status": "error", "error": "Unauthorized"})
c.Abort()
return
}
// Komari is a single user system
_, err = accounts.GetSession(session)
if err != nil {
c.JSON(http.StatusUnauthorized, gin.H{"status": "error", "error": "Unauthorized."})
c.Abort()
return
}
c.Next()
}
}
-81
View File
@@ -1,81 +0,0 @@
package api
import (
"bytes"
"database/sql"
"encoding/json"
"io"
"net/http"
"github.com/komari-monitor/komari/database/clients"
"gorm.io/gorm"
"github.com/gin-gonic/gin"
)
// TokenAuthMiddleware creates a Gin middleware that validates a token from query parameters or request body.
func TokenAuthMiddleware() gin.HandlerFunc {
return func(c *gin.Context) {
var token string
// Step 1: Check query parameter for token
token = c.Query("token")
// Step 2: If no token in query, check request body for non-GET requests
if token == "" && c.Request.Method != http.MethodGet {
// Read the body
bodyBytes, err := io.ReadAll(c.Request.Body)
if err != nil {
c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"status": "error", "error": "failed to read request body"})
return
}
// Restore the body for downstream handlers
c.Request.Body = io.NopCloser(bytes.NewReader(bodyBytes))
// Try to extract token from JSON body
var bodyMap map[string]interface{}
if len(bodyBytes) > 0 {
if err := json.Unmarshal(bodyBytes, &bodyMap); err == nil {
if tokenVal, exists := bodyMap["token"]; exists {
if str, ok := tokenVal.(string); ok && str != "" {
token = str
}
}
}
}
}
if token == "" {
c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"status": "error", "error": "token is required"})
return
}
exists, err := checkTokenExists(token)
if err != nil {
c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"status": "error", "error": "failed to validate token"})
return
}
if !exists {
c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"status": "error", "error": "invalid token"})
return
}
c.Next()
}
}
func checkTokenExists(token string) (bool, error) {
_, err := clients.GetClientUUIDByToken(token)
if err == sql.ErrNoRows {
return false, nil
}
if err == gorm.ErrRecordNotFound {
return false, nil
}
if err != nil {
return false, err
}
return true, nil
}
-146
View File
@@ -1,146 +0,0 @@
package admin
import (
"net/http"
"time"
"github.com/gin-gonic/gin"
"github.com/komari-monitor/komari/common"
"github.com/komari-monitor/komari/database/clients"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/database/records"
)
func AddClient(c *gin.Context) {
uuid, token, err := clients.CreateClient()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"status": "error", "error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"status": "success", "uuid": uuid, "token": token})
}
func EditClient(c *gin.Context) {
var req struct {
UUID string `json:"uuid" binding:"required"`
ClientName string `json:"name,omitempty"`
Token string `json:"token,omitempty"`
Weigth int `json:"weight,omitempty"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"status": "error", "error": err.Error()})
return
}
db := dbcore.GetDBInstance()
var err error
if req.ClientName != "" {
err = db.Model(&common.ClientInfo{}).Where("uuid = ?", req.UUID).
Updates(map[string]interface{}{"name": req.ClientName, "updated_at": time.Now()}).Error
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"status": "error", "error": err.Error()})
return
}
}
if req.Token != "" {
err = db.Model(&models.Client{}).Where("uuid = ?", req.UUID).
Updates(map[string]interface{}{"token": req.Token, "updated_at": time.Now()}).Error
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"status": "error", "error": err.Error()})
return
}
}
if req.Weigth != 0 {
err = db.Model(&common.ClientInfo{}).Where("uuid = ?", req.UUID).
Updates(map[string]interface{}{"weigth": req.Weigth, "updated_at": time.Now()}).Error
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"status": "error", "error": err.Error()})
return
}
}
c.JSON(http.StatusOK, gin.H{"status": "success"})
}
func RemoveClient(c *gin.Context) {
var req struct {
UUID string `json:"uuid" binding:"required"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(400, gin.H{
"status": "error",
"error": "Invalid or missing UUID",
})
}
err := clients.DeleteClient(req.UUID)
if err != nil {
c.JSON(500, gin.H{
"status": "error",
"error": "Failed to delete client" + err.Error(),
})
return
}
c.JSON(200, gin.H{"status": "success"})
}
func ClearRecord(c *gin.Context) {
if err := records.DeleteAll(); err != nil {
c.JSON(500, gin.H{
"status": "error",
"error": "Failed to delete Record" + err.Error(),
})
return
}
c.JSON(200, gin.H{"status": "success"})
}
func GetClient(c *gin.Context) {
uuid := c.Query("uuid")
if uuid == "" {
c.JSON(400, gin.H{
"status": "error",
"error": "Invalid or missing UUID",
})
return
}
result, err := clients.GetClientByUUID(uuid)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"status": "error", "error": err.Error()})
return
}
c.JSON(http.StatusOK, result)
}
func ListClients(c *gin.Context) {
cls, err := clients.GetAllClientBasicInfo()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"status": "error", "error": err.Error()})
return
}
c.JSON(http.StatusOK, cls)
}
func GetClientToken(c *gin.Context) {
uuid := c.Query("uuid")
if uuid == "" {
c.JSON(400, gin.H{
"status": "error",
"error": "Invalid or missing UUID",
})
return
}
token, err := clients.GetClientTokenByUUID(uuid)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"status": "error", "error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"status": "success", "token": token})
}
-46
View File
@@ -1,46 +0,0 @@
package admin
import (
"github.com/komari-monitor/komari/database/accounts"
"github.com/gin-gonic/gin"
)
func GetSessions(c *gin.Context) {
ss, err := accounts.GetAllSessions()
if err != nil {
c.JSON(500, gin.H{
"status": "error",
"error": "Failed to get sessions",
})
return
}
c.JSON(200, ss)
}
func DeleteSession(c *gin.Context) {
var req struct {
Session string `json:"session" binding:"required"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(400, gin.H{
"status": "error",
"error": "Invalid or missing session",
})
return
}
err := accounts.DeleteSession(req.Session)
if err != nil {
c.JSON(500, gin.H{
"status": "error",
"error": "Failed to delete session",
})
return
}
c.JSON(200, gin.H{
"status": "success",
"message": "Session deleted successfully",
})
}
-53
View File
@@ -1,53 +0,0 @@
package admin
import (
"database/sql"
"github.com/komari-monitor/komari/database/config"
"github.com/komari-monitor/komari/database/models"
"github.com/gin-gonic/gin"
)
// GetSettings 获取自定义配置
func GetSettings(c *gin.Context) {
cst, err := config.Get()
if err != nil {
if err == sql.ErrNoRows {
//override
cst = models.Config{Sitename: "Komari"}
cst.ID = 1
config.Save(cst)
c.JSON(200, cst)
return
}
c.JSON(500, gin.H{
"status": "error",
"error": "Internal Server Error: " + err.Error(),
})
}
c.JSON(200, cst)
}
// EditSettings 更新自定义配置
// 务必传全数据
func EditSettings(c *gin.Context) {
cfg := models.Config{}
if err := c.ShouldBindJSON(&cfg); err != nil {
c.JSON(400, gin.H{
"status": "error",
"error": "Bad Request: " + err.Error(),
})
return
}
cfg.ID = 1 // Only one record
if err := config.Save(cfg); err != nil {
c.JSON(500, gin.H{
"status": "error",
"error": "Internal Server Error: " + err.Error(),
})
return
}
c.JSON(200, gin.H{"status": "success"})
}
-136
View File
@@ -1,136 +0,0 @@
package client
import (
"bytes"
"encoding/json"
"fmt"
"io"
"time"
"log"
"net/http"
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
"github.com/komari-monitor/komari/common"
"github.com/komari-monitor/komari/database/clients"
"github.com/komari-monitor/komari/ws"
)
func UploadReport(c *gin.Context) {
bodyBytes, err := io.ReadAll(c.Request.Body)
if err != nil {
log.Println("Failed to read request body:", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
return
}
var data map[string]interface{}
err = json.Unmarshal(bodyBytes, &data)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
return
}
// Save report to database
var report common.Report
err = json.Unmarshal(bodyBytes, &report)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
return
}
err = clients.SaveClientReport(report.UUID, report)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("%v", err)})
return
}
// Update report with method and token
report.UpdatedAt = time.Now()
ws.LatestReport[report.UUID] = &report
c.Request.Body = io.NopCloser(bytes.NewBuffer(bodyBytes)) // Restore the body for further use
c.JSON(200, gin.H{"status": "success"})
}
func WebSocketReport(c *gin.Context) {
// 升级ws
if !websocket.IsWebSocketUpgrade(c.Request) {
c.JSON(http.StatusBadRequest, gin.H{"status": "error", "error": "Require WebSocket upgrade"})
return
}
upgrader := websocket.Upgrader{
CheckOrigin: func(r *http.Request) bool {
return true
},
}
// Upgrade the HTTP connection to a WebSocket connection
conn, err := upgrader.Upgrade(c.Writer, c.Request, nil)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"status": "error", "error": "Failed to upgrade to WebSocket"})
return
}
defer conn.Close()
_, message, err := conn.ReadMessage()
if err != nil {
log.Println("Error reading message:", err)
return
}
// 第一次数据拿token
data := map[string]interface{}{}
err = json.Unmarshal(message, &data)
if err != nil {
conn.WriteJSON(gin.H{"status": "error", "error": "Invalid JSON"})
return
}
// it should ok,token was verfied in the middleware
token := ""
var errMsg string
// 优先检查查询参数中的 token
token = c.Query("token")
// 如果 token 为空,返回错误
if token == "" {
conn.WriteJSON(gin.H{"status": "error", "error": errMsg})
return
}
uuid, err := clients.GetClientUUIDByToken(token)
if err != nil {
conn.WriteJSON(gin.H{"status": "error", "error": errMsg})
return
}
// 只允许一个客户端的连接
if _, exists := ws.ConnectedClients[uuid]; exists {
conn.WriteJSON(gin.H{"status": "error", "error": "Token already in use"})
return
}
ws.ConnectedClients[uuid] = conn
defer func() {
delete(ws.ConnectedClients, uuid)
}()
for {
_, message, err := conn.ReadMessage()
if err != nil {
break
}
report := common.Report{}
err = json.Unmarshal(message, &report)
if err != nil {
break
}
err = clients.SaveClientReport(uuid, report)
if err != nil {
conn.WriteJSON(gin.H{"status": "error", "error": fmt.Sprintf("%v", err)})
}
report.UpdatedAt = time.Now()
ws.LatestReport[uuid] = &report
}
}
-52
View File
@@ -1,52 +0,0 @@
package client
import (
"net"
"github.com/komari-monitor/komari/common"
"github.com/komari-monitor/komari/database/clients"
"github.com/komari-monitor/komari/database/config"
"github.com/komari-monitor/komari/utils/geoip"
"github.com/gin-gonic/gin"
)
func UploadBasicInfo(c *gin.Context) {
var cbi = common.ClientInfo{}
if err := c.ShouldBindJSON(&cbi); err != nil {
c.JSON(400, gin.H{"status": "error", "error": "Invalid or missing data"})
return
}
token := c.Query("token")
uuid, err := clients.GetClientUUIDByToken(token)
if uuid == "" || err != nil {
c.JSON(400, gin.H{"status": "error", "error": "Invalid token"})
return
}
cbi.UUID = uuid
if cfg, err := config.Get(); err == nil && cfg.GeoIpEnabled {
if cbi.IPv4 != "" {
ip4 := net.ParseIP(cbi.IPv4)
ip4_record, _ := geoip.GetGeoIpInfo(ip4)
if ip4_record != nil {
cbi.Region = geoip.GetRegionUnicodeEmoji(ip4_record.Country.ISOCode)
}
} else if cbi.IPv6 != "" {
ip6 := net.ParseIP(cbi.IPv6)
ip6_record, _ := geoip.GetGeoIpInfo(ip6)
if ip6_record != nil {
cbi.Region = geoip.GetRegionUnicodeEmoji(ip6_record.Country.ISOCode)
}
}
}
if err := clients.UpdateOrInsertBasicInfo(cbi); err != nil {
c.JSON(500, gin.H{"status": "error", "error": err})
return
}
c.JSON(200, gin.H{"status": "success"})
}
-51
View File
@@ -1,51 +0,0 @@
package api
import (
"encoding/json"
"io"
"net/http"
"github.com/komari-monitor/komari/database/accounts"
"github.com/gin-gonic/gin"
)
type LoginRequest struct {
Username string `json:"username"`
Password string `json:"password"`
}
func Login(c *gin.Context) {
// TODO: Settings -> Disable password login when using OAuth
bodyBytes, err := io.ReadAll(c.Request.Body)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"status": "error", "error": "Invalid request body"})
return
}
var data LoginRequest
err = json.Unmarshal(bodyBytes, &data)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"status": "error", "error": "Invalid request body"})
return
}
if data.Username == "" || data.Password == "" {
c.JSON(http.StatusBadRequest, gin.H{"status": "error", "error": "Invalid request body"})
return
}
if uuid, success := accounts.CheckPassword(data.Username, data.Password); success {
session, err := accounts.CreateSession(uuid, 2592000)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"status": "error", "error": "Failed to create session" + err.Error()})
return
}
c.SetCookie("session_token", session, 2592000, "/", "", false, true)
c.JSON(200, gin.H{"set-cookie": map[string]interface{}{"session_token": session}})
return
} else {
c.JSON(http.StatusUnauthorized, gin.H{"status": "error", "error": "Invalid credentials"})
}
}
-27
View File
@@ -1,27 +0,0 @@
package api
import (
"github.com/komari-monitor/komari/database/accounts"
"github.com/gin-gonic/gin"
)
func GetMe(c *gin.Context) {
session, err := c.Cookie("session_token")
if err != nil {
c.JSON(200, gin.H{"username": "Guest", "logged_in": false})
return
}
uuid, err := accounts.GetSession(session)
if err != nil {
c.JSON(200, gin.H{"username": "Guest", "logged_in": false})
return
}
user, err := accounts.GetUserByUUID(uuid)
if err != nil {
c.JSON(200, gin.H{"username": "Guest", "logged_in": false})
return
}
c.JSON(200, gin.H{"username": user.Username, "logged_in": true, "uuid": user.UUID})
}
-95
View File
@@ -1,95 +0,0 @@
package api
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"github.com/gin-gonic/gin"
"github.com/komari-monitor/komari/database/accounts"
"github.com/stretchr/testify/assert"
)
func TestGetMe(t *testing.T) {
// 设置测试模式
gin.SetMode(gin.TestMode)
// 创建测试用户
user, err := accounts.CreateAccount("testuser", "password")
if err != nil {
t.Fatalf("Failed to create test user: %v", err)
}
uuid := user.UUID
sessionToken, _ := accounts.CreateSession(uuid, 2592000)
tests := []struct {
name string
sessionToken string
expectedStatus int
expectedBody map[string]interface{}
}{
{
name: "有效的会话",
sessionToken: sessionToken,
expectedStatus: http.StatusOK,
expectedBody: map[string]interface{}{
"username": "testuser",
},
},
{
name: "无效的会话",
sessionToken: "invalid_token",
expectedStatus: http.StatusOK,
expectedBody: map[string]interface{}{
"username": "Guest",
},
},
{
name: "无会话",
sessionToken: "",
expectedStatus: http.StatusOK,
expectedBody: map[string]interface{}{
"username": "Guest",
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// 创建测试路由
router := gin.New()
router.GET("/me", GetMe)
// 创建测试请求
req, _ := http.NewRequest("GET", "/me", nil)
if tt.sessionToken != "" {
req.AddCookie(&http.Cookie{
Name: "session_token",
Value: tt.sessionToken,
})
}
// 创建响应记录器
w := httptest.NewRecorder()
// 执行请求
router.ServeHTTP(w, req)
// 断言状态码
assert.Equal(t, tt.expectedStatus, w.Code)
// 解析响应体
var response map[string]interface{}
err := json.Unmarshal(w.Body.Bytes(), &response)
assert.NoError(t, err)
// 断言响应体
assert.Equal(t, tt.expectedBody, response)
})
}
// 清除测试数据
accounts.DeleteAccountByUsername("testuser")
accounts.DeleteAllSessions()
}
-23
View File
@@ -1,23 +0,0 @@
package api
import (
"github.com/gin-gonic/gin"
"github.com/komari-monitor/komari/database/clients"
)
func GetNodesInformation(c *gin.Context) {
clientList, err := clients.GetAllClientBasicInfo()
if err != nil {
c.JSON(500, gin.H{"status": "error", "error": err.Error()})
return
}
count := len(clientList)
// 公开信息不展示IP地址
for i := 0; i < count; i++ {
clientList[i].IPv4 = ""
clientList[i].IPv6 = ""
}
c.JSON(200, gin.H{"status": "success", "data": clientList})
}
+9 -6
View File
@@ -5,11 +5,13 @@ import (
"github.com/komari-monitor/komari/cmd/flags"
"github.com/komari-monitor/komari/database/accounts"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"github.com/spf13/cobra"
)
var (
Username string
//Username string
NewPassword string
)
@@ -17,7 +19,7 @@ var ChpasswdCmd = &cobra.Command{
Use: "chpasswd",
Short: "Force change password",
Long: `Force change password`,
Example: `komari chpasswd -u <username> -p <password>`,
Example: `komari chpasswd -p <password>`,
Run: func(cmd *cobra.Command, args []string) {
if NewPassword == "" {
cmd.Help()
@@ -27,9 +29,10 @@ var ChpasswdCmd = &cobra.Command{
cmd.Println("Database file does not exist.")
return
}
cmd.Println("Changing password for user:", Username)
if err := accounts.ForceResetPassword(Username, NewPassword); err != nil {
user := &models.User{}
dbcore.GetDBInstance().Model(&models.User{}).First(user)
cmd.Println("Changing password for user:", user.Username)
if err := accounts.ForceResetPassword(user.Username, NewPassword); err != nil {
cmd.Println("Error:", err)
return
}
@@ -45,7 +48,7 @@ var ChpasswdCmd = &cobra.Command{
}
func init() {
ChpasswdCmd.PersistentFlags().StringVarP(&Username, "user", "u", "admin", "The username of the account to change password")
//ChpasswdCmd.PersistentFlags().StringVarP(&Username, "user", "u", "admin", "The username of the account to change password")
ChpasswdCmd.PersistentFlags().StringVarP(&NewPassword, "password", "p", "", "New password")
RootCmd.AddCommand(ChpasswdCmd)
}
+33
View File
@@ -0,0 +1,33 @@
package cmd
import (
"os"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"github.com/spf13/cobra"
"gorm.io/gorm"
)
var Disable2FA = &cobra.Command{
Use: "disable-2fa",
Short: "Force disable 2FA",
Long: `Force disable 2FA`,
Run: func(cmd *cobra.Command, args []string) {
db := dbcore.GetDBInstance()
err := db.Transaction(func(tx *gorm.DB) error {
return tx.Model(&models.User{}).Where("two_factor != ?", "").
Update("two_factor", "").Error
})
if err != nil {
cmd.Println("Error:", err)
os.Exit(1)
}
cmd.Println("2FA has been disabled.")
cmd.Println("Please restart the server to apply the changes.")
},
}
func init() {
RootCmd.AddCommand(Disable2FA)
}
+38 -3
View File
@@ -1,6 +1,41 @@
package flags
var (
DatabaseFile string
Listen string
import "strings"
const (
DatabaseTypeSQLite = "sqlite"
)
var (
// 数据库配置
DatabaseType string // 数据库类型:sqlite
DatabaseFile string // SQLite数据库文件路径
DatabaseHost string // 保留的兼容参数,当前未使用
DatabasePort string // 保留的兼容参数,当前未使用
DatabaseUser string // 保留的兼容参数,当前未使用
DatabasePass string // 保留的兼容参数,当前未使用
DatabaseName string // 保留的兼容参数,当前未使用
Listen string
)
func NormalizeDatabaseType(databaseType string) string {
databaseType = strings.ToLower(strings.TrimSpace(databaseType))
if databaseType == "" {
return DatabaseTypeSQLite
}
return databaseType
}
func ApplyDatabaseTypeNormalization() string {
DatabaseType = NormalizeDatabaseType(DatabaseType)
return DatabaseType
}
func IsSQLite() bool {
return NormalizeDatabaseType(DatabaseType) == DatabaseTypeSQLite
}
func SupportedDatabaseTypes() string {
return DatabaseTypeSQLite
}
+20
View File
@@ -0,0 +1,20 @@
package flags
import "testing"
func TestNormalizeDatabaseType(t *testing.T) {
tests := map[string]string{
"": DatabaseTypeSQLite,
"sqlite": DatabaseTypeSQLite,
" SQLite ": DatabaseTypeSQLite,
"SQLITE": DatabaseTypeSQLite,
"postgres": "postgres",
" postgres": "postgres",
}
for input, want := range tests {
if got := NormalizeDatabaseType(input); got != want {
t.Fatalf("NormalizeDatabaseType(%q) = %q, want %q", input, got, want)
}
}
}
+28
View File
@@ -0,0 +1,28 @@
package cmd
import (
"os"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/pkg/config"
"github.com/spf13/cobra"
)
var PermitPasswordLoginCmd = &cobra.Command{
Use: "permit-login",
Short: "Force permit password login",
Long: `Force permit password login`,
Run: func(cmd *cobra.Command, args []string) {
dbcore.GetDBInstance()
if err := config.Set(config.DisablePasswordLoginKey, false); err != nil {
cmd.Println("Error:", err)
os.Exit(1)
}
cmd.Println("Password login has been permitted.")
cmd.Println("Please restart the server to apply the changes.")
},
}
func init() {
RootCmd.AddCommand(PermitPasswordLoginCmd)
}
+30 -1
View File
@@ -9,6 +9,25 @@ import (
"github.com/spf13/cobra"
)
func GetEnv(key, defaultValue string) string {
value := os.Getenv(key)
if value == "" {
return defaultValue
}
return value
}
// 从环境变量获取默认值
var (
dbTypeEnv = GetEnv("KOMARI_DB_TYPE", "sqlite")
dbFileEnv = GetEnv("KOMARI_DB_FILE", "./data/komari.db")
dbHostEnv = GetEnv("KOMARI_DB_HOST", "localhost")
dbPortEnv = GetEnv("KOMARI_DB_PORT", "3306")
dbUserEnv = GetEnv("KOMARI_DB_USER", "root")
dbPassEnv = GetEnv("KOMARI_DB_PASS", "")
dbNameEnv = GetEnv("KOMARI_DB_NAME", "komari")
)
var RootCmd = &cobra.Command{
Use: "Komari",
Short: "Komari is a simple server monitoring tool",
@@ -28,5 +47,15 @@ func Execute() {
}
func init() {
RootCmd.PersistentFlags().StringVarP(&flags.DatabaseFile, "database", "d", "./data/komari.db", "Database file")
// 设置命令行参数,提供环境变量作为默认值
RootCmd.PersistentFlags().StringVarP(&flags.DatabaseType, "db-type", "t", dbTypeEnv, "Database type (sqlite) [env: KOMARI_DB_TYPE]")
RootCmd.PersistentFlags().StringVarP(&flags.DatabaseFile, "database", "d", dbFileEnv, "SQLite database file path [env: KOMARI_DB_FILE]")
RootCmd.PersistentFlags().StringVar(&flags.DatabaseHost, "db-host", dbHostEnv, "Reserved database host parameter [env: KOMARI_DB_HOST]")
RootCmd.PersistentFlags().StringVar(&flags.DatabasePort, "db-port", dbPortEnv, "Reserved database port parameter [env: KOMARI_DB_PORT]")
RootCmd.PersistentFlags().StringVar(&flags.DatabaseUser, "db-user", dbUserEnv, "Reserved database username parameter [env: KOMARI_DB_USER]")
RootCmd.PersistentFlags().StringVar(&flags.DatabasePass, "db-pass", dbPassEnv, "Reserved database password parameter [env: KOMARI_DB_PASS]")
RootCmd.PersistentFlags().StringVar(&flags.DatabaseName, "db-name", dbNameEnv, "Reserved database name parameter [env: KOMARI_DB_NAME]")
for _, name := range []string{"db-host", "db-port", "db-user", "db-pass", "db-name"} {
_ = RootCmd.PersistentFlags().MarkHidden(name)
}
}
+204 -99
View File
@@ -1,23 +1,40 @@
package cmd
import (
"context"
"fmt"
"log"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"github.com/komari-monitor/komari/api"
"github.com/komari-monitor/komari/api/admin"
"github.com/komari-monitor/komari/api/client"
"github.com/komari-monitor/komari/cmd/flags"
"github.com/komari-monitor/komari/database/accounts"
"github.com/komari-monitor/komari/database/config"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/records"
"github.com/komari-monitor/komari/public"
"github.com/komari-monitor/komari/utils/geoip"
"github.com/komari-monitor/komari/ws"
"github.com/gin-contrib/cors"
"github.com/gin-gonic/gin"
"github.com/komari-monitor/komari/cmd/flags"
"github.com/komari-monitor/komari/pkg/corn"
"github.com/komari-monitor/komari/web/api"
"github.com/komari-monitor/komari/database"
"github.com/komari-monitor/komari/database/accounts"
"github.com/komari-monitor/komari/database/auditlog"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
d_notification "github.com/komari-monitor/komari/database/notification"
"github.com/komari-monitor/komari/database/records"
"github.com/komari-monitor/komari/database/tasks"
"github.com/komari-monitor/komari/pkg/config"
"github.com/komari-monitor/komari/utils"
"github.com/komari-monitor/komari/utils/cloudflared"
"github.com/komari-monitor/komari/utils/geoip"
logutil "github.com/komari-monitor/komari/utils/log"
"github.com/komari-monitor/komari/utils/messageSender"
"github.com/komari-monitor/komari/utils/notifier"
"github.com/komari-monitor/komari/web/nezha"
"github.com/komari-monitor/komari/web/oauth"
report_cache "github.com/komari-monitor/komari/web/report"
"github.com/komari-monitor/komari/web/router"
"github.com/komari-monitor/komari/web/security"
"github.com/spf13/cobra"
)
@@ -26,93 +43,137 @@ var ServerCmd = &cobra.Command{
Short: "Start the server",
Long: `Start the server`,
Run: func(cmd *cobra.Command, args []string) {
InitDatabase()
go DoRecordsWork()
r := gin.Default()
cfg, err := config.Get()
if err != nil {
log.Fatalln("Failed to get config:", err)
}
if cfg.AllowCros {
r.Use(cors.New(cors.Config{
AllowOrigins: []string{"*"},
AllowMethods: []string{"GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS"},
AllowHeaders: []string{"Origin", "Content-Length", "Content-Type", "Authorization"},
ExposeHeaders: []string{"Content-Length"},
AllowCredentials: true,
MaxAge: 12 * time.Hour,
}))
}
if cfg.GeoIpEnabled {
geoip.InitGeoIp()
go func() {
ticker := time.NewTicker(time.Hour * 24)
for range ticker.C {
geoip.UpdateGeoIpDatabase()
}
}()
}
r.Any("/ping", func(c *gin.Context) {
c.String(200, "pong")
})
r.POST("/api/login", api.Login)
r.GET("/api/me", api.GetMe)
r.GET("/api/clients", ws.GetClients)
r.GET("/api/nodes", api.GetNodesInformation)
tokenAuthrized := r.Group("/api/clients", api.TokenAuthMiddleware())
{
tokenAuthrized.GET("/report", client.WebSocketReport) // websocket
tokenAuthrized.POST("/uploadBasicInfo", client.UploadBasicInfo)
tokenAuthrized.POST("/report", client.UploadReport)
}
adminAuthrized := r.Group("/api/admin", api.AdminAuthMiddleware())
{
// clients
adminAuthrized.POST("/addClient", admin.AddClient)
adminAuthrized.POST("/editClient", admin.EditClient)
adminAuthrized.GET("/listClients", admin.ListClients)
adminAuthrized.GET("/getClient", admin.GetClient)
adminAuthrized.POST("/clearRecord", admin.ClearRecord)
adminAuthrized.POST("/removeClient", admin.RemoveClient)
adminAuthrized.GET("/clientToken", admin.GetClientToken)
// settings
adminAuthrized.GET("/settings", admin.GetSettings)
adminAuthrized.POST("/settings", admin.EditSettings)
}
public.Static(r.Group("/"), func(handlers ...gin.HandlerFunc) {
r.NoRoute(handlers...)
})
go func() {
cfg, err := config.Get()
if err != nil {
log.Fatalln("Failed to get config:", err)
}
public.UpdateIndex(cfg)
}()
r.Run(flags.Listen)
RunServer()
},
}
func init() {
ServerCmd.PersistentFlags().StringVarP(&flags.Listen, "listen", "l", "0.0.0.0:25774", "Listen address")
// 从环境变量获取监听地址
listenAddr := GetEnv("KOMARI_LISTEN", "0.0.0.0:25774")
ServerCmd.PersistentFlags().StringVarP(&flags.Listen, "listen", "l", listenAddr, "监听地址 [env: KOMARI_LISTEN]")
RootCmd.AddCommand(ServerCmd)
}
func RunServer() {
// #region 初始化
if err := os.MkdirAll("./data/theme", os.ModePerm); err != nil {
log.Fatalf("Failed to create theme directory: %v", err)
}
InitDatabase()
if utils.VersionHash != "unknown" {
gin.SetMode(gin.ReleaseMode)
}
conf, err := config.GetManyAs[config.Settings]()
if err != nil {
log.Fatal(err)
}
go geoip.InitGeoIp()
go DoScheduledWork()
go messageSender.Initialize()
// oidcInit
go oauth.Initialize()
if conf.NezhaCompatEnabled {
go func() {
if err := nezha.StartNezhaCompat(conf.NezhaCompatListen); err != nil {
log.Printf("Nezha compat server error: %v", err)
auditlog.EventLog("error", fmt.Sprintf("Nezha compat server error: %v", err))
}
}()
}
config.Subscribe(func(event config.ConfigEvent) {
if ok, t := config.IsChangedT[string](event, config.OAuthProviderKey); ok {
if t == "" || t == "none" {
t = "github"
}
oidcProvider, err := database.GetOidcConfigByName(t)
if err != nil {
log.Printf("Failed to get OIDC provider config: %v", err)
} else {
log.Printf("Using %s as OIDC provider", oidcProvider.Name)
}
err = oauth.LoadProvider(oidcProvider.Name, oidcProvider.Addition)
if err != nil {
auditlog.EventLog("error", fmt.Sprintf("Failed to load OIDC provider: %v", err))
}
}
if ok, t := config.IsChangedT[bool](event, config.NezhaCompatEnabledKey); ok {
if t {
l, _ := config.GetAs[string](config.NezhaCompatListenKey)
if err := nezha.StartNezhaCompat(l); err != nil {
log.Printf("start Nezha compat server error: %v", err)
auditlog.EventLog("error", fmt.Sprintf("start Nezha compat server error: %v", err))
}
} else {
if err := nezha.StopNezhaCompat(); err != nil {
log.Printf("stop Nezha compat server error: %v", err)
auditlog.EventLog("error", fmt.Sprintf("stop Nezha compat server error: %v", err))
}
}
}
})
// 初始化 cloudflared
if err := cloudflared.AutoStart(GetEnv("KOMARI_CLOUDFLARED_TOKEN", "")); err != nil {
log.Printf("failed to auto start cloudflared: %v", err)
}
r := gin.New()
r.Use(logutil.GinLogger())
r.Use(logutil.GinRecovery())
config.Subscribe(func(event config.ConfigEvent) {
if event.IsChanged(config.GeoIpProviderKey) {
go geoip.InitGeoIp()
}
if event.IsChanged(config.NotificationMethodKey) {
go messageSender.Initialize()
}
})
r.Use(security.CorsMiddleware(conf.CorsOriginCheckEnabled, conf.CorsAllowedOrigins))
r.Use(api.IdentityMiddleware())
r.Use(api.PrivateSiteMiddleware())
r.Use(func(c *gin.Context) {
if len(c.Request.URL.Path) >= 4 && c.Request.URL.Path[:4] == "/api" {
c.Header("Cache-Control", "no-store")
}
c.Next()
})
router.Register(r)
srv := &http.Server{
Addr: flags.Listen,
Handler: r,
}
log.Printf("Starting server on %s ...", flags.Listen)
go func() {
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
OnFatal(err)
log.Fatalf("listen: %s\n", err)
}
}()
quit := make(chan os.Signal, 1)
signal.Notify(quit, os.Interrupt, syscall.SIGTERM)
<-quit
OnShutdown()
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
log.Fatalf("Server forced to shutdown: %v", err)
}
}
func InitDatabase() {
if !dbcore.InitDatabase() {
var count int64 = 0
if dbcore.GetDBInstance().Model(&models.User{}).Count(&count); count == 0 {
user, passwd, err := accounts.CreateDefaultAdminAccount()
if err != nil {
panic(err)
@@ -121,12 +182,56 @@ func InitDatabase() {
}
}
func DoRecordsWork() {
ticker := time.NewTicker(time.Minute * 30)
records.DeleteRecordBefore(time.Now().Add(-time.Hour * 24 * 30))
records.CompactRecord()
for range ticker.C {
records.DeleteRecordBefore(time.Now().Add(-time.Hour * 24 * 30))
records.CompactRecord()
// #region 定时任务
func DoScheduledWork() {
if err := tasks.ReloadPingSchedule(); err != nil {
log.Println("Failed to reload ping schedule:", err)
}
if err := d_notification.ReloadLoadNotificationSchedule(); err != nil {
log.Println("Failed to reload load notification schedule:", err)
}
records.CompactRecord()
if err := corn.AddFunc("records:cleanup", "@every 30m", cleanupScheduledData); err != nil {
log.Println("Failed to add cleanup scheduled task:", err)
}
if err := corn.AddFunc("records:minute", "@every 1m", minuteScheduledWork); err != nil {
log.Println("Failed to add minute scheduled task:", err)
}
if err := corn.AddFunc("notifier:expire", "0 0 9 * * *", notifier.CheckExpireScheduledWork); err != nil {
log.Println("Failed to add expire notification scheduled task:", err)
}
notifier.InitTrafficReportSchedule()
}
func cleanupScheduledData() {
cfg, _ := config.GetManyAs[config.Settings]()
records.DeleteRecordBefore(time.Now().Add(-time.Hour * time.Duration(cfg.RecordPreserveTime)))
records.CompactRecord()
tasks.ClearTaskResultsByTimeBefore(time.Now().Add(-time.Hour * time.Duration(cfg.RecordPreserveTime)))
tasks.DeletePingRecordsBefore(time.Now().Add(-time.Hour * time.Duration(cfg.PingRecordPreserveTime)))
auditlog.RemoveOldLogs()
accounts.RemoveExpiredSessions()
}
func minuteScheduledWork() {
cfg, _ := config.GetManyAs[config.Settings]()
report_cache.SaveClientReportToDB()
if !cfg.RecordEnabled {
records.DeleteAll()
tasks.DeleteAllPingRecords()
}
// 每分钟检查一次流量提醒
notifier.CheckTraffic()
}
func OnShutdown() {
auditlog.Log("", "", "server is shutting down", "info")
corn.StopAll()
cloudflared.Shutdown()
}
func OnFatal(err error) {
auditlog.Log("", "", "server encountered a fatal error: "+err.Error(), "error")
cloudflared.Shutdown()
}
-111
View File
@@ -1,111 +0,0 @@
package common
import "time"
type Message struct {
Type string `json:"type"`
Content string `json:"content"`
Sender string `json:"sender"`
Timestamp int64 `json:"timestamp"`
}
type ClientConfig struct {
ClientUUID string `json:"client_uuid" gorm:"type:uuid;primaryKey;foreignKey:ClientUUID;references:UUID;constraint:OnDelete:CASCADE"`
CPU bool `json:"cpu" gorm:"default:true"`
GPU bool `json:"gpu" gorm:"default:true"`
RAM bool `json:"ram" gorm:"default:true"`
SWAP bool `json:"swap" gorm:"default:true"`
LOAD bool `json:"load" gorm:"default:true"`
UPTIME bool `json:"uptime" gorm:"default:true"`
TEMP bool `json:"temp" gorm:"default:true"`
OS bool `json:"os" gorm:"default:true"`
DISK bool `json:"disk" gorm:"default:true"`
NET bool `json:"net" gorm:"default:true"`
PROCESS bool `json:"process" gorm:"default:true"`
Connections bool `json:"connections" gorm:"default:true"`
Interval int `json:"interval" gorm:"default:3;check:Interval >= 1"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
// ClientInfo stores static client information
type ClientInfo struct {
UUID string `json:"uuid,omitempty" gorm:"type:uuid;primaryKey;foreignKey:ClientUUID;references:UUID;constraint:OnDelete:CASCADE"`
Name string `json:"name" gorm:"type:varchar(100);not null"`
CpuName string `json:"cpu_name" gorm:"type:varchar(100)"`
Arch string `json:"arch" gorm:"type:varchar(50)"`
CpuCores int `json:"cpu_cores" gorm:"type:int"`
OS string `json:"os" gorm:"type:varchar(100)"`
GpuName string `json:"gpu_name" gorm:"type:varchar(100)"`
IPv4 string `json:"ipv4,omitempty" gorm:"type:varchar(100)"`
IPv6 string `json:"ipv6,omitempty" gorm:"type:varchar(100)"`
Region string `json:"region" gorm:"type:varchar(100)"`
Remark string `json:"remark,omitempty" gorm:"type:longtext"`
MemTotal int64 `json:"mem_total" gorm:"type:bigint"`
SwapTotal int64 `json:"swap_total" gorm:"type:bigint"`
DiskTotal int64 `json:"disk_total" gorm:"type:bigint"`
Version string `json:"version" gorm:"type:varchar(100)"`
Weigth int `json:"weigth" gorm:"type:int"`
CreatedAt time.Time
UpdatedAt time.Time
}
type IPAddress struct {
Ipv4 string `json:"ipv4"`
Ipv6 string `json:"ipv6"`
}
type Report struct {
UUID string `json:"uuid,omitempty"`
CPU CPUReport `json:"cpu"`
Ram RamReport `json:"ram"`
Swap RamReport `json:"swap"`
Load LoadReport `json:"load"`
Disk DiskReport `json:"disk"`
Network NetworkReport `json:"network"`
Connections ConnectionsReport `json:"connections"`
Uptime int64 `json:"uptime"`
Process int `json:"process"`
Message string `json:"message"`
Method string `json:"method,omitempty"`
UpdatedAt time.Time `json:"updated_at"`
}
type CPUReport struct {
Name string `json:"name,omitempty"`
Cores int `json:"cores,omitempty"`
Arch string `json:"arch,omitempty"`
Usage float64 `json:"usage,omitempty"`
}
type GPUReport struct {
Name string `json:"name,omitempty"`
Usage float64 `json:"usage,omitempty"`
}
type RamReport struct {
Total int64 `json:"total"`
Used int64 `json:"used"`
}
type LoadReport struct {
Load1 float64 `json:"load1"`
Load5 float64 `json:"load5"`
Load15 float64 `json:"load15"`
}
type DiskReport struct {
Total int64 `json:"total"`
Used int64 `json:"used"`
}
type NetworkReport struct {
Up int64 `json:"up"`
Down int64 `json:"down"`
TotalUp int64 `json:"totalUp"`
TotalDown int64 `json:"totalDown"`
}
type ConnectionsReport struct {
TCP int `json:"tcp"`
UDP int `json:"udp"`
}
+58
View File
@@ -0,0 +1,58 @@
package accounts
import (
"image"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"github.com/pquerna/otp/totp"
)
var (
TwoFactorIssuer = "Komari Monitor"
)
func Generate2Fa() (string, image.Image, error) {
otp, err := totp.Generate(totp.GenerateOpts{
Issuer: TwoFactorIssuer,
AccountName: "komari",
})
if err != nil {
return "", nil, err
}
img, err := otp.Image(250, 250)
if err != nil {
return "", nil, err
}
return otp.Secret(), img, nil
}
func Enable2Fa(uuid, secret string) error {
db := dbcore.GetDBInstance()
return db.Model(&models.User{}).Where("uuid = ?", uuid).Update("two_factor", secret).Error
}
func Verify2Fa(uuid, code string) (bool, error) {
db := dbcore.GetDBInstance()
var user models.User
err := db.Where("uuid = ?", uuid).First(&user).Error
if err != nil {
return false, err
}
if user.TwoFactor == "" {
return false, nil // 用户未启用2FA
}
valid := totp.Validate(code, user.TwoFactor)
if !valid {
return false, nil
}
return true, nil
}
func Disable2Fa(uuid string) error {
db := dbcore.GetDBInstance()
return db.Model(&models.User{}).Where("uuid = ?", uuid).Update("two_factor", "").Error
}
+52 -4
View File
@@ -100,8 +100,8 @@ func CreateDefaultAdminAccount() (username, passwd string, err error) {
Username: username,
Passwd: hashedPassword,
SSOID: "",
CreatedAt: time.Now(),
UpdatedAt: time.Now(),
CreatedAt: models.FromTime(time.Now()),
UpdatedAt: models.FromTime(time.Now()),
}
err = db.Create(&user).Error
@@ -122,7 +122,7 @@ func GetUserByUUID(uuid string) (user models.User, err error) {
}
// 通过 SSO 信息获取用户
func GetUserBySSO(ssoID, username string) (user models.User, err error) {
func GetUserBySSO(ssoID string) (user models.User, err error) {
db := dbcore.GetDBInstance()
// 首先尝试查找已存在的用户
@@ -131,5 +131,53 @@ func GetUserBySSO(ssoID, username string) (user models.User, err error) {
return user, nil
}
return user, nil
// 如果找不到用户,返回明确的错误信息
return models.User{}, fmt.Errorf("用户不存在:%s", ssoID)
}
func BindingExternalAccount(uuid string, sso_id string) error {
db := dbcore.GetDBInstance()
err := db.Model(&models.User{}).Where("uuid = ?", uuid).Update("sso_id", sso_id).Error
if err != nil {
return err
}
return nil
}
func UnbindExternalAccount(uuid string) error {
db := dbcore.GetDBInstance()
err := db.Model(&models.User{}).Where("uuid = ?", uuid).Update("sso_id", "").Error
if err != nil {
return err
}
return nil
}
func UpdateUser(uuid string, name, password, sso_type *string) error {
db := dbcore.GetDBInstance()
// Check if user exists
var existingUser models.User
result := db.Where("uuid = ?", uuid).First(&existingUser)
if result.Error != nil {
return fmt.Errorf("user not found: %s", uuid)
}
updates := make(map[string]interface{})
if name != nil {
updates["username"] = *name
}
if password != nil {
updates["passwd"] = hashPasswd(*password)
}
if sso_type != nil {
updates["sso_type"] = *sso_type
}
updates["updated_at"] = time.Now()
err := db.Model(&models.User{}).Where("uuid = ?", uuid).Updates(updates).Error
if err != nil {
return err
}
if password != nil {
DeleteAllSessions()
}
return nil
}
+64 -5
View File
@@ -2,11 +2,17 @@ package accounts
import (
"errors"
"fmt"
"net"
"time"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
messageevent "github.com/komari-monitor/komari/database/models/messageEvent"
"github.com/komari-monitor/komari/pkg/config"
"github.com/komari-monitor/komari/utils"
"github.com/komari-monitor/komari/utils/geoip"
"github.com/komari-monitor/komari/utils/messageSender"
)
// GetAllSessions 获取所有会话
@@ -20,15 +26,36 @@ func GetAllSessions() (sessions []models.Session, err error) {
}
// CreateSession 创建新会话
func CreateSession(uuid string, expires int) (string, error) {
func CreateSession(uuid string, expires int, userAgent, ip, login_method string) (string, error) {
db := dbcore.GetDBInstance()
session := utils.GenerateRandomString(32)
sessionRecord := models.Session{
UUID: uuid,
Session: session,
Expires: time.Now().Add(time.Duration(expires) * time.Second),
UUID: uuid,
Session: session,
Expires: models.FromTime(time.Now().Add(time.Duration(expires) * time.Second)),
UserAgent: userAgent,
Ip: ip,
LoginMethod: login_method,
LatestOnline: models.FromTime(time.Now()),
}
go func() {
LoginNotification, _ := config.GetAs[bool](config.LoginNotificationKey, false)
if LoginNotification {
ipAddr := net.ParseIP(ip)
ipinfo, _ := geoip.GetGeoInfo(ipAddr)
loc := "unknown"
if ipinfo != nil && ipinfo.Name != "" {
loc = ipinfo.Name
}
messageSender.SendEvent(models.EventMessage{
Event: messageevent.Login,
Time: time.Now(),
Message: fmt.Sprintf("%s: %s (%s)\n%s", login_method, ip, loc, userAgent),
Emoji: "🔑",
})
}
}()
err := db.Create(&sessionRecord).Error
if err != nil {
@@ -46,7 +73,7 @@ func GetSession(session string) (uuid string, err error) {
return "", err
}
if time.Now().After(sessionRecord.Expires) {
if time.Now().After(sessionRecord.Expires.ToTime()) {
// 会话已过期,删除它
_ = DeleteSession(session)
return "", errors.New("session expired")
@@ -83,3 +110,35 @@ func DeleteAllSessions() error {
}
return nil
}
func UpdateLatestOnline(session string) error {
db := dbcore.GetDBInstance()
return db.Model(&models.Session{}).Where("session = ?", session).Update("latest_online", time.Now()).Error
}
func UpdateLatestUserAgent(session, userAgent string) error {
db := dbcore.GetDBInstance()
return db.Model(&models.Session{}).Where("session = ?", session).Update("latest_user_agent", userAgent).Error
}
func UpdateLatestIp(session, ip string) error {
db := dbcore.GetDBInstance()
return db.Model(&models.Session{}).Where("session = ?", session).Update("latest_ip", ip).Error
}
func UpdateLatest(session, useragent, ip string) error {
db := dbcore.GetDBInstance()
return db.Model(&models.Session{}).Where("session = ?", session).Updates(map[string]interface{}{
"latest_online": time.Now(),
"latest_user_agent": useragent,
"latest_ip": ip,
}).Error
}
func RemoveExpiredSessions() error {
db := dbcore.GetDBInstance()
result := db.Where("expires < ?", time.Now()).Delete(&models.Session{})
if result.Error != nil {
return result.Error
}
return nil
}
+35
View File
@@ -0,0 +1,35 @@
package auditlog
import (
"log"
"time"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
)
func Log(ip, uuid, message, msgType string) {
now := time.Now()
db := dbcore.GetDBInstance()
logEntry := &models.Log{
IP: ip,
UUID: uuid,
Message: message,
MsgType: msgType,
Time: models.FromTime(now),
}
db.Create(logEntry)
}
func EventLog(eventType, message string) {
Log("", "", message, eventType)
}
// Delete logs older than 30 days
func RemoveOldLogs() {
db := dbcore.GetDBInstance()
threshold := time.Now().AddDate(0, 0, -30)
if err := db.Where("time < ?", threshold).Delete(&models.Log{}).Error; err != nil {
log.Println("Failed to remove old logs:", err)
}
}
+153 -89
View File
@@ -1,94 +1,121 @@
package clients
import (
"encoding/json"
"fmt"
"log"
"math"
"time"
"github.com/komari-monitor/komari/common"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/database/tasks"
"github.com/komari-monitor/komari/utils"
"gorm.io/gorm/clause"
"gorm.io/gorm"
"github.com/google/uuid"
)
// Deprecated: DeleteClientConfig is deprecated and will be removed in a future release. Use DeleteClient instead.
func DeleteClientConfig(clientUuid string) error {
db := dbcore.GetDBInstance()
err := db.Delete(&common.ClientConfig{ClientUUID: clientUuid}).Error
if err != nil {
return err
}
return nil
}
func DeleteClient(clientUuid string) error {
db := dbcore.GetDBInstance()
err := db.Delete(&models.Client{}, "uuid = ?", clientUuid).Error
if err != nil {
return err
}
err = db.Delete(&common.ClientInfo{}, "uuid = ?", clientUuid).Error
if err != nil {
return err
}
return nil
}
// 更新或插入客户端基本信息
func UpdateOrInsertBasicInfo(cbi common.ClientInfo) error {
func SaveClientInfo(update map[string]interface{}) error {
db := dbcore.GetDBInstance()
updates := make(map[string]interface{})
clientUUID, ok := update["uuid"].(string)
if !ok || clientUUID == "" {
return fmt.Errorf("invalid client UUID")
}
if cbi.Name != "" {
updates["name"] = cbi.Name
// 确保更新的字段不为空
if len(update) == 0 {
return fmt.Errorf("no fields to update")
}
if cbi.CpuName != "" {
updates["cpu_name"] = cbi.CpuName
}
if cbi.Arch != "" {
updates["arch"] = cbi.Arch
}
if cbi.CpuCores != 0 {
updates["cpu_cores"] = cbi.CpuCores
}
if cbi.OS != "" {
updates["os"] = cbi.OS
}
if cbi.GpuName != "" {
updates["gpu_name"] = cbi.GpuName
}
if cbi.IPv4 != "" {
updates["ipv4"] = cbi.IPv4
}
if cbi.IPv6 != "" {
updates["ipv6"] = cbi.IPv6
}
if cbi.Region != "" {
updates["region"] = cbi.Region
}
if cbi.Remark != "" {
updates["remark"] = cbi.Remark
}
updates["mem_total"] = cbi.MemTotal
updates["swap_total"] = cbi.SwapTotal
updates["disk_total"] = cbi.DiskTotal
updates["version"] = cbi.Version
updates["updated_at"] = time.Now()
err := db.Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "uuid"}},
DoUpdates: clause.Assignments(updates),
}).Create(&cbi).Error
if err != nil {
update["updated_at"] = time.Now()
toFloat64 := func(value interface{}) (float64, bool) {
switch typed := value.(type) {
case float64:
return typed, true
case float32:
return float64(typed), true
case int:
return float64(typed), true
case int8:
return float64(typed), true
case int16:
return float64(typed), true
case int32:
return float64(typed), true
case int64:
return float64(typed), true
case uint:
return float64(typed), true
case uint8:
return float64(typed), true
case uint16:
return float64(typed), true
case uint32:
return float64(typed), true
case uint64:
return float64(typed), true
case json.Number:
parsed, err := typed.Float64()
if err != nil {
return 0, false
}
return parsed, true
default:
return 0, false
}
}
checkOptionalInt := func(name, key string, maxValue float64) error {
value, exists := update[key]
if !exists || value == nil {
return nil
}
numericValue, ok := toFloat64(value)
if !ok {
return fmt.Errorf("%s must be a valid number", name)
}
if numericValue < 0 || numericValue > maxValue {
return fmt.Errorf("%s must be a valid non-negative number: %v", name, value)
}
return nil
}
verify := func(update map[string]interface{}) error {
if err := checkOptionalInt("Cpu.Cores", "cpu_cores", math.MaxInt-1); err != nil {
return err
}
if err := checkOptionalInt("Cpu.PhysicalCores", "cpu_physical_cores", math.MaxInt-1); err != nil {
return err
}
if err := checkOptionalInt("Ram.Total", "mem_total", math.MaxInt64-1); err != nil {
return err
}
if err := checkOptionalInt("Swap.Total", "swap_total", math.MaxInt64-1); err != nil {
return err
}
if err := checkOptionalInt("Disk.Total", "disk_total", math.MaxInt64-1); err != nil {
return err
}
return nil
}
if err := verify(update); err != nil {
return err
}
return nil
}
// 更新客户端设置
func UpdateClientConfig(config common.ClientConfig) error {
db := dbcore.GetDBInstance()
err := db.Save(&config).Error
err := db.Model(&models.Client{}).Where("uuid = ?", clientUUID).Updates(update).Error
if err != nil {
return err
}
@@ -97,28 +124,13 @@ func UpdateClientConfig(config common.ClientConfig) error {
func EditClientName(clientUUID, clientName string) error {
db := dbcore.GetDBInstance()
err := db.Model(&models.Client{}).Where("uuid = ?", clientUUID).Update("client_name", clientName).Error
err := db.Model(&models.Client{}).Where("uuid = ?", clientUUID).Update("name", clientName).Error
if err != nil {
return err
}
return nil
}
/*
// UpdateClientByUUID 更新指定 UUID 的客户端配置
func UpdateClientByUUID(config common.ClientConfig) error {
db := dbcore.GetDBInstance()
result := db.Model(&common.ClientConfig{}).Where("client_uuid = ?", config.ClientUUID).Updates(config)
if result.Error != nil {
return result.Error
}
if result.RowsAffected == 0 {
return gorm.ErrRecordNotFound
}
return nil
}
*/
func EditClientToken(clientUUID, token string) error {
db := dbcore.GetDBInstance()
err := db.Model(&models.Client{}).Where("uuid = ?", clientUUID).Update("token", token).Error
@@ -137,22 +149,43 @@ func CreateClient() (clientUUID, token string, err error) {
client := models.Client{
UUID: clientUUID,
Token: token,
CreatedAt: time.Now(),
UpdatedAt: time.Now(),
Name: "client_" + clientUUID[0:8],
CreatedAt: models.FromTime(time.Now()),
UpdatedAt: models.FromTime(time.Now()),
}
err = db.Create(&client).Error
if err != nil {
return "", "", err
}
clientInfo := common.ClientInfo{
UUID: clientUUID,
Name: "client_" + clientUUID[0:8],
if err := tasks.AddDefaultOnClientUUID(clientUUID); err != nil {
log.Println("Failed to apply default-on ping tasks to new client:", err)
}
err = db.Create(&clientInfo).Error
return clientUUID, token, nil
}
func CreateClientWithName(name string) (clientUUID, token string, err error) {
if name == "" {
return CreateClient()
}
db := dbcore.GetDBInstance()
token = utils.GenerateToken()
clientUUID = uuid.New().String()
client := models.Client{
UUID: clientUUID,
Token: token,
Name: name,
CreatedAt: models.FromTime(time.Now()),
UpdatedAt: models.FromTime(time.Now()),
}
err = db.Create(&client).Error
if err != nil {
return "", "", err
}
if err := tasks.AddDefaultOnClientUUID(clientUUID); err != nil {
log.Println("Failed to apply default-on ping tasks to new client:", err)
}
return clientUUID, token, nil
}
@@ -178,13 +211,15 @@ func GetClientByUUID(uuid string) (client models.Client, err error) {
}
// GetClientBasicInfo 获取指定 UUID 的客户端基本信息
func GetClientBasicInfo(uuid string) (client common.ClientInfo, err error) {
func GetClientBasicInfo(uuid string) (client models.Client, err error) {
db := dbcore.GetDBInstance()
err = db.Where("uuid = ?", uuid).First(&client).Error
if err != nil {
return client, err
if err == gorm.ErrRecordNotFound {
return models.Client{}, fmt.Errorf("客户端不存在: %s", uuid)
}
return models.Client{}, err
}
return client, nil
}
@@ -198,7 +233,7 @@ func GetClientTokenByUUID(uuid string) (token string, err error) {
return client.Token, nil
}
func GetAllClientBasicInfo() (clients []common.ClientInfo, err error) {
func GetAllClientBasicInfo() (clients []models.Client, err error) {
db := dbcore.GetDBInstance()
err = db.Find(&clients).Error
if err != nil {
@@ -206,3 +241,32 @@ func GetAllClientBasicInfo() (clients []common.ClientInfo, err error) {
}
return clients, nil
}
func SaveClient(updates map[string]interface{}) error {
db := dbcore.GetDBInstance()
clientUUID, ok := updates["uuid"].(string)
if !ok || clientUUID == "" {
return fmt.Errorf("invalid client UUID")
}
// 确保更新的字段不为空
if len(updates) == 0 {
return fmt.Errorf("no fields to update")
}
if v, exists := updates["traffic_limit"]; exists {
if val, ok := v.(float64); ok {
if val < 0 || val > math.MaxInt64-1 {
return fmt.Errorf("traffic_limit must be a valid non-negative int64 value, got %v", val)
}
}
}
updates["updated_at"] = time.Now()
err := db.Model(&models.Client{}).Where("uuid = ?", clientUUID).Updates(updates).Error
if err != nil {
return err
}
return nil
}
+184 -23
View File
@@ -2,16 +2,48 @@ package clients
import (
"encoding/json"
"errors"
"fmt"
"math"
"sync"
"time"
"github.com/komari-monitor/komari/common"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
v1 "github.com/komari-monitor/komari/protocol/v1"
"github.com/komari-monitor/komari/utils"
"gorm.io/gorm"
)
var reportSaveLocks sync.Map
func getReportSaveLock(clientUUID string) *sync.Mutex {
lock, _ := reportSaveLocks.LoadOrStore(clientUUID, &sync.Mutex{})
return lock.(*sync.Mutex)
}
func getLatestTrafficRecord(tx *gorm.DB, clientUUID string) (*models.Record, error) {
var record models.Record
err := tx.Where("client = ?", clientUUID).Order("time DESC").First(&record).Error
if err == nil {
return &record, nil
}
if !errors.Is(err, gorm.ErrRecordNotFound) {
return nil, err
}
err = tx.Table("records_long_term").Where("client = ?", clientUUID).Order("time DESC").First(&record).Error
if err == nil {
return &record, nil
}
if !errors.Is(err, gorm.ErrRecordNotFound) {
return nil, err
}
return nil, nil
}
// Report 表示客户端报告数据
// SaveReport 保存报告数据
func SaveReport(uuid string, data map[string]interface{}) (err error) {
@@ -39,49 +71,178 @@ func GetClientUUIDByToken(token string) (clientUUID string, err error) {
return client.UUID, nil
}
func ParseReport(data map[string]interface{}) (report common.Report, err error) {
func ParseReport(data map[string]interface{}) (report v1.Report, err error) {
jsonData, err := json.Marshal(data)
if err != nil {
return common.Report{}, err
return v1.Report{}, err
}
err = json.Unmarshal(jsonData, &report)
if err != nil {
return common.Report{}, err
return v1.Report{}, err
}
return report, nil
}
// 检查数据防止异常数据导致数据库损坏
func ReportVerify(report v1.Report) error {
// 防止输入不合理范围
if report.CPU.Usage < 0 || report.CPU.Usage > 100 {
return fmt.Errorf("CPU.Usage must be between 0 and 100")
}
if report.Load.Load1 < 0 || report.Load.Load1 > 1000 {
return fmt.Errorf("Load.Load1 must be non-negative, got %.2f", report.Load.Load1)
}
checkFloat64 := func(name string, val float64) error {
if val > math.MaxFloat64-1 || val < -math.MaxFloat64+1 {
return fmt.Errorf("%s value exceeds float64 range: %g", name, val)
}
return nil
}
// [float64] 防止数据溢出
if err := checkFloat64("CPU.Usage", report.CPU.Usage); err != nil {
return err
}
if err := checkFloat64("Load.Load1", report.Load.Load1); err != nil {
return err
}
checkInt64 := func(name string, val int64) error {
if val < 0 {
return fmt.Errorf("%s must be non-negative, got %d", name, val)
}
if val > math.MaxInt64-1 {
return fmt.Errorf("%s exceeds int64 max limit: %d", name, val)
}
return nil
}
// [int64] 防止数据溢出
// Ram 验证
if err := checkInt64("Ram.Used", report.Ram.Used); err != nil {
return err
}
if err := checkInt64("Ram.Total", report.Ram.Total); err != nil {
return err
}
// Swap 验证
if err := checkInt64("Swap.Used", report.Swap.Used); err != nil {
return err
}
if err := checkInt64("Swap.Total", report.Swap.Total); err != nil {
return err
}
// Disk 验证
if err := checkInt64("Disk.Used", report.Disk.Used); err != nil {
return err
}
if err := checkInt64("Disk.Total", report.Disk.Total); err != nil {
return err
}
// Network 验证
if err := checkInt64("Network.Up", report.Network.Up); err != nil {
return err
}
if err := checkInt64("Network.Down", report.Network.Down); err != nil {
return err
}
if err := checkInt64("Network.TotalUp", report.Network.TotalUp); err != nil {
return err
}
if err := checkInt64("Network.TotalDown", report.Network.TotalDown); err != nil {
return err
}
// 拒绝所有负数Int
if report.Process < 0 {
return fmt.Errorf("Process must be non-negative: %d", report.Process)
}
if report.Connections.TCP < 0 {
return fmt.Errorf("Connections.TCP must be non-negative: %d", report.Connections.TCP)
}
if report.Connections.UDP < 0 {
return fmt.Errorf("Connections.UDP must be non-negative: %d", report.Connections.UDP)
}
return nil
}
// SaveClientReport 保存客户端报告到 Record 表
func SaveClientReport(clientUUID string, report common.Report) (err error) {
func SaveClientReport(clientUUID string, report v1.Report) (err error) {
db := dbcore.GetDBInstance()
lock := getReportSaveLock(clientUUID)
lock.Lock()
defer lock.Unlock()
if err := ReportVerify(report); err != nil {
return fmt.Errorf("failed to save Record: %w", err)
}
// 保存GPU详细记录到独立表
currentTime := time.Now()
if report.GPU != nil && len(report.GPU.DetailedInfo) > 0 {
for idx, gpu := range report.GPU.DetailedInfo {
gpuRecord := models.GPURecord{
Client: clientUUID,
Time: models.FromTime(currentTime),
DeviceIndex: idx,
DeviceName: gpu.Name,
MemTotal: gpu.MemoryTotal,
MemUsed: gpu.MemoryUsed,
Utilization: float32(gpu.Utilization),
Temperature: gpu.Temperature,
}
if err := db.Create(&gpuRecord).Error; err != nil {
return fmt.Errorf("failed to save GPU record: %w", err)
}
}
}
// 计算平均GPU使用率,用于向后兼容
averageGPUUsage := float32(0)
if report.GPU != nil && len(report.GPU.DetailedInfo) > 0 {
averageGPUUsage = float32(report.GPU.AverageUsage)
}
Record := models.Record{
Client: clientUUID,
Time: time.Now(),
CPU: float32(report.CPU.Usage),
GPU: 0, // Report 未提供 GPU Usage,设为 0(与原 nil 行为类似)
RAM: report.Ram.Used,
RAMTotal: report.Ram.Total,
SWAP: report.Swap.Used,
SWAPTotal: report.Swap.Total,
LOAD: float32(report.Load.Load1), // 使用 Load1 作为主要负载指标
TEMP: 0, // Report 未提供 TEMP,设为 0(与原 nil 行为类似)
DISK: report.Disk.Used,
DISKTotal: report.Disk.Total,
NETIn: report.Network.Down,
NETOut: report.Network.Up,
NETTotalUp: report.Network.TotalUp,
NETTotalDown: report.Network.TotalDown,
PROCESS: report.Process,
Time: models.FromTime(currentTime),
Cpu: float32(report.CPU.Usage),
Gpu: averageGPUUsage, // 使用平均GPU使用率
Ram: report.Ram.Used,
RamTotal: report.Ram.Total,
Swap: report.Swap.Used,
SwapTotal: report.Swap.Total,
Load: float32(report.Load.Load1), // 使用 Load1 作为主要负载指标
Temp: 0, // Report 未提供 TEMP,设为 0(与原 nil 行为类似)
Disk: report.Disk.Used,
DiskTotal: report.Disk.Total,
NetIn: report.Network.Down,
NetOut: report.Network.Up,
NetTotalUp: report.Network.TotalUp,
NetTotalDown: report.Network.TotalDown,
TrafficUp: 0,
TrafficDown: 0,
Process: report.Process,
Connections: report.Connections.TCP,
ConnectionsUDP: report.Connections.UDP,
ConnectionsUdp: report.Connections.UDP,
//Uptime: report.Uptime,
}
// 使用事务确保 Record 和 ClientsInfo 一致性
err = db.Transaction(func(tx *gorm.DB) error {
previous, err := getLatestTrafficRecord(tx, clientUUID)
if err != nil {
return fmt.Errorf("failed to load previous Record: %w", err)
}
if previous != nil {
Record.TrafficUp = utils.ComputeTrafficDelta(report.Network.TotalUp, previous.NetTotalUp)
Record.TrafficDown = utils.ComputeTrafficDelta(report.Network.TotalDown, previous.NetTotalDown)
}
// 保存 Record
if err := tx.Create(&Record).Error; err != nil {
return fmt.Errorf("failed to save Record: %v", err)
return fmt.Errorf("failed to save Record: %w", err)
}
return nil
})
+83
View File
@@ -0,0 +1,83 @@
package clients
import (
"sync"
"testing"
"time"
"github.com/komari-monitor/komari/utils"
"github.com/stretchr/testify/assert"
)
func TestComputeTrafficDeltaHandlesZeroAndReset(t *testing.T) {
tests := []struct {
name string
current int64
previous int64
want int64
}{
{name: "previous zero counts current delta", current: 120, previous: 0, want: 120},
{name: "monotonic counter uses difference", current: 250, previous: 200, want: 50},
{name: "counter reset uses current", current: 15, previous: 250, want: 15},
{name: "negative previous remains guarded", current: 15, previous: -1, want: 0},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.Equal(t, test.want, utils.ComputeTrafficDelta(test.current, test.previous))
})
}
}
func TestReportSaveLockSerializesSameClient(t *testing.T) {
clientUUID := "client-lock-same"
lock := getReportSaveLock(clientUUID)
lock.Lock()
defer lock.Unlock()
acquired := make(chan struct{})
go func() {
secondLock := getReportSaveLock(clientUUID)
secondLock.Lock()
defer secondLock.Unlock()
close(acquired)
}()
select {
case <-acquired:
t.Fatal("same-client report save lock was acquired while already held")
case <-time.After(20 * time.Millisecond):
}
lock.Unlock()
select {
case <-acquired:
case <-time.After(time.Second):
t.Fatal("same-client report save lock was not released")
}
lock.Lock()
}
func TestReportSaveLockAllowsDifferentClients(t *testing.T) {
firstLock := getReportSaveLock("client-lock-a")
firstLock.Lock()
defer firstLock.Unlock()
var waitGroup sync.WaitGroup
waitGroup.Add(1)
acquired := make(chan struct{})
go func() {
defer waitGroup.Done()
secondLock := getReportSaveLock("client-lock-b")
secondLock.Lock()
defer secondLock.Unlock()
close(acquired)
}()
select {
case <-acquired:
case <-time.After(time.Second):
t.Fatal("different-client report save lock should not block")
}
waitGroup.Wait()
}
+58
View File
@@ -0,0 +1,58 @@
package clipboard
import (
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
)
// CreateClipboard 创建剪贴板记录
func CreateClipboard(cb *models.Clipboard) error {
db := dbcore.GetDBInstance()
return db.Create(cb).Error
}
// GetClipboardByID 根据ID获取剪贴板记录
func GetClipboardByID(id int) (*models.Clipboard, error) {
var cb models.Clipboard
db := dbcore.GetDBInstance()
if err := db.First(&cb, id).Error; err != nil {
return nil, err
}
return &cb, nil
}
// UpdateClipboardFields 更新剪贴板记录
func UpdateClipboardFields(id int, fields map[string]interface{}) error {
db := dbcore.GetDBInstance()
return db.Model(&models.Clipboard{}).Where("id = ?", id).Updates(fields).Error
}
// DeleteClipboard 删除剪贴板记录
func DeleteClipboard(id int) error {
db := dbcore.GetDBInstance()
// Check if record exists first
var cb models.Clipboard
if err := db.First(&cb, id).Error; err != nil {
return err // Record not found or other error
}
return db.Delete(&cb).Error
}
// DeleteClipboardBatch 批量删除剪贴板记录
func DeleteClipboardBatch(ids []int) error {
if len(ids) == 0 {
return nil
}
db := dbcore.GetDBInstance()
return db.Where("id IN ?", ids).Delete(&models.Clipboard{}).Error
}
// ListClipboard 列出所有剪贴板记录
func ListClipboard() ([]models.Clipboard, error) {
var list []models.Clipboard
db := dbcore.GetDBInstance()
if err := db.Find(&list).Error; err != nil {
return nil, err
}
return list, nil
}
-36
View File
@@ -1,36 +0,0 @@
package config
import (
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/public"
"gorm.io/gorm"
)
func Get() (models.Config, error) {
db := dbcore.GetDBInstance()
var config models.Config
if err := db.First(&config).Error; err != nil {
if err == gorm.ErrRecordNotFound {
config = models.Config{
ID: 1,
}
Save(config)
return config, nil
}
return config, err
}
return config, nil
}
func Save(cst models.Config) error {
db := dbcore.GetDBInstance()
var config models.Config
// Only one records
config.ID = 1
if err := db.Save(&config).Error; err != nil {
return err
}
go public.UpdateIndex(config)
return nil
}
+314 -30
View File
@@ -1,67 +1,351 @@
package dbcore
import (
"archive/zip"
"fmt"
"io"
"log"
"os"
"path/filepath"
"strings"
"sync"
"time"
"github.com/komari-monitor/komari/cmd/flags"
"github.com/komari-monitor/komari/common"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/pkg/config"
"github.com/komari-monitor/komari/pkg/migrations"
logutil "github.com/komari-monitor/komari/utils/log"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
)
// zipDirectoryExcluding 将 srcDir 打包为 dstZip,exclude 是绝对路径集合需要排除
func zipDirectoryExcluding(srcDir, dstZip string, exclude map[string]struct{}) error {
// 规范化排除路径为绝对路径
normExclude := make(map[string]struct{}, len(exclude))
for p := range exclude {
abs, _ := filepath.Abs(p)
normExclude[abs] = struct{}{}
}
out, err := os.Create(dstZip)
if err != nil {
return err
}
defer out.Close()
zw := zip.NewWriter(out)
defer zw.Close()
absSrc, _ := filepath.Abs(srcDir)
walkErr := filepath.Walk(absSrc, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
// 排除 backup.zip 本身
if _, ok := normExclude[path]; ok {
if info.IsDir() {
return filepath.SkipDir
}
return nil
}
// 计算 zip 内相对路径
rel, err := filepath.Rel(absSrc, path)
if err != nil {
return err
}
// 根目录跳过
if rel == "." {
return nil
}
// 替换为正斜杠
zipName := filepath.ToSlash(rel)
if info.IsDir() {
_, err := zw.Create(zipName + "/")
return err
}
// 普通文件
fh, err := os.Open(path)
if err != nil {
return err
}
w, err := zw.Create(zipName)
if err != nil {
fh.Close()
return err
}
if _, err := io.Copy(w, fh); err != nil {
fh.Close()
return err
}
fh.Close()
return nil
})
if walkErr != nil {
return walkErr
}
return zw.Close()
}
// removeAllInDirExcept 删除 dir 下除 exclude 指定绝对路径外的所有文件和文件夹
func removeAllInDirExcept(dir string, exclude map[string]struct{}) error {
absDir, err := filepath.Abs(dir)
if err != nil {
return err
}
normExclude := make(map[string]struct{}, len(exclude))
for p := range exclude {
abs, _ := filepath.Abs(p)
normExclude[abs] = struct{}{}
}
entries, err := os.ReadDir(absDir)
if err != nil {
return err
}
for _, e := range entries {
full := filepath.Join(absDir, e.Name())
if _, ok := normExclude[full]; ok {
continue
}
if err := os.RemoveAll(full); err != nil {
return err
}
}
return nil
}
// unzipToDir 将 zipPath 解压到 dstDir,包含路径遍历保护
func unzipToDir(zipPath, dstDir string) error {
zr, err := zip.OpenReader(zipPath)
if err != nil {
return err
}
defer zr.Close()
if err := os.MkdirAll(dstDir, 0755); err != nil {
return err
}
absDst, _ := filepath.Abs(dstDir)
for _, f := range zr.File {
// 构造目标路径并做路径遍历保护
cleanName := filepath.Clean(f.Name)
targetPath := filepath.Join(absDst, cleanName)
if !strings.HasPrefix(targetPath, absDst+string(os.PathSeparator)) && targetPath != absDst {
return fmt.Errorf("illegal file path in zip: %s", f.Name)
}
if f.FileInfo().IsDir() {
if err := os.MkdirAll(targetPath, 0755); err != nil {
return err
}
continue
}
if err := os.MkdirAll(filepath.Dir(targetPath), 0755); err != nil {
return err
}
rc, err := f.Open()
if err != nil {
return err
}
out, err := os.Create(targetPath)
if err != nil {
rc.Close()
return err
}
if _, err := io.Copy(out, rc); err != nil {
out.Close()
rc.Close()
return err
}
out.Close()
rc.Close()
}
return nil
}
var (
instance *gorm.DB
once sync.Once
)
// true if database file exists
//
// false if database file does not exist
func InitDatabase() bool {
if _, err := os.Stat(flags.DatabaseFile); os.IsNotExist(err) {
log.Printf("Database file %q does not exist, creating...", flags.DatabaseFile)
dbDir := filepath.Dir(flags.DatabaseFile)
if dbDir != "" {
if err := os.MkdirAll(dbDir, 0755); err != nil {
log.Fatalf("Failed to create directory %q for database file: %v", dbDir, err)
}
}
file, err := os.Create(flags.DatabaseFile)
if err != nil {
log.Fatalf("Failed to create SQLite3 database file %q: %v", flags.DatabaseFile, err)
}
if err := file.Close(); err != nil {
log.Fatalf("Failed to close database file %q: %v", flags.DatabaseFile, err)
}
return false
} else if err != nil {
log.Fatalf("Failed to check database file %q: %v", flags.DatabaseFile, err)
func buildSQLiteDSN(databaseFile string) string {
if databaseFile == "" {
databaseFile = "./data/komari.db"
}
return true
params := "_busy_timeout=5000&_txlock=immediate&_journal_mode=WAL&_synchronous=NORMAL"
separator := "?"
if strings.Contains(databaseFile, "?") {
separator = "&"
}
if strings.HasPrefix(databaseFile, "file:") {
return databaseFile + separator + params
}
if databaseFile == ":memory:" {
return "file::memory:?cache=shared&" + params
}
return "file:" + filepath.ToSlash(databaseFile) + separator + params
}
func GetDBInstance() *gorm.DB {
once.Do(func() {
var err error
instance, err = gorm.Open(sqlite.Open(flags.DatabaseFile), &gorm.Config{})
if err != nil {
log.Fatalf("Failed to connect to SQLite3 database: %v", err)
// 在数据库初始化前执行:如果存在 ./data/backup.zip,则进行恢复逻辑
func() {
backupZipPath := filepath.Join(".", "data", "backup.zip")
if _, statErr := os.Stat(backupZipPath); statErr == nil {
// 4. 把除了 ./data/backup.zip 之外的所有文件压缩到 ./backup/{time}.zip
if err := os.MkdirAll("./backup", 0755); err != nil {
log.Printf("[restore] failed to create backup dir: %v", err)
} else {
tsName := time.Now().Format("20060102-150405")
bakPath := filepath.Join("./backup", fmt.Sprintf("%s.zip", tsName))
if zipErr := zipDirectoryExcluding("./data", bakPath, map[string]struct{}{backupZipPath: {}}); zipErr != nil {
log.Printf("[restore] failed to zip current data: %v", zipErr)
} else {
log.Printf("[restore] current data zipped to %s", bakPath)
}
}
// 5. 删除除了 ./data/backup.zip 之外的所有文件
if delErr := removeAllInDirExcept("./data", map[string]struct{}{backupZipPath: {}}); delErr != nil {
log.Printf("[restore] failed to cleanup data dir: %v", delErr)
}
// 6. 解压 ./data/backup.zip 到 ./data
if unzipErr := unzipToDir(backupZipPath, "./data"); unzipErr != nil {
log.Printf("[restore] failed to unzip backup into data: %v", unzipErr)
} else {
log.Printf("[restore] backup.zip extracted to ./data")
}
// 7. 删除 ./data/backup.zip
if rmErr := os.Remove(backupZipPath); rmErr != nil {
log.Printf("[restore] failed to remove backup.zip: %v", rmErr)
} else {
log.Printf("[restore] backup.zip removed")
}
// 8. 删除标记
if rmErr := os.Remove("./data/komari-backup-markup"); rmErr != nil {
log.Printf("[restore] failed to remove komari-backup-markup: %v", rmErr)
} else {
log.Printf("[restore] komari-backup-markup removed")
}
}
}()
logConfig := &gorm.Config{
Logger: logutil.NewGormLogger(),
}
// 根据数据库类型选择不同的连接方式
switch flags.ApplyDatabaseTypeNormalization() {
case flags.DatabaseTypeSQLite:
// SQLite 连接
// 通过 DSN 传入 _busy_timeout / _txlock 等参数,确保连接池中的每一条连接
// 都生效:
// - _busy_timeout=5000:遇到写锁时最多等待 5s 再返回,避免瞬时
// "database is locked" 直接失败(仅靠后续 PRAGMA Exec 只作用于
// 当时执行该语句的单条连接,池内其它连接不生效)。
// - _txlock=immediate:事务一开始即获取写锁,避免「先 SELECT 后写」
// 的锁升级在并发写入下产生死锁式的立即 SQLITE_BUSY。
// - _journal_mode=WAL / _synchronous=NORMAL:与下方 PRAGMA 保持一致,
// 在 DSN 层为所有连接预设。
dsn := buildSQLiteDSN(flags.DatabaseFile)
instance, err = gorm.Open(sqlite.Open(dsn), logConfig)
if err != nil {
log.Fatalf("Failed to connect to SQLite3 database: %v", err)
}
log.Printf("Using SQLite database file: %s", flags.DatabaseFile)
if sqlDB, dbErr := instance.DB(); dbErr == nil {
// SQLite 同一时刻只允许一个写者;限制连接数可避免连接池层面的写写竞争。
// 负载历史每分钟会执行包含读和写的事务,若连接池允许多个连接,容易与
// ping 结果等短写入撞锁并导致整批负载记录回滚。
sqlDB.SetMaxOpenConns(1)
sqlDB.SetMaxIdleConns(1)
sqlDB.SetConnMaxLifetime(0)
} else {
log.Printf("Failed to access underlying sql.DB for SQLite tuning: %v", dbErr)
}
instance.Exec("PRAGMA wal = ON;")
if err := instance.Exec("PRAGMA journal_mode = WAL;").Error; err != nil {
log.Printf("Failed to enable WAL mode for SQLite: %v", err)
}
instance.Exec("PRAGMA synchronous = NORMAL;")
instance.Exec("PRAGMA busy_timeout = 5000;")
instance.Exec("PRAGMA cache_size = -65536;")
instance.Exec("PRAGMA temp_store = MEMORY;")
instance.Exec("PRAGMA wal_checkpoint(TRUNCATE);")
default:
log.Fatalf("Unsupported database type: %s (supported: %s)", flags.DatabaseType, flags.SupportedDatabaseTypes())
}
if err := migrations.Run(migrations.Context{DB: instance}); err != nil {
log.Fatalf("Failed to run startup migrations: %v", err)
}
config.SetDb(instance)
// 自动迁移模型
err = instance.AutoMigrate(
&models.User{},
&models.Client{},
&models.Session{},
&models.Record{},
&common.ClientInfo{},
&models.Config{},
&models.GPURecord{},
&models.Log{},
&models.Clipboard{},
&models.LoadNotification{},
&models.OfflineNotification{},
&models.TrafficReportNotification{},
&models.PingRecord{},
&models.PingTask{},
&models.OidcProvider{},
&models.MessageSenderProvider{},
&models.ThemeConfiguration{},
)
if err != nil {
log.Fatalf("Failed to create tables: %v", err)
}
err = instance.Table("records_long_term").AutoMigrate(
&models.Record{},
)
if err != nil {
log.Printf("Failed to create records_long_term table, it may already exist: %v", err)
}
err = instance.Table("gpu_records_long_term").AutoMigrate(
&models.GPURecord{},
)
if err != nil {
log.Printf("Failed to create gpu_records_long_term table, it may already exist: %v", err)
}
err = instance.AutoMigrate(
&models.Session{},
)
if err != nil {
log.Printf("Failed to create Session table, it may already exist: %v", err)
}
err = instance.AutoMigrate(
&models.Task{},
&models.TaskResult{},
)
if err != nil {
log.Printf("Failed to create Task and TaskResult table, it may already exist: %v", err)
}
// Manually create composite indexes
if flags.IsSQLite() {
instance.Exec("CREATE INDEX IF NOT EXISTS idx_record_client_time ON records(client, time)")
instance.Exec("CREATE INDEX IF NOT EXISTS idx_record_lt_client_time ON records_long_term(client, time)")
instance.Exec("CREATE INDEX IF NOT EXISTS idx_gpu_record_client_time ON gpu_records(client, time)")
instance.Exec("CREATE INDEX IF NOT EXISTS idx_gpu_record_lt_client_time ON gpu_records_long_term(client, time)")
instance.Exec("CREATE INDEX IF NOT EXISTS idx_ping_record_client_time ON ping_records(client, time)")
}
})
return instance
}
+29
View File
@@ -0,0 +1,29 @@
package database
import (
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
)
func GetAllMessageSenderConfigs() []models.MessageSenderProvider {
db := dbcore.GetDBInstance()
var result []models.MessageSenderProvider
if err := db.Find(&result).Error; err != nil {
return nil
}
return result
}
func GetMessageSenderConfigByName(name string) (*models.MessageSenderProvider, error) {
db := dbcore.GetDBInstance()
var config models.MessageSenderProvider
if err := db.Where("name = ?", name).First(&config).Error; err != nil {
return nil, err
}
return &config, nil
}
func SaveMessageSenderConfig(config *models.MessageSenderProvider) error {
db := dbcore.GetDBInstance()
return db.Save(config).Error
}
+121
View File
@@ -0,0 +1,121 @@
package models
import (
"database/sql/driver"
"fmt"
"log"
"os"
"strings"
"sync"
"time"
)
// LocalTime is a custom time type for GORM to handle time.Time correctly.
// It ensures that all times are stored and retrieved based on the application's configured timezone.
type LocalTime time.Time
// Value implements the driver.Valuer interface.
// It converts UTCTime to a TEXT format that the database understands.
// The time is formatted as a string in the application's local timezone, without timezone information.
func (t LocalTime) Value() (driver.Value, error) {
if time.Time(t).IsZero() {
return nil, nil
}
return time.Time(t).In(GetAppLocation()).Format("2006-01-02 15:04:05.0000000"), nil
}
// Scan implements the sql.Scanner interface.
// It scans a value from the database into a UTCTime object.
func (t *LocalTime) Scan(v interface{}) error {
if v == nil {
*t = LocalTime(time.Time{})
return nil
}
loc := GetAppLocation()
switch val := v.(type) {
case time.Time:
// CRITICAL FIX: When the driver reads a timezone-less string (e.g., "16:00:00"),
// it often incorrectly assumes it's UTC. We must correct this by re-interpreting
// the date and clock values in the application's actual timezone.
year, month, day := val.Date()
hour, min, sec := val.Clock()
nanosec := val.Nanosecond()
*t = LocalTime(time.Date(year, month, day, hour, min, sec, nanosec, loc))
return nil
case []byte:
return t.parseTime(string(val), loc)
case string:
return t.parseTime(val, loc)
default:
return fmt.Errorf("UTCTime scan source was not string, []byte or time.Time: %T (%v)", v, v)
}
}
// parseTime handles parsing a string into UTCTime.
func (t *LocalTime) parseTime(timeStr string, loc *time.Location) error {
timeStr = strings.TrimSpace(timeStr)
if timeStr == "" {
*t = LocalTime(time.Time{})
return nil
}
layouts := []string{
time.RFC3339Nano, time.RFC3339,
"2006-01-02 15:04:05.0000000-07:00", "2006-01-02 15:04:05-07:00",
"2006-01-02 15:04:05.0000000", "2006-01-02 15:04:05", "2006-01-02",
}
for _, layout := range layouts {
if parsedTime, err := time.ParseInLocation(layout, timeStr, loc); err == nil {
*t = LocalTime(parsedTime)
return nil
}
}
return fmt.Errorf("unable to parse time string '%s' into UTCTime", timeStr)
}
// MarshalJSON implements the json.Marshaler interface.
// Serializes UTCTime to JSON in RFC3339 format with the correct timezone offset.
func (t LocalTime) MarshalJSON() ([]byte, error) {
if time.Time(t).IsZero() {
return []byte("null"), nil
}
formattedTime := time.Time(t).In(GetAppLocation()).Format(time.RFC3339)
return []byte(fmt.Sprintf(`"%s"`, formattedTime)), nil
}
// ToTime converts UTCTime to Go's native time.Time type.
func (t LocalTime) ToTime() time.Time { return time.Time(t) }
// FromTime converts Go's native time.Time type to UTCTime.
func FromTime(t time.Time) LocalTime { return LocalTime(t) }
// Now returns the current time in the application's configured location.
func Now() LocalTime { return LocalTime(time.Now().In(GetAppLocation())) }
var (
appLocation *time.Location
locationOnce sync.Once
)
// GetAppLocation retrieves the application's global timezone from the "TZ" environment variable.
func GetAppLocation() *time.Location {
locationOnce.Do(func() {
tz := os.Getenv("TZ")
if tz == "" {
tz = "UTC"
}
loc, err := time.LoadLocation(tz)
if err != nil {
log.Printf("Warning: Failed to load timezone '%s', falling back to UTC. Error: %v", tz, err)
appLocation = time.UTC
} else {
appLocation = loc
}
time.Local = appLocation
log.Printf("Application timezone is set to '%s'.", appLocation.String())
})
return appLocation
}
+11
View File
@@ -0,0 +1,11 @@
package models
type Clipboard struct {
Id int `json:"id" gorm:"primaryKey;autoIncrement;unique"`
Text string `json:"text" gorm:"type:longtext"`
Name string `json:"name" gorm:"type:varchar(255)"`
Weight int `json:"weight" gorm:"type:int"`
Remark string `json:"remark" gorm:"type:text"`
CreatedAt LocalTime `json:"created_at"`
UpdatedAt LocalTime `json:"updated_at"`
}
+10
View File
@@ -0,0 +1,10 @@
package models
type Log struct {
ID uint `json:"id,omitempty" gorm:"primaryKey;autoIncrement"`
IP string `json:"ip" gorm:"type:varchar(45);"` // IPv4 or IPv6
UUID string `json:"uuid" gorm:"type:varchar(36);"`
Message string `json:"message" gorm:"type:text;not null"`
MsgType string `json:"msg_type" gorm:"type:varchar(20);not null"`
Time LocalTime `json:"time" gorm:"autoCreateTime;not null"`
}
@@ -0,0 +1,14 @@
package messageevent
const (
Offline = "Offline"
Online = "Online"
Expire = "Expire"
Renew = "Renew"
Login = "Login"
Alert = "Alert"
Traffic = "Traffic"
DReport = "DReport" // 日报
WReport = "WReport" // 周报
MReport = "MReport" // 月报
)
+16
View File
@@ -0,0 +1,16 @@
package models
import "time"
type MessageSenderProvider struct {
Name string `json:"name" gorm:"primaryKey;unique;not null"`
Addition string `json:"addition" gorm:"type:longtext" default:"{}"`
}
type EventMessage struct {
Event string `json:"event"`
Clients []Client `json:"clients"`
Time time.Time `json:"time"`
Message string `json:"message"`
Emoji string `json:"emoji"`
}
+111 -48
View File
@@ -1,74 +1,137 @@
package models
import (
"time"
"database/sql/driver"
"encoding/json"
"fmt"
)
// Client represents a registered client device
type Client struct {
UUID string `json:"uuid,omitempty" gorm:"type:uuid;primaryKey"`
Token string `json:"token,omitempty" gorm:"type:varchar(255);unique;not null"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
UUID string `json:"uuid,omitempty" gorm:"type:varchar(36);primaryKey"`
Token string `json:"token,omitempty" gorm:"type:varchar(255);unique;not null"`
Name string `json:"name" gorm:"type:varchar(100)"`
CpuName string `json:"cpu_name" gorm:"type:varchar(100)"`
Virtualization string `json:"virtualization" gorm:"type:varchar(50)"`
Arch string `json:"arch" gorm:"type:varchar(50)"`
CpuCores int `json:"cpu_cores" gorm:"type:int"`
CpuPhysicalCores int `json:"cpu_physical_cores" gorm:"type:int"`
OS string `json:"os" gorm:"type:varchar(100)"`
KernelVersion string `json:"kernel_version" gorm:"type:varchar(100)"`
GpuName string `json:"gpu_name" gorm:"type:varchar(100)"`
IPv4 string `json:"ipv4,omitempty" gorm:"type:varchar(100)"`
IPv6 string `json:"ipv6,omitempty" gorm:"type:varchar(100)"`
Region string `json:"region" gorm:"type:varchar(100)"`
Remark string `json:"remark,omitempty" gorm:"type:longtext"`
PublicRemark string `json:"public_remark,omitempty" gorm:"type:longtext"`
MemTotal int64 `json:"mem_total" gorm:"type:bigint"`
SwapTotal int64 `json:"swap_total" gorm:"type:bigint"`
DiskTotal int64 `json:"disk_total" gorm:"type:bigint"`
Version string `json:"version,omitempty" gorm:"type:varchar(100)"`
Weight int `json:"weight" gorm:"type:int"`
Price float64 `json:"price"`
BillingCycle int `json:"billing_cycle"`
AutoRenewal bool `json:"auto_renewal" gorm:"default:false"` // 是否自动续费
Currency string `json:"currency" gorm:"type:varchar(20);default:'$'"`
ExpiredAt LocalTime `json:"expired_at" gorm:"type:timestamp"`
Group string `json:"group" gorm:"type:varchar(100)"`
Tags string `json:"tags" gorm:"type:text"` // split by ';'
Hidden bool `json:"hidden" gorm:"default:false"`
TrafficLimit int64 `json:"traffic_limit" gorm:"type:bigint"`
TrafficLimitType string `json:"traffic_limit_type" gorm:"type:varchar(10);default:'max'"` // 流量阈值类型:sum max min up down
CreatedAt LocalTime `json:"created_at"`
UpdatedAt LocalTime `json:"updated_at"`
}
// User represents an authenticated user
type User struct {
UUID string `json:"uuid,omitempty" gorm:"type:uuid;primaryKey"`
UUID string `json:"uuid,omitempty" gorm:"type:varchar(36);primaryKey"`
Username string `json:"username" gorm:"type:varchar(50);unique;not null"`
Passwd string `json:"passwd,omitempty" gorm:"type:varchar(255);not null"` // Hashed password
SSOType string `json:"sso_type" gorm:"type:varchar(20)"` // e.g., "github", "google"
SSOID string `json:"sso_id" gorm:"type:varchar(100)"` // OAuth provider's user ID
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
TwoFactor string `json:"two_factor,omitempty" gorm:"type:varchar(255)"` // 2FA secret
Sessions []Session `json:"sessions,omitempty" gorm:"foreignKey:UUID;references:UUID;constraint:OnDelete:CASCADE,OnUpdate:CASCADE"`
CreatedAt LocalTime `json:"created_at"`
UpdatedAt LocalTime `json:"updated_at"`
}
// Session manages user sessions
type Session struct {
UUID string `gorm:"type:uuid;foreignKey:UserUUID;references:UUID;constraint:OnDelete:CASCADE"`
Session string `gorm:"type:varchar(255);unique;not null"`
Expires time.Time `gorm:"not null"`
CreatedAt time.Time
UUID string `json:"uuid" gorm:"type:varchar(36)"`
Session string `json:"session" gorm:"type:varchar(255);primaryKey;uniqueIndex:idx_sessions_session;not null"`
UserAgent string `json:"user_agent" gorm:"type:text"`
Ip string `json:"ip" gorm:"type:varchar(100)"`
LoginMethod string `json:"login_method" gorm:"type:varchar(50)"`
LatestOnline LocalTime `json:"latest_online" gorm:"type:timestamp"`
LatestUserAgent string `json:"latest_user_agent" gorm:"type:text"`
LatestIp string `json:"latest_ip" gorm:"type:varchar(100)"`
Expires LocalTime `json:"expires" gorm:"not null"`
CreatedAt LocalTime `json:"created_at"`
}
// Record logs client metrics over time
type Record struct {
Client string `gorm:"type:uuid;index;foreignKey:ClientUUID;references:UUID;constraint:OnDelete:CASCADE"`
Time time.Time `gorm:"index;default:CURRENT_TIMESTAMP"`
CPU float32 `gorm:"type:decimal(5,2)"` // e.g., 75.50%
GPU float32 `gorm:"type:decimal(5,2)"`
RAM int64 `gorm:"type:bigint"`
RAMTotal int64 `gorm:"type:bigint"`
SWAP int64 `gorm:"type:bigint"`
SWAPTotal int64 `gorm:"type:bigint"`
LOAD float32 `gorm:"type:decimal(5,2)"`
TEMP float32 `gorm:"type:decimal(5,2)"`
DISK int64 `gorm:"type:bigint"`
DISKTotal int64 `gorm:"type:bigint"`
NETIn int64 `gorm:"type:bigint"`
NETOut int64 `gorm:"type:bigint"`
NETTotalUp int64 `gorm:"type:bigint"`
NETTotalDown int64 `gorm:"type:bigint"`
PROCESS int
Connections int
ConnectionsUDP int
Client string `json:"client" gorm:"type:varchar(36);index"`
Time LocalTime `json:"time" gorm:"index"`
Cpu float32 `json:"cpu" gorm:"type:decimal(5,2)"` // e.g., 75.50%
Gpu float32 `json:"gpu" gorm:"type:decimal(5,2)"`
Ram int64 `json:"ram" gorm:"type:bigint"`
RamTotal int64 `json:"ram_total" gorm:"type:bigint"`
Swap int64 `json:"swap" gorm:"type:bigint"`
SwapTotal int64 `json:"swap_total" gorm:"type:bigint"`
Load float32 `json:"load" gorm:"type:decimal(5,2)"`
Temp float32 `json:"temp" gorm:"type:decimal(5,2)"`
Disk int64 `json:"disk" gorm:"type:bigint"`
DiskTotal int64 `json:"disk_total" gorm:"type:bigint"`
NetIn int64 `json:"net_in" gorm:"type:bigint"`
NetOut int64 `json:"net_out" gorm:"type:bigint"`
NetTotalUp int64 `json:"net_total_up" gorm:"type:bigint"`
NetTotalDown int64 `json:"net_total_down" gorm:"type:bigint"`
TrafficUp int64 `json:"traffic_up" gorm:"type:bigint"`
TrafficDown int64 `json:"traffic_down" gorm:"type:bigint"`
Process int `json:"process"`
Connections int `json:"connections"`
ConnectionsUdp int `json:"connections_udp"`
//Uptime int64 `json:"uptime" gorm:"type:bigint"`
}
// Config stores site-wide settings
type Config struct {
ID uint `json:"id,omitempty" gorm:"primaryKey;autoIncrement"`
Sitename string `json:"sitename" gorm:"type:varchar(100);not null"`
Description string `json:"desc" gorm:"type:text"`
AllowCros bool `json:"allow_cros" gorm:"default:false"`
// GeoIP 配置
GeoIpEnabled bool `json:"geoip_enable" gorm:"default:true"`
GeoIpProvider string `json:"geoip_provider" gorm:"type:varchar(20);default:'mmdb'"` // mmdb, bilibili, ip-api. 暂时只实现了mmdb
// OAuth 配置
OAuthClientID string `json:"oauth_id" gorm:"type:varchar(255);not null"`
OAuthClientSecret string `json:"oauth_secret" gorm:"type:varchar(255);not null"`
OAuthEnabled bool `json:"oauth_enable" gorm:"default:false"`
// 自定义美化
CustomHead string `json:"custom_head" gorm:"type:longtext"`
CreatedAt time.Time
UpdatedAt time.Time
// GPURecord logs individual GPU metrics over time
type GPURecord struct {
Client string `json:"client" gorm:"type:varchar(36);index"` // 客户端UUID
Time LocalTime `json:"time" gorm:"index"` // 记录时间
DeviceIndex int `json:"device_index" gorm:"index"` // GPU设备索引 (0,1,2...)
DeviceName string `json:"device_name" gorm:"type:varchar(100)"` // GPU型号
MemTotal int64 `json:"mem_total" gorm:"type:bigint"` // 显存总量(字节)
MemUsed int64 `json:"mem_used" gorm:"type:bigint"` // 显存使用(字节)
Utilization float32 `json:"utilization" gorm:"type:decimal(5,2)"` // GPU使用率(%)
Temperature int `json:"temperature"` // GPU温度(°C)
}
// StringArray represents a slice of strings stored as JSON in the database
// StringArray 存储为 JSON 的字符串切片类型
type StringArray []string
func (sa *StringArray) Scan(value interface{}) error {
var bytes []byte
switch v := value.(type) {
case nil:
*sa = StringArray{}
return nil
case []byte:
bytes = v
case string:
bytes = []byte(v)
default:
return fmt.Errorf("failed to scan StringArray: unsupported value type %T", value)
}
if len(bytes) == 0 {
*sa = StringArray{}
return nil
}
return json.Unmarshal(bytes, sa)
}
func (sa StringArray) Value() (driver.Value, error) {
return json.Marshal(sa)
}
+30
View File
@@ -0,0 +1,30 @@
package models
import "testing"
func TestStringArrayScanAcceptsSQLiteTextShapes(t *testing.T) {
tests := map[string]interface{}{
"bytes": []byte(`["a","b"]`),
"text": `["a","b"]`,
"empty": "",
"nil": nil,
}
for name, input := range tests {
t.Run(name, func(t *testing.T) {
var got StringArray
if err := got.Scan(input); err != nil {
t.Fatalf("scan StringArray: %v", err)
}
if input == "" || input == nil {
if len(got) != 0 {
t.Fatalf("expected empty StringArray, got %v", got)
}
return
}
if len(got) != 2 || got[0] != "a" || got[1] != "b" {
t.Fatalf("unexpected StringArray: %v", got)
}
})
}
}
+33
View File
@@ -0,0 +1,33 @@
package models
// Notification 定义了通知相关的数据库模型
type OfflineNotification struct {
Client string `json:"client" gorm:"type:varchar(36);not null;index;unique;constraint:OnDelete:CASCADE,OnUpdate:CASCADE;foreignKey:client;references:UUID"`
ClientInfo Client `json:"client_info,omitempty" gorm:"foreignKey:Client;references:UUID"`
Enable bool `json:"enable" gorm:"type:boolean;default:false"`
//Cooldown int `json:"cooldown" gorm:"type:int;not null;default:1800"` // 冷却时间(秒),默认 30 分钟
GracePeriod int `json:"grace_period" gorm:"type:int;not null;default:180"` // 宽限期(秒),默认 3 分钟
LastNotified LocalTime `json:"last_notified"` // 上次通知时间
}
// LoadNotification 定义了基于资源占用达标时间比的负载通知规则
type LoadNotification struct {
Id uint `json:"id,omitempty" gorm:"primaryKey;autoIncrement"`
Name string `json:"name" gorm:"type:varchar(255)"`
Clients StringArray `json:"clients" gorm:"type:longtext"`
Metric string `json:"metric" gorm:"type:varchar(50);not null;default:'cpu'"` // 监控指标,如 cpu, ram, load
Threshold float32 `json:"threshold" gorm:"type:decimal(5,2);not null;default:80.00"` // 阈值百分比
Ratio float32 `json:"ratio" gorm:"type:decimal(5,2);not null;default:0.80"` // 达标时间比
Interval int `json:"interval" gorm:"type:int;not null;default:15"` // 监测间隔(分钟)
LastNotified LocalTime `json:"last_notified"` // 上次通知时间
}
// TrafficReportNotification 定义了流量定时报告的数据库模型
type TrafficReportNotification struct {
Client string `json:"client" gorm:"type:varchar(36);not null;index;unique;constraint:OnDelete:CASCADE,OnUpdate:CASCADE;foreignKey:client;references:UUID"`
ClientInfo Client `json:"client_info,omitempty" gorm:"foreignKey:Client;references:UUID"`
Enable bool `json:"enable" gorm:"type:boolean;default:false"`
Daily bool `json:"daily" gorm:"type:boolean;default:false"` // 日报
Weekly bool `json:"weekly" gorm:"type:boolean;default:false"` // 周报
Monthly bool `json:"monthly" gorm:"type:boolean;default:false"` // 月报
}
+6
View File
@@ -0,0 +1,6 @@
package models
type OidcProvider struct {
Name string `json:"name" gorm:"primaryKey;unique;not null"`
Addition string `json:"addition" gorm:"type:longtext" default:"{}"`
}
+35
View File
@@ -0,0 +1,35 @@
package models
type PingRecord struct {
Client string `json:"client" gorm:"type:varchar(36);not null;index"`
ClientInfo Client `json:"client_info" gorm:"foreignKey:Client;references:UUID;constraint:OnDelete:CASCADE,OnUpdate:CASCADE"`
TaskId uint `json:"task_id" gorm:"not null;index"`
Task PingTask `json:"task" gorm:"foreignKey:TaskId;references:Id;constraint:OnDelete:CASCADE,OnUpdate:CASCADE;"`
Time LocalTime `json:"time" gorm:"index;not null"`
Value int `json:"value" gorm:"type:int;not null"` // Ping 值,单位毫秒
}
// PingTask 表示一次延迟监测任务配置。
type PingTask struct {
Id uint `json:"id,omitempty" gorm:"primaryKey;autoIncrement"`
Weight int `json:"weight" gorm:"type:int;not null;default:0;index"`
Name string `json:"name" gorm:"type:varchar(255);not null;index"`
Clients StringArray `json:"clients" gorm:"type:longtext"`
DefaultOn bool `json:"default_on" gorm:"column:all_clients;not null;default:false"` // 新加入的服务器是否自动开启此监测;现有服务器不受此字段影响
Type string `json:"type" gorm:"type:varchar(12);not null;default:'icmp'"` // icmp tcp http
Target string `json:"target" gorm:"type:varchar(255);not null"` // Ping 目标地址
Interval int `json:"interval" gorm:"type:int;not null;default:60"` // 间隔时间
}
// AppliesToClient 判断当前 PingTask 是否适用于指定服务器。
func (task PingTask) AppliesToClient(uuid string) bool {
if uuid == "" {
return false
}
for _, client := range task.Clients {
if client == uuid {
return true
}
}
return false
}
+18
View File
@@ -0,0 +1,18 @@
package models
type Task struct {
TaskId string `json:"task_id" gorm:"type:varchar(36);primaryKey;unique"`
Clients StringArray `json:"clients" gorm:"type:longtext"`
Command string `json:"command" gorm:"type:text"`
Results []TaskResult `gorm:"foreignKey:TaskId;references:TaskId;constraint:OnDelete:CASCADE,OnUpdate:CASCADE"`
}
type TaskResult struct {
TaskId string `json:"task_id" gorm:"type:varchar(36);index"`
Client string `json:"client" gorm:"type:varchar(36)"`
ClientInfo Client `json:"client_info" gorm:"foreignKey:Client;references:UUID;constraint:OnDelete:CASCADE,OnUpdate:CASCADE"`
Result string `json:"result" gorm:"type:longtext"`
ExitCode *int `json:"exit_code" gorm:"type:int"`
FinishedAt *LocalTime `json:"finished_at" gorm:"type:timestamp"`
CreatedAt LocalTime `json:"created_at" gorm:"type:timestamp"`
}
+146
View File
@@ -0,0 +1,146 @@
package models
import (
"fmt"
"net/url"
"path"
"strings"
)
const (
ThemeConfigurationManaged = "managed"
ThemeConfigurationRaw = "raw"
ThemeConfigurationRedirect = "redirect"
)
// Theme represents a komari theme information
type Theme struct {
Name string `json:"name"` // 主题名称
Short string `json:"short"` // 短名称,用作文件夹名
Description string `json:"description"` // 主题描述
Version string `json:"version"` // 版本号
Author string `json:"author"` // 作者
URL string `json:"url"` // 主题URL
Preview string `json:"preview"` // 预览图片相对路径
Configuration Configuration `json:"configuration"` // 声明配置项
}
type Configuration struct {
Type string `json:"type"` // managed raw redirect
Icon string `json:"icon"` // 图标
Name any `json:"name"`
Data any `json:"data"` // 配置数据
}
type ManagedThemeConfigurationItem struct {
Key string `json:"key"`
Name any `json:"name"`
Required bool `json:"required"`
Type string `json:"type"` // string number select switch title
Options string `json:"options"`
Default any `json:"default"`
Help any `json:"help"`
}
type ThemeConfiguration struct {
Short string `json:"short" gorm:"primaryKey;unique;not null"`
Data string `json:"data" gorm:"type:longtext" default:"{}"`
}
func (t Theme) ConfigurationType() string {
typ := strings.ToLower(strings.TrimSpace(t.Configuration.Type))
if typ == "" {
return ThemeConfigurationManaged
}
return typ
}
func (t Theme) RawHTML() (string, bool) {
if t.ConfigurationType() != ThemeConfigurationRaw {
return "", false
}
return configurationDataString(t.Configuration.Data)
}
func (t Theme) RedirectTarget() (string, bool) {
if t.ConfigurationType() != ThemeConfigurationRedirect {
return "", false
}
return NormalizeThemeRedirectTarget(t.Configuration.Data)
}
func (t Theme) ValidateConfiguration() error {
switch t.ConfigurationType() {
case ThemeConfigurationManaged:
return nil
case ThemeConfigurationRaw:
html, ok := t.RawHTML()
if !ok || strings.TrimSpace(html) == "" {
return fmt.Errorf("raw 类型主题需要在 configuration.data 中提供 HTML 字符串")
}
return nil
case ThemeConfigurationRedirect:
if _, ok := t.RedirectTarget(); !ok {
return fmt.Errorf("redirect 类型主题需要在 configuration.data 中提供站内相对路径")
}
return nil
default:
return fmt.Errorf("不支持的主题类型: %s", t.Configuration.Type)
}
}
func NormalizeThemeRedirectTarget(data any) (string, bool) {
target, ok := configurationDataString(data)
if !ok {
return "", false
}
target = strings.TrimSpace(target)
if target == "" || strings.Contains(target, "\\") || strings.HasPrefix(target, "//") {
return "", false
}
parsed, err := url.Parse(target)
if err != nil || parsed.IsAbs() || parsed.Host != "" {
return "", false
}
cleanInputPath := parsed.Path
if strings.HasPrefix(cleanInputPath, "/") {
cleanInputPath = strings.TrimLeft(cleanInputPath, "/")
} else {
for strings.HasPrefix(cleanInputPath, "../") {
cleanInputPath = strings.TrimPrefix(cleanInputPath, "../")
}
}
for _, segment := range strings.Split(cleanInputPath, "/") {
if segment == ".." {
return "", false
}
}
cleanPath := cleanInputPath
if cleanPath == "" {
cleanPath = "/"
} else {
cleanPath = path.Clean(cleanPath)
if cleanPath == "." {
cleanPath = "/"
} else {
cleanPath = "/" + strings.TrimPrefix(cleanPath, "/")
}
}
normalized := url.URL{
Path: cleanPath,
RawQuery: parsed.RawQuery,
Fragment: parsed.Fragment,
}
return normalized.String(), true
}
func configurationDataString(data any) (string, bool) {
value, ok := data.(string)
return value, ok
}
+141
View File
@@ -0,0 +1,141 @@
package models
import "testing"
func TestThemeValidateConfiguration(t *testing.T) {
tests := map[string]struct {
theme Theme
wantErr bool
}{
"managed empty data": {
theme: Theme{
Configuration: Configuration{Type: ThemeConfigurationManaged},
},
},
"missing type defaults to managed": {
theme: Theme{},
},
"raw html": {
theme: Theme{
Configuration: Configuration{
Type: ThemeConfigurationRaw,
Data: "<!doctype html><title>raw</title>",
},
},
},
"raw rejects empty html": {
theme: Theme{
Configuration: Configuration{
Type: ThemeConfigurationRaw,
Data: " ",
},
},
wantErr: true,
},
"redirect relative path": {
theme: Theme{
Configuration: Configuration{
Type: ThemeConfigurationRedirect,
Data: "/dashboard?tab=nodes",
},
},
},
"redirect rejects absolute url": {
theme: Theme{
Configuration: Configuration{
Type: ThemeConfigurationRedirect,
Data: "https://example.com",
},
},
wantErr: true,
},
"redirect rejects parent traversal": {
theme: Theme{
Configuration: Configuration{
Type: ThemeConfigurationRedirect,
Data: "/../admin",
},
},
wantErr: true,
},
"unknown type": {
theme: Theme{
Configuration: Configuration{Type: "hosted"},
},
wantErr: true,
},
}
for name, tt := range tests {
t.Run(name, func(t *testing.T) {
err := tt.theme.ValidateConfiguration()
if tt.wantErr && err == nil {
t.Fatal("expected error, got nil")
}
if !tt.wantErr && err != nil {
t.Fatalf("unexpected error: %v", err)
}
})
}
}
func TestNormalizeThemeRedirectTarget(t *testing.T) {
tests := map[string]struct {
input string
want string
ok bool
}{
"leading slash": {
input: "/nodes",
want: "/nodes",
ok: true,
},
"missing leading slash": {
input: "nodes",
want: "/nodes",
ok: true,
},
"parent prefix points to root path": {
input: "../settings",
want: "/settings",
ok: true,
},
"multiple parent prefixes point to root path": {
input: "../../settings/site",
want: "/settings/site",
ok: true,
},
"query and fragment": {
input: "nodes?id=1#cpu",
want: "/nodes?id=1#cpu",
ok: true,
},
"absolute parent traversal": {
input: "/../admin",
},
"nested parent traversal": {
input: "nodes/../admin",
},
"protocol relative": {
input: "//example.com",
},
"backslash": {
input: `\admin`,
},
"absolute url": {
input: "https://example.com",
},
}
for name, tt := range tests {
t.Run(name, func(t *testing.T) {
got, ok := NormalizeThemeRedirectTarget(tt.input)
if ok != tt.ok {
t.Fatalf("ok = %v, want %v", ok, tt.ok)
}
if got != tt.want {
t.Fatalf("target = %q, want %q", got, tt.want)
}
})
}
}
+68
View File
@@ -0,0 +1,68 @@
package notification
import (
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/utils/notifier"
"gorm.io/gorm"
)
func AddLoadNotification(clients []string, name string, metric string, threshold float32, ratio float32, interval int) (uint, error) {
db := dbcore.GetDBInstance()
notification := models.LoadNotification{
Clients: clients,
Name: name,
Metric: metric,
Threshold: threshold,
Ratio: ratio,
Interval: interval,
}
if err := db.Create(&notification).Error; err != nil {
return 0, err
}
return notification.Id, ReloadLoadNotificationSchedule()
}
func DeleteLoadNotification(id []uint) error {
db := dbcore.GetDBInstance()
result := db.Where("id IN ?", id).Delete(&models.LoadNotification{})
if result.RowsAffected == 0 {
return gorm.ErrRecordNotFound
}
return ReloadLoadNotificationSchedule()
}
func EditLoadNotification(notifications []*models.LoadNotification) error {
db := dbcore.GetDBInstance()
for _, notification := range notifications {
result := db.Model(&models.LoadNotification{}).Where("id = ?", notification.Id).Updates(notification)
if result.RowsAffected == 0 {
return gorm.ErrRecordNotFound
}
}
return ReloadLoadNotificationSchedule()
}
func GetAllLoadNotifications() ([]models.LoadNotification, error) {
db := dbcore.GetDBInstance()
var notifications []models.LoadNotification
if err := db.Find(&notifications).Error; err != nil {
return nil, err
}
return notifications, nil
}
func SaveLoadNotification(record models.LoadNotification) error {
db := dbcore.GetDBInstance()
return db.Create(&record).Error
}
func ReloadLoadNotificationSchedule() error {
db := dbcore.GetDBInstance()
var loadNotifications []models.LoadNotification
if err := db.Find(&loadNotifications).Error; err != nil {
return err
}
return notifier.ReloadLoadNotificationSchedule(loadNotifications)
}
+144
View File
@@ -0,0 +1,144 @@
package notification
import (
"fmt"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"gorm.io/gorm/clause"
)
func validateTrafficReportNotification(notification models.TrafficReportNotification) error {
if notification.Client == "" {
return fmt.Errorf("client UUID cannot be empty")
}
if notification.Enable && !notification.Daily && !notification.Weekly && !notification.Monthly {
return fmt.Errorf("at least one cadence must be selected when enabling traffic reports")
}
return nil
}
func ValidateTrafficReportNotifications(notifications []models.TrafficReportNotification) error {
for _, notification := range notifications {
if err := validateTrafficReportNotification(notification); err != nil {
return err
}
}
return nil
}
func buildEnabledTrafficReportNotifications(uuids []string, existing []models.TrafficReportNotification) ([]models.TrafficReportNotification, error) {
if len(uuids) == 0 {
return nil, fmt.Errorf("at least one client UUID is required")
}
existingByClient := make(map[string]models.TrafficReportNotification, len(existing))
for _, notification := range existing {
existingByClient[notification.Client] = notification
}
notifications := make([]models.TrafficReportNotification, 0, len(uuids))
for _, uuid := range uuids {
if uuid == "" {
return nil, fmt.Errorf("client UUID cannot be empty")
}
existingNotification, ok := existingByClient[uuid]
if !ok || (!existingNotification.Daily && !existingNotification.Weekly && !existingNotification.Monthly) {
return nil, fmt.Errorf("at least one cadence must be selected when enabling traffic reports")
}
notifications = append(notifications, models.TrafficReportNotification{
Client: uuid,
Enable: true,
})
}
return notifications, nil
}
// ListTrafficReportNotifications 获取所有流量定时报告配置(关联客户端信息)
func ListTrafficReportNotifications() ([]models.TrafficReportNotification, error) {
db := dbcore.GetDBInstance()
var notifications []models.TrafficReportNotification
err := db.Model(&models.TrafficReportNotification{}).Preload("ClientInfo").Find(&notifications).Error
return notifications, err
}
// EditTrafficReportNotifications 批量更新流量定时报告配置
func EditTrafficReportNotifications(notifications []models.TrafficReportNotification) error {
if err := ValidateTrafficReportNotifications(notifications); err != nil {
return err
}
db := dbcore.GetDBInstance()
return db.Model(&models.TrafficReportNotification{}).
Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "client"}},
DoUpdates: clause.AssignmentColumns([]string{"enable", "daily", "weekly", "monthly"}),
}).
Select("client", "enable", "daily", "weekly", "monthly").
Create(notifications).Error
}
// EnableTrafficReportNotifications 批量启用(仅更新 enable 字段)
func EnableTrafficReportNotifications(uuids []string) error {
if len(uuids) == 0 {
return fmt.Errorf("at least one client UUID is required")
}
db := dbcore.GetDBInstance()
var existing []models.TrafficReportNotification
if err := db.Where("client IN ?", uuids).Find(&existing).Error; err != nil {
return err
}
notifications, err := buildEnabledTrafficReportNotifications(uuids, existing)
if err != nil {
return err
}
return db.Model(&models.TrafficReportNotification{}).
Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "client"}},
DoUpdates: clause.AssignmentColumns([]string{"enable"}),
}).
Select("client", "enable").
Create(notifications).Error
}
// DisableTrafficReportNotifications 批量禁用
func DisableTrafficReportNotifications(uuids []string) error {
if len(uuids) == 0 {
return fmt.Errorf("at least one client UUID is required")
}
db := dbcore.GetDBInstance()
var notifications []models.TrafficReportNotification
for _, uuid := range uuids {
if uuid == "" {
return fmt.Errorf("client UUID cannot be empty")
}
notifications = append(notifications, models.TrafficReportNotification{
Client: uuid,
Enable: false,
})
}
return db.Model(&models.TrafficReportNotification{}).
Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "client"}},
DoUpdates: clause.AssignmentColumns([]string{"enable"}),
}).
Select("client", "enable").
Create(notifications).Error
}
// GetEnabledTrafficReportByType 获取启用了指定类型报告的客户端配置
func GetEnabledTrafficReportByType(daily, weekly, monthly bool) ([]models.TrafficReportNotification, error) {
db := dbcore.GetDBInstance()
var notifications []models.TrafficReportNotification
query := db.Model(&models.TrafficReportNotification{}).Where("enable = ?", true)
if daily {
query = query.Where("daily = ?", true)
} else if weekly {
query = query.Where("weekly = ?", true)
} else if monthly {
query = query.Where("monthly = ?", true)
}
err := query.Find(&notifications).Error
return notifications, err
}
@@ -0,0 +1,36 @@
package notification
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/komari-monitor/komari/database/models"
)
func TestValidateTrafficReportNotificationsRejectsEnabledWithoutCadence(t *testing.T) {
err := ValidateTrafficReportNotifications([]models.TrafficReportNotification{{
Client: "client-a",
Enable: true,
}})
assert.Error(t, err)
}
func TestBuildEnabledTrafficReportNotificationsRequiresExistingCadence(t *testing.T) {
_, err := buildEnabledTrafficReportNotifications([]string{"client-a"}, nil)
assert.Error(t, err)
_, err = buildEnabledTrafficReportNotifications([]string{"client-a"}, []models.TrafficReportNotification{{Client: "client-a"}})
assert.Error(t, err)
notifications, err := buildEnabledTrafficReportNotifications([]string{"client-a"}, []models.TrafficReportNotification{{
Client: "client-a",
Daily: true,
}})
require.NoError(t, err)
require.Len(t, notifications, 1)
assert.Equal(t, "client-a", notifications[0].Client)
assert.True(t, notifications[0].Enable)
}
+32
View File
@@ -0,0 +1,32 @@
package database
import (
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
)
func GetAllOidcConfigs() []models.OidcProvider {
db := dbcore.GetDBInstance()
var result []models.OidcProvider
if err := db.Find(&result).Error; err != nil {
return nil
}
return result
}
func GetOidcConfigByName(name string) (*models.OidcProvider, error) {
db := dbcore.GetDBInstance()
var config models.OidcProvider
if err := db.Where("name = ?", name).First(&config).Error; err != nil {
return nil, err
}
return &config, nil
}
func SaveOidcConfig(config *models.OidcProvider) error {
db := dbcore.GetDBInstance()
if err := db.Save(config).Error; err != nil {
return err
}
return nil
}
+534 -153
View File
@@ -1,13 +1,18 @@
package records
import (
"errors"
"log"
"sort"
"strings"
"time"
"gorm.io/gorm"
"github.com/komari-monitor/komari/cmd/flags"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/utils"
)
func RecordOne(rec models.Record) error {
@@ -15,9 +20,59 @@ func RecordOne(rec models.Record) error {
return db.Create(&rec).Error
}
func RecordGPU(rec models.GPURecord) error {
db := dbcore.GetDBInstance()
return db.Create(&rec).Error
}
func DeleteAll() error {
db := dbcore.GetDBInstance()
return db.Exec("DELETE FROM Record").Error
if err := db.Exec("DELETE FROM records_long_term").Error; err != nil {
return err
}
if err := db.Exec("DELETE FROM gpu_records_long_term").Error; err != nil {
return err
}
if err := db.Exec("DELETE FROM gpu_records").Error; err != nil {
return err
}
return db.Exec("DELETE FROM records").Error
}
// GetGPURecordsByClientAndTime 获取GPU记录数据
func GetGPURecordsByClientAndTime(uuid string, start, end time.Time) ([]models.GPURecord, error) {
db := dbcore.GetDBInstance()
var records []models.GPURecord
fourHoursAgo := time.Now().Add(-4*time.Hour - time.Minute)
var recentRecords []models.GPURecord
recentStart := start
if end.After(fourHoursAgo) {
if recentStart.Before(fourHoursAgo) {
recentStart = fourHoursAgo
}
err := db.Where("client = ? AND time >= ? AND time <= ?", uuid, recentStart, end).
Order("time ASC, device_index ASC").Find(&recentRecords).Error
if err != nil {
log.Printf("Error fetching recent GPU records for client %s between %s and %s: %v", uuid, recentStart, end, err)
return nil, err
}
}
var longTermRecords []models.GPURecord
err := db.Table("gpu_records_long_term").Where("client = ? AND time >= ? AND time <= ?", uuid, start, end).
Order("time ASC, device_index ASC").Find(&longTermRecords).Error
if err != nil {
log.Printf("Error fetching long-term GPU records for client %s between %s and %s: %v", uuid, start, end, err)
return recentRecords, nil
}
// 合并结果 - 不再需要类型转换
records = append(records, recentRecords...)
records = append(records, longTermRecords...)
return records, nil
}
func GetLatestRecord(uuid string) (Record []models.Record, err error) {
@@ -28,201 +83,527 @@ func GetLatestRecord(uuid string) (Record []models.Record, err error) {
func DeleteRecordBefore(before time.Time) error {
db := dbcore.GetDBInstance()
db.Table("records_long_term").Where("time < ?", before).Delete(&models.Record{})
db.Table("gpu_records_long_term").Where("time < ?", before).Delete(&models.GPURecord{})
db.Where("time < ?", before).Delete(&models.GPURecord{})
return db.Where("time < ?", before).Delete(&models.Record{}).Error
}
// 计算区间 [0.02, 0.98] 的平均值
func QuantileMean(values []float32) float32 {
if len(values) == 0 {
return 0
}
// 排序数据
sorted := make([]float32, len(values))
copy(sorted, values)
for i := 0; i < len(sorted)-1; i++ {
for j := i + 1; j < len(sorted); j++ {
if sorted[i] > sorted[j] {
sorted[i], sorted[j] = sorted[j], sorted[i]
}
func GetRecordsByClientAndTime(uuid string, start, end time.Time) ([]models.Record, error) {
db := dbcore.GetDBInstance()
var records []models.Record
fourHoursAgo := time.Now().Add(-4*time.Hour - time.Minute)
var recentRecords []models.Record
recentStart := start
if end.After(fourHoursAgo) {
if recentStart.Before(fourHoursAgo) {
recentStart = fourHoursAgo
}
err := db.Where("client = ? AND time >= ? AND time <= ?", uuid, recentStart, end).Order("time ASC").Find(&recentRecords).Error
if err != nil {
log.Printf("Error fetching recent records for client %s between %s and %s: %v", uuid, recentStart, end, err)
return nil, err
}
}
// 区间边缘index
lower := int(0.02 * float64(len(sorted)))
upper := int(0.98 * float64(len(sorted)))
if lower >= upper || lower >= len(sorted) {
return 0
var long_term []models.Record
err := db.Table("records_long_term").Where("client = ? AND time >= ? AND time <= ?", uuid, start, end).Order("time ASC").Find(&long_term).Error
if err != nil {
log.Printf("Error fetching long-term records for client %s between %s and %s: %v", uuid, start, end, err)
return recentRecords, nil
}
sum := float32(0)
count := 0
for i := lower; i < upper && i < len(sorted); i++ {
sum += sorted[i]
count++
if len(long_term) == 0 {
// 没有查到long_term,返回全部recentRecords
records = append(records, recentRecords...)
return records, nil
}
if count == 0 {
return 0
// 查到了long_term,recentRecords按15分钟分组,每组只保留一条(取最新一条)
grouped := make(map[string]models.Record)
for _, rec := range recentRecords {
key := rec.Time.ToTime().Truncate(15 * time.Minute).Format(time.RFC3339)
if old, ok := grouped[key]; !ok || rec.Time.ToTime().After(old.Time.ToTime()) {
grouped[key] = rec
}
}
return sum / float32(count)
var groupedList []models.Record
for _, rec := range grouped {
groupedList = append(groupedList, rec)
}
sort.Slice(groupedList, func(i, j int) bool {
return groupedList[i].Time.ToTime().Before(groupedList[j].Time.ToTime())
})
records = append(records, groupedList...)
records = append(records, long_term...)
return records, nil
}
// 压缩数据库,针对每个ClientUUID,10分钟内的数据不动,24小时内的数据精简为每15分钟一条,3天为每30分钟一条,7天为每1小时一条,30天为每12小时一条。所有精简的数据是取 [0.02,0.98] 区间内的平均值
func GetAllRecords() ([]models.Record, error) {
db := dbcore.GetDBInstance()
var records []models.Record
var long_term []models.Record
err := db.Table("records").Order("time ASC").Find(&records).Error
if err != nil {
log.Printf("Error fetching all records: %v", err)
return nil, err
}
err = db.Table("records_long_term").Order("time ASC").Find(&long_term).Error
if err != nil {
log.Printf("Error fetching long-term records: %v", err)
return records, nil
}
records = append(records, long_term...)
return records, nil
}
// 压缩数据库
func CompactRecord() error {
db := dbcore.GetDBInstance()
log.Printf("Compacting records...")
var clientUUIDs []string
if err := db.Model(&models.Record{}).Distinct("client").Pluck("client", &clientUUIDs).Error; err != nil {
err := migrateOldRecords(db)
if err != nil {
log.Printf("Error migrating old records: %v", err)
return err
}
now := time.Now()
tenMinutesAgo := now.Add(-10 * time.Minute)
oneDayAgo := now.Add(-24 * time.Hour)
threeDaysAgo := now.Add(-3 * 24 * time.Hour)
sevenDaysAgo := now.Add(-7 * 24 * time.Hour)
thirtyDaysAgo := now.Add(-30 * 24 * time.Hour)
for _, clientUUID := range clientUUIDs {
// Process each time range with specific aggregation intervals
if err := processTimeRange(db, clientUUID, thirtyDaysAgo, sevenDaysAgo, 12*time.Hour, now); err != nil {
log.Printf("Error compacting 30d-7d for client %s: %v", clientUUID, err)
continue
}
if err := processTimeRange(db, clientUUID, sevenDaysAgo, threeDaysAgo, 1*time.Hour, now); err != nil {
log.Printf("Error compacting 7d-3d for client %s: %v", clientUUID, err)
continue
}
if err := processTimeRange(db, clientUUID, threeDaysAgo, oneDayAgo, 30*time.Minute, now); err != nil {
log.Printf("Error compacting 3d-1d for client %s: %v", clientUUID, err)
continue
}
if err := processTimeRange(db, clientUUID, oneDayAgo, tenMinutesAgo, 15*time.Minute, now); err != nil {
log.Printf("Error compacting 1d-10m for client %s: %v", clientUUID, err)
continue
}
err = migrateGPURecords(db)
if err != nil {
log.Printf("Error migrating GPU records: %v", err)
return err
}
if err := db.Exec("VACUUM").Error; err != nil {
log.Printf("Error vacuuming the database: %v", err)
if flags.IsSQLite() {
db.Exec("PRAGMA wal_checkpoint(PASSIVE);")
}
log.Printf("Record compaction completed")
//log.Printf("Record compaction completed")
return nil
}
func processTimeRange(db *gorm.DB, clientUUID string, start, end time.Time, interval time.Duration, now time.Time) error {
// Round start time to the nearest interval
start = start.Truncate(interval)
end = end.Truncate(interval)
func migrateOldRecords(db *gorm.DB) error {
return migrateOldRecordsAt(db, time.Now())
}
// Get all records in the time range
func compactRecordCutoff(now time.Time) time.Time {
return now.Add(-4 * time.Hour).Truncate(15 * time.Minute)
}
func migrateOldRecordsAt(db *gorm.DB, now time.Time) error {
cutoff := compactRecordCutoff(now)
// 查询 records 表中超过 4 小时的记录
var records []models.Record
if err := db.Where("client = ? AND time >= ? AND time < ?", clientUUID, start, end).
Order("time ASC").
Find(&records).Error; err != nil {
if err := db.Table("records").Where("time < ?", cutoff).Find(&records).Error; err != nil {
return err
}
if len(records) == 0 {
return nil
}
// Group records by interval
type aggregatedRecord struct {
StartTime time.Time
Records []models.Record
previousByClient, err := getPreviousTrafficRecordsBefore(db, records)
if err != nil {
return err
}
var aggregatedRecords []aggregatedRecord
currentGroup := aggregatedRecord{StartTime: start}
currentGroup.Records = make([]models.Record, 0)
repairZeroTrafficDeltas(records, previousByClient)
// 按 Client 和 15 分钟时间段分组,并存储所有记录以计算分位数
type groupData struct {
Cpu []float32
Gpu []float32
Load []float32
Temp []float32
Ram []int64
RamTotal []int64
Swap []int64
SwapTotal []int64
Disk []int64
DiskTotal []int64
NetIn []int64
NetOut []int64
NetTotalUp []int64
NetTotalDown []int64
TrafficUp int64
TrafficDown int64
LatestTime time.Time
LatestTotalUp int64
LatestTotalDown int64
Process []int
Connections []int
ConnectionsUdp []int
Uptime []int64
}
groupedRecords := make(map[string]*groupData)
for _, record := range records {
groupTime := record.Time.Truncate(interval)
if !groupTime.Equal(currentGroup.StartTime) {
if len(currentGroup.Records) > 0 {
aggregatedRecords = append(aggregatedRecords, currentGroup)
key := record.Client + "_" + record.Time.ToTime().Truncate(15*time.Minute).Format(time.RFC3339)
if _, ok := groupedRecords[key]; !ok {
groupedRecords[key] = &groupData{}
}
data := groupedRecords[key]
data.Cpu = append(data.Cpu, record.Cpu)
data.Gpu = append(data.Gpu, record.Gpu)
data.Load = append(data.Load, record.Load)
data.Temp = append(data.Temp, record.Temp)
data.Ram = append(data.Ram, record.Ram)
data.RamTotal = append(data.RamTotal, record.RamTotal)
data.Swap = append(data.Swap, record.Swap)
data.SwapTotal = append(data.SwapTotal, record.SwapTotal)
data.Disk = append(data.Disk, record.Disk)
data.DiskTotal = append(data.DiskTotal, record.DiskTotal)
data.NetIn = append(data.NetIn, record.NetIn)
data.NetOut = append(data.NetOut, record.NetOut)
data.NetTotalUp = append(data.NetTotalUp, record.NetTotalUp)
data.NetTotalDown = append(data.NetTotalDown, record.NetTotalDown)
data.TrafficUp += record.TrafficUp
data.TrafficDown += record.TrafficDown
if data.LatestTime.IsZero() || record.Time.ToTime().After(data.LatestTime) {
data.LatestTime = record.Time.ToTime()
data.LatestTotalUp = record.NetTotalUp
data.LatestTotalDown = record.NetTotalDown
}
data.Process = append(data.Process, record.Process)
data.Connections = append(data.Connections, record.Connections)
data.ConnectionsUdp = append(data.ConnectionsUdp, record.ConnectionsUdp)
//data.Uptime = append(data.Uptime, record.Uptime)
}
getPercentile := func(values []float64, percentile float64) float64 {
if len(values) == 0 {
return 0
}
sortedValues := make([]float64, len(values))
copy(sortedValues, values)
sort.Float64s(sortedValues)
index := float64(len(sortedValues)-1) * percentile
lowerIndex := int(index)
if lowerIndex >= len(sortedValues)-1 {
return sortedValues[len(sortedValues)-1]
}
frac := index - float64(lowerIndex)
return sortedValues[lowerIndex] + frac*(sortedValues[lowerIndex+1]-sortedValues[lowerIndex])
}
getIntPercentile := func(values []int64, percentile float64) int64 {
if len(values) == 0 {
return 0
}
floats := make([]float64, len(values))
for i, v := range values {
floats[i] = float64(v)
}
return int64(getPercentile(floats, percentile))
}
getInt32Percentile := func(values []int, percentile float64) int {
if len(values) == 0 {
return 0
}
floats := make([]float64, len(values))
for i, v := range values {
floats[i] = float64(v)
}
return int(getPercentile(floats, percentile))
}
return db.Transaction(func(tx *gorm.DB) error {
for key, data := range groupedRecords {
// 解析 Client 和时间
parts := strings.Split(key, "_")
clientUUID := parts[0]
timeSlot, err := time.Parse(time.RFC3339, strings.Join(parts[1:], "_"))
if err != nil {
return err
}
currentGroup = aggregatedRecord{StartTime: groupTime}
currentGroup.Records = []models.Record{record}
} else {
currentGroup.Records = append(currentGroup.Records, record)
cpuFloats := make([]float64, len(data.Cpu))
for i, v := range data.Cpu {
cpuFloats[i] = float64(v)
}
gpuFloats := make([]float64, len(data.Gpu))
for i, v := range data.Gpu {
gpuFloats[i] = float64(v)
}
loadFloats := make([]float64, len(data.Load))
for i, v := range data.Load {
loadFloats[i] = float64(v)
}
tempFloats := make([]float64, len(data.Temp))
for i, v := range data.Temp {
tempFloats[i] = float64(v)
}
// 取高位
high_percentile := 0.7
// 检查 records_long_term 表中是否已存在相同的记录
var existingCount int64
if err := tx.Table("records_long_term").Where("client = ? AND time = ?", clientUUID, timeSlot).Count(&existingCount).Error; err != nil {
return err
}
newRec := models.Record{
Client: clientUUID,
Time: models.FromTime(timeSlot),
Cpu: float32(getPercentile(cpuFloats, high_percentile)),
Gpu: float32(getPercentile(gpuFloats, high_percentile)),
Load: float32(getPercentile(loadFloats, high_percentile)),
Temp: float32(getPercentile(tempFloats, high_percentile)),
Ram: getIntPercentile(data.Ram, high_percentile),
RamTotal: getIntPercentile(data.RamTotal, high_percentile),
Swap: getIntPercentile(data.Swap, high_percentile),
SwapTotal: getIntPercentile(data.SwapTotal, high_percentile),
Disk: getIntPercentile(data.Disk, high_percentile),
DiskTotal: getIntPercentile(data.DiskTotal, high_percentile),
NetIn: getIntPercentile(data.NetIn, 0.2),
NetOut: getIntPercentile(data.NetOut, 0.2),
NetTotalUp: data.LatestTotalUp,
NetTotalDown: data.LatestTotalDown,
TrafficUp: data.TrafficUp,
TrafficDown: data.TrafficDown,
Process: getInt32Percentile(data.Process, high_percentile),
Connections: getInt32Percentile(data.Connections, high_percentile),
ConnectionsUdp: getInt32Percentile(data.ConnectionsUdp, high_percentile),
//Uptime: getIntPercentile(data.Uptime, high_percentile),
}
// 如果记录已存在则更新,否则创建新记录
if existingCount > 0 {
if err := tx.Table("records_long_term").Where("client = ? AND time = ?", clientUUID, timeSlot).Updates(&newRec).Error; err != nil {
return err
}
} else {
if err := tx.Table("records_long_term").Create(&newRec).Error; err != nil {
return err
}
}
}
// 删除 records 表中的旧数据
if err := tx.Table("records").Where("time < ?", cutoff.Add(-1*time.Hour)).Delete(&models.Record{}).Error; err != nil {
return err
}
return nil
})
}
func getPreviousTrafficRecordsBefore(db *gorm.DB, records []models.Record) (map[string]*models.Record, error) {
firstTimeByClient := make(map[string]time.Time)
for _, record := range records {
recordTime := record.Time.ToTime()
firstTime, ok := firstTimeByClient[record.Client]
if !ok || recordTime.Before(firstTime) {
firstTimeByClient[record.Client] = recordTime
}
}
if len(currentGroup.Records) > 0 {
aggregatedRecords = append(aggregatedRecords, currentGroup)
previousByClient := make(map[string]*models.Record, len(firstTimeByClient))
for clientUUID, firstTime := range firstTimeByClient {
previous, err := getPreviousTrafficRecordBefore(db, clientUUID, firstTime)
if err != nil {
return nil, err
}
if previous != nil {
previousByClient[clientUUID] = previous
}
}
return previousByClient, nil
}
func getPreviousTrafficRecordBefore(db *gorm.DB, clientUUID string, before time.Time) (*models.Record, error) {
var latest *models.Record
for _, table := range []string{"records", "records_long_term"} {
var record models.Record
queryBefore := before
if table == "records_long_term" {
queryBefore = before.Truncate(15 * time.Minute)
}
err := db.Table(table).
Where("client = ? AND time < ?", clientUUID, models.FromTime(queryBefore)).
Order("time DESC").
First(&record).Error
if err == nil {
if latest == nil || record.Time.ToTime().After(latest.Time.ToTime()) {
latest = &record
}
continue
}
if !errors.Is(err, gorm.ErrRecordNotFound) {
return nil, err
}
}
return latest, nil
}
func repairZeroTrafficDeltas(records []models.Record, previousByClient map[string]*models.Record) {
recordsByClient := make(map[string][]*models.Record)
for i := range records {
recordsByClient[records[i].Client] = append(recordsByClient[records[i].Client], &records[i])
}
// Begin transaction
tx := db.Begin()
if tx.Error != nil {
return tx.Error
for _, clientRecords := range recordsByClient {
sort.Slice(clientRecords, func(i, j int) bool {
return clientRecords[i].Time.ToTime().Before(clientRecords[j].Time.ToTime())
})
var previous *models.Record
if previousByClient != nil {
previous = previousByClient[clientRecords[0].Client]
}
for _, current := range clientRecords {
if previous == nil {
previous = current
continue
}
if current.TrafficUp == 0 {
current.TrafficUp = utils.ComputeTrafficDelta(current.NetTotalUp, previous.NetTotalUp)
}
if current.TrafficDown == 0 {
current.TrafficDown = utils.ComputeTrafficDelta(current.NetTotalDown, previous.NetTotalDown)
}
previous = current
}
}
}
// Delete original records in the time range
if err := tx.Where("client = ? AND time >= ? AND time < ?", clientUUID, start, end).Delete(&models.Record{}).Error; err != nil {
tx.Rollback()
// migrateGPURecords 压缩GPU记录数据
func migrateGPURecords(db *gorm.DB) error {
cutoff := compactRecordCutoff(time.Now())
// 查询超过4小时的GPU记录
var gpuRecords []models.GPURecord
if err := db.Where("time < ?", cutoff).Find(&gpuRecords).Error; err != nil {
return err
}
// Insert aggregated records
for _, agg := range aggregatedRecords {
if len(agg.Records) == 0 {
continue
}
// Calculate averages and sums
var sumCPU, sumGPU, sumLOAD, sumTEMP float32
var sumRAM, sumRAMTotal, sumSWAP, sumSWAPTotal, sumDISK, sumDISKTotal int64
var sumNETIn, sumNETOut, sumNETTotalUp, sumNETTotalDown int64
var sumPROCESS, sumConnections, sumConnectionsUDP int
count := len(agg.Records)
for _, r := range agg.Records {
sumCPU += r.CPU
sumGPU += r.GPU
sumLOAD += r.LOAD
sumTEMP += r.TEMP
sumRAM += r.RAM
sumRAMTotal += r.RAMTotal
sumSWAP += r.SWAP
sumSWAPTotal += r.SWAPTotal
sumDISK += r.DISK
sumDISKTotal += r.DISKTotal
sumNETIn += r.NETIn
sumNETOut += r.NETOut
sumNETTotalUp += r.NETTotalUp
sumNETTotalDown += r.NETTotalDown
sumPROCESS += r.PROCESS
sumConnections += r.Connections
sumConnectionsUDP += r.ConnectionsUDP
}
// Create new aggregated record
newRecord := models.Record{
Client: clientUUID,
Time: agg.StartTime,
CPU: sumCPU / float32(count),
GPU: sumGPU / float32(count),
RAM: sumRAM / int64(count),
RAMTotal: sumRAMTotal / int64(count),
SWAP: sumSWAP / int64(count),
SWAPTotal: sumSWAPTotal / int64(count),
LOAD: sumLOAD / float32(count),
TEMP: sumTEMP / float32(count),
DISK: sumDISK / int64(count),
DISKTotal: sumDISKTotal / int64(count),
NETIn: sumNETIn / int64(count),
NETOut: sumNETOut / int64(count),
NETTotalUp: sumNETTotalUp / int64(count),
NETTotalDown: sumNETTotalDown / int64(count),
PROCESS: sumPROCESS / count,
Connections: sumConnections / count,
ConnectionsUDP: sumConnectionsUDP / count,
}
if err := tx.Create(&newRecord).Error; err != nil {
tx.Rollback()
return err
}
if len(gpuRecords) == 0 {
return nil
}
return tx.Commit().Error
// 按Client + DeviceIndex + 15分钟时间窗口分组
type gpuGroupKey struct {
Client string
DeviceIndex int
TimeSlot time.Time
DeviceName string
}
type gpuGroupData struct {
MemTotal []int64
MemUsed []int64
Utilization []float32
Temperature []int
}
groupedGPUs := make(map[gpuGroupKey]*gpuGroupData)
for _, record := range gpuRecords {
key := gpuGroupKey{
Client: record.Client,
DeviceIndex: record.DeviceIndex,
TimeSlot: record.Time.ToTime().Truncate(15 * time.Minute),
DeviceName: record.DeviceName,
}
if _, ok := groupedGPUs[key]; !ok {
groupedGPUs[key] = &gpuGroupData{}
}
data := groupedGPUs[key]
data.MemTotal = append(data.MemTotal, record.MemTotal)
data.MemUsed = append(data.MemUsed, record.MemUsed)
data.Utilization = append(data.Utilization, record.Utilization)
data.Temperature = append(data.Temperature, record.Temperature)
}
// 百分位数计算函数 (复用传统Record压缩逻辑)
getPercentile := func(values []float64, percentile float64) float64 {
if len(values) == 0 {
return 0
}
sortedValues := make([]float64, len(values))
copy(sortedValues, values)
sort.Float64s(sortedValues)
index := float64(len(sortedValues)-1) * percentile
lowerIndex := int(index)
if lowerIndex >= len(sortedValues)-1 {
return sortedValues[len(sortedValues)-1]
}
frac := index - float64(lowerIndex)
return sortedValues[lowerIndex] + frac*(sortedValues[lowerIndex+1]-sortedValues[lowerIndex])
}
getIntPercentile := func(values []int64, percentile float64) int64 {
if len(values) == 0 {
return 0
}
floats := make([]float64, len(values))
for i, v := range values {
floats[i] = float64(v)
}
return int64(getPercentile(floats, percentile))
}
// 温度数据转换辅助函数
convertIntToInt64 := func(values []int) []int64 {
result := make([]int64, len(values))
for i, v := range values {
result[i] = int64(v)
}
return result
}
getFloat32Percentile := func(values []float32, percentile float64) float32 {
if len(values) == 0 {
return 0
}
floats := make([]float64, len(values))
for i, v := range values {
floats[i] = float64(v)
}
return float32(getPercentile(floats, percentile))
}
// 保持与传统Record压缩的一致性
high_percentile := 0.7
return db.Transaction(func(tx *gorm.DB) error {
for key, data := range groupedGPUs {
// 检查是否已存在记录
var existingCount int64
if err := tx.Table("gpu_records_long_term").Where("client = ? AND device_index = ? AND time = ?",
key.Client, key.DeviceIndex, key.TimeSlot).Count(&existingCount).Error; err != nil {
return err
}
compressedGPU := models.GPURecord{
Client: key.Client,
DeviceIndex: key.DeviceIndex,
Time: models.FromTime(key.TimeSlot),
DeviceName: key.DeviceName,
MemTotal: getIntPercentile(data.MemTotal, high_percentile),
MemUsed: getIntPercentile(data.MemUsed, high_percentile),
Utilization: getFloat32Percentile(data.Utilization, high_percentile),
Temperature: int(getIntPercentile(convertIntToInt64(data.Temperature), high_percentile)),
}
if existingCount > 0 {
// 更新已存在记录
if err := tx.Table("gpu_records_long_term").Where("client = ? AND device_index = ? AND time = ?",
key.Client, key.DeviceIndex, key.TimeSlot).Updates(&compressedGPU).Error; err != nil {
return err
}
} else {
// 创建新记录
if err := tx.Table("gpu_records_long_term").Create(&compressedGPU).Error; err != nil {
return err
}
}
}
// 删除已压缩的原始GPU数据
if err := tx.Where("time < ?", cutoff.Add(-1*time.Hour)).Delete(&models.GPURecord{}).Error; err != nil {
return err
}
return nil
})
}
+321
View File
@@ -0,0 +1,321 @@
package records
import (
"encoding/csv"
"os"
"strconv"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
"github.com/komari-monitor/komari/database/models"
)
var uuid = "7901508c-304f-49aa-b84f-957c33ae6f8a"
var _ = func() bool {
// 确保 Test 环境中使用 sqlite 内存数据库
return true
}()
// TestCompactRecord tests the database compaction logic by inserting 4h30m of data (one record per minute),
// then running migrateOldRecords and verifying the aggregation and cleanup.
func TestCompactRecord(t *testing.T) {
const totalMinutes = 12*60 + 30
loc := models.GetAppLocation()
now := time.Date(2026, 6, 15, 14, 30, 0, 0, loc)
threshold := compactRecordCutoff(now)
overlapCutoff := threshold.Add(-1 * time.Hour)
// 使用 sqlite 内存数据库并迁移表结构
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
assert.NoError(t, err)
assert.NoError(t, db.AutoMigrate(&models.Record{}))
assert.NoError(t, db.Table("records_long_term").AutoMigrate(&models.Record{}))
expectedGroups := make(map[time.Time]struct{})
expectedRawRemain := 0
// 插入数据
for i := 0; i < totalMinutes; i++ {
recTime := now.Add(-time.Duration(i) * time.Minute)
rec := models.Record{Client: uuid, Time: models.FromTime(recTime), Cpu: float32(i), Gpu: float32(i), Load: float32(i), Temp: float32(i), Ram: int64(i)}
err := db.Create(&rec).Error
assert.NoError(t, err)
if recTime.Before(threshold) {
slot := recTime.Truncate(15 * time.Minute)
expectedGroups[slot] = struct{}{}
}
if !recTime.Before(overlapCutoff) {
expectedRawRemain++
}
}
// 导出原始数据到 CSV
os.MkdirAll("../../data", 0755)
var origRecs []models.Record
db.Order("time desc").Find(&origRecs)
fOrig, err := os.Create("../../data/original.csv")
assert.NoError(t, err)
defer fOrig.Close()
wOrig := csv.NewWriter(fOrig)
defer wOrig.Flush()
wOrig.Write([]string{"Client", "Time", "Cpu", "Gpu", "Load", "Temp", "Ram"})
for _, r := range origRecs {
wOrig.Write([]string{
r.Client,
r.Time.ToTime().Format(time.RFC3339),
strconv.FormatFloat(float64(r.Cpu), 'f', -1, 32),
strconv.FormatFloat(float64(r.Gpu), 'f', -1, 32),
strconv.FormatFloat(float64(r.Load), 'f', -1, 32),
strconv.FormatFloat(float64(r.Temp), 'f', -1, 32),
strconv.FormatInt(r.Ram, 10),
})
}
// 运行压缩(迁移)逻辑
err = migrateOldRecordsAt(db, now)
assert.NoError(t, err)
// 验证 long-term 表中的聚合记录数
var longCount int64
assert.NoError(t, db.Table("records_long_term").Count(&longCount).Error)
assert.Equal(t, int64(len(expectedGroups)), longCount)
// 验证原始表中剩余记录数
var remainCount int64
assert.NoError(t, db.Table("records").Count(&remainCount).Error)
assert.Equal(t, int64(expectedRawRemain), remainCount)
// 导出压缩后的数据到 CSV
var compRecs []models.Record
db.Table("records_long_term").Order("time desc").Find(&compRecs)
fComp, err := os.Create("../../data/compressed.csv")
assert.NoError(t, err)
defer fComp.Close()
wComp := csv.NewWriter(fComp)
defer wComp.Flush()
wComp.Write([]string{"Client", "Time", "Cpu", "Gpu", "Load", "Temp", "Ram"})
for _, r := range compRecs {
wComp.Write([]string{
r.Client,
r.Time.ToTime().Format(time.RFC3339),
strconv.FormatFloat(float64(r.Cpu), 'f', -1, 32),
strconv.FormatFloat(float64(r.Gpu), 'f', -1, 32),
strconv.FormatFloat(float64(r.Load), 'f', -1, 32),
strconv.FormatFloat(float64(r.Temp), 'f', -1, 32),
strconv.FormatInt(r.Ram, 10),
})
}
db.Table("records").Order("time desc").Find(&compRecs)
fComp, err = os.Create("../../data/compressed_records.csv")
assert.NoError(t, err)
defer fComp.Close()
wComp = csv.NewWriter(fComp)
defer wComp.Flush()
wComp.Write([]string{"Client", "Time", "Cpu", "Gpu", "Load", "Temp", "Ram"})
for _, r := range compRecs {
wComp.Write([]string{
r.Client,
r.Time.ToTime().Format(time.RFC3339),
strconv.FormatFloat(float64(r.Cpu), 'f', -1, 32),
strconv.FormatFloat(float64(r.Gpu), 'f', -1, 32),
strconv.FormatFloat(float64(r.Load), 'f', -1, 32),
strconv.FormatFloat(float64(r.Temp), 'f', -1, 32),
strconv.FormatInt(r.Ram, 10),
})
}
}
func TestCompactRecordPreservesExactTrafficDelta(t *testing.T) {
loc := models.GetAppLocation()
currentTime := time.Date(2026, 6, 15, 14, 30, 0, 0, loc)
now := currentTime.Truncate(15 * time.Minute).Add(-5*time.Hour + time.Minute)
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
assert.NoError(t, err)
assert.NoError(t, db.AutoMigrate(&models.Record{}))
assert.NoError(t, db.Table("records_long_term").AutoMigrate(&models.Record{}))
records := []models.Record{
{
Client: uuid,
Time: models.FromTime(now),
NetTotalUp: 100,
NetTotalDown: 200,
TrafficUp: 0,
TrafficDown: 0,
},
{
Client: uuid,
Time: models.FromTime(now.Add(5 * time.Minute)),
NetTotalUp: 150,
NetTotalDown: 260,
TrafficUp: 50,
TrafficDown: 60,
},
{
Client: uuid,
Time: models.FromTime(now.Add(10 * time.Minute)),
NetTotalUp: 10,
NetTotalDown: 30,
TrafficUp: 10,
TrafficDown: 30,
},
}
for _, rec := range records {
assert.NoError(t, db.Create(&rec).Error)
}
assert.NoError(t, migrateOldRecordsAt(db, currentTime))
var compacted []models.Record
assert.NoError(t, db.Table("records_long_term").Find(&compacted).Error)
require.Len(t, compacted, 1)
assert.Equal(t, int64(60), compacted[0].TrafficUp)
assert.Equal(t, int64(90), compacted[0].TrafficDown)
assert.Equal(t, int64(10), compacted[0].NetTotalUp)
assert.Equal(t, int64(30), compacted[0].NetTotalDown)
assert.True(t, compacted[0].Time.ToTime().Equal(records[2].Time.ToTime().Truncate(15*time.Minute)))
}
func TestRepairZeroTrafficDeltasPreservesRawResetDetailBeforeCompaction(t *testing.T) {
loc := models.GetAppLocation()
start := time.Date(2026, 6, 6, 0, 0, 0, 0, loc)
records := []models.Record{
{Client: uuid, Time: models.FromTime(start), NetTotalUp: 100, NetTotalDown: 200},
{Client: uuid, Time: models.FromTime(start.Add(5 * time.Minute)), NetTotalUp: 140, NetTotalDown: 260},
{Client: uuid, Time: models.FromTime(start.Add(10 * time.Minute)), NetTotalUp: 10, NetTotalDown: 20},
{Client: uuid, Time: models.FromTime(start.Add(15 * time.Minute)), NetTotalUp: 25, NetTotalDown: 35},
}
repairZeroTrafficDeltas(records, nil)
assert.Equal(t, int64(0), records[0].TrafficUp)
assert.Equal(t, int64(0), records[0].TrafficDown)
assert.Equal(t, int64(40), records[1].TrafficUp)
assert.Equal(t, int64(60), records[1].TrafficDown)
assert.Equal(t, int64(10), records[2].TrafficUp)
assert.Equal(t, int64(20), records[2].TrafficDown)
assert.Equal(t, int64(15), records[3].TrafficUp)
assert.Equal(t, int64(15), records[3].TrafficDown)
}
func TestRepairZeroTrafficDeltasUsesPreviousPersistedBaseline(t *testing.T) {
loc := models.GetAppLocation()
now := time.Date(2026, 6, 7, 12, 0, 0, 0, loc)
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
assert.NoError(t, err)
assert.NoError(t, db.AutoMigrate(&models.Record{}))
assert.NoError(t, db.Table("records_long_term").AutoMigrate(&models.Record{}))
assert.NoError(t, db.Table("records_long_term").Create(&models.Record{
Client: uuid,
Time: models.FromTime(now.Add(-10 * time.Minute)),
NetTotalUp: 100,
NetTotalDown: 200,
}).Error)
rawRecords := []models.Record{
{Client: uuid, Time: models.FromTime(now), NetTotalUp: 150, NetTotalDown: 260},
{Client: uuid, Time: models.FromTime(now.Add(5 * time.Minute)), NetTotalUp: 175, NetTotalDown: 300},
}
previousByClient, err := getPreviousTrafficRecordsBefore(db, rawRecords)
assert.NoError(t, err)
repairZeroTrafficDeltas(rawRecords, previousByClient)
assert.Equal(t, int64(50), rawRecords[0].TrafficUp)
assert.Equal(t, int64(60), rawRecords[0].TrafficDown)
assert.Equal(t, int64(25), rawRecords[1].TrafficUp)
assert.Equal(t, int64(40), rawRecords[1].TrafficDown)
}
func TestRepairZeroTrafficDeltasIgnoresSameSlotLongTermBaseline(t *testing.T) {
loc := models.GetAppLocation()
now := time.Date(2026, 6, 7, 12, 0, 30, 0, loc)
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
assert.NoError(t, err)
assert.NoError(t, db.AutoMigrate(&models.Record{}))
assert.NoError(t, db.Table("records_long_term").AutoMigrate(&models.Record{}))
assert.NoError(t, db.Table("records_long_term").Create(&models.Record{
Client: uuid,
Time: models.FromTime(now.Truncate(15 * time.Minute)),
NetTotalUp: 900,
NetTotalDown: 900,
}).Error)
rawRecords := []models.Record{
{Client: uuid, Time: models.FromTime(now), NetTotalUp: 150, NetTotalDown: 260},
{Client: uuid, Time: models.FromTime(now.Add(5 * time.Minute)), NetTotalUp: 175, NetTotalDown: 300},
}
previousByClient, err := getPreviousTrafficRecordsBefore(db, rawRecords)
assert.NoError(t, err)
repairZeroTrafficDeltas(rawRecords, previousByClient)
assert.Equal(t, int64(0), rawRecords[0].TrafficUp)
assert.Equal(t, int64(0), rawRecords[0].TrafficDown)
assert.Equal(t, int64(25), rawRecords[1].TrafficUp)
assert.Equal(t, int64(40), rawRecords[1].TrafficDown)
}
func TestCompactRecordRetainsOneHourOverlapWindow(t *testing.T) {
loc := models.GetAppLocation()
now := time.Date(2026, 6, 7, 12, 7, 0, 0, loc)
cutoff := compactRecordCutoff(now)
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
assert.NoError(t, err)
assert.NoError(t, db.AutoMigrate(&models.Record{}))
assert.NoError(t, db.Table("records_long_term").AutoMigrate(&models.Record{}))
records := []models.Record{
{Client: uuid, Time: models.FromTime(cutoff.Add(-time.Hour - time.Minute)), TrafficUp: 5},
{Client: uuid, Time: models.FromTime(cutoff.Add(-30 * time.Minute)), TrafficUp: 7},
{Client: uuid, Time: models.FromTime(now.Add(-3 * time.Hour)), TrafficUp: 11},
}
for _, rec := range records {
assert.NoError(t, db.Create(&rec).Error)
}
assert.NoError(t, migrateOldRecordsAt(db, now))
var remainTimes []models.Record
assert.NoError(t, db.Table("records").Order("time ASC").Find(&remainTimes).Error)
require.Len(t, remainTimes, 2)
assert.True(t, remainTimes[0].Time.ToTime().Equal(records[1].Time.ToTime()))
assert.True(t, remainTimes[1].Time.ToTime().Equal(records[2].Time.ToTime()))
}
func TestCompactRecordOnlyMigratesCompleteFifteenMinuteBuckets(t *testing.T) {
loc := models.GetAppLocation()
now := time.Date(2026, 6, 7, 12, 7, 0, 0, loc)
cutoff := compactRecordCutoff(now)
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
assert.NoError(t, err)
assert.NoError(t, db.AutoMigrate(&models.Record{}))
assert.NoError(t, db.Table("records_long_term").AutoMigrate(&models.Record{}))
compactable := models.Record{Client: uuid, Time: models.FromTime(cutoff.Add(-time.Minute)), TrafficUp: 5}
partialSlot := models.Record{Client: uuid, Time: models.FromTime(cutoff.Add(time.Minute)), TrafficUp: 7}
assert.NoError(t, db.Create(&compactable).Error)
assert.NoError(t, db.Create(&partialSlot).Error)
assert.NoError(t, migrateOldRecordsAt(db, now))
var compacted []models.Record
assert.NoError(t, db.Table("records_long_term").Order("time ASC").Find(&compacted).Error)
require.Len(t, compacted, 1)
assert.True(t, compacted[0].Time.ToTime().Equal(compactable.Time.ToTime().Truncate(15*time.Minute)))
var rawCount int64
assert.NoError(t, db.Table("records").Where("time = ?", partialSlot.Time).Count(&rawCount).Error)
assert.Equal(t, int64(1), rawCount)
}
+217
View File
@@ -0,0 +1,217 @@
package tasks
import (
"time"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/utils"
"gorm.io/gorm"
)
// AddPingTask 创建延迟监测任务。defaultOn 表示新加入的服务器是否自动开启此监测。
func AddPingTask(clients []string, defaultOn bool, name string, target, task_type string, interval int) (uint, error) {
db := dbcore.GetDBInstance()
normalizedClients := normalizePingClients(models.StringArray(clients))
task := models.PingTask{
Clients: normalizedClients,
DefaultOn: defaultOn,
Name: name,
Type: task_type,
Target: target,
Interval: interval,
}
err := db.Transaction(func(tx *gorm.DB) error {
if err := tx.Create(&task).Error; err != nil {
return err
}
// Append by id to avoid races between concurrent create requests.
result := tx.Model(&models.PingTask{}).Where("id = ?", task.Id).Update("weight", int(task.Id))
if result.Error != nil {
return result.Error
}
if result.RowsAffected == 0 {
return gorm.ErrRecordNotFound
}
return nil
})
if err != nil {
return 0, err
}
ReloadPingSchedule()
return task.Id, nil
}
func DeletePingTask(id []uint) error {
db := dbcore.GetDBInstance()
result := db.Where("id IN ?", id).Delete(&models.PingTask{})
if result.RowsAffected == 0 {
return gorm.ErrRecordNotFound
}
ReloadPingSchedule()
return result.Error
}
// EditPingTask 批量更新延迟监测任务配置。
func EditPingTask(tasks []*models.PingTask) error {
db := dbcore.GetDBInstance()
for _, task := range tasks {
task.Clients = normalizePingClients(task.Clients)
// 使用 map 显式更新,避免 GORM struct Updates 跳过 false/0/空切片等零值。
updates := map[string]interface{}{
"name": task.Name,
"clients": task.Clients,
"all_clients": task.DefaultOn,
"type": task.Type,
"target": task.Target,
"interval": task.Interval,
}
result := db.Model(&models.PingTask{}).Where("id = ?", task.Id).Updates(updates)
if result.RowsAffected == 0 {
return gorm.ErrRecordNotFound
}
}
ReloadPingSchedule()
return nil
}
// normalizePingClients 保持 clients 字段序列化为 JSON 数组,避免空值变成 null。
func normalizePingClients(clients models.StringArray) models.StringArray {
if clients == nil {
return models.StringArray{}
}
return clients
}
func GetAllPingTasks() ([]models.PingTask, error) {
db := dbcore.GetDBInstance()
var tasks []models.PingTask
if err := db.Order("weight ASC").Order("id ASC").Find(&tasks).Error; err != nil {
return nil, err
}
return tasks, nil
}
// GetPingTasksByClient 获取指定服务器需要执行的延迟监测任务。
func GetPingTasksByClient(uuid string) []models.PingTask {
db := dbcore.GetDBInstance()
var tasks []models.PingTask
if err := db.Where("clients LIKE ?", `%"`+uuid+`"%`).Find(&tasks).Error; err != nil {
return nil
}
return tasks
}
func UpdatePingTaskOrder(order map[uint]int) error {
db := dbcore.GetDBInstance()
err := db.Transaction(func(tx *gorm.DB) error {
for id, weight := range order {
result := tx.Model(&models.PingTask{}).Where("id = ?", id).Update("weight", weight)
if result.Error != nil {
return result.Error
}
if result.RowsAffected == 0 {
return gorm.ErrRecordNotFound
}
}
return nil
})
if err != nil {
return err
}
ReloadPingSchedule()
return nil
}
func SavePingRecord(record models.PingRecord) error {
db := dbcore.GetDBInstance()
return db.Create(&record).Error
}
func DeletePingRecordsBefore(time time.Time) error {
db := dbcore.GetDBInstance()
err := db.Where("time < ?", time).Delete(&models.PingRecord{}).Error
return err
}
func DeletePingRecords(id []uint) error {
db := dbcore.GetDBInstance()
result := db.Where("task_id IN ?", id).Delete(&models.PingRecord{})
if result.RowsAffected == 0 {
return gorm.ErrRecordNotFound
}
return result.Error
}
func DeleteAllPingRecords() error {
db := dbcore.GetDBInstance()
result := db.Exec("DELETE FROM ping_records")
if result.RowsAffected == 0 {
return gorm.ErrRecordNotFound
}
return result.Error
}
func ReloadPingSchedule() error {
db := dbcore.GetDBInstance()
var pingTasks []models.PingTask
if err := db.Find(&pingTasks).Error; err != nil {
return err
}
return utils.ReloadPingSchedule(pingTasks)
}
// AddDefaultOnClientUUID 在新客户端注册后,把该 UUID 追加到所有 default_on=true 的任务的 clients 中(去重)。
func AddDefaultOnClientUUID(uuid string) error {
if uuid == "" {
return nil
}
db := dbcore.GetDBInstance()
var tasks []models.PingTask
if err := db.Where("all_clients = ?", true).Find(&tasks).Error; err != nil {
return err
}
if len(tasks) == 0 {
return nil
}
changed := false
for _, task := range tasks {
exists := false
for _, c := range task.Clients {
if c == uuid {
exists = true
break
}
}
if exists {
continue
}
next := append(models.StringArray{}, task.Clients...)
next = append(next, uuid)
if err := db.Model(&models.PingTask{}).Where("id = ?", task.Id).Update("clients", next).Error; err != nil {
return err
}
changed = true
}
if changed {
return ReloadPingSchedule()
}
return nil
}
func GetPingRecords(uuid string, taskId int, start, end time.Time) ([]models.PingRecord, error) {
db := dbcore.GetDBInstance()
var records []models.PingRecord
dbQuery := db.Model(&models.PingRecord{})
if uuid != "" {
dbQuery = dbQuery.Where("client = ?", uuid)
}
if taskId >= 0 {
dbQuery = dbQuery.Where("task_id = ?", uint(taskId))
}
if err := dbQuery.Where("time >= ? AND time <= ?", start, end).Order("time DESC").Find(&records).Error; err != nil {
return nil, err
}
return records, nil
}
+99
View File
@@ -0,0 +1,99 @@
package tasks
import (
"time"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
)
func CreateTask(taskId string, clients []string, command string) error {
db := dbcore.GetDBInstance()
// Create a new task in the database with clients as JSON array
task := models.Task{
TaskId: taskId,
Clients: models.StringArray(clients),
Command: command,
}
if err := db.Create(&task).Error; err != nil {
return err
}
var taskResults []models.TaskResult
for _, client := range clients {
taskResults = append(taskResults, models.TaskResult{
TaskId: taskId,
Client: client,
Result: "",
ExitCode: nil,
FinishedAt: nil,
CreatedAt: models.FromTime(time.Now()),
})
}
if len(taskResults) > 0 {
return db.Create(&taskResults).Error
}
return nil
}
func GetTaskByTaskId(taskId string) (*models.Task, error) {
var task models.Task
if err := dbcore.GetDBInstance().Where("task_id = ?", taskId).First(&task).Error; err != nil {
return nil, err
}
return &task, nil
}
func GetTasksByClientId(clientId string) ([]models.Task, error) {
var tasks []models.Task
if err := dbcore.GetDBInstance().Where("clients LIKE ?", "%"+clientId+"%").Find(&tasks).Error; err != nil {
return nil, err
}
return tasks, nil
}
func GetSpecificTaskResult(taskId, clientId string) (*models.TaskResult, error) {
var result models.TaskResult
if err := dbcore.GetDBInstance().Where("task_id = ? AND client = ?", taskId, clientId).First(&result).Error; err != nil {
return nil, err
}
return &result, nil
}
func GetAllTasksResultByUUID(uuid string) ([]models.TaskResult, error) {
var results []models.TaskResult
if err := dbcore.GetDBInstance().Where("client = ?", uuid).Find(&results).Error; err != nil {
return nil, err
}
return results, nil
}
func GetAllTasks() ([]models.Task, error) {
var tasks []models.Task
if err := dbcore.GetDBInstance().Find(&tasks).Error; err != nil {
return nil, err
}
return tasks, nil
}
func GetTaskResultsByTaskId(taskId string) ([]models.TaskResult, error) {
var results []models.TaskResult
if err := dbcore.GetDBInstance().Where("task_id = ?", taskId).Find(&results).Error; err != nil {
return nil, err
}
return results, nil
}
func DeleteTaskByTaskId(taskId string) error {
return dbcore.GetDBInstance().Where("task_id = ?", taskId).Delete(&models.Task{}).Error
}
func SaveTaskResult(taskId, clientId, result string, exitCode int, timestamp models.LocalTime) error {
return dbcore.GetDBInstance().
Model(&models.TaskResult{}).
Where("task_id = ? AND client = ?", taskId, clientId).
Updates(map[string]interface{}{
"result": result,
"exit_code": exitCode,
"finished_at": timestamp,
}).Error
}
func ClearTaskResultsByTimeBefore(before time.Time) error {
return dbcore.GetDBInstance().Where("created_at < ?", before.Format(time.RFC3339)).Delete(&models.TaskResult{}).Error
}
+145
View File
@@ -0,0 +1,145 @@
package database
import (
"encoding/json"
"log"
"os"
"path/filepath"
"strings"
"github.com/gin-gonic/gin"
"github.com/komari-monitor/komari/database/dbcore"
"github.com/komari-monitor/komari/database/models"
"github.com/komari-monitor/komari/pkg/config"
)
func GetPublicInfo() (map[string]interface{}, error) {
cstPtr, err := config.GetManyAs[config.Settings]()
if err != nil {
return nil, err
}
cst := *cstPtr
all, allErr := config.GetAll()
hasKey := func(k string) bool {
if allErr != nil {
return false
}
_, ok := all[k]
return ok
}
// Apply defaults only when a key is missing.
if !hasKey("sitename") {
cst.Sitename = "Komari"
}
if !hasKey("description") {
cst.Description = "Komari Monitor, a simple server monitoring tool."
}
if !hasKey("theme") {
cst.Theme = "default"
}
if !hasKey("o_auth_provider") {
cst.OAuthProvider = "github"
}
if !hasKey("record_enabled") {
cst.RecordEnabled = true
}
if !hasKey("record_preserve_time") {
cst.RecordPreserveTime = 720
}
if !hasKey("ping_record_preserve_time") {
cst.PingRecordPreserveTime = 24
}
// Fallback defaults if we couldn't enumerate keys.
if allErr != nil {
if cst.Sitename == "" {
cst.Sitename = "Komari"
}
if cst.Description == "" {
cst.Description = "Komari Monitor, a simple server monitoring tool."
}
}
db := dbcore.GetDBInstance()
tc := models.ThemeConfiguration{}
err = db.Model(&models.ThemeConfiguration{}).Where("short = ?", cst.Theme).First(&tc).Error
if err != nil {
tc.Data = "{}"
}
tc_data := gin.H{}
err = json.Unmarshal([]byte(tc.Data), &tc_data)
if err != nil {
log.Printf("%v", err)
}
// Try to load theme declaration file and merge defaults for managed configuration
// Theme declarations live in ./data/theme/<short>/komari-theme.json
if cst.Theme != "" && cst.Theme != "default" {
themeConfigPath := filepath.Join("./data/theme", cst.Theme, "komari-theme.json")
if _, err := os.Stat(themeConfigPath); err == nil {
b, err := os.ReadFile(themeConfigPath)
if err == nil {
var themeDecl struct {
Configuration struct {
Type string `json:"type"`
Data []models.ManagedThemeConfigurationItem `json:"data"`
} `json:"configuration"`
}
if err := json.Unmarshal(b, &themeDecl); err == nil {
if themeDecl.Configuration.Type == "managed" {
for _, item := range themeDecl.Configuration.Data {
if item.Key == "" {
continue
}
// missing
if _, exists := tc_data[item.Key]; !exists {
var def any = item.Default
// select
if item.Type == "select" {
if def == nil || def == "" {
if item.Options != "" {
opts := strings.Split(item.Options, ",")
if len(opts) > 0 {
def = strings.TrimSpace(opts[0])
}
}
}
}
// number->0, string->"", switch->false
if def == nil {
switch item.Type {
case "number":
def = 0
case "switch":
def = false
default:
def = ""
}
}
tc_data[item.Key] = def
}
}
}
}
}
}
}
return gin.H{
"sitename": cst.Sitename,
"description": cst.Description,
"custom_head": cst.CustomHead,
"custom_body": cst.CustomBody,
"oauth_enable": cst.OAuthEnabled,
"oauth_provider": cst.OAuthProvider,
"disable_password_login": cst.DisablePasswordLogin,
"cors_origin_check_enabled": cst.CorsOriginCheckEnabled,
"record_enabled": cst.RecordEnabled,
"record_preserve_time": cst.RecordPreserveTime,
"ping_record_preserve_time": cst.PingRecordPreserveTime,
"private_site": cst.PrivateSite,
"theme": cst.Theme,
"theme_settings": tc_data,
}, nil
}
+148
View File
@@ -0,0 +1,148 @@
# Komari
![Badge](https://hitscounter.dev/api/hit?url=https%3A%2F%2Fgithub.com%2Fkomari-monitor%2Fkomari&label=&icon=github&color=%23a370f7&message=&style=flat&tz=UTC)
![komari](https://socialify.git.ci/komari-monitor/komari/image?description=1&font=Inter&forks=1&issues=1&language=1&logo=https%3A%2F%2Fraw.githubusercontent.com%2Fkomari-monitor%2Fkomari-web%2Fd54ce1288df41ead08aa19f8700186e68028a889%2Fpublic%2Ffavicon.png&name=1&owner=1&pattern=Plus&pulls=1&stargazers=1&theme=Auto)
Komariは、サーバーのパフォーマンスを監視するためのシンプルで効率的なソリューションを提供することを目的とした、軽量の自己ホスト型サーバー監視ツールです。Webインターフェースを介してサーバーのステータスを表示し、軽量エージェントを介してデータを収集します。
> [!WARNING]
> Komariは、自己ホスト型の監視/制御プログラムであり、所有しているシステム、または管理権限を得ているシステムでのみ使用してください。Komariを武器化したり、許可なく展開、アクセス、永続化、コマンド実行、その他の不正利用に使用したりしないでください。実際の悪用リスクについては、Huntressの分析をご参照ください:[Komari C2 agent abuse](https://www.huntress.com/blog/komari-c2-agent-abuse)。
> Komariの展開および運用方法については、利用者自身が責任を負います。開発者は、無許可または不正な利用、およびその結果について責任を負いません。
> Windows端末でリモートコントロールを有効にした場合、クライアントはユーザーがログインするたびに、KomariがリモートコントロールソフトウェアであることをWindows通知で知らせます。
[ドキュメント](https://komari-document.pages.dev/) | [Telegramグループ](https://t.me/komari_monitor)
## 特徴
- **軽量で効率的**: リソース消費が少なく、あらゆる規模のサーバーに適しています。
- **自己ホスト型**: データプライバシーを完全に制御でき、展開も簡単です。
- **Webインターフェース**: 直感的な監視ダッシュボードで、使いやすいです。
## クイックスタート
### 0. クラウドホスティングによるワンクリック展開
- Rainyun - CNY 4.5/月
[![](https://rainyun-apps.cn-nb1.rains3.com/materials/deploy-on-rainyun-cn.svg)](https://app.rainyun.com/apps/rca/store/6780/NzYxNzAz_)
- 1Panel アプリストア
1Panel アプリストアで利用可能です。「アプリストア」>「ユーティリティ」>「Komari」からインストールしてください。
### 1. ワンクリックインストールスクリプトを使用する
systemdを使用するディストリビューション(Ubuntu、Debianなど)に適しています。
```bash
curl -fsSL https://raw.githubusercontent.com/komari-monitor/komari/main/install-komari.sh -o install-komari.sh
chmod +x install-komari.sh
sudo ./install-komari.sh
```
### 2. Docker展開
1. データディレクトリを作成します:
```bash
mkdir -p ./data
```
2. Dockerコンテナを実行します:
```bash
docker run -d \
-p 25774:25774 \
-v $(pwd)/data:/app/data \
--name komari \
ghcr.io/komari-monitor/komari:latest
```
3. デフォルトのユーザー名とパスワードを表示します:
```bash
docker logs komari
```
4. ブラウザで `http://<your_server_ip>:25774` にアクセスします。
> [!NOTE]
> 環境変数 `ADMIN_USERNAME` と `ADMIN_PASSWORD` を使用して、初期のユーザー名とパスワードをカスタマイズすることもできます。
### 3. バイナリファイル展開
1. Komariの[GitHubリリース](https://github.com/komari-monitor/komari/releases)ページにアクセスして、お使いのオペレーティングシステム用の最新のバイナリをダウンロードします。
2. Komariを実行します:
```bash
./komari server -l 0.0.0.0:25774
```
3. ブラウザで `http://<your_server_ip>:25774` にアクセスします。デフォルトのポートは `25774` です。
4. デフォルトのユーザー名とパスワードは、起動ログで確認するか、環境変数 `ADMIN_USERNAME` と `ADMIN_PASSWORD` を介して設定できます。
> [!NOTE]
> バイナリに実行権限があることを確認してください(`chmod +x komari`)。データは実行ディレクトリの `data` フォルダに保存されます。
### 手動ビルド
#### 依存関係
- Go 1.18+ および Node.js 20+(手動ビルド用)
1. フロントエンドの静的ファイルをビルドします:
```bash
git clone https://github.com/komari-monitor/komari-web
cd komari-web
npm install
npm run build
```
2. バックエンドをビルドします:
```bash
git clone https://github.com/komari-monitor/komari
cd komari
```
ステップ1で生成された静的ファイルを `komari` プロジェクトのルートにある `/web/public/defaultTheme/dist` フォルダにコピーし、`komari-theme.json` と `preview.png`/`perview.png` を `/web/public/defaultTheme` にコピーします。
```bash
go build -o komari
```
3. 実行:
```bash
./komari server -l 0.0.0.0:25774
```
デフォルトのリスニングポートは `25774` です。`http://localhost:25774` にアクセスします。
## フロントエンド開発ガイド
[Komariテーマ開発ガイド | Komari](https://komari-document.pages.dev/dev/theme.html)
[CrowdinでKomariを翻訳する](https://crowdin.com/project/komari/invite?h=cd051bf172c9a9f7f1360e87ffb521692507706)
## クライアントエージェント開発ガイド
[Komariエージェント情報レポートおよびイベント処理ドキュメント](https://komari-document.pages.dev/dev/agent.html)
## 貢献
IssueやPull Requestを歓迎します!
## 謝辞
### 破碎工坊云 (Crash Work)
[破碎工坊云 - 効率的で安定した、安全な高防御サーバーとCDNソリューションを提供する専門的なクラウドコンピューティングプラットフォーム](https://www.crash.work/)
### DreamCloud
[DreamCloud - コストパフォーマンスに優れたアジア太平洋向け高防御直通](https://as211392.com/)
### 🚀 SharonNetworks スポンサー
[![Sharon Networks](https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/sharon-networks.webp)](https://sharon.io)
SharonNetworks は、あなたのビジネスの離陸を力強くサポートします!
アジア太平洋のデータセンターから中国最適化ネットワーク接続を提供。低レイテンシー & 高帯域幅、Tbps 級のローカル洗浄 DDoS 防御で、ビジネスとお客様の体験を守ります。コミュニティ [Telegram グループ](https://t.me/SharonNetwork) に参加すると、チャリティまたはグループ内抽選で無料利用のチャンスがあります。
### オープンソースコミュニティ
PR を送ってくれた方、テーマを作成してくれた全ての開発者
—— そして:こんなに暇でいられる自分に感謝
## Star履歴
[![Star History Chart](https://api.star-history.com/svg?repos=komari-monitor/komari&type=Date)](https://www.star-history.com/#komari-monitor/komari&Date)
+148
View File
@@ -0,0 +1,148 @@
# Komari
![Badge](https://hitscounter.dev/api/hit?url=https%3A%2F%2Fgithub.com%2Fkomari-monitor%2Fkomari&label=&icon=github&color=%23a370f7&message=&style=flat&tz=UTC)
![komari](https://socialify.git.ci/komari-monitor/komari/image?description=1&font=Inter&forks=1&issues=1&language=1&logo=https%3A%2F%2Fraw.githubusercontent.com%2Fkomari-monitor%2Fkomari-web%2Fd54ce1288df41ead08aa19f8700186e68028a889%2Fpublic%2Ffavicon.png&name=1&owner=1&pattern=Plus&pulls=1&stargazers=1&theme=Auto)
Komari 是一款輕量級的自託管伺服器監控工具,旨在提供簡單、高效的伺服器性能監控解決方案。它支援透過 Web 介面查看伺服器狀態,並透過輕量級 Agent 收集數據。
> [!WARNING]
> Komari 是一款自託管的監控/控制程式,僅應部署在你擁有或已獲授權管理的系統上。請勿將 Komari 武器化,或在未獲授權的情況下部署、存取、持久化、執行命令及從事其他濫用行為。關於現實中的濫用風險,可參考 Huntress 的分析:[Komari C2 agent abuse](https://www.huntress.com/blog/komari-c2-agent-abuse)。
> 使用者需要自行承擔部署和使用 Komari 的責任。開發者不對未經授權或濫用行為及其後果承擔責任。
> 在 Windows 端啟用遠端控制後,客戶端會在每次使用者登入時透過 Windows 通知提醒使用者 Komari 是一款遠端控制軟體。
[文檔](https://komari-document.pages.dev/) | [Telegram 群組](https://t.me/komari_monitor)
## 特性
- **輕量高效**:低資源佔用,適合各種規模的伺服器。
- **自託管**:完全掌控數據隱私,部署簡單。
- **Web 介面**:直觀的監控儀表盤,易於使用。
## 快速開始
### 0. 容器雲一鍵部署
- 雨雲雲應用 - CNY 4.5/月
[![](https://rainyun-apps.cn-nb1.rains3.com/materials/deploy-on-rainyun-cn.svg)](https://app.rainyun.com/apps/rca/store/6780/NzYxNzAz_)
- 1Panel 應用商店
已上架 1Panel 應用商店,應用商店-實用工具-Komari 即可安裝
### 1. 使用一鍵安裝腳本
適用於使用了 systemd 的發行版(Ubuntu、Debian...)。
```bash
curl -fsSL https://raw.githubusercontent.com/komari-monitor/komari/main/install-komari.sh -o install-komari.sh
chmod +x install-komari.sh
sudo ./install-komari.sh
```
### 2. Docker 部署
1. 建立資料目錄:
```bash
mkdir -p ./data
```
2. 執行 Docker 容器:
```bash
docker run -d \
-p 25774:25774 \
-v $(pwd)/data:/app/data \
--name komari \
ghcr.io/komari-monitor/komari:latest
```
3. 查看預設帳號和密碼:
```bash
docker logs komari
```
4. 在瀏覽器中存取 `http://<your_server_ip>:25774`。
> [!NOTE]
> 你也可以透過環境變數 `ADMIN_USERNAME` 和 `ADMIN_PASSWORD` 自訂初始使用者名稱和密碼。
### 3. 二進位檔案部署
1. 存取 Komari 的 [GitHub Release 頁面](https://github.com/komari-monitor/komari/releases) 下載適用於你作業系統的最新二進位檔案。
2. 執行 Komari:
```bash
./komari server -l 0.0.0.0:25774
```
3. 在瀏覽器中存取 `http://<your_server_ip>:25774`,預設監聽 `25774` 連接埠。
4. 預設帳號和密碼可在啟動日誌中查看,或透過環境變數 `ADMIN_USERNAME` 和 `ADMIN_PASSWORD` 設定。
> [!NOTE]
> 確保二進位檔案具有可執行權限(`chmod +x komari`)。資料將保存在執行目錄下的 `data` 資料夾中。
### 手工建置
#### 依賴
- Go 1.18+ 和 Node.js 20+(手工建置)
1. 建置前端靜態檔案:
```bash
git clone https://github.com/komari-monitor/komari-web
cd komari-web
npm install
npm run build
```
2. 建置後端:
```bash
git clone https://github.com/komari-monitor/komari
cd komari
```
將步驟1中產生的靜態檔案複製到 `komari` 專案根目錄下的 `/web/public/defaultTheme/dist` 資料夾,並將 `komari-theme.json` 與 `preview.png`/`perview.png` 複製到 `/web/public/defaultTheme`。
```bash
go build -o komari
```
3. 執行:
```bash
./komari server -l 0.0.0.0:25774
```
預設監聽 `25774` 連接埠,存取 `http://localhost:25774`。
## 前端開發指南
[Komari 主題開發指南 | Komari](https://komari-document.pages.dev/dev/theme.html)
[在 Crowdin 上翻譯 Komari](https://crowdin.com/project/komari/invite?h=cd051bf172c9a9f7f1360e87ffb521692507706)
## 客戶端 Agent 開發指南
[Komari Agent 資訊上報與事件處理文檔](https://komari-document.pages.dev/dev/agent.html)
## 貢獻
歡迎提交 Issue 或 Pull Request!
## 鳴謝
### 破碎工坊雲
[破碎工坊雲 - 專業雲計算服務平台,提供高效、穩定、安全的高防伺服器與CDN解決方案](https://www.crash.work/)
### DreamCloud
[DreamCloud - 極高性價比解鎖直連亞太高防](https://as211392.com/)
### 🚀 由 SharonNetworks 贊助
[![Sharon Networks](https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/sharon-networks.webp)](https://sharon.io)
SharonNetworks 為您的業務起飛保駕護航!
亞太資料中心提供頂級的中國優化網路接入 · 低延遲 & 高頻寬 & 提供 Tbps 級本地清洗高防服務,為您的業務保駕護航,為您的客戶提供極致體驗。加入社群 [Telegram 群組](https://t.me/SharonNetwork) 可參與公益募捐或群內抽獎免費使用。
### 開源社群
提交 PR、製作主題的各位開發者
—— 以及:感謝我自己能這麼閒
## Star History
[![Star History Chart](https://api.star-history.com/svg?repos=komari-monitor/komari&type=Date)](https://www.star-history.com/#komari-monitor/komari&Date)
+148
View File
@@ -0,0 +1,148 @@
# Komari
![Badge](https://hitscounter.dev/api/hit?url=https%3A%2F%2Fgithub.com%2Fkomari-monitor%2Fkomari&label=&icon=github&color=%23a370f7&message=&style=flat&tz=UTC)
![komari](https://socialify.git.ci/komari-monitor/komari/image?description=1&font=Inter&forks=1&issues=1&language=1&logo=https%3A%2F%2Fraw.githubusercontent.com%2Fkomari-monitor%2Fkomari-web%2Fd54ce1288df41ead08aa19f8700186e68028a889%2Fpublic%2Ffavicon.png&name=1&owner=1&pattern=Plus&pulls=1&stargazers=1&theme=Auto)
Komari 是一款轻量级的自托管服务器监控工具,旨在提供简单、高效的服务器性能监控解决方案。它支持通过 Web 界面查看服务器状态,并通过轻量级 Agent 收集数据。
> [!WARNING]
> Komari 是一款自托管的监控/控制程序,仅应部署在你拥有或已获得授权管理的系统上。请勿将 Komari 武器化,或在未获授权的情况下部署、访问、持久化、执行命令及从事其他滥用行为。关于现实中的滥用风险,可参考 Huntress 的分析:[Komari C2 agent abuse](https://www.huntress.com/blog/komari-c2-agent-abuse)。
> 用户需要自行承担部署和使用 Komari 的责任。开发者不对未经授权或滥用行为及其后果承担责任。
> 在 Windows 端开启远程控制后,客户端会在每次用户登录时通过 Windows 通知提醒用户 Komari 是一款远程控制软件。
[文档](https://komari-document.pages.dev/) | [文档(镜像站 By Geekertao)](https://www.komari.wiki) | [Telegram 群组](https://t.me/komari_monitor)
## 特性
- **轻量高效**:低资源占用,适合各种规模的服务器。
- **自托管**:完全掌控数据隐私,部署简单。
- **Web 界面**:直观的监控仪表盘,易于使用。
## 快速开始
### 0. 容器云一键部署
- 雨云云应用 - CNY 4.5/月
[![](https://rainyun-apps.cn-nb1.rains3.com/materials/deploy-on-rainyun-cn.svg)](https://app.rainyun.com/apps/rca/store/6780/NzYxNzAz_)
- 1Panel 应用商店
已上架1Panel应用商店,应用商店-实用工具-Komari 即可安装
### 1. 使用一键安装脚本
适用于使用了 systemd 的发行版(Ubuntu、Debian...)。
```bash
curl -fsSL https://raw.githubusercontent.com/komari-monitor/komari/main/install-komari.sh -o install-komari.sh
chmod +x install-komari.sh
sudo ./install-komari.sh
```
### 2. Docker 部署
1. 创建数据目录:
```bash
mkdir -p ./data
```
2. 运行 Docker 容器:
```bash
docker run -d \
-p 25774:25774 \
-v $(pwd)/data:/app/data \
--name komari \
ghcr.io/komari-monitor/komari:latest
```
3. 查看默认账号和密码:
```bash
docker logs komari
```
4. 在浏览器中访问 `http://<your_server_ip>:25774`。
> [!NOTE]
> 你也可以通过环境变量 `ADMIN_USERNAME` 和 `ADMIN_PASSWORD` 自定义初始用户名和密码。
### 3. 二进制文件部署
1. 访问 Komari 的 [GitHub Release 页面](https://github.com/komari-monitor/komari/releases) 下载适用于你操作系统的最新二进制文件。
2. 运行 Komari:
```bash
./komari server -l 0.0.0.0:25774
```
3. 在浏览器中访问 `http://<your_server_ip>:25774`,默认监听 `25774` 端口。
4. 默认账号和密码可在启动日志中查看,或通过环境变量 `ADMIN_USERNAME` 和 `ADMIN_PASSWORD` 设置。
> [!NOTE]
> 确保二进制文件具有可执行权限(`chmod +x komari`)。数据将保存在运行目录下的 `data` 文件夹中。
### 手工构建
#### 依赖
- Go 1.18+ 和 Node.js 20+(手工构建)
1. 构建前端静态文件:
```bash
git clone https://github.com/komari-monitor/komari-web
cd komari-web
npm install
npm run build
```
2. 构建后端:
```bash
git clone https://github.com/komari-monitor/komari
cd komari
```
将步骤1中生成的静态文件复制到 `komari` 项目根目录下的 `/web/public/defaultTheme/dist` 文件夹,并将 `komari-theme.json` 与 `preview.png`/`perview.png` 复制到 `/web/public/defaultTheme`。
```bash
go build -o komari
```
3. 运行:
```bash
./komari server -l 0.0.0.0:25774
```
默认监听 `25774` 端口,访问 `http://localhost:25774`。
## 前端开发指南
[Komari 主题开发指南 | Komari](https://komari-document.pages.dev/dev/theme.html)
[在 Crowdin 上翻译 Komari](https://crowdin.com/project/komari/invite?h=cd051bf172c9a9f7f1360e87ffb521692507706)
## 客户端 Agent 开发指南
[Komari Agent 信息上报与事件处理文档](https://komari-document.pages.dev/dev/agent.html)
## 贡献
欢迎提交 Issue 或 Pull Request!
## 鸣谢
### 破碎工坊云
[破碎工坊云 - 专业云计算服务平台,提供高效、稳定、安全的高防服务器与CDN解决方案](https://www.crash.work/)
### DreamCloud
[DreamCloud - 极高性价比解锁直连亚太高防](https://as211392.com/)
### 🚀 由 SharonNetworks 赞助
[![Sharon Networks](https://raw.githubusercontent.com/komari-monitor/public/refs/heads/main/images/sharon-networks.webp)](https://sharon.io)
SharonNetworks 为您的业务起飞保驾护航!
亚太数据中心提供顶级的中国优化网络接入 · 低延时 & 高带宽 & 提供 Tbps 级本地清洗高防服务,为您的业务保驾护航,为您的客户提供极致体验。加入社区 [Telegram 群组](https://t.me/SharonNetwork) 可参与公益募捐或群内抽奖免费使用。
### 开源社区
提交 PR、制作主题的各位开发者
—— 以及:感谢我自己能这么闲
## Star History
[![Star History Chart](https://api.star-history.com/svg?repos=komari-monitor/komari&type=Date)](https://www.star-history.com/#komari-monitor/komari&Date)
+31 -12
View File
@@ -1,41 +1,58 @@
module github.com/komari-monitor/komari
go 1.23.2
go 1.25.0
require (
github.com/gin-contrib/cors v1.7.5
github.com/coreos/go-oidc/v3 v3.15.0
github.com/dop251/goja v0.0.0-20251008123653-cf18d89f3cf6
github.com/dop251/goja_nodejs v0.0.0-20251015164255-5e94316bedaf
github.com/gin-gonic/gin v1.10.0
github.com/go-sql-driver/mysql v1.10.0
github.com/google/uuid v1.6.0
github.com/gorilla/websocket v1.5.3
github.com/jackc/pgx/v5 v5.10.0
github.com/mattn/go-sqlite3 v1.14.27
github.com/oschwald/maxminddb-golang v1.13.1
github.com/patrickmn/go-cache v2.1.0+incompatible
github.com/pquerna/otp v1.5.0
github.com/spf13/cobra v1.9.1
github.com/stretchr/testify v1.10.0
github.com/stretchr/testify v1.11.1
golang.org/x/image v0.35.0
google.golang.org/grpc v1.75.0
google.golang.org/protobuf v1.36.6
gorm.io/driver/sqlite v1.5.7
gorm.io/gorm v1.25.12
gorm.io/gorm v1.26.1
)
// Misuse of ServerConfig.PublicKeyCallback may cause authorization bypass in golang.org/x/crypto #1
// golang.org/x/crypto Vulnerable to Denial of Service (DoS) via Slow or Incomplete Key Exchange #3
require golang.org/x/crypto v0.37.0 // indirect
require golang.org/x/crypto v0.39.0 // indirect
// HTTP Proxy bypass using IPv6 Zone IDs in golang.org/x/net #2
// golang.org/x/net vulnerable to Cross-site Scripting #4
require golang.org/x/net v0.39.0 // indirect
require golang.org/x/net v0.41.0 // indirect
require (
filippo.io/edwards25519 v1.2.0 // indirect
github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc // indirect
github.com/bytedance/sonic v1.13.2 // indirect
github.com/bytedance/sonic/loader v0.2.4 // indirect
github.com/cloudwego/base64x v0.1.5 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dlclark/regexp2 v1.11.4 // indirect
github.com/gabriel-vasile/mimetype v1.4.8 // indirect
github.com/gin-contrib/sse v1.0.0 // indirect
github.com/go-jose/go-jose/v4 v4.1.1 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.26.0 // indirect
github.com/go-sourcemap/sourcemap v2.1.4+incompatible // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/google/go-cmp v0.6.0 // indirect
github.com/google/pprof v0.0.0-20240727154555-813a5fbdbec8 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.5 // indirect
github.com/json-iterator/go v1.1.12 // indirect
@@ -43,17 +60,19 @@ require (
github.com/kr/text v0.2.0 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-sqlite3 v1.14.27 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/pelletier/go-toml/v2 v2.2.3 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rogpeppe/go-internal v1.8.0 // indirect
github.com/spf13/pflag v1.0.6 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.2.12 // indirect
golang.org/x/arch v0.15.0 // indirect
golang.org/x/sys v0.32.0 // indirect
golang.org/x/text v0.24.0 // indirect
google.golang.org/protobuf v1.36.6 // indirect
golang.org/x/oauth2 v0.30.0 // indirect
golang.org/x/sync v0.19.0 // indirect
golang.org/x/sys v0.33.0 // indirect
golang.org/x/text v0.33.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250707201910-8d1bb00bc6a7 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
+86 -15
View File
@@ -1,3 +1,9 @@
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/Masterminds/semver/v3 v3.2.1 h1:RN9w6+7QoMeJVGyfmbcgs28Br8cvmnucEXnY0rYXWg0=
github.com/Masterminds/semver/v3 v3.2.1/go.mod h1:qvl/7zhW3nngYb5+80sSMF+FG2BjYrf8m9wsX0PNOMQ=
github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc h1:biVzkmvwrH8WK8raXaxBx6fRVTlJILwEwQGL1I/ByEI=
github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
github.com/bytedance/sonic v1.13.2 h1:8/H1FempDZqC4VqjptGo14QQlJx8VdZJegxs6wwfqpQ=
github.com/bytedance/sonic v1.13.2/go.mod h1:o68xyaF9u2gvVBuGHPlUVCy+ZfmNNO5ETf1+KgkJhz4=
github.com/bytedance/sonic/loader v0.1.1/go.mod h1:ncP89zfokxS5LZrJxl5z0UJcsk4M4yY2JpfqGeCtNLU=
@@ -6,19 +12,31 @@ github.com/bytedance/sonic/loader v0.2.4/go.mod h1:N8A3vUdtUebEY2/VQC0MyhYeKUFos
github.com/cloudwego/base64x v0.1.5 h1:XPciSp1xaq2VCSt6lF0phncD4koWyULpl5bUxbfCyP4=
github.com/cloudwego/base64x v0.1.5/go.mod h1:0zlkT4Wn5C6NdauXdJRhSKRlJvmclQ1hhJgA0rcu/8w=
github.com/cloudwego/iasm v0.2.0/go.mod h1:8rXZaNYT2n95jn+zTI1sDr+IgcD2GVs0nlbbQPiEFhY=
github.com/coreos/go-oidc/v3 v3.15.0 h1:R6Oz8Z4bqWR7VFQ+sPSvZPQv4x8M+sJkDO5ojgwlyAg=
github.com/coreos/go-oidc/v3 v3.15.0/go.mod h1:HaZ3szPaZ0e4r6ebqvsLWlk2Tn+aejfmrfah6hnSYEU=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dlclark/regexp2 v1.11.4 h1:rPYF9/LECdNymJufQKmri9gV604RvvABwgOA8un7yAo=
github.com/dlclark/regexp2 v1.11.4/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/dop251/goja v0.0.0-20251008123653-cf18d89f3cf6 h1:6dE1TmjqkY6tehR4A67gDNhvDtuZ54ocu7ab4K9o540=
github.com/dop251/goja v0.0.0-20251008123653-cf18d89f3cf6/go.mod h1:MxLav0peU43GgvwVgNbLAj1s/bSGboKkhuULvq/7hx4=
github.com/dop251/goja_nodejs v0.0.0-20251015164255-5e94316bedaf h1:gbmvliZnCut4NjaPSNOQlfqBoZ9C5Dpf72mHMMYhgVE=
github.com/dop251/goja_nodejs v0.0.0-20251015164255-5e94316bedaf/go.mod h1:Tb7Xxye4LX7cT3i8YLvmPMGCV92IOi4CDZvm/V8ylc0=
github.com/gabriel-vasile/mimetype v1.4.8 h1:FfZ3gj38NjllZIeJAmMhr+qKL8Wu+nOoI3GqacKw1NM=
github.com/gabriel-vasile/mimetype v1.4.8/go.mod h1:ByKUIKGjh1ODkGM1asKUbQZOLGrPjydw3hYPU2YU9t8=
github.com/gin-contrib/cors v1.7.5 h1:cXC9SmofOrRg0w9PigwGlHG3ztswH6bqq4vJVXnvYMk=
github.com/gin-contrib/cors v1.7.5/go.mod h1:4q3yi7xBEDDWKapjT2o1V7mScKDDr8k+jZ0fSquGoy0=
github.com/gin-contrib/sse v1.0.0 h1:y3bT1mUWUxDpW4JLQg/HnTqV4rozuW4tC9eFKTxYI9E=
github.com/gin-contrib/sse v1.0.0/go.mod h1:zNuFdwarAygJBht0NTKiSi3jRf6RbqeILZ9Sp6Slhe0=
github.com/gin-gonic/gin v1.10.0 h1:nTuyha1TYqgedzytsKYqna+DfLos46nTv2ygFy86HFU=
github.com/gin-gonic/gin v1.10.0/go.mod h1:4PMNQiOhvDRa013RKVbsiNwoyezlm2rm0uX/T7kzp5Y=
github.com/go-jose/go-jose/v4 v4.1.1 h1:JYhSgy4mXXzAdF3nUx3ygx347LRXJRrpgyU3adRmkAI=
github.com/go-jose/go-jose/v4 v4.1.1/go.mod h1:BdsZGqgdO3b6tTc6LSE56wcDbMMLuPsw5d4ZD5f94kA=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
@@ -27,17 +45,33 @@ github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJn
github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
github.com/go-playground/validator/v10 v10.26.0 h1:SP05Nqhjcvz81uJaRfEV0YBSSSGMc/iMaVtFbr3Sw2k=
github.com/go-playground/validator/v10 v10.26.0/go.mod h1:I5QpIEbmr8On7W0TktmJAumgzX4CA1XNl4ZmDuVHKKo=
github.com/go-sourcemap/sourcemap v2.1.4+incompatible h1:a+iTbH5auLKxaNwQFg0B+TCYl6lbukKPc7b5x0n1s6Q=
github.com/go-sourcemap/sourcemap v2.1.4+incompatible/go.mod h1:F8jJfvm2KbVjc5NqelyYJmf/v5J0dwNLS2mL4sNA1Jg=
github.com/go-sql-driver/mysql v1.10.0 h1:Q+1LV8DkHJvSYAdR83XzuhDaTykuDx0l6fkXxoWCWfw=
github.com/go-sql-driver/mysql v1.10.0/go.mod h1:M+cqaI7+xxXGG9swrdeUIoPG3Y3KCkF0pZej+SK+nWk=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/pprof v0.0.0-20240727154555-813a5fbdbec8 h1:FKHo8hFI3A+7w0aUQuYXQ+6EN5stWmeY/AZqtM8xk9k=
github.com/google/pprof v0.0.0-20240727154555-813a5fbdbec8/go.mod h1:K1liHPHnj73Fdn/EKuT8nrFqBihUSKXoLYU0BuatOYo=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ=
@@ -48,8 +82,11 @@ github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa02
github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE=
github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/knz/go-libedit v1.10.1/go.mod h1:MZTVkCWyz0oBc7JOWP3wNAzd002ZbM/5hgShxwh4x8M=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
@@ -65,10 +102,15 @@ github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9G
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/oschwald/maxminddb-golang v1.13.1 h1:G3wwjdN9JmIK2o/ermkHM+98oX5fS+k5MbwsmL4MRQE=
github.com/oschwald/maxminddb-golang v1.13.1/go.mod h1:K4pgV9N/GcK694KSTmVSDTODk4IsCNThNdTmnaBZ/F8=
github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
github.com/pelletier/go-toml/v2 v2.2.3 h1:YmeHyLY8mFWbdkNWwpr+qIL2bEqT0o95WSdkNHvL12M=
github.com/pelletier/go-toml/v2 v2.2.3/go.mod h1:MfCQTFTvCcUyyvvwm1+G6H/jORL20Xlb6rzQu9GuUkc=
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pquerna/otp v1.5.0 h1:NMMR+WrmaqXU4EzdGJEE1aUUI0AMRzsp96fFFWNPwxs=
github.com/pquerna/otp v1.5.0/go.mod h1:dkJfzwRKNiegxyNb54X/3fLwhCynbMspSyWKnvi1AEg=
github.com/rogpeppe/go-internal v1.8.0 h1:FCbCCtXNOY3UtUuHUYaghJg4y7Fd14rXifAYUAtL9R8=
github.com/rogpeppe/go-internal v1.8.0/go.mod h1:WmiCO8CzOY8rg0OYDC4/i/2WRWAB6poM+XZ2dLUbcbE=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
@@ -86,33 +128,62 @@ github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS4MhqMhdFk5YI=
github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08=
github.com/ugorji/go/codec v1.2.12 h1:9LC83zGrHhuUA9l16C9AHXAqEV/2wBQ4nkvumAE65EE=
github.com/ugorji/go/codec v1.2.12/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg=
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
go.opentelemetry.io/otel v1.37.0 h1:9zhNfelUvx0KBfu/gb+ZgeAfAgtWrfHJZcAqFC228wQ=
go.opentelemetry.io/otel v1.37.0/go.mod h1:ehE/umFRLnuLa/vSccNq9oS1ErUlkkK71gMcN34UG8I=
go.opentelemetry.io/otel/metric v1.37.0 h1:mvwbQS5m0tbmqML4NqK+e3aDiO02vsf/WgbsdpcPoZE=
go.opentelemetry.io/otel/metric v1.37.0/go.mod h1:04wGrZurHYKOc+RKeye86GwKiTb9FKm1WHtO+4EVr2E=
go.opentelemetry.io/otel/sdk v1.37.0 h1:ItB0QUqnjesGRvNcmAcU0LyvkVyGJ2xftD29bWdDvKI=
go.opentelemetry.io/otel/sdk v1.37.0/go.mod h1:VredYzxUvuo2q3WRcDnKDjbdvmO0sCzOvVAiY+yUkAg=
go.opentelemetry.io/otel/sdk/metric v1.37.0 h1:90lI228XrB9jCMuSdA0673aubgRobVZFhbjxHHspCPc=
go.opentelemetry.io/otel/sdk/metric v1.37.0/go.mod h1:cNen4ZWfiD37l5NhS+Keb5RXVWZWpRE+9WyVCpbo5ps=
go.opentelemetry.io/otel/trace v1.37.0 h1:HLdcFNbRQBE2imdSEgm/kwqmQj1Or1l/7bW6mxVK7z4=
go.opentelemetry.io/otel/trace v1.37.0/go.mod h1:TlgrlQ+PtQO5XFerSPUYG0JSgGyryXewPGyayAWSBS0=
golang.org/x/arch v0.15.0 h1:QtOrQd0bTUnhNVNndMpLHNWrDmYzZ2KDqSrEymqInZw=
golang.org/x/arch v0.15.0/go.mod h1:JmwW7aLIoRUKgaTzhkiEFxvcEiQGyOg9BMonBJUS7EE=
golang.org/x/crypto v0.37.0 h1:kJNSjF/Xp7kU0iB2Z+9viTPMW4EqqsrywMXLJOOsXSE=
golang.org/x/crypto v0.37.0/go.mod h1:vg+k43peMZ0pUMhYmVAWysMK35e6ioLh3wB8ZCAfbVc=
golang.org/x/net v0.39.0 h1:ZCu7HMWDxpXpaiKdhzIfaltL9Lp31x/3fCP11bc6/fY=
golang.org/x/net v0.39.0/go.mod h1:X7NRbYVEA+ewNkCNyJ513WmMdQ3BineSwVtN2zD/d+E=
golang.org/x/crypto v0.39.0 h1:SHs+kF4LP+f+p14esP5jAoDpHU8Gu/v9lFRK6IT5imM=
golang.org/x/crypto v0.39.0/go.mod h1:L+Xg3Wf6HoL4Bn4238Z6ft6KfEpN0tJGo53AAPC632U=
golang.org/x/image v0.35.0 h1:LKjiHdgMtO8z7Fh18nGY6KDcoEtVfsgLDPeLyguqb7I=
golang.org/x/image v0.35.0/go.mod h1:MwPLTVgvxSASsxdLzKrl8BRFuyqMyGhLwmC+TO1Sybk=
golang.org/x/net v0.41.0 h1:vBTly1HeNPEn3wtREYfy4GZ/NECgw2Cnl+nK6Nz3uvw=
golang.org/x/net v0.41.0/go.mod h1:B/K4NNqkfmg07DQYrbwvSluqCJOOXwUjeb/5lOisjbA=
golang.org/x/oauth2 v0.30.0 h1:dnDm7JmhM45NNpd8FDDeLhK6FwqbOf4MLCM9zb1BOHI=
golang.org/x/oauth2 v0.30.0/go.mod h1:B++QgG3ZKulg6sRPGD/mqlHQs5rB3Ml9erfeDY7xKlU=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.32.0 h1:s77OFDvIQeibCmezSnk/q6iAfkdiQaJi4VzroCFrN20=
golang.org/x/sys v0.32.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/text v0.24.0 h1:dd5Bzh4yt5KYA8f9CJHCP4FB4D51c2c6JvN37xJJkJ0=
golang.org/x/text v0.24.0/go.mod h1:L8rBsPeo2pSS+xqN0d5u2ikmjtmoJbDBT1b7nHvFCdU=
golang.org/x/sys v0.33.0 h1:q3i8TbbEz+JRD9ywIRlyRAQbM0qF7hu24q3teo2hbuw=
golang.org/x/sys v0.33.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/text v0.33.0 h1:B3njUFyqtHDUI5jMn1YIr5B0IE2U0qck04r6d4KPAxE=
golang.org/x/text v0.33.0/go.mod h1:LuMebE6+rBincTi9+xWTY8TztLzKHc/9C1uBCG27+q8=
gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk=
gonum.org/v1/gonum v0.16.0/go.mod h1:fef3am4MQ93R2HHpKnLk4/Tbh/s0+wqD5nfa6Pnwy4E=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250707201910-8d1bb00bc6a7 h1:pFyd6EwwL2TqFf8emdthzeX+gZE1ElRq3iM8pui4KBY=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250707201910-8d1bb00bc6a7/go.mod h1:qQ0YXyHHx3XkvlzUtpXDkS29lDSafHMZBAZDc03LQ3A=
google.golang.org/grpc v1.75.0 h1:+TW+dqTd2Biwe6KKfhE5JpiYIBWq865PhKGSXiivqt4=
google.golang.org/grpc v1.75.0/go.mod h1:JtPAzKiq4v1xcAB2hydNlWI2RnF85XXcV0mhKXr2ecQ=
google.golang.org/protobuf v1.36.6 h1:z1NpPI8ku2WgiWnf+t9wTPsn6eP1L7ksHUlkfLvd9xY=
google.golang.org/protobuf v1.36.6/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gorm.io/driver/sqlite v1.5.7 h1:8NvsrhP0ifM7LX9G4zPB97NwovUakUxc+2V2uuf3Z1I=
gorm.io/driver/sqlite v1.5.7/go.mod h1:U+J8craQU6Fzkcvu8oLeAQmi50TkwPEhHDEjQZXDah4=
gorm.io/gorm v1.25.12 h1:I0u8i2hWQItBq1WfE0o2+WuL9+8L21K9e2HHSTE/0f8=
gorm.io/gorm v1.25.12/go.mod h1:xh7N7RHfYlNc5EmcI/El95gXusucDrQnHXe0+CgWcLQ=
gorm.io/gorm v1.26.1 h1:ghB2gUI9FkS46luZtn6DLZ0f6ooBJ5IbVej2ENFDjRw=
gorm.io/gorm v1.26.1/go.mod h1:8Z33v652h4//uMA76KjeDH8mJXPm1QNCYrMeatR0DOE=
nullprogram.com/x/optparse v1.0.0/go.mod h1:KdyPE+Igbe0jQUrVfMqDMeJQIJZEuyV7pjYmp6pbG50=
+635
View File
@@ -0,0 +1,635 @@
#!/bin/bash
# Color definitions for terminal output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
NC='\033[0m' # No Color
# Logging functions
log_info() {
echo -e "$1"
}
log_success() {
echo -e "${GREEN}$1${NC}"
}
log_error() {
echo -e "${RED}$1${NC}"
}
log_step() {
echo -e "${YELLOW}$1${NC}"
}
# Global variables
INSTALL_DIR="/opt/komari"
DATA_DIR="/opt/komari"
SERVICE_NAME="komari"
BINARY_PATH="$INSTALL_DIR/komari"
DEFAULT_PORT="25774"
LISTEN_PORT=""
REPO="komari-monitor/komari"
# 发布通道: stable(稳定版)或 snapshot(快照版)
CHANNEL="stable"
# TUI 工具: whiptail / dialog / 空(回退纯文本)
TUI_TOOL=""
# ==========================================================
# TUI / 交互层
# ==========================================================
# 检测可用的 TUI 工具
detect_tui() {
if command -v whiptail >/dev/null 2>&1; then
TUI_TOOL="whiptail"
elif command -v dialog >/dev/null 2>&1; then
TUI_TOOL="dialog"
else
TUI_TOOL=""
fi
}
# 是否启用 TUI
tui_enabled() {
[ -n "$TUI_TOOL" ]
}
# 菜单选择
# 用法: ui_menu "标题" "提示" tag1 "item1" tag2 "item2" ...
# 返回: 选中的 tag(输出到 stdout),取消返回非零
ui_menu() {
local title="$1"; shift
local prompt="$1"; shift
if tui_enabled; then
$TUI_TOOL --title "$title" --menu "$prompt" 20 70 10 "$@" 3>&1 1>&2 2>&3
return $?
fi
# 纯文本回退
{
echo
echo "=============================================================="
echo " $title"
echo "=============================================================="
echo "$prompt"
echo
local tag item
local args=("$@")
local i=0
while [ $i -lt ${#args[@]} ]; do
tag="${args[$i]}"
item="${args[$((i + 1))]}"
echo " $tag) $item"
i=$((i + 2))
done
echo
} >&2
local choice
read -r -p "输入选项: " choice >&2
echo "$choice"
}
# 输入框
# 用法: ui_input "标题" "提示" "默认值"
# 返回: 输入内容(输出到 stdout),取消返回非零
ui_input() {
local title="$1"
local prompt="$2"
local default="$3"
if tui_enabled; then
$TUI_TOOL --title "$title" --inputbox "$prompt" 12 70 "$default" 3>&1 1>&2 2>&3
return $?
fi
local input
read -r -p "$prompt [默认: $default]: " input >&2
if [ -z "$input" ]; then
echo "$default"
else
echo "$input"
fi
}
# 是/否确认
# 用法: ui_yesno "标题" "提示"
# 返回: 0 表示 是,1 表示 否
ui_yesno() {
local title="$1"
local prompt="$2"
if tui_enabled; then
$TUI_TOOL --title "$title" --yesno "$prompt" 12 70
return $?
fi
local confirm
read -r -p "$prompt (Y/n): " confirm >&2
if [[ $confirm =~ ^[Nn]$ ]]; then
return 1
fi
return 0
}
# 信息提示框
# 用法: ui_msgbox "标题" "内容"
ui_msgbox() {
local title="$1"
local content="$2"
if tui_enabled; then
$TUI_TOOL --title "$title" --msgbox "$content" 20 72
return
fi
echo
echo "=============================================================="
echo " $title"
echo "=============================================================="
echo -e "$content"
echo "=============================================================="
read -r -p "按回车键继续..." _
}
# 显示横幅(仅纯文本模式)
show_banner() {
if tui_enabled; then
return
fi
clear
echo "=============================================================="
echo " Komari Monitoring System Installer"
echo " https://github.com/komari-monitor/komari"
echo "=============================================================="
echo
}
# 选择发布通道,结果写入全局变量 CHANNEL
select_channel() {
local choice
choice=$(ui_menu "选择发布通道" "请选择要使用的发布通道:" \
"stable" "稳定版 (推荐)" \
"snapshot" "快照版 (最新功能)")
case "$choice" in
snapshot|2)
CHANNEL="snapshot"
;;
stable|1|"")
CHANNEL="stable"
;;
*)
CHANNEL="stable"
;;
esac
log_info "已选择通道: $CHANNEL"
}
# ==========================================================
# 基础检查
# ==========================================================
# Check if running as root
check_root() {
if [ "$EUID" -ne 0 ]; then
log_error "请使用 root 权限运行此脚本"
exit 1
fi
}
# Check for systemd
check_systemd() {
if ! command -v systemctl >/dev/null 2>&1; then
return 1
else
return 0
fi
}
# Detect system architecture
detect_arch() {
local arch=$(uname -m)
case $arch in
x86_64)
echo "amd64"
;;
aarch64)
echo "arm64"
;;
i386|i686)
echo "386"
;;
riscv64)
echo "riscv64"
;;
*)
log_error "不支持的架构: $arch"
exit 1
;;
esac
}
# Check if Komari is already installed
is_installed() {
if [ -f "$BINARY_PATH" ]; then
return 0 # 0 means true in bash exit codes
else
return 1 # 1 means false
fi
}
# Install dependencies
install_dependencies() {
log_step "检查并安装依赖..."
if ! command -v curl >/dev/null 2>&1; then
if command -v apt >/dev/null 2>&1; then
log_info "使用 apt 安装依赖..."
apt update
apt install -y curl
elif command -v yum >/dev/null 2>&1; then
log_info "使用 yum 安装依赖..."
yum install -y curl
elif command -v apk >/dev/null 2>&1; then
log_info "使用 apk 安装依赖..."
apk add curl
else
log_error "未找到支持的包管理器 (apt/yum/apk)"
exit 1
fi
fi
}
# Get download URL based on channel
get_download_url() {
local arch=$1
local file_name="komari-linux-${arch}"
if [ "$CHANNEL" = "snapshot" ]; then
# 获取最新的 snapshot 预发布版本
log_info "获取最新 snapshot 版本..." >&2
local latest_snapshot=$(curl -s "https://api.github.com/repos/${REPO}/releases" | grep '"tag_name"' | grep 'Snapshot-' | head -1 | sed -e 's/.*"tag_name": *"//' -e 's/".*//')
if [ -z "$latest_snapshot" ]; then
log_error "未找到 snapshot 版本" >&2
return 1
fi
log_info "最新 snapshot 版本: $latest_snapshot" >&2
echo "https://github.com/${REPO}/releases/download/${latest_snapshot}/${file_name}"
else
# 稳定版:使用 latest
echo "https://github.com/${REPO}/releases/latest/download/${file_name}"
fi
}
# ==========================================================
# 业务操作
# ==========================================================
# Binary installation
install_binary() {
log_step "开始二进制安装..."
if is_installed; then
ui_msgbox "提示" "Komari 已安装。\n如需升级,请使用主菜单中的升级选项。"
return
fi
# 选择发布通道
select_channel
# 监听端口输入,校验范围 1-65535
while true; do
local input_port
input_port=$(ui_input "监听端口" "请输入 Komari 的监听端口 (1-65535):" "$DEFAULT_PORT")
# 取消输入
if [ $? -ne 0 ]; then
log_info "安装已取消"
return
fi
if [[ -z "$input_port" ]]; then
LISTEN_PORT="$DEFAULT_PORT"
break
elif [[ "$input_port" =~ ^[0-9]+$ ]] && (( input_port >= 1 && input_port <= 65535 )); then
LISTEN_PORT="$input_port"
break
else
ui_msgbox "错误" "端口号无效,请输入 1-65535 之间的数字。"
fi
done
install_dependencies
local arch=$(detect_arch)
log_info "检测到架构: $arch"
log_step "创建安装目录: $INSTALL_DIR"
mkdir -p "$INSTALL_DIR"
log_step "创建数据目录: $DATA_DIR"
mkdir -p "$DATA_DIR"
local download_url=$(get_download_url "$arch")
if [ $? -ne 0 ]; then
ui_msgbox "错误" "获取下载链接失败,请检查网络连接或稍后重试。"
return 1
fi
log_step "下载 Komari 二进制文件..."
log_info "URL: $download_url"
if ! curl -L -o "$BINARY_PATH" "$download_url"; then
ui_msgbox "错误" "下载失败,请检查网络连接。"
return 1
fi
chmod +x "$BINARY_PATH"
log_success "Komari 二进制文件安装完成: $BINARY_PATH"
if ! check_systemd; then
ui_msgbox "安装完成" "警告:未检测到 systemd,已跳过服务创建。\n\n您可以手动运行 Komari:\n $BINARY_PATH server -l 0.0.0.0:$LISTEN_PORT"
return
fi
create_systemd_service "$LISTEN_PORT"
systemctl daemon-reload
systemctl enable ${SERVICE_NAME}.service
systemctl start ${SERVICE_NAME}.service
if systemctl is-active --quiet ${SERVICE_NAME}.service; then
log_success "Komari 服务启动成功"
log_step "正在获取初始密码..."
sleep 5
local password=$(journalctl -u ${SERVICE_NAME} --since "1 minute ago" | grep "admin account created." | tail -n 1 | sed -e 's/.*admin account created.//')
if [ -z "$password" ]; then
log_error "未能获取初始密码,请检查日志"
fi
show_access_info "$password" "$LISTEN_PORT"
else
ui_msgbox "错误" "Komari 服务启动失败。\n\n查看日志: journalctl -u ${SERVICE_NAME} -f"
return 1
fi
}
# Create systemd service file
create_systemd_service() {
local port="$1"
log_step "创建 systemd 服务..."
local service_file="/etc/systemd/system/${SERVICE_NAME}.service"
cat > "$service_file" << EOF
[Unit]
Description=Komari Monitor Service
After=network.target
[Service]
Type=simple
ExecStart=${BINARY_PATH} server -l 0.0.0.0:${port}
WorkingDirectory=${DATA_DIR}
Restart=always
User=root
[Install]
WantedBy=multi-user.target
EOF
log_success "systemd 服务文件创建完成"
}
# Show access information
show_access_info() {
local password=$1
local port=${2:-$DEFAULT_PORT}
local ip=$(hostname -I | awk '{print $1}')
local content="安装完成!\n\n"
content+="访问信息:\n"
content+=" URL: http://${ip}:${port}\n"
if [ -n "$password" ]; then
content+=" 初始登录信息(仅显示一次): ${password}\n"
fi
content+="\n服务管理命令:\n"
content+=" 状态: systemctl status $SERVICE_NAME\n"
content+=" 启动: systemctl start $SERVICE_NAME\n"
content+=" 停止: systemctl stop $SERVICE_NAME\n"
content+=" 重启: systemctl restart $SERVICE_NAME\n"
content+=" 日志: journalctl -u $SERVICE_NAME -f"
ui_msgbox "安装完成" "$content"
}
# Upgrade function
upgrade_komari() {
log_step "升级 Komari..."
if ! is_installed; then
ui_msgbox "错误" "Komari 未安装。请先安装它。"
return 1
fi
if ! check_systemd; then
ui_msgbox "错误" "未检测到 systemd。无法管理服务。"
return 1
fi
# 选择发布通道
select_channel
log_step "停止 Komari 服务..."
systemctl stop ${SERVICE_NAME}.service
log_step "备份当前二进制文件..."
cp "$BINARY_PATH" "${BINARY_PATH}.backup.$(date +%Y%m%d_%H%M%S)"
local arch=$(detect_arch)
local download_url=$(get_download_url "$arch")
if [ $? -ne 0 ]; then
log_error "获取下载链接失败,正在从备份恢复"
mv "${BINARY_PATH}.backup."* "$BINARY_PATH"
systemctl start ${SERVICE_NAME}.service
ui_msgbox "错误" "获取下载链接失败,已从备份恢复。"
return 1
fi
log_step "下载最新版本..."
if ! curl -L -o "$BINARY_PATH" "$download_url"; then
log_error "下载失败,正在从备份恢复"
mv "${BINARY_PATH}.backup."* "$BINARY_PATH"
systemctl start ${SERVICE_NAME}.service
ui_msgbox "错误" "下载失败,已从备份恢复。"
return 1
fi
chmod +x "$BINARY_PATH"
log_step "重启 Komari 服务..."
systemctl start ${SERVICE_NAME}.service
if systemctl is-active --quiet ${SERVICE_NAME}.service; then
ui_msgbox "升级完成" "Komari 升级成功 (通道: $CHANNEL)。"
else
ui_msgbox "错误" "服务在升级后未能启动,请检查日志。"
fi
}
# Uninstall function
uninstall_komari() {
log_step "卸载 Komari..."
if ! is_installed; then
ui_msgbox "提示" "Komari 未安装。"
return 0
fi
if ! ui_yesno "确认卸载" "这将删除 Komari 二进制文件和服务。\n\n您确定要继续吗?"; then
log_info "卸载已取消"
return 0
fi
if check_systemd; then
log_step "停止并禁用服务..."
systemctl stop ${SERVICE_NAME}.service >/dev/null 2>&1
systemctl disable ${SERVICE_NAME}.service >/dev/null 2>&1
rm -f "/etc/systemd/system/${SERVICE_NAME}.service"
systemctl daemon-reload
log_success "systemd 服务已删除"
fi
log_step "删除二进制文件..."
rm -f "$BINARY_PATH"
# 尝试在目录为空时删除该目录
rmdir "$INSTALL_DIR" 2>/dev/null || log_info "数据目录 $INSTALL_DIR 不为空,未删除"
log_success "Komari 二进制文件已删除"
ui_msgbox "卸载完成" "Komari 卸载完成。\n\n数据文件保留在 $DATA_DIR"
}
# Show service status
show_status() {
if ! is_installed; then
ui_msgbox "错误" "Komari 未安装。"
return
fi
if ! check_systemd; then
ui_msgbox "错误" "未检测到 systemd。无法获取服务状态。"
return
fi
if tui_enabled; then
local status_output
status_output=$(systemctl status ${SERVICE_NAME}.service --no-pager -l 2>&1)
ui_msgbox "服务状态" "$status_output"
else
log_step "Komari 服务状态:"
systemctl status ${SERVICE_NAME}.service --no-pager -l
read -r -p "按回车键继续..." _
fi
}
# Show service logs
show_logs() {
if ! is_installed; then
ui_msgbox "错误" "Komari 未安装。"
return
fi
if ! check_systemd; then
ui_msgbox "错误" "未检测到 systemd。无法获取服务日志。"
return
fi
# 日志为实时流,直接在终端显示
if tui_enabled; then
clear
fi
log_step "查看 Komari 服务日志 (按 Ctrl+C 退出)..."
journalctl -u ${SERVICE_NAME} -f --no-pager
}
# Restart service
restart_service() {
if ! is_installed; then
ui_msgbox "错误" "Komari 未安装。"
return
fi
if ! check_systemd; then
ui_msgbox "错误" "未检测到 systemd。无法重启服务。"
return
fi
log_step "重启 Komari 服务..."
systemctl restart ${SERVICE_NAME}.service
if systemctl is-active --quiet ${SERVICE_NAME}.service; then
ui_msgbox "成功" "服务重启成功。"
else
ui_msgbox "错误" "服务重启失败,请检查日志。"
fi
}
# Stop service
stop_service() {
if ! is_installed; then
ui_msgbox "错误" "Komari 未安装。"
return
fi
if ! check_systemd; then
ui_msgbox "错误" "未检测到 systemd。无法停止服务。"
return
fi
log_step "停止 Komari 服务..."
systemctl stop ${SERVICE_NAME}.service
ui_msgbox "成功" "服务已停止。"
}
# Main menu
main_menu() {
while true; do
show_banner
local choice
choice=$(ui_menu "Komari 监控系统安装器" "请选择操作:" \
"1" "安装 Komari" \
"2" "升级 Komari" \
"3" "卸载 Komari" \
"4" "查看状态" \
"5" "查看日志" \
"6" "重启服务" \
"7" "停止服务" \
"8" "退出")
# 用户在 TUI 中取消(ESC/Cancel)则退出
if [ $? -ne 0 ] && tui_enabled; then
clear
exit 0
fi
case $choice in
1) install_binary ;;
2) upgrade_komari ;;
3) uninstall_komari ;;
4) show_status ;;
5) show_logs ;;
6) restart_service ;;
7) stop_service ;;
8)
tui_enabled && clear
exit 0
;;
*) ui_msgbox "错误" "无效选项" ;;
esac
# 纯文本模式下单次执行后退出循环(保持原有行为,避免输出被覆盖)
if ! tui_enabled; then
break
fi
done
}
# Main execution
check_root
detect_tui
main_menu
+13
View File
@@ -1,9 +1,22 @@
package main
import (
"log"
"log/slog"
"github.com/komari-monitor/komari/cmd"
"github.com/komari-monitor/komari/utils"
logutil "github.com/komari-monitor/komari/utils/log"
)
func main() {
if utils.VersionHash == "unknown" {
logutil.SetupGlobalLogger(slog.LevelDebug)
} else {
logutil.SetupGlobalLogger(slog.LevelInfo)
}
log.Printf("Komari Monitor %s (hash: %s)", utils.CurrentVersion, utils.VersionHash)
cmd.Execute()
}
+635
View File
@@ -0,0 +1,635 @@
package config
import (
"encoding/json"
"fmt"
"log/slog"
"reflect"
"strings"
"sync"
"gorm.io/gorm"
"gorm.io/gorm/clause"
)
type ConfigItem struct {
Key string `gorm:"primaryKey;column:key;type:text"`
Value string `gorm:"column:value;type:text"` // 存 JSON 字符串
}
func (ConfigItem) TableName() string {
return "configs"
}
var (
db *gorm.DB
SetDb = func(gdb *gorm.DB) {
db = gdb
if err := db.AutoMigrate(&ConfigItem{}); err != nil {
panic("failed to migrate config item table: " + err.Error())
}
}
)
// Get 获取原始值 (反序列化为 interface{})
func Get(key string, defaul ...any) (any, error) {
var item ConfigItem
err := db.First(&item, "key = ?", key).Error
if err != nil {
if len(defaul) > 0 {
v := defaul[0]
err = Set(key, v)
return v, err
}
return nil, err
}
var val any
if err := json.Unmarshal([]byte(item.Value), &val); err != nil {
return nil, err
}
return val, nil
}
// GetAs 获取并转换为指定类型 (泛型),支持数值类型自动转换
func GetAs[T any](key string, defaul ...any) (T, error) {
var t T
var item ConfigItem
err := db.First(&item, "key = ?", key).Error
if err != nil {
if len(defaul) > 0 {
// 尝试直接类型断言
if v, ok := defaul[0].(T); ok {
err = Set(key, v)
return v, err
}
// 尝试类型转换
val := reflect.ValueOf(&t).Elem()
if err := convertAndSet(defaul[0], val); err != nil {
return t, fmt.Errorf("default value type mismatch: expected %T, got %T", t, defaul[0])
}
err = Set(key, t)
return t, err
}
return t, err
}
// 先尝试直接反序列化
if err = json.Unmarshal([]byte(item.Value), &t); err != nil {
// 尝试通用解析后转换
var generic any
if err := json.Unmarshal([]byte(item.Value), &generic); err != nil {
return t, err
}
val := reflect.ValueOf(&t).Elem()
if err := convertAndSet(generic, val); err != nil {
return t, err
}
}
return t, nil
}
// GetMany 获取多个配置项,keys 为 map[key]defaultValue
// 如果 defaultValue 为 nil,则数据库不存在时不写入
// 如果 defaultValue 不为 nil,则数据库不存在时写入默认值
func GetMany(keys map[string]any) (map[string]any, error) {
var items []ConfigItem
result := make(map[string]any)
keyList := make([]string, 0, len(keys))
for k := range keys {
keyList = append(keyList, k)
}
if len(keyList) == 0 {
return result, nil
}
if err := db.Where("key IN ?", keyList).Find(&items).Error; err != nil {
return nil, err
}
foundKeys := make(map[string]bool)
for _, item := range items {
var parsed any
if err := json.Unmarshal([]byte(item.Value), &parsed); err == nil {
result[item.Key] = parsed
foundKeys[item.Key] = true
}
}
// 收集需要写入数据库的默认值
var toInsert []ConfigItem
for k, def := range keys {
if _, found := foundKeys[k]; !found {
if def != nil {
result[k] = def
// 序列化后加入待写入列表
jsonBytes, err := json.Marshal(def)
if err != nil {
slog.Warn("marshal default value failed", "key", k, "error", err)
continue
}
toInsert = append(toInsert, ConfigItem{
Key: k,
Value: string(jsonBytes),
})
}
}
}
// 批量写入默认值到数据库
if len(toInsert) > 0 {
if err := db.Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "key"}},
DoUpdates: clause.AssignmentColumns([]string{"value"}),
}).Create(&toInsert).Error; err != nil {
slog.Warn("batch insert default config failed", "error", err)
}
}
return result, nil
}
// GetManyAs 将多个配置项映射到一个结构体中,json tag 作为 Key
// 支持 default tag 作为默认值,如果数据库中不存在且有 default tag 则写入数据库
// 没有 default tag 的字段使用零值,不写入数据库
func GetManyAs[T any]() (*T, error) {
var t T
val := reflect.ValueOf(&t).Elem()
typ := val.Type()
type fieldInfo struct {
index int
key string
hasDefault bool
defaultVal string
}
fields := make([]fieldInfo, 0)
keys := make([]string, 0)
for i := 0; i < val.NumField(); i++ {
field := typ.Field(i)
jsonTag := field.Tag.Get("json")
if jsonTag == "" || jsonTag == "-" {
continue
}
// 解析 json tag,处理 "key,omitempty" 格式
key := strings.Split(jsonTag, ",")[0]
if key == "" || key == "-" {
continue
}
defaultTag := field.Tag.Get("default")
hasDefault := defaultTag != "" || field.Tag.Get("default") == ""
// 检查是否显式定义了 default tag (即使值为空)
_, hasDefault = field.Tag.Lookup("default")
fields = append(fields, fieldInfo{
index: i,
key: key,
hasDefault: hasDefault,
defaultVal: defaultTag,
})
keys = append(keys, key)
}
if len(keys) == 0 {
return &t, nil
}
var items []ConfigItem
if err := db.Where("key IN ?", keys).Find(&items).Error; err != nil {
return nil, err
}
// 建立数据库中存在的 key 映射
foundItems := make(map[string]string) // key -> value
for _, item := range items {
foundItems[item.Key] = item.Value
}
// 需要写入数据库的新配置项
var toInsert []ConfigItem
for _, fi := range fields {
fieldVal := val.Field(fi.index)
if !fieldVal.CanSet() {
continue
}
if dbValue, found := foundItems[fi.key]; found {
// 数据库中存在,使用数据库值
if err := unmarshalToField(dbValue, fieldVal); err != nil {
slog.Warn("unmarshal config failed", "key", fi.key, "error", err)
}
} else if fi.hasDefault {
// 数据库中不存在,但有 default tag,解析默认值并写入数据库
if err := parseDefaultToField(fi.defaultVal, fieldVal); err != nil {
slog.Warn("parse default value failed", "key", fi.key, "error", err)
continue
}
// 序列化后写入数据库
jsonBytes, err := json.Marshal(fieldVal.Interface())
if err != nil {
slog.Warn("marshal default value failed", "key", fi.key, "error", err)
continue
}
toInsert = append(toInsert, ConfigItem{
Key: fi.key,
Value: string(jsonBytes),
})
}
// 没有 default tag 且数据库中不存在,保持零值,不写入数据库
}
// 批量写入默认值到数据库
if len(toInsert) > 0 {
if err := db.Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "key"}},
DoUpdates: clause.AssignmentColumns([]string{"value"}),
}).Create(&toInsert).Error; err != nil {
slog.Warn("batch insert default config failed", "error", err)
}
}
return &t, nil
}
// unmarshalToField 将 JSON 字符串反序列化到字段,支持数值类型转换
func unmarshalToField(jsonStr string, fieldVal reflect.Value) error {
target := reflect.New(fieldVal.Type()).Interface()
if err := json.Unmarshal([]byte(jsonStr), target); err != nil {
// 尝试通用解析后转换
var generic any
if err := json.Unmarshal([]byte(jsonStr), &generic); err != nil {
return err
}
return convertAndSet(generic, fieldVal)
}
fieldVal.Set(reflect.ValueOf(target).Elem())
return nil
}
// parseDefaultToField 解析 default tag 值到字段
func parseDefaultToField(defaultVal string, fieldVal reflect.Value) error {
kind := fieldVal.Kind()
switch kind {
case reflect.String:
fieldVal.SetString(defaultVal)
case reflect.Bool:
fieldVal.SetBool(defaultVal == "true" || defaultVal == "1")
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
var v int64
if defaultVal != "" {
if _, err := fmt.Sscanf(defaultVal, "%d", &v); err != nil {
// 尝试解析浮点数后转换
var f float64
if _, err := fmt.Sscanf(defaultVal, "%f", &f); err != nil {
return err
}
v = int64(f)
}
}
fieldVal.SetInt(v)
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
var v uint64
if defaultVal != "" {
if _, err := fmt.Sscanf(defaultVal, "%d", &v); err != nil {
var f float64
if _, err := fmt.Sscanf(defaultVal, "%f", &f); err != nil {
return err
}
v = uint64(f)
}
}
fieldVal.SetUint(v)
case reflect.Float32, reflect.Float64:
var v float64
if defaultVal != "" {
if _, err := fmt.Sscanf(defaultVal, "%f", &v); err != nil {
return err
}
}
fieldVal.SetFloat(v)
default:
// 对于复杂类型,尝试 JSON 解析
if defaultVal == "" {
return nil // 保持零值
}
target := reflect.New(fieldVal.Type()).Interface()
if err := json.Unmarshal([]byte(defaultVal), target); err != nil {
return err
}
fieldVal.Set(reflect.ValueOf(target).Elem())
}
return nil
}
// convertAndSet 通用类型转换并设置字段值
func convertAndSet(val any, fieldVal reflect.Value) error {
if val == nil {
return nil
}
targetType := fieldVal.Type()
v := reflect.ValueOf(val)
// 直接类型匹配
if v.Type().AssignableTo(targetType) {
fieldVal.Set(v)
return nil
}
// 类型可转换
if v.Type().ConvertibleTo(targetType) {
fieldVal.Set(v.Convert(targetType))
return nil
}
// 数值类型特殊处理 (JSON 数字默认解析为 float64)
if f, ok := val.(float64); ok {
switch fieldVal.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
fieldVal.SetInt(int64(f))
return nil
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
fieldVal.SetUint(uint64(f))
return nil
case reflect.Float32, reflect.Float64:
fieldVal.SetFloat(f)
return nil
}
}
// JSON 回环转换
b, err := json.Marshal(val)
if err != nil {
return err
}
target := reflect.New(targetType).Interface()
if err := json.Unmarshal(b, target); err != nil {
return err
}
fieldVal.Set(reflect.ValueOf(target).Elem())
return nil
}
func GetAll() (map[string]any, error) {
var items []ConfigItem
result := make(map[string]any)
if err := db.Find(&items).Error; err != nil {
return nil, err
}
for _, item := range items {
var parsed any
if err := json.Unmarshal([]byte(item.Value), &parsed); err == nil {
result[item.Key] = parsed
}
}
return result, nil
}
// Set 设置单个配置
func Set(key string, value any) error {
oldVal := map[string]any{}
{
var oldItem ConfigItem
if err := db.First(&oldItem, "key = ?", key).Error; err == nil {
var parsed any
if err := json.Unmarshal([]byte(oldItem.Value), &parsed); err == nil {
oldVal[key] = parsed
}
}
}
bytes, err := json.Marshal(value)
if err != nil {
return err
}
item := ConfigItem{
Key: key,
Value: string(bytes),
}
err = db.Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "key"}},
DoUpdates: clause.AssignmentColumns([]string{"value"}),
}).Create(&item).Error
if err != nil {
return err
}
newVal := map[string]any{key: value}
publishEvent(oldVal, newVal)
return nil
}
// SetMany 将结构体保存为多个配置项
func SetManyAs[T any](config T) error {
val := reflect.ValueOf(config)
if val.Kind() == reflect.Ptr {
val = val.Elem()
}
typ := val.Type()
var items []ConfigItem
for i := 0; i < val.NumField(); i++ {
fieldType := typ.Field(i)
tag := fieldType.Tag.Get("json")
if tag == "" || tag == "-" {
continue
}
fieldValue := val.Field(i).Interface()
bytes, err := json.Marshal(fieldValue)
if err != nil {
return fmt.Errorf("marshal field %s failed: %w", fieldType.Name, err)
}
items = append(items, ConfigItem{
Key: tag,
Value: string(bytes),
})
}
if len(items) == 0 {
return nil
}
keys := make([]string, 0, len(items))
newVal := make(map[string]any, len(items))
for _, it := range items {
keys = append(keys, it.Key)
var parsed any
if err := json.Unmarshal([]byte(it.Value), &parsed); err == nil {
newVal[it.Key] = parsed
}
}
oldVal := map[string]any{}
if len(keys) > 0 {
var oldItems []ConfigItem
if err := db.Where("key IN ?", keys).Find(&oldItems).Error; err == nil {
for _, oi := range oldItems {
var parsed any
if err := json.Unmarshal([]byte(oi.Value), &parsed); err == nil {
oldVal[oi.Key] = parsed
}
}
}
}
err := db.Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "key"}},
DoUpdates: clause.AssignmentColumns([]string{"value"}),
}).Create(&items).Error
if err != nil {
return err
}
publishEvent(oldVal, newVal)
return nil
}
func SetMany(cst map[string]any) error {
var items []ConfigItem
for k, v := range cst {
bytes, err := json.Marshal(v)
if err != nil {
return fmt.Errorf("marshal key %s failed: %w", k, err)
}
items = append(items, ConfigItem{
Key: k,
Value: string(bytes),
})
}
if len(items) == 0 {
return nil
}
keys := make([]string, 0, len(items))
newVal := make(map[string]any, len(items))
for _, it := range items {
keys = append(keys, it.Key)
var parsed any
if err := json.Unmarshal([]byte(it.Value), &parsed); err == nil {
newVal[it.Key] = parsed
}
}
oldVal := map[string]any{}
if len(keys) > 0 {
var oldItems []ConfigItem
if err := db.Where("key IN ?", keys).Find(&oldItems).Error; err == nil {
for _, oi := range oldItems {
var parsed any
if err := json.Unmarshal([]byte(oi.Value), &parsed); err == nil {
oldVal[oi.Key] = parsed
}
}
}
}
err := db.Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "key"}},
DoUpdates: clause.AssignmentColumns([]string{"value"}),
}).Create(&items).Error
if err != nil {
return err
}
publishEvent(oldVal, newVal)
return nil
}
type ConfigEvent struct {
Old map[string]any
New map[string]any
}
func (e ConfigEvent) IsChanged(key string) bool {
oldVal, oldOk := e.Old[key]
newVal, newOk := e.New[key]
if !oldOk && !newOk {
return false
}
if oldOk != newOk {
return true
}
return !reflect.DeepEqual(oldVal, newVal)
}
func IsChangedT[T any](e ConfigEvent, key string) (bool, T) {
changed := e.IsChanged(key)
var zero T
val, ok := e.New[key]
if !ok {
val, ok = e.Old[key]
if !ok {
return changed, zero
}
}
if val == nil {
return changed, zero
}
// Fast path: direct assertion.
if cast, ok := val.(T); ok {
return changed, cast
}
// Try reflection-based conversion (covers numeric conversions, etc.).
targetType := reflect.TypeOf((*T)(nil)).Elem()
v := reflect.ValueOf(val)
if v.IsValid() {
if v.Type().AssignableTo(targetType) {
return changed, v.Interface().(T)
}
if v.Type().ConvertibleTo(targetType) {
converted := v.Convert(targetType)
return changed, converted.Interface().(T)
}
}
// Fallback: JSON roundtrip for map/struct and other loosely typed values.
if b, err := json.Marshal(val); err == nil {
var out T
if err := json.Unmarshal(b, &out); err == nil {
return changed, out
}
}
return changed, zero
}
// ConfigSubscriber handles config events
type ConfigSubscriber func(event ConfigEvent)
var (
subscribersMu sync.RWMutex
subscribers []ConfigSubscriber
)
// Subscribe registers a subscriber for all config events.
func Subscribe(subscriber ConfigSubscriber) {
subscribersMu.Lock()
defer subscribersMu.Unlock()
subscribers = append(subscribers, subscriber)
}
// publishEvent notifies all subscribers of a config change.
func publishEvent(oldVal, newVal map[string]any) {
subscribersMu.RLock()
defer subscribersMu.RUnlock()
for _, sub := range subscribers {
event := ConfigEvent{Old: oldVal, New: newVal}
go sub(event)
}
}
+87
View File
@@ -0,0 +1,87 @@
package config
import "time"
type Settings struct {
ID uint `json:"id,omitempty"` // 1
Sitename string `json:"sitename" default:"Komari"` // 站点名称,默认 "Komari"
Description string `json:"description" default:"A simple server monitor tool."` // 站点描述
CorsOriginCheckEnabled bool `json:"cors_origin_check_enabled" default:"true"` // 是否启用 API CORS 跨域请求校验,默认 true
CorsAllowedOrigins string `json:"cors_allowed_origins" default:""` // API 跨域允许列表
WsOriginCheckEnabled bool `json:"ws_origin_check_enabled" default:"true"` // 是否校验 WebSocket Origin
WsAllowedOrigins string `json:"ws_allowed_origins" default:""` // WebSocket Origin 允许列表
Theme string `json:"theme" default:"default"` // 主题名称,默认 'default'
PrivateSite bool `json:"private_site" default:"false"` // 是否为私有站点,默认 false
ApiKey string `json:"api_key" default:""` // API 密钥,默认空字符串
AutoDiscoveryKey string `json:"auto_discovery_key" default:""` // 自动发现密钥
ScriptDomain string `json:"script_domain" default:""` // 自定义脚本域名
SendIpAddrToGuest bool `json:"send_ip_addr_to_guest" default:"false"` // 是否向访客页面发送 IP 地址,默认 false
EulaAccepted bool `json:"eula_accepted" default:"false"`
BaseScriptsURLKey string `json:"base_scripts_url" default:""`
// GeoIP 配置
GeoIpEnabled bool `json:"geo_ip_enabled" default:"true"`
GeoIpProvider string `json:"geo_ip_provider" default:"ipinfo"` // empty, mmdb, ip-api, geojs
// Nezha 兼容(Agent gRPC)
NezhaCompatEnabled bool `json:"nezha_compat_enabled" default:"false"`
NezhaCompatListen string `json:"nezha_compat_listen" default:""` // 例如 0.0.0.0:5555
// OAuth 配置
OAuthEnabled bool `json:"o_auth_enabled" default:"false"`
OAuthProvider string `json:"o_auth_provider" default:"github"`
DisablePasswordLogin bool `json:"disable_password_login" default:"false"`
CloudflareTunnelToken string `json:"cloudflare_tunnel_token" default:""`
// 自定义美化
CustomHead string `json:"custom_head" default:""`
CustomBody string `json:"custom_body" default:""`
// 通知
NotificationEnabled bool `json:"notification_enabled" default:"true"` // 通知总开关
NotificationMethod string `json:"notification_method" default:"none"`
NotificationTemplate string `json:"notification_template" default:"{{emoji}}{{emoji}}{{emoji}}\nEvent: {{event}}\nClients: {{client}}\nMessage: {{message}}\nTime: {{time}}"`
ExpireNotificationEnabled bool `json:"expire_notification_enabled" default:"true"` // 是否启用过期通知
ExpireNotificationLeadDays int `json:"expire_notification_lead_days" default:"7"` // 过期前多少天通知,默认7天
LoginNotification bool `json:"login_notification" default:"true"` // 登录通知
TrafficLimitPercentage float64 `json:"traffic_limit_percentage" default:"80.00"` // 流量限制百分比,默认80.00%
// Record
RecordEnabled bool `json:"record_enabled" default:"true"` // 是否启用记录功能
RecordPreserveTime int `json:"record_preserve_time" default:"720"` // 记录保留时间,单位小时,默认30天
PingRecordPreserveTime int `json:"ping_record_preserve_time" default:"24"` // Ping 记录保留时间,单位小时,默认1天
UpdatedAt time.Time
}
const (
SitenameKey = "sitename"
DescriptionKey = "description"
CorsOriginCheckEnabledKey = "cors_origin_check_enabled"
CorsAllowedOriginsKey = "cors_allowed_origins"
WsOriginCheckEnabledKey = "ws_origin_check_enabled"
WsAllowedOriginsKey = "ws_allowed_origins"
ThemeKey = "theme"
PrivateSiteKey = "private_site"
ApiKeyKey = "api_key"
AutoDiscoveryKeyKey = "auto_discovery_key"
ScriptDomainKey = "script_domain"
SendIpAddrToGuestKey = "send_ip_addr_to_guest"
EulaAcceptedKey = "eula_accepted"
BaseScriptsURLKey = "base_scripts_url"
GeoIpEnabledKey = "geo_ip_enabled"
GeoIpProviderKey = "geo_ip_provider"
NezhaCompatEnabledKey = "nezha_compat_enabled"
NezhaCompatListenKey = "nezha_compat_listen"
OAuthEnabledKey = "o_auth_enabled"
OAuthProviderKey = "o_auth_provider"
DisablePasswordLoginKey = "disable_password_login"
CloudflareTunnelTokenKey = "cloudflare_tunnel_token"
CustomHeadKey = "custom_head"
CustomBodyKey = "custom_body"
NotificationEnabledKey = "notification_enabled"
NotificationMethodKey = "notification_method"
NotificationTemplateKey = "notification_template"
ExpireNotificationEnabledKey = "expire_notification_enabled"
ExpireNotificationLeadDaysKey = "expire_notification_lead_days"
LoginNotificationKey = "login_notification"
TrafficLimitPercentageKey = "traffic_limit_percentage"
RecordEnabledKey = "record_enabled"
RecordPreserveTimeKey = "record_preserve_time"
PingRecordPreserveTimeKey = "ping_record_preserve_time"
UpdatedAtKey = "updated_at"
XtermjsSettingsKey = "xtermjs_settings"
)
+345
View File
@@ -0,0 +1,345 @@
package corn
import (
"context"
"fmt"
"log"
"strconv"
"strings"
"sync"
"time"
)
// Func 是 corn 调度器执行的任务函数。
// 调度器会为每次执行传入可取消的 context,便于任务在重载或关闭时尽快退出。
type Func func(ctx context.Context)
type job struct {
cancel context.CancelFunc
}
type schedule interface {
Next(time.Time) time.Time
}
type everySchedule struct {
interval time.Duration
}
func (s everySchedule) Next(t time.Time) time.Time {
return t.Add(s.interval)
}
type cronSchedule struct {
seconds map[int]struct{}
minutes map[int]struct{}
hours map[int]struct{}
dom map[int]struct{}
months map[int]struct{}
dow map[int]struct{}
}
func (s cronSchedule) Next(t time.Time) time.Time {
next := t.Truncate(time.Second).Add(time.Second)
limit := next.Add(366 * 24 * time.Hour)
for next.Before(limit) {
if s.match(next) {
return next
}
next = next.Add(time.Second)
}
return time.Time{}
}
func (s cronSchedule) match(t time.Time) bool {
_, okSecond := s.seconds[t.Second()]
_, okMinute := s.minutes[t.Minute()]
_, okHour := s.hours[t.Hour()]
_, okDay := s.dom[t.Day()]
_, okMonth := s.months[int(t.Month())]
_, okWeek := s.dow[int(t.Weekday())]
return okSecond && okMinute && okHour && okDay && okMonth && okWeek
}
type Manager struct {
mu sync.Mutex
jobs map[string]job
}
var defaultManager = NewManager()
func NewManager() *Manager {
return &Manager{jobs: make(map[string]job)}
}
// AddFunc 按 cron 表达式注册一个任务,fn 会在独立 goroutine 中执行。
// 支持 5 字段、6 字段 cron 表达式,以及 @every 1m 这类固定间隔表达式。
func AddFunc(name string, spec string, fn func()) error {
return AddContextFunc(name, spec, false, func(context.Context) { fn() })
}
// AddContextFunc 按 cron 表达式注册一个任务,支持传递带 context 的 func。
func AddContextFunc(name string, spec string, runImmediately bool, fn Func) error {
return defaultManager.AddContextFunc(name, spec, runImmediately, fn)
}
func Every(duration time.Duration) string {
return "@every " + duration.String()
}
func Remove(name string) {
defaultManager.Remove(name)
}
func RemovePrefix(prefix string) {
defaultManager.RemovePrefix(prefix)
}
func StopAll() {
defaultManager.StopAll()
}
func (m *Manager) AddFunc(name string, spec string, fn func()) error {
return m.AddContextFunc(name, spec, false, func(context.Context) { fn() })
}
func (m *Manager) AddContextFunc(name string, spec string, runImmediately bool, fn Func) error {
if name == "" {
return fmt.Errorf("corn job name is empty")
}
s, err := Parse(spec)
if err != nil {
return fmt.Errorf("corn job %q spec is invalid: %w", name, err)
}
if fn == nil {
return fmt.Errorf("corn job %q func is nil", name)
}
ctx, cancel := context.WithCancel(context.Background())
m.replace(name, cancel)
go m.run(ctx, name, s, runImmediately, fn)
return nil
}
func (m *Manager) Remove(name string) {
m.mu.Lock()
defer m.mu.Unlock()
if old, ok := m.jobs[name]; ok {
old.cancel()
delete(m.jobs, name)
}
}
func (m *Manager) RemovePrefix(prefix string) {
m.mu.Lock()
defer m.mu.Unlock()
for name, old := range m.jobs {
if strings.HasPrefix(name, prefix) {
old.cancel()
delete(m.jobs, name)
}
}
}
func (m *Manager) StopAll() {
m.mu.Lock()
defer m.mu.Unlock()
for name, old := range m.jobs {
old.cancel()
delete(m.jobs, name)
}
}
func (m *Manager) replace(name string, cancel context.CancelFunc) {
m.mu.Lock()
defer m.mu.Unlock()
if old, ok := m.jobs[name]; ok {
old.cancel()
}
m.jobs[name] = job{cancel: cancel}
}
func (m *Manager) run(ctx context.Context, name string, s schedule, runImmediately bool, fn Func) {
if runImmediately {
go safeRun(ctx, name, fn)
}
nextTick := s.Next(time.Now())
if nextTick.IsZero() {
log.Printf("corn job %s has no next run time", name)
return
}
timer := time.NewTimer(time.Until(nextTick))
defer timer.Stop()
for {
select {
case <-ctx.Done():
return
case <-timer.C:
go safeRun(ctx, name, fn)
nextTick = s.Next(nextTick)
if nextTick.IsZero() {
return
}
resetTimer(timer, time.Until(nextTick))
}
}
}
func resetTimer(timer *time.Timer, duration time.Duration) {
if duration < 0 {
duration = 0
}
timer.Reset(duration)
}
func safeRun(ctx context.Context, name string, fn Func) {
defer func() {
if r := recover(); r != nil {
log.Printf("corn job %s panic: %v", name, r)
}
}()
select {
case <-ctx.Done():
return
default:
fn(ctx)
}
}
// Parse 解析 corn 表达式。
// 支持:
// - 5 字段:minute hour day-of-month month day-of-week
// - 6 字段:second minute hour day-of-month month day-of-week
// - @every 1m / @every 30s
//
// 字段支持 *、*/n、a-b、a-b/n、逗号列表和具体数字。
func Parse(spec string) (schedule, error) {
spec = strings.TrimSpace(spec)
if spec == "" {
return nil, fmt.Errorf("empty spec")
}
if strings.HasPrefix(spec, "@every ") {
duration, err := time.ParseDuration(strings.TrimSpace(strings.TrimPrefix(spec, "@every ")))
if err != nil {
return nil, err
}
if duration <= 0 {
return nil, fmt.Errorf("@every duration must be positive")
}
return everySchedule{interval: duration}, nil
}
fields := strings.Fields(spec)
if len(fields) == 5 {
fields = append([]string{"0"}, fields...)
}
if len(fields) != 6 {
return nil, fmt.Errorf("expected 5 or 6 fields, got %d", len(fields))
}
seconds, err := parseField(fields[0], 0, 59)
if err != nil {
return nil, fmt.Errorf("second: %w", err)
}
minutes, err := parseField(fields[1], 0, 59)
if err != nil {
return nil, fmt.Errorf("minute: %w", err)
}
hours, err := parseField(fields[2], 0, 23)
if err != nil {
return nil, fmt.Errorf("hour: %w", err)
}
dom, err := parseField(fields[3], 1, 31)
if err != nil {
return nil, fmt.Errorf("day-of-month: %w", err)
}
months, err := parseField(fields[4], 1, 12)
if err != nil {
return nil, fmt.Errorf("month: %w", err)
}
dow, err := parseField(fields[5], 0, 7)
if err != nil {
return nil, fmt.Errorf("day-of-week: %w", err)
}
if _, ok := dow[7]; ok {
dow[0] = struct{}{}
delete(dow, 7)
}
return cronSchedule{seconds: seconds, minutes: minutes, hours: hours, dom: dom, months: months, dow: dow}, nil
}
func parseField(field string, min int, max int) (map[int]struct{}, error) {
values := make(map[int]struct{})
for _, part := range strings.Split(field, ",") {
part = strings.TrimSpace(part)
if part == "" {
return nil, fmt.Errorf("empty part")
}
base := part
step := 1
if strings.Contains(part, "/") {
parts := strings.Split(part, "/")
if len(parts) != 2 {
return nil, fmt.Errorf("invalid step %q", part)
}
base = parts[0]
parsedStep, err := strconv.Atoi(parts[1])
if err != nil || parsedStep <= 0 {
return nil, fmt.Errorf("invalid step %q", parts[1])
}
step = parsedStep
}
start, end, err := parseRange(base, min, max)
if err != nil {
return nil, err
}
for i := start; i <= end; i += step {
values[i] = struct{}{}
}
}
return values, nil
}
func parseRange(base string, min int, max int) (int, int, error) {
if base == "*" || base == "" {
return min, max, nil
}
if strings.Contains(base, "-") {
parts := strings.Split(base, "-")
if len(parts) != 2 {
return 0, 0, fmt.Errorf("invalid range %q", base)
}
start, err := strconv.Atoi(parts[0])
if err != nil {
return 0, 0, fmt.Errorf("invalid range start %q", parts[0])
}
end, err := strconv.Atoi(parts[1])
if err != nil {
return 0, 0, fmt.Errorf("invalid range end %q", parts[1])
}
if start > end || start < min || end > max {
return 0, 0, fmt.Errorf("range %q out of bounds %d-%d", base, min, max)
}
return start, end, nil
}
value, err := strconv.Atoi(base)
if err != nil {
return 0, 0, fmt.Errorf("invalid value %q", base)
}
if value < min || value > max {
return 0, 0, fmt.Errorf("value %d out of bounds %d-%d", value, min, max)
}
return value, value, nil
}
+270
View File
@@ -0,0 +1,270 @@
package metric
import (
"fmt"
"math"
"sort"
"time"
)
// AggregatePoints groups raw points into time buckets and computes aggregate values.
//
// AggregatePoints 在内存中按查询配置将原始点分桶并聚合。
func AggregatePoints(points []Point, query AggregateQuery) ([]AggregatePoint, error) {
if err := query.Validate(); err != nil {
return nil, err
}
if len(points) == 0 {
if query.FillEmpty {
return emptyBuckets(query), nil
}
return []AggregatePoint{}, nil
}
points = sortedPoints(points)
groups := make(map[int64][]Point)
for _, point := range points {
bucket := alignTime(point.Timestamp, query.Interval).UnixNano()
groups[bucket] = append(groups[bucket], point)
}
var buckets []int64
if query.FillEmpty {
for t := alignTime(query.Start, query.Interval); !t.After(query.End); t = t.Add(query.Interval) {
buckets = append(buckets, t.UnixNano())
}
} else {
for bucket := range groups {
buckets = append(buckets, bucket)
}
sort.Slice(buckets, func(i, j int) bool { return buckets[i] < buckets[j] })
}
out := make([]AggregatePoint, 0, len(buckets))
for _, bucket := range buckets {
group := groups[bucket]
value, err := aggregateValue(group, query.Aggregation)
if err != nil {
return nil, err
}
out = append(out, AggregatePoint{
MetricName: query.MetricName,
EntityID: query.EntityID,
Bucket: time.Unix(0, bucket).UTC(),
Value: value,
Count: len(group),
})
}
return out, nil
}
// CalculateStats computes summary statistics for a point series.
//
// CalculateStats 基于一组点计算统计摘要,包括均值、百分位、首尾值和标准差。
func CalculateStats(points []Point) (Stats, error) {
if len(points) == 0 {
// Distinguish "the metric/range yielded no samples" from "the metric
// definition does not exist" (which surfaces as ErrNotFound elsewhere).
return Stats{}, ErrNoData
}
points = sortedPoints(points)
avg, _ := aggregateValue(points, AggAvg)
min, _ := aggregateValue(points, AggMin)
max, _ := aggregateValue(points, AggMax)
sum, _ := aggregateValue(points, AggSum)
p50, _ := aggregateValue(points, AggP50)
p95, _ := aggregateValue(points, AggP95)
p99, _ := aggregateValue(points, AggP99)
first, _ := aggregateValue(points, AggFirst)
last, _ := aggregateValue(points, AggLast)
rate, _ := aggregateValue(points, AggRate)
var variance float64
for _, p := range points {
diff := p.Value - avg
variance += diff * diff
}
return Stats{
Count: len(points),
Min: min,
Max: max,
Avg: avg,
Sum: sum,
P50: p50,
P95: p95,
P99: p99,
First: first,
Last: last,
Rate: rate,
Start: points[0].Timestamp,
End: points[len(points)-1].Timestamp,
StdDev: math.Sqrt(variance / float64(len(points))),
}, nil
}
// aggregateValue computes one aggregation over a point group.
//
// aggregateValue 对一组点执行单个聚合类型的计算。
func aggregateValue(points []Point, agg Aggregation) (float64, error) {
if len(points) == 0 {
return 0, nil
}
switch agg {
case AggAvg:
sum, _ := aggregateValue(points, AggSum)
return sum / float64(len(points)), nil
case AggMin:
v := points[0].Value
for _, p := range points[1:] {
v = math.Min(v, p.Value)
}
return v, nil
case AggMax:
v := points[0].Value
for _, p := range points[1:] {
v = math.Max(v, p.Value)
}
return v, nil
case AggSum:
var sum float64
for _, p := range points {
sum += p.Value
}
return sum, nil
case AggCount:
return float64(len(points)), nil
case AggFirst:
// Callers (AggregatePoints, CalculateStats) pass time-ordered slices.
return points[0].Value, nil
case AggLast:
return points[len(points)-1].Value, nil
case AggRate:
return counterRate(points), nil
case AggStdDev:
return stdDevPop(points), nil
default:
// Any percentile (p50, p95, p99, and arbitrary pXX / pXX.X) is computed
// here via linear interpolation over the sorted values. The fixed
// AggP50/AggP95/AggP99 constants are just common cases of this.
if frac, ok := parsePercentile(agg); ok {
return percentile(points, frac), nil
}
return 0, fmt.Errorf("%w: unsupported aggregation %q", ErrInvalidArgument, agg)
}
}
// counterRate computes a per-second rate of change that is resilient to counter
// resets. It walks the time-ordered series and sums only positive deltas; a
// decrease is treated as a reset (the counter restarted) and contributes zero
// rather than a negative spike. For a strictly increasing counter this equals
// (last-first)/seconds. For a gauge it yields the total upward movement per
// second, which is a stable definition for an otherwise ill-defined quantity.
//
// counterRate 计算能抵抗计数器重置的每秒变化率。它会遍历按时间排序的序列,
// 并且只累加正向增量;当值下降时会被视为重置(计数器重新开始),该段贡献为零,
// 而不是产生负向尖峰。对于严格递增的计数器,这等于 (last-first)/seconds。
// 对于 gauge,它表示每秒总上升量,为这种本来定义不明确的量提供稳定定义。
func counterRate(points []Point) float64 {
if len(points) < 2 {
return 0
}
// Input is time-ordered by the callers; avoid an extra sort here.
seconds := points[len(points)-1].Timestamp.Sub(points[0].Timestamp).Seconds()
if seconds <= 0 {
return 0
}
var increase float64
for i := 1; i < len(points); i++ {
delta := points[i].Value - points[i-1].Value
if delta > 0 {
increase += delta
}
}
return increase / seconds
}
// stdDevPop computes the population standard deviation (dividing by N), matching
// SQL STDDEV_POP and the StdDev field produced by CalculateStats.
//
// stdDevPop 计算总体标准差(除以 N),与 SQL 的 STDDEV_POP 以及
// CalculateStats 返回的 StdDev 语义保持一致。
func stdDevPop(points []Point) float64 {
if len(points) == 0 {
return 0
}
var sum float64
for _, p := range points {
sum += p.Value
}
mean := sum / float64(len(points))
var variance float64
for _, p := range points {
d := p.Value - mean
variance += d * d
}
return math.Sqrt(variance / float64(len(points)))
}
// percentile computes a quantile from point values.
//
// percentile 提取点值、排序,并计算指定小数形式的百分位。
func percentile(points []Point, p float64) float64 {
values := make([]float64, len(points))
for i, point := range points {
values[i] = point.Value
}
sort.Float64s(values)
return percentileSorted(values, p)
}
// percentileSorted returns the linear-interpolation percentile of an
// already-sorted slice. Shared by percentile() and the raw-value paths so the
// interpolation method stays identical everywhere.
//
// percentileSorted 基于已排序切片用线性插值计算百分位,供原始值路径和
// percentile 共用,确保所有路径的插值方法一致。
func percentileSorted(values []float64, p float64) float64 {
if len(values) == 0 {
return 0
}
if len(values) == 1 {
return values[0]
}
pos := p * float64(len(values)-1)
lower := int(math.Floor(pos))
upper := int(math.Ceil(pos))
if lower == upper {
return values[lower]
}
weight := pos - float64(lower)
return values[lower]*(1-weight) + values[upper]*weight
}
// alignTime floors a timestamp to the start of its interval bucket.
//
// alignTime 将时间向下对齐到指定间隔的桶起点。
func alignTime(t time.Time, interval time.Duration) time.Time {
nano := t.UTC().UnixNano()
size := interval.Nanoseconds()
// Floor division toward negative infinity so timestamps before the Unix
// epoch (negative nanos) align to the bucket start rather than rounding up.
// Go's % returns a remainder with the dividend's sign, so normalize it.
rem := ((nano % size) + size) % size
return time.Unix(0, nano-rem).UTC()
}
// emptyBuckets builds zero-count aggregate buckets for an empty range.
//
// emptyBuckets 根据查询范围生成空聚合桶,用于 FillEmpty 场景。
func emptyBuckets(query AggregateQuery) []AggregatePoint {
var out []AggregatePoint
for t := alignTime(query.Start, query.Interval); !t.After(query.End); t = t.Add(query.Interval) {
out = append(out, AggregatePoint{
MetricName: query.MetricName,
EntityID: query.EntityID,
Bucket: t,
})
}
return out
}
+485
View File
@@ -0,0 +1,485 @@
package metric
import (
"database/sql"
"fmt"
"net/url"
"path/filepath"
"strings"
"time"
)
// Config describes database, pooling, migration, retention, and rollup settings.
//
// Config 描述 Store 的数据库后端、连接池、迁移、保留和 rollup 配置。
type Config struct {
// Driver selects the database backend.
//
// Driver 选择数据库后端。
Driver Driver
// DSN is the database connection string when DB is not supplied.
//
// DSN 是未传入 DB 时使用的数据库连接字符串。
DSN string
// DB can be supplied by the host app when it owns the connection pool.
// When DB is set, DSN is ignored and Close leaves the supplied DB open.
//
// 当宿主程序自己管理连接池时可以传入 DB;设置 DB 后会忽略 DSN,
// Close 也不会关闭这个外部传入的连接池。
DB *sql.DB
// TablePrefix prefixes every table managed by the store.
//
// TablePrefix 是 Store 管理的所有表名前缀。
TablePrefix string
// DefaultRetentionDays is used when a metric definition omits retention.
//
// DefaultRetentionDays 在指标定义未设置保留时间时使用。
DefaultRetentionDays int
// AutoMigrate controls whether Open creates or updates the schema.
//
// AutoMigrate 控制 Open 是否创建或更新表结构。
AutoMigrate bool
// MaxOpenConns sets the primary pool's maximum open connections.
//
// MaxOpenConns 设置主连接池的最大打开连接数。
MaxOpenConns int
// MaxIdleConns sets the primary pool's maximum idle connections.
//
// MaxIdleConns 设置主连接池的最大空闲连接数。
MaxIdleConns int
// ConnMaxLifetime sets the maximum lifetime for pooled connections.
//
// ConnMaxLifetime 设置池化连接的最大生命周期。
ConnMaxLifetime time.Duration
// ConnectTimeout bounds the initial database ping in Open.
//
// ConnectTimeout 限制 Open 中首次 ping 数据库的时间。
ConnectTimeout time.Duration
// SQLite holds SQLite-specific options.
//
// SQLite 保存 SQLite 专用选项。
SQLite SQLiteOptions
// RollupPolicy configures downsampling tiers and tiered retention. When it
// defines tiers, Store.Compact materializes them and enforces every
// retention window. The zero value disables rollups (raw points only).
//
// RollupPolicy 配置降采样层级和分层保留时间;定义层级后,
// Store.Compact 会生成 rollup 并执行保留策略。零值表示禁用 rollup。
RollupPolicy RollupPolicy
}
// SQLiteOptions contains SQLite-specific performance and read-concurrency settings.
//
// SQLiteOptions 保存 SQLite 专用的性能和并发读取选项。
type SQLiteOptions struct {
// PerformanceProfile selects the SQLite synchronous durability preset.
//
// PerformanceProfile 选择 SQLite synchronous 持久化预设。
PerformanceProfile SQLitePerformanceProfile
// BusyTimeout is applied to SQLite busy_timeout.
//
// BusyTimeout 会应用到 SQLite busy_timeout。
BusyTimeout time.Duration
// CacheSizeKB sets the SQLite page cache size in KB.
//
// CacheSizeKB 设置 SQLite 页缓存大小,单位为 KB。
CacheSizeKB int
// PageSize sets SQLite page_size when positive.
//
// PageSize 为正数时设置 SQLite page_size。
PageSize int
// TempStoreMemory enables memory-backed temporary storage.
//
// TempStoreMemory 启用基于内存的临时存储。
TempStoreMemory bool
// MMapSizeBytes sets SQLite mmap_size in bytes.
//
// MMapSizeBytes 设置 SQLite mmap_size,单位为字节。
MMapSizeBytes int64
// WALAutoCheckpoint sets the SQLite WAL auto-checkpoint page count.
//
// WALAutoCheckpoint 设置 SQLite WAL 自动 checkpoint 页数。
WALAutoCheckpoint int
// ReadPoolSize, when > 0, opens a second read-only connection pool with
// this many connections for SELECT-style calls. SQLite serializes writes,
// but WAL lets readers run concurrently, so a read pool lifts read
// throughput while keeping writes on the single primary connection.
// 0 (default) means all calls share the single primary pool, preserving
// the previous single-connection behavior.
//
// ReadPoolSize 大于 0 时会为 SELECT 类调用打开独立只读连接池。
// SQLite 写入会被串行化,但 WAL 允许读取并发执行,因此读池可以在写入仍走
// 单主连接的同时提升读取吞吐。0(默认值)表示所有调用共享单个主连接池,
// 保留之前的单连接行为。
ReadPoolSize int
}
// SQLitePerformanceProfile names SQLite durability and performance presets.
//
// SQLitePerformanceProfile 表示 SQLite synchronous 等持久化策略预设。
type SQLitePerformanceProfile string
const (
// SQLiteProfileDefault uses the package default SQLite profile.
//
// SQLiteProfileDefault 使用包默认的 SQLite 配置档。
SQLiteProfileDefault SQLitePerformanceProfile = ""
// SQLiteProfileBalanced uses NORMAL synchronous mode.
//
// SQLiteProfileBalanced 使用 NORMAL synchronous 模式。
SQLiteProfileBalanced SQLitePerformanceProfile = "balanced"
// SQLiteProfilePerformance favors throughput over durability.
//
// SQLiteProfilePerformance 优先吞吐而非持久性。
SQLiteProfilePerformance SQLitePerformanceProfile = "performance"
// SQLiteProfileDurable favors durability over write throughput.
//
// SQLiteProfileDurable 优先持久性而非写入吞吐。
SQLiteProfileDurable SQLitePerformanceProfile = "durable"
)
// Option mutates a Config value.
//
// Option 是用于修改 Config 的函数式选项。
type Option func(*Config)
// DefaultConfig returns the default configuration for a backend.
//
// DefaultConfig 返回指定后端的默认配置。
func DefaultConfig(driver Driver, dsn string) Config {
return Config{
Driver: driver,
DSN: dsn,
TablePrefix: "metric_",
DefaultRetentionDays: 90,
AutoMigrate: true,
MaxOpenConns: 25,
MaxIdleConns: 5,
ConnMaxLifetime: time.Hour,
ConnectTimeout: 10 * time.Second,
SQLite: SQLiteOptions{
PerformanceProfile: SQLiteProfileBalanced,
BusyTimeout: 5 * time.Second,
CacheSizeKB: 64 * 1024,
TempStoreMemory: true,
MMapSizeBytes: 256 * 1024 * 1024,
WALAutoCheckpoint: 1000,
},
}
}
// Backend builds a generic backend configuration and applies options.
//
// Backend 构造通用后端配置,并应用额外选项。
func Backend(driver Driver, dsn string, opts ...Option) Config {
cfg := DefaultConfig(driver, dsn)
applyOptions(&cfg, opts...)
return cfg
}
// SQLite builds a SQLite backend configuration.
//
// SQLite 构造 SQLite 后端配置。
func SQLite(dsn string, opts ...Option) Config {
cfg := SQLiteConfig(dsn)
applyOptions(&cfg, opts...)
return cfg
}
// SQLiteInDir builds a managed SQLite file configuration rooted at a directory.
//
// SQLiteInDir 构造由 package 管理目录的 SQLite 文件数据库配置。
func SQLiteInDir(dir string, opts ...Option) Config {
cfg := SQLiteConfig(sqliteFileDSN(filepath.Join(dir, "metrics.db")))
cfg.MaxOpenConns = 1
cfg.MaxIdleConns = 1
applyOptions(&cfg, opts...)
return cfg
}
// MySQL builds a MySQL backend configuration.
//
// MySQL 构造 MySQL 后端配置。
func MySQL(dsn string, opts ...Option) Config {
cfg := MySQLConfig(dsn)
applyOptions(&cfg, opts...)
return cfg
}
// PostgreSQL builds a PostgreSQL backend configuration.
//
// PostgreSQL 构造 PostgreSQL 后端配置。
func PostgreSQL(dsn string, opts ...Option) Config {
cfg := PostgreSQLConfig(dsn)
applyOptions(&cfg, opts...)
return cfg
}
// SQLiteConfig returns the default SQLite configuration.
//
// SQLiteConfig 返回 SQLite 后端的默认配置。
func SQLiteConfig(dsn string) Config {
cfg := DefaultConfig(DriverSQLite, dsn)
if cfg.DSN == "" {
cfg.DSN = "file:metric?mode=memory&cache=shared"
}
// SQLite serializes writes, so a single connection is the safe default and
// avoids "database is locked" contention. Callers can still override with
// WithMaxOpenConns/WithMaxIdleConns. Setting it here (rather than inferring
// it later from the 25/5 defaults) means an explicit WithMaxOpenConns(25)
// is honored instead of being mistaken for "unset".
cfg.MaxOpenConns = 1
cfg.MaxIdleConns = 1
return cfg
}
// MySQLConfig returns the default MySQL configuration.
//
// MySQLConfig 返回 MySQL 后端的默认配置。
func MySQLConfig(dsn string) Config {
return DefaultConfig(DriverMySQL, dsn)
}
// PostgreSQLConfig returns the default PostgreSQL configuration.
//
// PostgreSQLConfig 返回 PostgreSQL 后端的默认配置。
func PostgreSQLConfig(dsn string) Config {
return DefaultConfig(DriverPostgreSQL, dsn)
}
// WithDB sets a caller-owned database connection pool.
//
// WithDB 设置由调用方提供的数据库连接池。
func WithDB(db *sql.DB) Option {
return func(c *Config) {
c.DB = db
}
}
// WithTablePrefix sets the table-name prefix used by metric tables.
//
// WithTablePrefix 设置 metric 私有表的表名前缀。
func WithTablePrefix(prefix string) Option {
return func(c *Config) {
c.TablePrefix = prefix
}
}
// WithDefaultRetention sets the default metric retention in days.
//
// WithDefaultRetention 设置指标默认保留天数。
func WithDefaultRetention(days int) Option {
return func(c *Config) {
c.DefaultRetentionDays = days
}
}
// WithAutoMigrate controls whether Open creates the schema automatically.
//
// WithAutoMigrate 控制 Open 时是否自动创建表结构。
func WithAutoMigrate(enabled bool) Option {
return func(c *Config) {
c.AutoMigrate = enabled
}
}
// WithMaxOpenConns sets the maximum number of open database connections.
//
// WithMaxOpenConns 设置底层数据库连接池的最大打开连接数。
func WithMaxOpenConns(n int) Option {
return func(c *Config) {
c.MaxOpenConns = n
}
}
// WithMaxIdleConns sets the maximum number of idle database connections.
//
// WithMaxIdleConns 设置底层数据库连接池的最大空闲连接数。
func WithMaxIdleConns(n int) Option {
return func(c *Config) {
c.MaxIdleConns = n
}
}
// WithConnMaxLifetime sets the maximum lifetime for pooled connections.
//
// WithConnMaxLifetime 设置连接最大复用时间。
func WithConnMaxLifetime(d time.Duration) Option {
return func(c *Config) {
c.ConnMaxLifetime = d
}
}
// WithConnectTimeout sets the timeout used while opening the store.
//
// WithConnectTimeout 设置打开 Store 时 ping 数据库的超时时间。
func WithConnectTimeout(d time.Duration) Option {
return func(c *Config) {
c.ConnectTimeout = d
}
}
// WithSQLiteProfile sets the SQLite durability and performance profile.
//
// WithSQLiteProfile 设置 SQLite 持久化性能预设。
func WithSQLiteProfile(profile SQLitePerformanceProfile) Option {
return func(c *Config) {
c.SQLite.PerformanceProfile = profile
}
}
// WithSQLiteBusyTimeout sets SQLite busy_timeout.
//
// WithSQLiteBusyTimeout 设置 SQLite busy_timeout。
func WithSQLiteBusyTimeout(d time.Duration) Option {
return func(c *Config) {
c.SQLite.BusyTimeout = d
}
}
// WithSQLiteCacheSizeKB sets the SQLite page cache size in KB.
//
// WithSQLiteCacheSizeKB 设置 SQLite 页缓存大小,单位为 KB。
func WithSQLiteCacheSizeKB(kb int) Option {
return func(c *Config) {
c.SQLite.CacheSizeKB = kb
}
}
// WithSQLiteMMapSize sets SQLite mmap_size.
//
// WithSQLiteMMapSize 设置 SQLite mmap_size。
func WithSQLiteMMapSize(bytes int64) Option {
return func(c *Config) {
c.SQLite.MMapSizeBytes = bytes
}
}
// WithSQLitePageSize sets SQLite page_size.
//
// WithSQLitePageSize 设置 SQLite page_size。
func WithSQLitePageSize(bytes int) Option {
return func(c *Config) {
c.SQLite.PageSize = bytes
}
}
// WithSQLiteTempStoreMemory controls whether SQLite uses memory for temporary storage.
//
// WithSQLiteTempStoreMemory 设置 SQLite 是否使用内存临时存储。
func WithSQLiteTempStoreMemory(enabled bool) Option {
return func(c *Config) {
c.SQLite.TempStoreMemory = enabled
}
}
// WithSQLiteWALAutoCheckpoint sets the SQLite WAL auto-checkpoint page count.
//
// WithSQLiteWALAutoCheckpoint 设置 SQLite WAL 自动 checkpoint 页数。
func WithSQLiteWALAutoCheckpoint(pages int) Option {
return func(c *Config) {
c.SQLite.WALAutoCheckpoint = pages
}
}
// WithSQLiteReadPool enables a dedicated read-only connection pool of n
// connections for SQLite. Writes stay on the single primary connection
// (SQLite serializes them); reads fan out across the pool, which WAL mode
// allows to run concurrently. Pass n <= 1 to disable (the default).
//
// WithSQLiteReadPool 为 SQLite 启用包含 n 个连接的独立只读连接池。写入仍走
// 单主连接(SQLite 会串行化写入);读取会分散到该连接池中,WAL 模式允许它们
// 并发执行。传入 n <= 1 会禁用该功能(默认行为)。
func WithSQLiteReadPool(n int) Option {
return func(c *Config) {
c.SQLite.ReadPoolSize = n
}
}
// WithRollupPolicy sets the downsampling/retention ladder. Compact uses it to
// build rollup tiers (each progressively coarser and longer-lived) and to age
// out raw points and expired tiers.
//
// WithRollupPolicy 设置降采样与保留时间阶梯;Compact 会据此构建逐级
// 更粗、保留更久的 rollup,并清理过期原始点和过期层级。
func WithRollupPolicy(p RollupPolicy) Option {
return func(c *Config) {
c.RollupPolicy = p
}
}
// applyOptions applies configuration options in order.
//
// applyOptions 将一组选项依次应用到 Config。
func applyOptions(cfg *Config, opts ...Option) {
for _, opt := range opts {
if opt != nil {
opt(cfg)
}
}
}
// Validate checks whether the value is well formed.
//
// Validate 检查 Config 的后端、表名前缀、保留时间和 rollup 策略是否合法。
func (c Config) Validate() error {
switch c.Driver {
case DriverSQLite, DriverMySQL, DriverPostgreSQL:
default:
return fmt.Errorf("%w: unsupported driver %q", ErrInvalidArgument, c.Driver)
}
if c.DB == nil && strings.TrimSpace(c.DSN) == "" {
return fmt.Errorf("%w: dsn is required when db is not supplied", ErrInvalidArgument)
}
if c.TablePrefix == "" {
return fmt.Errorf("%w: table prefix cannot be empty", ErrInvalidArgument)
}
for _, r := range c.TablePrefix {
if !(r == '_' || r >= '0' && r <= '9' || r >= 'A' && r <= 'Z' || r >= 'a' && r <= 'z') {
return fmt.Errorf("%w: table prefix must contain only letters, digits, and underscores", ErrInvalidArgument)
}
}
if c.DefaultRetentionDays <= 0 {
return fmt.Errorf("%w: default retention days must be positive", ErrInvalidArgument)
}
if err := c.RollupPolicy.Validate(); err != nil {
return err
}
return nil
}
// driverName returns the database/sql driver name for the configured backend.
//
// driverName 返回 database/sql 注册使用的驱动名称。
func (c Config) driverName() string {
switch c.Driver {
case DriverSQLite:
return "sqlite3"
case DriverMySQL:
return "mysql"
case DriverPostgreSQL:
return "pgx"
default:
return string(c.Driver)
}
}
// sqliteFileDSN converts a filesystem path into a SQLite file DSN.
//
// sqliteFileDSN 将文件路径转换为 SQLite file: DSN。
func sqliteFileDSN(path string) string {
return "file:" + filepath.ToSlash(path) + "?cache=shared&mode=rwc"
}
// appendSQLiteDSNParam appends a query parameter to a SQLite DSN.
//
// appendSQLiteDSNParam 向 SQLite DSN 追加查询参数。
func appendSQLiteDSNParam(dsn, key, value string) string {
if strings.Contains(dsn, "?") {
return dsn + "&" + url.QueryEscape(key) + "=" + url.QueryEscape(value)
}
return dsn + "?" + url.QueryEscape(key) + "=" + url.QueryEscape(value)
}
+68
View File
@@ -0,0 +1,68 @@
package metric
import (
"testing"
"time"
)
// TestBackendBuildersApplyOptions verifies backend builders apply options.
//
// TestBackendBuildersApplyOptions 验证各后端构造器会正确应用选项。
func TestBackendBuildersApplyOptions(t *testing.T) {
cfg := SQLite(
"file:metrics.db?cache=shared",
WithTablePrefix("x_metric_"),
WithDefaultRetention(180),
WithAutoMigrate(false),
WithMaxOpenConns(10),
WithMaxIdleConns(2),
WithConnMaxLifetime(30*time.Minute),
WithConnectTimeout(2*time.Second),
)
if cfg.Driver != DriverSQLite {
t.Fatalf("expected sqlite driver, got %q", cfg.Driver)
}
if cfg.TablePrefix != "x_metric_" {
t.Fatalf("table prefix option was not applied: %q", cfg.TablePrefix)
}
if cfg.DefaultRetentionDays != 180 {
t.Fatalf("default retention option was not applied: %d", cfg.DefaultRetentionDays)
}
if cfg.AutoMigrate {
t.Fatalf("auto migrate option was not applied")
}
if cfg.MaxOpenConns != 10 || cfg.MaxIdleConns != 2 {
t.Fatalf("pool options were not applied: %#v", cfg)
}
if cfg.ConnMaxLifetime != 30*time.Minute || cfg.ConnectTimeout != 2*time.Second {
t.Fatalf("timeout options were not applied: %#v", cfg)
}
}
// TestSQLiteInDirBuildsFileConfig verifies SQLiteInDir file configuration.
//
// TestSQLiteInDirBuildsFileConfig 验证 SQLiteInDir 会生成预期的文件数据库配置。
func TestSQLiteInDirBuildsFileConfig(t *testing.T) {
cfg := SQLiteInDir(
"data/metrics",
WithSQLiteProfile(SQLiteProfilePerformance),
WithSQLiteCacheSizeKB(128*1024),
)
if cfg.Driver != DriverSQLite {
t.Fatalf("expected sqlite driver, got %q", cfg.Driver)
}
if cfg.MaxOpenConns != 1 || cfg.MaxIdleConns != 1 {
t.Fatalf("sqlite directory config should default to a single connection: %#v", cfg)
}
if cfg.SQLite.PerformanceProfile != SQLiteProfilePerformance {
t.Fatalf("sqlite profile option was not applied: %q", cfg.SQLite.PerformanceProfile)
}
if cfg.SQLite.CacheSizeKB != 128*1024 {
t.Fatalf("sqlite cache option was not applied: %d", cfg.SQLite.CacheSizeKB)
}
if cfg.DSN != "file:data/metrics/metrics.db?cache=shared&mode=rwc" {
t.Fatalf("unexpected sqlite dir dsn: %q", cfg.DSN)
}
}
+444
View File
@@ -0,0 +1,444 @@
package metric
import (
"database/sql"
"fmt"
"strings"
)
// dialect abstracts SQL differences between supported databases.
//
// dialect 抽象不同数据库后端的 SQL 差异。
type dialect interface {
placeholder(n int) string
// jsonPlaceholder returns the bind placeholder for a JSON column value.
// PostgreSQL needs an explicit ::jsonb cast because the driver sends Go
// strings with a text OID, which the server refuses to assign to a jsonb
// column. SQLite and MySQL store JSON as text/native and need no cast.
//
// jsonPlaceholder 返回 JSON 列值的绑定占位符。PostgreSQL 需要显式的
// ::jsonb 转换,因为驱动会以 text OID 发送 Go 字符串,服务器拒绝把它赋给
// jsonb 列。SQLite 和 MySQL 将 JSON 存为文本或原生类型,不需要转换。
jsonPlaceholder(n int) string
jsonType() string
autoIncrementPrimaryKey() string
nowExpr() string
insertDefinitionSQL(t tables) string
upsertPointSQL(t tables, rowCount int) string
// jsonExtractEquals renders a boolean predicate comparing the text value at
// key inside a JSON column to a bind placeholder. The key is interpolated
// into the SQL (callers must restrict it to a safe charset); the compared
// value is always bound.
//
// jsonExtractEquals 渲染一个布尔谓词,用于比较 JSON 列内 key 对应的文本值和
// 绑定占位符。key 会插入 SQL 中(调用方必须限制到安全字符集);被比较的值
// 始终通过绑定参数传入。
jsonExtractEquals(column, key, placeholder string) string
// blobType is the column type for the rollup t-digest sketch.
//
// blobType 是 rollup t-digest sketch 的列类型。
blobType() string
// upsertRollupSQL builds a single-row upsert for one rollup bucket cell.
//
// upsertRollupSQL 为单个 rollup 桶单元构造单行 upsert。
upsertRollupSQL(t tables) string
// compactTxOptions returns the transaction options used for the compaction
// transaction. PostgreSQL/MySQL escalate to SERIALIZABLE so the raw scan and
// the raw deletion share one snapshot and a point inserted between them
// cannot be deleted without first being rolled up. SQLite returns nil
// (its default) because a single connection already serializes writes.
//
// compactTxOptions 返回 compaction 事务使用的事务选项。PostgreSQL/MySQL 会
// 提升到 SERIALIZABLE,使 raw 扫描和 raw 删除共享同一个快照,让两者之间写入
// 的点不会在尚未进入 rollup 前被删除。SQLite 返回 nil(默认),因为单连接
// 已经串行化写入。
compactTxOptions() *sql.TxOptions
}
// tables stores the physical table names used by a Store.
//
// tables 保存 Store 使用的实际表名。
type tables struct {
// definitions is the metric definitions table.
//
// definitions 是指标定义表。
definitions string
// points is the raw points table.
//
// points 是原始采样点表。
points string
// rollups is the downsampled rollups table.
//
// rollups 是降采样 rollup 表。
rollups string
}
// newDialect returns the SQL dialect implementation for a backend.
//
// newDialect 根据数据库后端创建对应 SQL 方言实现。
func newDialect(driver Driver) dialect {
switch driver {
case DriverPostgreSQL:
return postgresDialect{}
case DriverMySQL:
return mysqlDialect{}
default:
return sqliteDialect{}
}
}
// sqliteDialect implements SQL rendering for SQLite.
//
// sqliteDialect 实现 SQLite 方言。
type sqliteDialect struct{}
// placeholder returns a bind placeholder for this SQL dialect.
//
// placeholder 返回 SQLite 的绑定参数占位符。
func (sqliteDialect) placeholder(int) string { return "?" }
// jsonPlaceholder returns a bind placeholder for JSON values.
//
// jsonPlaceholder 返回 SQLite JSON 值的绑定参数占位符。
func (sqliteDialect) jsonPlaceholder(int) string { return "?" }
// jsonType returns the SQL column type used for JSON values.
//
// jsonType 返回 SQLite 使用的 JSON 存储类型。
func (sqliteDialect) jsonType() string { return "TEXT" }
// autoIncrementPrimaryKey returns the SQL definition for an auto-incrementing primary key.
//
// autoIncrementPrimaryKey 返回 SQLite 自增主键定义。
func (sqliteDialect) autoIncrementPrimaryKey() string { return "INTEGER PRIMARY KEY AUTOINCREMENT" }
// nowExpr returns the SQL expression for the current time.
//
// nowExpr 返回 SQLite 当前时间表达式。
func (sqliteDialect) nowExpr() string { return "CURRENT_TIMESTAMP" }
// insertDefinitionSQL builds SQL for inserting or updating a metric definition.
//
// insertDefinitionSQL 构造 SQLite 指标定义 upsert SQL。
func (sqliteDialect) insertDefinitionSQL(t tables) string {
return fmt.Sprintf(`INSERT INTO %s
(name, type, unit, description, retention_days, metadata, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(name) DO UPDATE SET
type = excluded.type,
unit = excluded.unit,
description = excluded.description,
retention_days = excluded.retention_days,
metadata = excluded.metadata,
updated_at = excluded.updated_at`, t.definitions)
}
// upsertPointSQL builds SQL for inserting or updating metric points.
//
// upsertPointSQL 构造 SQLite 采样点批量 upsert SQL。
func (sqliteDialect) upsertPointSQL(t tables, rowCount int) string {
return buildInsertPointsSQL(t.points, sqliteDialect{}, rowCount, `ON CONFLICT(metric_name, entity_id, tags_hash, ts_nano) DO UPDATE SET
value = excluded.value,
tags = excluded.tags,
labels = excluded.labels,
created_at = excluded.created_at`)
}
// mysqlDialect implements SQL rendering for MySQL.
//
// mysqlDialect 实现 MySQL 方言。
type mysqlDialect struct{}
// placeholder returns a bind placeholder for this SQL dialect.
//
// placeholder 返回 MySQL 的绑定参数占位符。
func (mysqlDialect) placeholder(int) string { return "?" }
// jsonPlaceholder returns a bind placeholder for JSON values.
//
// jsonPlaceholder 返回 MySQL JSON 值的绑定参数占位符。
func (mysqlDialect) jsonPlaceholder(int) string { return "?" }
// jsonType returns the SQL column type used for JSON values.
//
// jsonType 返回 MySQL 使用的 JSON 存储类型。
func (mysqlDialect) jsonType() string { return "JSON" }
// autoIncrementPrimaryKey returns the SQL definition for an auto-incrementing primary key.
//
// autoIncrementPrimaryKey 返回 MySQL 自增主键定义。
func (mysqlDialect) autoIncrementPrimaryKey() string { return "BIGINT AUTO_INCREMENT PRIMARY KEY" }
// nowExpr returns the SQL expression for the current time.
//
// nowExpr 返回 MySQL 当前时间表达式。
func (mysqlDialect) nowExpr() string { return "CURRENT_TIMESTAMP" }
// insertDefinitionSQL builds SQL for inserting or updating a metric definition.
//
// insertDefinitionSQL 构造 MySQL 指标定义 upsert SQL。
func (mysqlDialect) insertDefinitionSQL(t tables) string {
return fmt.Sprintf(`INSERT INTO %s
(name, type, unit, description, retention_days, metadata, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON DUPLICATE KEY UPDATE
type = VALUES(type),
unit = VALUES(unit),
description = VALUES(description),
retention_days = VALUES(retention_days),
metadata = VALUES(metadata),
updated_at = VALUES(updated_at)`, t.definitions)
}
// upsertPointSQL builds SQL for inserting or updating metric points.
//
// upsertPointSQL 构造 MySQL 采样点批量 upsert SQL。
func (mysqlDialect) upsertPointSQL(t tables, rowCount int) string {
return buildInsertPointsSQL(t.points, mysqlDialect{}, rowCount, `ON DUPLICATE KEY UPDATE
value = VALUES(value),
tags = VALUES(tags),
labels = VALUES(labels),
created_at = VALUES(created_at)`)
}
// postgresDialect implements SQL rendering for PostgreSQL.
//
// postgresDialect 实现 PostgreSQL 方言。
type postgresDialect struct{}
// placeholder returns a bind placeholder for this SQL dialect.
//
// placeholder 返回 PostgreSQL 的编号绑定参数占位符。
func (postgresDialect) placeholder(n int) string { return fmt.Sprintf("$%d", n) }
// jsonPlaceholder returns a bind placeholder for JSON values.
//
// jsonPlaceholder 返回 PostgreSQL JSONB 值的绑定参数占位符。
func (postgresDialect) jsonPlaceholder(n int) string { return fmt.Sprintf("$%d::jsonb", n) }
// jsonType returns the SQL column type used for JSON values.
//
// jsonType 返回 PostgreSQL 使用的 JSONB 存储类型。
func (postgresDialect) jsonType() string { return "JSONB" }
// autoIncrementPrimaryKey returns the SQL definition for an auto-incrementing primary key.
//
// autoIncrementPrimaryKey 返回 PostgreSQL 自增主键定义。
func (postgresDialect) autoIncrementPrimaryKey() string { return "BIGSERIAL PRIMARY KEY" }
// nowExpr returns the SQL expression for the current time.
//
// nowExpr 返回 PostgreSQL 当前时间表达式。
func (postgresDialect) nowExpr() string { return "CURRENT_TIMESTAMP" }
// insertDefinitionSQL builds SQL for inserting or updating a metric definition.
//
// insertDefinitionSQL 构造 PostgreSQL 指标定义 upsert SQL。
func (postgresDialect) insertDefinitionSQL(t tables) string {
return fmt.Sprintf(`INSERT INTO %s
(name, type, unit, description, retention_days, metadata, created_at, updated_at)
VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8)
ON CONFLICT(name) DO UPDATE SET
type = EXCLUDED.type,
unit = EXCLUDED.unit,
description = EXCLUDED.description,
retention_days = EXCLUDED.retention_days,
metadata = EXCLUDED.metadata,
updated_at = EXCLUDED.updated_at`, t.definitions)
}
// upsertPointSQL builds SQL for inserting or updating metric points.
//
// upsertPointSQL 构造 PostgreSQL 采样点批量 upsert SQL。
func (postgresDialect) upsertPointSQL(t tables, rowCount int) string {
return buildInsertPointsSQL(t.points, postgresDialect{}, rowCount, `ON CONFLICT(metric_name, entity_id, tags_hash, ts_nano) DO UPDATE SET
value = EXCLUDED.value,
tags = EXCLUDED.tags,
labels = EXCLUDED.labels,
created_at = EXCLUDED.created_at`)
}
// buildInsertPointsSQL builds a multi-row insert statement for metric points.
//
// buildInsertPointsSQL 构造多行采样点 INSERT/UPSERT SQL。
func buildInsertPointsSQL(table string, d dialect, rowCount int, suffix string) string {
var b strings.Builder
b.WriteString("INSERT INTO ")
b.WriteString(table)
b.WriteString(" (metric_name, entity_id, tags_hash, ts_nano, value, tags, labels, created_at) VALUES ")
arg := 1
rows := make([]string, rowCount)
for i := 0; i < rowCount; i++ {
parts := make([]string, 8)
for j := range parts {
// Columns are (metric_name, entity_id, tags_hash, ts_nano, value, tags,
// labels, created_at); positions 5 and 6 (0-indexed) are JSON values.
// tags_hash makes the tag set part of a point's identity, so points
// that share entity+timestamp but differ in tags no longer collide.
if j == 5 || j == 6 {
parts[j] = d.jsonPlaceholder(arg)
} else {
parts[j] = d.placeholder(arg)
}
arg++
}
rows[i] = "(" + strings.Join(parts, ", ") + ")"
}
b.WriteString(strings.Join(rows, ", "))
if suffix != "" {
b.WriteByte(' ')
b.WriteString(suffix)
}
return b.String()
}
// tableName combines a table prefix and logical table name.
//
// tableName 用表名前缀和逻辑表名生成实际表名。
func tableName(prefix, name string) string {
return prefix + name
}
// insertDefinitionOnlySQL builds a plain INSERT (no upsert clause) for a metric
// definition. CreateMetric uses it so a duplicate name surfaces as a unique
// constraint violation instead of silently overwriting the existing row.
//
// insertDefinitionOnlySQL 构造普通 INSERT(不带 upsert),供
// CreateMetric 保持“只创建”语义;重复名称会暴露为唯一约束错误。
func insertDefinitionOnlySQL(d dialect, t tables) string {
cols := "(name, type, unit, description, retention_days, metadata, created_at, updated_at)"
ph := []string{
d.placeholder(1),
d.placeholder(2),
d.placeholder(3),
d.placeholder(4),
d.placeholder(5),
d.jsonPlaceholder(6),
d.placeholder(7),
d.placeholder(8),
}
return fmt.Sprintf("INSERT INTO %s %s VALUES (%s)", t.definitions, cols, strings.Join(ph, ", "))
}
// sqlSingleQuote escapes a string for safe inclusion inside a single-quoted SQL
// string literal by doubling embedded single quotes.
//
// sqlSingleQuote 通过把单引号加倍,安全地转义可放入 SQL 单引号字符串
// 字面量的内容。
func sqlSingleQuote(s string) string {
return strings.ReplaceAll(s, "'", "''")
}
// sqlJSONPathQuoted builds a quoted JSON path member access ($."key") for the
// given object key and returns it ready to embed inside a single-quoted SQL
// string literal. Using the quoted form ($."key") rather than the bare form
// ($.key) is required for keys that contain dots, hyphens, spaces or other
// characters the JSON path grammar would otherwise interpret (e.g. a dot is a
// member separator, so the bare path "$.region.zone" would look up nested
// member "zone" of object "region" instead of the flat key "region.zone").
//
// Escaping is applied in two layers, innermost first:
// 1. JSON path string escaping inside the double quotes: backslash and double
// quote are backslash-escaped.
// 2. SQL string-literal escaping of the whole path: single quotes doubled.
//
// sqlJSONPathQuoted 为给定对象键构造带引号的 JSON path 成员访问($."key"),
// 并返回可嵌入 SQL 单引号字符串字面量的结果。必须使用带引号形式($."key"),
// 而不是裸形式($.key),因为包含点号、连字符、空格或其他 JSON path 语法字符的
// key 会被误解;例如点号是成员分隔符,所以裸路径 "$.region.zone" 会查找对象
// "region" 的嵌套成员 "zone",而不是平铺 key "region.zone"。
//
// 转义按从内到外两层应用:
// 1. 双引号内的 JSON path 字符串转义:反斜杠和双引号使用反斜杠转义。
// 2. 整个 path 的 SQL 字符串字面量转义:单引号加倍。
func sqlJSONPathQuoted(key string) string {
escaped := strings.ReplaceAll(key, "\\", "\\\\")
escaped = strings.ReplaceAll(escaped, "\"", "\\\"")
return sqlSingleQuote("$.\"" + escaped + "\"")
}
// jsonExtractEquals renders a JSON equality predicate.
//
// jsonExtractEquals 构造 SQLite JSON 字段等值过滤表达式。
func (sqliteDialect) jsonExtractEquals(column, key, placeholder string) string {
return fmt.Sprintf("json_extract(%s, '%s') = %s", column, sqlJSONPathQuoted(key), placeholder)
}
// jsonExtractEquals renders a JSON equality predicate.
//
// jsonExtractEquals 构造 MySQL JSON 字段等值过滤表达式。
func (mysqlDialect) jsonExtractEquals(column, key, placeholder string) string {
return fmt.Sprintf("JSON_UNQUOTE(JSON_EXTRACT(%s, '%s')) = %s", column, sqlJSONPathQuoted(key), placeholder)
}
// jsonExtractEquals renders a JSON equality predicate.
//
// jsonExtractEquals 构造 PostgreSQL JSONB 字段等值过滤表达式。
func (postgresDialect) jsonExtractEquals(column, key, placeholder string) string {
// PostgreSQL's ->> takes the key as a plain text literal (not a JSON path),
// so a dot or hyphen in the key is harmless; only single quotes need SQL
// escaping.
return fmt.Sprintf("%s->>'%s' = %s", column, sqlSingleQuote(key), placeholder)
}
// sqlAggValueExpr returns the SQL value expression that computes agg over the
// "value" column for the given driver, and whether that aggregation can be
// pushed down to this backend. Aggregations that need the ordered raw series
// (first/last/rate) are never pushed down here. Percentiles and population
// standard deviation are pushed down only where the backend has a portable,
// semantics-matching function:
//
// - STDDEV_POP: MySQL and PostgreSQL (matches the in-memory population stddev,
// which divides by N). SQLite has no built-in, so it falls back to memory.
// - percentile_cont(p) WITHIN GROUP (ORDER BY value): PostgreSQL only. This
// matches the in-memory linear-interpolation percentile for ANY p (p50,
// p95, p99, p99.9, ...). MySQL lacks a portable continuous-percentile
// function and SQLite has none, so both fall back to memory.
//
// The simple reductions (avg/min/max/sum/count) are portable everywhere.
//
// sqlAggValueExpr 返回给定驱动上对 "value" 列计算 agg 的 SQL 值表达式,
// 以及该聚合是否能下推到该后端。需要有序原始序列的聚合(first/last/rate)
// 永远不会在这里下推。百分位和总体标准差只会在后端提供可移植且语义匹配的函数时
// 下推:
//
// - STDDEV_POP:MySQL 和 PostgreSQL 支持(匹配内存中的总体标准差,
// 即除以 N)。SQLite 没有内置函数,因此回退到内存计算。
// - percentile_cont(p) WITHIN GROUP (ORDER BY value):仅 PostgreSQL 支持。
// 它匹配内存中任意 p(p50、p95、p99、p99.9 等)的线性插值百分位。
// MySQL 没有可移植的连续百分位函数,SQLite 也没有,因此两者都回退到内存计算。
//
// 简单归约(avg/min/max/sum/count)在所有后端都可移植。
func sqlAggValueExpr(driver Driver, agg Aggregation) (string, bool) {
switch agg {
case AggAvg:
return "AVG(value)", true
case AggMin:
return "MIN(value)", true
case AggMax:
return "MAX(value)", true
case AggSum:
return "SUM(value)", true
case AggCount:
return "COUNT(*)", true
case AggStdDev:
switch driver {
case DriverMySQL, DriverPostgreSQL:
return "STDDEV_POP(value)", true
default:
return "", false
}
default:
// Any percentile (fixed p50/p95/p99 and arbitrary pXX / pXX.X all land
// here). PostgreSQL has a portable continuous percentile that matches the
// in-memory linear-interpolation percentile; other backends fall back to
// the in-memory path.
if frac, ok := percentileFractionString(agg); ok {
if driver == DriverPostgreSQL {
return fmt.Sprintf("percentile_cont(%s) WITHIN GROUP (ORDER BY value)", frac), true
}
return "", false
}
return "", false
}
}
+160
View File
@@ -0,0 +1,160 @@
package metric
import (
"database/sql"
"fmt"
"strings"
)
// rollupColumns is the fixed column list for the rollups table, shared by the
// upsert builder and the row scanner so they cannot drift apart. tags_hash is a
// stable fingerprint of the (canonical) tag set; tags carries the tag map itself
// so reads can return it and tag filters can be pushed down. Including the tag
// dimension in the key is what keeps each distinct tag combination (e.g. a GPU
// device_index) as its own rollup series instead of being merged together.
//
// rollupColumns 是 rollups 表固定列清单,由 upsert 构造器和行扫描器共用,
// 避免二者漂移。tags_hash 是(规范化)标签集合的稳定指纹;tags 携带标签 map
// 本身,因此读取时可以返回它,也能下推标签过滤。把标签维度纳入 key,正是让
// 每个不同标签组合(例如 GPU 的 device_index)保留为自己的 rollup 序列、
// 而不是被合并在一起的原因。
const rollupColumns = "metric_name, entity_id, tags_hash, tags, resolution_nano, bucket_nano, " +
"count, sum, sum_sq, min_val, max_val, first_val, first_ts, last_val, last_ts, digest, created_at"
// rollupColumnCount is the number of columns in rollupColumns.
//
// rollupColumnCount 是 rollupColumns 中的列数量。
const rollupColumnCount = 17
// rollupTagsArgIndex is the 1-based position of the `tags` JSON column in
// rollupColumns. PostgreSQL needs a ::jsonb cast on that bind placeholder.
//
// rollupTagsArgIndex 是 tags JSON 列在 rollupColumns 中的 1 基位置;
// PostgreSQL 需要在对应占位符上追加 ::jsonb 转换。
const rollupTagsArgIndex = 4
// blobType returns the column type used to store the t-digest sketch.
//
// blobType 返回用于保存 t-digest 摘要的二进制列类型。
func (sqliteDialect) blobType() string { return "BLOB" }
// blobType returns the binary column type used for t-digest blobs.
//
// blobType 返回 MySQL 用于保存 t-digest 摘要的二进制列类型。
func (mysqlDialect) blobType() string { return "LONGBLOB" }
// blobType returns the binary column type used for t-digest blobs.
//
// blobType 返回 PostgreSQL 用于保存 t-digest 摘要的二进制列类型。
func (postgresDialect) blobType() string { return "BYTEA" }
// compactTxOptions returns nil so SQLite uses its default isolation. A single
// write connection already serializes the compaction transaction against
// concurrent writers, so the raw scan and the raw deletion observe a stable
// view without escalating the isolation level.
//
// compactTxOptions 返回 nil,让 SQLite 使用默认隔离级别。单个写连接已经把
// compaction 事务与并发写入串行化,因此 raw 扫描和 raw 删除无需提升隔离级别
// 即可观察到稳定视图。
func (sqliteDialect) compactTxOptions() *sql.TxOptions { return nil }
// compactTxOptions escalates to SERIALIZABLE so a point inserted between the raw
// scan and the raw deletion cannot be deleted without first being rolled up.
// InnoDB's SERIALIZABLE turns the scan into a locking read that blocks such a
// concurrent insert in the scanned range until the compaction commits.
//
// compactTxOptions 提升到 SERIALIZABLE,使在 raw 扫描和 raw 删除之间写入的点
// 不会在尚未进入 rollup 前被删除。InnoDB 的 SERIALIZABLE 会把扫描变为加锁读,
// 在 compaction 提交前阻塞扫描范围内的此类并发插入。
func (mysqlDialect) compactTxOptions() *sql.TxOptions {
return &sql.TxOptions{Isolation: sql.LevelSerializable}
}
// compactTxOptions escalates to SERIALIZABLE so the raw scan and the raw
// deletion share one snapshot. A concurrent insert into the scanned range that
// would otherwise be deleted without a rollup instead triggers a serialization
// failure that CompactMetric retries on a fresh snapshot.
//
// compactTxOptions 提升到 SERIALIZABLE,使 raw 扫描和 raw 删除共享同一个快照。
// 对扫描范围内的并发插入(否则会被删除却没有进入 rollup)会触发序列化失败,
// 由 CompactMetric 在新快照上重试。
func (postgresDialect) compactTxOptions() *sql.TxOptions {
return &sql.TxOptions{Isolation: sql.LevelSerializable}
}
// upsertRollupSQL builds a single-row upsert for a rollup cell, keyed by
// (metric_name, entity_id, tags_hash, resolution_nano, bucket_nano). Compact
// recomputes a bucket wholesale from its inputs each run, so the conflict action
// replaces every summary column rather than trying to merge in SQL.
//
// upsertRollupSQL 构造单个 rollup 单元的 upsert SQL,其 key 为
// (metric_name, entity_id, tags_hash, resolution_nano, bucket_nano)。Compact
// 每次都会从输入整体重新计算一个桶,因此冲突动作会替换每个摘要列,
// 而不是尝试在 SQL 中合并。
func (d sqliteDialect) upsertRollupSQL(t tables) string {
return buildUpsertRollupSQL(t.rollups, d, sqliteRollupConflict)
}
// upsertRollupSQL builds backend-specific SQL for upserting a rollup cell.
//
// upsertRollupSQL 构造 MySQL rollup 单元 upsert SQL。
func (d mysqlDialect) upsertRollupSQL(t tables) string {
return buildUpsertRollupSQL(t.rollups, d, mysqlRollupConflict)
}
// upsertRollupSQL builds backend-specific SQL for upserting a rollup cell.
//
// upsertRollupSQL 构造 PostgreSQL rollup 单元 upsert SQL。
func (d postgresDialect) upsertRollupSQL(t tables) string {
return buildUpsertRollupSQL(t.rollups, d, postgresRollupConflict)
}
// sqliteRollupConflict is the SQLite conflict clause for replacing rollup cells.
//
// sqliteRollupConflict 是 SQLite 用于替换 rollup 单元的冲突处理子句。
const sqliteRollupConflict = `ON CONFLICT(metric_name, entity_id, tags_hash, resolution_nano, bucket_nano) DO UPDATE SET
tags = excluded.tags,
count = excluded.count, sum = excluded.sum, sum_sq = excluded.sum_sq,
min_val = excluded.min_val, max_val = excluded.max_val,
first_val = excluded.first_val, first_ts = excluded.first_ts,
last_val = excluded.last_val, last_ts = excluded.last_ts,
digest = excluded.digest, created_at = excluded.created_at`
// postgresRollupConflict is the PostgreSQL conflict clause for replacing rollup cells.
//
// postgresRollupConflict 是 PostgreSQL 用于替换 rollup 单元的冲突处理子句。
const postgresRollupConflict = `ON CONFLICT(metric_name, entity_id, tags_hash, resolution_nano, bucket_nano) DO UPDATE SET
tags = EXCLUDED.tags,
count = EXCLUDED.count, sum = EXCLUDED.sum, sum_sq = EXCLUDED.sum_sq,
min_val = EXCLUDED.min_val, max_val = EXCLUDED.max_val,
first_val = EXCLUDED.first_val, first_ts = EXCLUDED.first_ts,
last_val = EXCLUDED.last_val, last_ts = EXCLUDED.last_ts,
digest = EXCLUDED.digest, created_at = EXCLUDED.created_at`
// mysqlRollupConflict is the MySQL conflict clause for replacing rollup cells.
//
// mysqlRollupConflict 是 MySQL 用于替换 rollup 单元的冲突处理子句。
const mysqlRollupConflict = `ON DUPLICATE KEY UPDATE
tags = VALUES(tags),
count = VALUES(count), sum = VALUES(sum), sum_sq = VALUES(sum_sq),
min_val = VALUES(min_val), max_val = VALUES(max_val),
first_val = VALUES(first_val), first_ts = VALUES(first_ts),
last_val = VALUES(last_val), last_ts = VALUES(last_ts),
digest = VALUES(digest), created_at = VALUES(created_at)`
// buildUpsertRollupSQL builds a single-row rollup upsert statement.
//
// buildUpsertRollupSQL 构造单行 rollup upsert SQL。
func buildUpsertRollupSQL(table string, d dialect, conflict string) string {
ph := make([]string, rollupColumnCount)
for i := 0; i < rollupColumnCount; i++ {
if i+1 == rollupTagsArgIndex {
// The tags column is JSON; PostgreSQL requires an explicit ::jsonb cast.
ph[i] = d.jsonPlaceholder(i + 1)
} else {
ph[i] = d.placeholder(i + 1)
}
}
return fmt.Sprintf("INSERT INTO %s (%s) VALUES (%s) %s",
table, rollupColumns, strings.Join(ph, ", "), conflict)
}
+60
View File
@@ -0,0 +1,60 @@
package metric
import (
"strings"
"testing"
)
// TestDialectsGenerateBackendSpecificSQL verifies backend-specific SQL rendering.
//
// TestDialectsGenerateBackendSpecificSQL 验证不同数据库后端生成各自的 SQL。
func TestDialectsGenerateBackendSpecificSQL(t *testing.T) {
tests := []struct {
name string
driver Driver
placeholder string
upsert string
jsonType string
}{
{
name: "sqlite",
driver: DriverSQLite,
placeholder: "?",
upsert: "ON CONFLICT(metric_name, entity_id, tags_hash, ts_nano)",
jsonType: "TEXT",
},
{
name: "mysql",
driver: DriverMySQL,
placeholder: "?",
upsert: "ON DUPLICATE KEY UPDATE",
jsonType: "JSON",
},
{
name: "postgresql",
driver: DriverPostgreSQL,
placeholder: "$1",
upsert: "ON CONFLICT(metric_name, entity_id, tags_hash, ts_nano)",
jsonType: "JSONB",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
d := newDialect(tt.driver)
if got := d.placeholder(1); got != tt.placeholder {
t.Fatalf("placeholder: expected %q, got %q", tt.placeholder, got)
}
if got := d.jsonType(); got != tt.jsonType {
t.Fatalf("json type: expected %q, got %q", tt.jsonType, got)
}
sql := d.upsertPointSQL(tables{points: "metric_points"}, 2)
if !strings.Contains(sql, "metric_points") {
t.Fatalf("upsert sql should reference metric_points: %s", sql)
}
if !strings.Contains(sql, tt.upsert) {
t.Fatalf("upsert sql should contain %q: %s", tt.upsert, sql)
}
})
}
}
+26
View File
@@ -0,0 +1,26 @@
package metric
import "errors"
var (
// ErrInvalidArgument reports an invalid caller-supplied argument.
//
// ErrInvalidArgument 表示调用方传入了无效参数。
ErrInvalidArgument = errors.New("metric: invalid argument")
// ErrNotFound reports a missing metric or record.
//
// ErrNotFound 表示指标或记录不存在。
ErrNotFound = errors.New("metric: not found")
// ErrNoData reports that a valid query range contained no samples.
//
// ErrNoData 表示有效查询范围内没有采样。
ErrNoData = errors.New("metric: no data in range")
// ErrAlreadyExists reports that a create-only operation found an existing row.
//
// ErrAlreadyExists 表示只创建操作遇到了已存在的行。
ErrAlreadyExists = errors.New("metric: already exists")
// ErrClosed reports that the store has already been closed.
//
// ErrClosed 表示 Store 已经关闭。
ErrClosed = errors.New("metric: store is closed")
)
+347
View File
@@ -0,0 +1,347 @@
package metric_test
import (
"context"
"fmt"
"log"
"strings"
"time"
"github.com/komari-monitor/komari/pkg/metric"
)
// Example_rollupTags demonstrates tagged rollups with automatic series routing.
//
// Example_rollupTags 演示带标签的 rollup 以及 Series 自动路由读取。
func Example_rollupTags() {
ctx := context.Background()
base := time.Date(2026, 6, 18, 10, 0, 0, 0, time.UTC)
store, err := metric.Open(ctx, metric.SQLite(
"file:metric-example?mode=memory&cache=shared",
metric.WithRollupPolicy(metric.RollupPolicy{
RawRetention: 2 * time.Minute,
Tiers: []metric.RollupTier{
{Interval: time.Minute, Retention: 24 * time.Hour},
},
}),
))
if err != nil {
log.Fatal(err)
}
defer store.Close()
if err := store.CreateMetric(ctx, metric.Definition{
Name: "gpu.usage",
Type: metric.TypeGauge,
Unit: "%",
}); err != nil {
log.Fatal(err)
}
var points []metric.Point
for i := 0; i < 10; i++ {
ts := base.Add(time.Duration(i) * time.Second)
points = append(points,
metric.Point{
MetricName: "gpu.usage",
EntityID: "host-1",
Timestamp: ts,
Value: float64(10 + i),
Tags: map[string]string{"device_index": "0"},
},
metric.Point{
MetricName: "gpu.usage",
EntityID: "host-1",
Timestamp: ts,
Value: float64(80 + i),
Tags: map[string]string{"device_index": "1"},
},
)
}
if err := store.WriteBatch(ctx, points); err != nil {
log.Fatal(err)
}
// Compact builds one rollup series per tag set. Raw points older than the
// policy's RawRetention are deleted after their rollups are materialized.
now := base.Add(time.Hour)
if _, err := store.Compact(ctx, now); err != nil {
log.Fatal(err)
}
series, err := store.Series(ctx, metric.AggregateQuery{
Query: metric.Query{
MetricName: "gpu.usage",
EntityID: "host-1",
Start: base,
End: base.Add(time.Minute),
Tags: map[string]string{"device_index": "0"},
},
Aggregation: metric.AggAvg,
Interval: time.Minute,
}, now)
if err != nil {
log.Fatal(err)
}
fmt.Printf("device 0: count=%d avg=%.1f\n", series[0].Count, series[0].Value)
// Output:
// device 0: count=10 avg=14.5
}
// Example_agentFleetLifecycle demonstrates a realistic Komari-style fleet:
// 50 agents report system metrics every second, ping tasks are represented as a
// tagged ping metric, and lifecycle events clean up task and agent series.
//
// Example_agentFleetLifecycle 演示贴近 Komari 的实际场景:50 个 agent 每秒上报系统
// 指标,ping task 用带标签的 ping 指标表示,并通过生命周期事件清理 task 和 agent 序列。
func Example_agentFleetLifecycle() {
ctx := context.Background()
base := time.Date(2026, 6, 18, 10, 0, 0, 0, time.UTC)
store, err := metric.Open(ctx, metric.SQLite(
"file:metric-agent-fleet?mode=memory&cache=shared",
metric.WithRollupPolicy(metric.RollupPolicy{
RawRetention: 2 * time.Minute,
Tiers: []metric.RollupTier{
{Interval: time.Minute, Retention: 24 * time.Hour},
{Interval: 5 * time.Minute, Retention: 30 * 24 * time.Hour},
},
}),
))
if err != nil {
log.Fatal(err)
}
defer store.Close()
for _, def := range []metric.Definition{
{Name: "cpu.usage", Type: metric.TypeGauge, Unit: "%"},
{Name: "mem.usage", Type: metric.TypeGauge, Unit: "%"},
{Name: "disk.usage", Type: metric.TypeGauge, Unit: "%"},
{Name: "swap.usage", Type: metric.TypeGauge, Unit: "%"},
{Name: "net.in.bytes", Type: metric.TypeCounter, Unit: "bytes"},
{Name: "net.out.bytes", Type: metric.TypeCounter, Unit: "bytes"},
{Name: "process.count", Type: metric.TypeGauge, Unit: "count"},
{Name: "gpu.usage", Type: metric.TypeGauge, Unit: "%"},
{Name: "ping.latency_ms", Type: metric.TypeGauge, Unit: "ms"},
} {
if err := store.UpsertMetric(ctx, def); err != nil {
log.Fatal(err)
}
}
agents := makeAgents(50)
pingTasks := []string{"pingtask1", "pingtask2"}
for sec := 0; sec < 120; sec++ {
ts := base.Add(time.Duration(sec) * time.Second)
switch sec {
case 45:
// A ping task is deleted by the admin. Because task_id is a tag, one
// DeleteSeries removes that task's records for every agent, including
// already materialized rollups.
if _, err := store.DeleteSeries(ctx, metric.Query{
MetricName: "ping.latency_ms",
Tags: map[string]string{"task_id": "pingtask2"},
}); err != nil {
log.Fatal(err)
}
pingTasks = []string{"pingtask1"}
case 70:
// The admin clears all ping task records while keeping the task
// definitions themselves. Deleting the metric data is the simplest
// reset; UpsertMetric recreates the definition before new samples land.
if err := store.DeleteMetric(ctx, "ping.latency_ms"); err != nil {
log.Fatal(err)
}
if err := store.UpsertMetric(ctx, metric.Definition{
Name: "ping.latency_ms", Type: metric.TypeGauge, Unit: "ms",
}); err != nil {
log.Fatal(err)
}
case 85:
// An agent is removed. DeleteEntity clears every metric series for
// that agent, raw and rollup, across the whole store.
if _, err := store.DeleteEntity(ctx, "agent-007"); err != nil {
log.Fatal(err)
}
agents = removeAgent(agents, "agent-007")
case 95:
// A new agent is added. No schema change is needed; the first write
// creates its series naturally.
agents = append(agents, "agent-051")
}
points := make([]metric.Point, 0, len(agents)*(7+2+len(pingTasks)))
for _, agentID := range agents {
points = append(points, systemPoints(agentID, ts, sec)...)
for _, gpu := range []string{"0", "1"} {
points = append(points, metric.Point{
MetricName: "gpu.usage",
EntityID: agentID,
Timestamp: ts,
Value: gpuUsage(agentID, gpu, sec),
Tags: map[string]string{"device_index": gpu},
})
}
for _, taskID := range pingTasks {
points = append(points, metric.Point{
MetricName: "ping.latency_ms",
EntityID: agentID,
Timestamp: ts,
Value: pingLatency(agentID, taskID, sec),
Tags: map[string]string{"task_id": taskID},
})
}
}
if err := store.WriteBatch(ctx, points); err != nil {
log.Fatal(err)
}
if sec%30 == 29 {
if _, err := store.Compact(ctx, ts); err != nil {
log.Fatal(err)
}
}
}
now := base.Add(2 * time.Hour)
if _, err := store.Compact(ctx, now); err != nil {
log.Fatal(err)
}
activeCPU, err := store.Series(ctx, metric.AggregateQuery{
Query: metric.Query{
MetricName: "cpu.usage",
Start: base,
End: base.Add(2 * time.Minute),
},
Aggregation: metric.AggAvg,
Interval: time.Minute,
}, now)
if err != nil {
log.Fatal(err)
}
deletedAgentCPU, err := store.Series(ctx, metric.AggregateQuery{
Query: metric.Query{
MetricName: "cpu.usage",
EntityID: "agent-007",
Start: base,
End: base.Add(2 * time.Minute),
},
Aggregation: metric.AggAvg,
Interval: time.Minute,
}, now)
if err != nil {
log.Fatal(err)
}
task1, err := store.Series(ctx, metric.AggregateQuery{
Query: metric.Query{
MetricName: "ping.latency_ms",
Start: base,
End: base.Add(2 * time.Minute),
Tags: map[string]string{"task_id": "pingtask1"},
},
Aggregation: metric.AggAvg,
Interval: time.Minute,
}, now)
if err != nil {
log.Fatal(err)
}
task2, err := store.Series(ctx, metric.AggregateQuery{
Query: metric.Query{
MetricName: "ping.latency_ms",
Start: base,
End: base.Add(2 * time.Minute),
Tags: map[string]string{"task_id": "pingtask2"},
},
Aggregation: metric.AggAvg,
Interval: time.Minute,
}, now)
if err != nil {
log.Fatal(err)
}
fmt.Printf("agents now=%d\n", len(agents))
fmt.Printf("fleet cpu buckets=%d\n", len(nonEmpty(activeCPU)))
fmt.Printf("deleted agent buckets=%d\n", len(nonEmpty(deletedAgentCPU)))
fmt.Printf("pingtask1 buckets=%d\n", len(nonEmpty(task1)))
fmt.Printf("pingtask2 buckets=%d\n", len(nonEmpty(task2)))
// Output:
// agents now=50
// fleet cpu buckets=2
// deleted agent buckets=0
// pingtask1 buckets=1
// pingtask2 buckets=0
}
func makeAgents(n int) []string {
out := make([]string, 0, n)
for i := 1; i <= n; i++ {
out = append(out, fmt.Sprintf("agent-%03d", i))
}
return out
}
func removeAgent(agents []string, deleted string) []string {
out := agents[:0]
for _, agentID := range agents {
if agentID != deleted {
out = append(out, agentID)
}
}
return out
}
func systemPoints(agentID string, ts time.Time, sec int) []metric.Point {
id := numericSuffix(agentID)
return []metric.Point{
{MetricName: "cpu.usage", EntityID: agentID, Timestamp: ts, Value: float64((id*3 + sec) % 100)},
{MetricName: "mem.usage", EntityID: agentID, Timestamp: ts, Value: 35 + float64((id+sec)%40)},
{MetricName: "disk.usage", EntityID: agentID, Timestamp: ts, Value: 50 + float64(id%20)/2},
{MetricName: "swap.usage", EntityID: agentID, Timestamp: ts, Value: float64((id + sec/10) % 12)},
{MetricName: "net.in.bytes", EntityID: agentID, Timestamp: ts, Value: float64(sec * (1000 + id))},
{MetricName: "net.out.bytes", EntityID: agentID, Timestamp: ts, Value: float64(sec * (700 + id))},
{MetricName: "process.count", EntityID: agentID, Timestamp: ts, Value: float64(90 + id%25)},
}
}
func gpuUsage(agentID, deviceIndex string, sec int) float64 {
id := numericSuffix(agentID)
device := 0
if deviceIndex == "1" {
device = 37
}
return float64((id*5 + device + sec*2) % 100)
}
func pingLatency(agentID, taskID string, sec int) float64 {
id := numericSuffix(agentID)
offset := 0
if strings.HasSuffix(taskID, "2") {
offset = 35
}
return float64(20 + (id+sec+offset)%80)
}
func numericSuffix(agentID string) int {
var n int
for _, r := range agentID {
if r >= '0' && r <= '9' {
n = n*10 + int(r-'0')
}
}
return n
}
func nonEmpty(points []metric.AggregatePoint) []metric.AggregatePoint {
out := points[:0]
for _, p := range points {
if p.Count > 0 {
out = append(out, p)
}
}
return out
}
+565
View File
@@ -0,0 +1,565 @@
package metric
import (
"context"
"errors"
"math"
"path/filepath"
"testing"
"time"
)
// newMemStore opens an isolated in-memory store for tests.
//
// newMemStore 打开一个用于测试的隔离内存 Store。
func newMemStore(t *testing.T) *Store {
t.Helper()
store, err := Open(context.Background(), SQLite("file:imp-test?mode=memory&cache=shared"))
if err != nil {
t.Fatalf("open store: %v", err)
}
t.Cleanup(func() { store.Close() })
return store
}
// TestCreateMetricRejectsDuplicate verifies create-only metric semantics.
//
// TestCreateMetricRejectsDuplicate 验证 CreateMetric 遇到重复指标时会拒绝。
func TestCreateMetricRejectsDuplicate(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
def := Definition{Name: "dup.metric", Type: TypeGauge}
if err := s.CreateMetric(ctx, def); err != nil {
t.Fatalf("first create: %v", err)
}
err := s.CreateMetric(ctx, def)
if !errors.Is(err, ErrAlreadyExists) {
t.Fatalf("expected ErrAlreadyExists on duplicate, got %v", err)
}
}
// TestUpsertMetricOverwrites verifies upsert updates mutable definition fields.
//
// TestUpsertMetricOverwrites 验证 UpsertMetric 会更新指标定义的可变字段。
func TestUpsertMetricOverwrites(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "m", Type: TypeGauge, Unit: "ms"}); err != nil {
t.Fatalf("create: %v", err)
}
if err := s.UpsertMetric(ctx, Definition{Name: "m", Type: TypeCounter, Unit: "count"}); err != nil {
t.Fatalf("upsert: %v", err)
}
got, err := s.GetMetric(ctx, "m")
if err != nil {
t.Fatalf("get: %v", err)
}
if got.Type != TypeCounter || got.Unit != "count" {
t.Fatalf("upsert did not overwrite: %#v", got)
}
}
// TestTagFilterPushdownWithPaging verifies tag filtering happens before paging.
//
// TestTagFilterPushdownWithPaging 验证标签过滤会先于分页在 SQL 中执行。
func TestTagFilterPushdownWithPaging(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "req", Type: TypeCounter}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
var batch []Point
for i := 0; i < 6; i++ {
env := "prod"
if i%2 == 1 {
env = "stage"
}
batch = append(batch, Point{
MetricName: "req", EntityID: "n1",
Timestamp: base.Add(time.Duration(i) * time.Minute),
Value: float64(i),
Tags: map[string]string{"env": env},
})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
// 3 prod points (i=0,2,4); with Limit=2 the SQL LIMIT must apply to the
// tag-filtered set, returning the first two prod points in time order.
got, err := s.Query(ctx, Query{
MetricName: "req", EntityID: "n1",
Start: base, End: base.Add(time.Hour),
Tags: map[string]string{"env": "prod"},
Limit: 2,
})
if err != nil {
t.Fatalf("query: %v", err)
}
if len(got) != 2 {
t.Fatalf("expected 2 paged prod points, got %d", len(got))
}
if got[0].Value != 0 || got[1].Value != 2 {
t.Fatalf("unexpected paged values: %#v", got)
}
}
// TestSQLAggregateMatchesInMemory compares SQL and in-memory aggregation.
//
// TestSQLAggregateMatchesInMemory 对比 SQL 下推聚合和内存聚合结果。
func TestSQLAggregateMatchesInMemory(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "g", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
var batch []Point
vals := []float64{5, 15, 25, 35, 100, 200}
for i, v := range vals {
batch = append(batch, Point{
MetricName: "g", EntityID: "n1",
Timestamp: base.Add(time.Duration(i) * time.Minute),
Value: v,
})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
q := AggregateQuery{
Query: Query{MetricName: "g", EntityID: "n1", Start: base, End: base.Add(time.Hour)},
Aggregation: AggAvg,
Interval: 3 * time.Minute,
}
// SQL pushdown path.
sqlRes, err := s.Aggregate(ctx, q)
if err != nil {
t.Fatalf("sql aggregate: %v", err)
}
// In-memory reference over the same raw points.
pts, err := s.Query(ctx, q.Query)
if err != nil {
t.Fatalf("query: %v", err)
}
memRes, err := AggregatePoints(pts, q)
if err != nil {
t.Fatalf("mem aggregate: %v", err)
}
if len(sqlRes) != len(memRes) {
t.Fatalf("bucket count mismatch: sql=%d mem=%d", len(sqlRes), len(memRes))
}
for i := range sqlRes {
if !sqlRes[i].Bucket.Equal(memRes[i].Bucket) || sqlRes[i].Value != memRes[i].Value || sqlRes[i].Count != memRes[i].Count {
t.Fatalf("bucket %d mismatch: sql=%#v mem=%#v", i, sqlRes[i], memRes[i])
}
}
}
// TestCounterRateHandlesReset verifies reset-aware counter rate calculation.
//
// TestCounterRateHandlesReset 验证计数器重置时速率计算仍然稳定。
func TestCounterRateHandlesReset(t *testing.T) {
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
// Counter goes 0 -> 10 -> reset -> 5; naive (last-first)/sec would give a
// positive-but-wrong 5/30s. Correct counter rate sums positive deltas:
// (10-0) + (5-0 after reset) = 15 over 30s = 0.5/s.
pts := []Point{
{Timestamp: base, Value: 0},
{Timestamp: base.Add(10 * time.Second), Value: 10},
{Timestamp: base.Add(20 * time.Second), Value: 0},
{Timestamp: base.Add(30 * time.Second), Value: 5},
}
rate := counterRate(pts)
if rate != 0.5 {
t.Fatalf("expected reset-aware rate 0.5/s, got %v", rate)
}
}
// TestAlignTimeNegativeTimestamp verifies pre-epoch bucket alignment.
//
// TestAlignTimeNegativeTimestamp 验证 Unix epoch 之前的时间也能正确对齐桶。
func TestAlignTimeNegativeTimestamp(t *testing.T) {
interval := time.Minute
// 30s before the epoch should align down to -60s, not up to 0.
tm := time.Unix(-30, 0).UTC()
got := alignTime(tm, interval)
want := time.Unix(-60, 0).UTC()
if !got.Equal(want) {
t.Fatalf("alignTime negative: got %v want %v", got, want)
}
}
// TestLatestReturnsMostRecent verifies latest-point ordering.
//
// TestLatestReturnsMostRecent 验证 Latest 返回最新采样点。
func TestLatestReturnsMostRecent(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "l", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
for i := 0; i < 5; i++ {
if err := s.Write(ctx, Point{MetricName: "l", EntityID: "n1", Timestamp: base.Add(time.Duration(i) * time.Minute), Value: float64(i)}); err != nil {
t.Fatalf("write: %v", err)
}
}
got, err := s.Latest(ctx, "l", "n1", 2)
if err != nil {
t.Fatalf("latest: %v", err)
}
if len(got) != 2 || got[0].Value != 4 || got[1].Value != 3 {
t.Fatalf("unexpected latest: %#v", got)
}
}
// TestStatsDistinguishesNoDataFromUnknownMetric verifies stats error semantics.
//
// TestStatsDistinguishesNoDataFromUnknownMetric 验证 Stats 能区分无数据和未知指标。
func TestStatsDistinguishesNoDataFromUnknownMetric(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
// Unknown metric -> ErrNotFound.
_, err := s.Stats(ctx, Query{MetricName: "nope", Start: base, End: base.Add(time.Hour)})
if !errors.Is(err, ErrNotFound) {
t.Fatalf("unknown metric: expected ErrNotFound, got %v", err)
}
// Known metric but empty window -> ErrNoData.
if err := s.CreateMetric(ctx, Definition{Name: "known", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
_, err = s.Stats(ctx, Query{MetricName: "known", Start: base, End: base.Add(time.Hour)})
if !errors.Is(err, ErrNoData) {
t.Fatalf("empty window: expected ErrNoData, got %v", err)
}
}
// TestStdDevPopMatchesCalculateStats verifies population standard deviation.
//
// TestStdDevPopMatchesCalculateStats 验证总体标准差与统计摘要一致。
func TestStdDevPopMatchesCalculateStats(t *testing.T) {
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
var pts []Point
for i, v := range []float64{10, 20, 30, 40, 50} {
pts = append(pts, Point{Timestamp: base.Add(time.Duration(i) * time.Minute), Value: v})
}
st, err := CalculateStats(pts)
if err != nil {
t.Fatalf("stats: %v", err)
}
got, _ := aggregateValue(pts, AggStdDev)
if math.Abs(got-st.StdDev) > 1e-9 {
t.Fatalf("AggStdDev %v != Stats.StdDev %v", got, st.StdDev)
}
// Known value: population stddev of 10..50 step 10 is sqrt(200) ~= 14.142135.
if math.Abs(got-14.142135623730951) > 1e-9 {
t.Fatalf("unexpected population stddev: %v", got)
}
}
// TestAggregateStdDevSQLiteUsesMemoryPath verifies SQLite stddev fallback.
//
// TestAggregateStdDevSQLiteUsesMemoryPath 验证 SQLite 标准差聚合会回退到内存路径。
func TestAggregateStdDevSQLiteUsesMemoryPath(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "sd", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
var batch []Point
for i, v := range []float64{10, 20, 30} {
batch = append(batch, Point{MetricName: "sd", EntityID: "n1", Timestamp: base.Add(time.Duration(i) * time.Minute), Value: v})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
res, err := s.Aggregate(ctx, AggregateQuery{
Query: Query{MetricName: "sd", EntityID: "n1", Start: base, End: base.Add(time.Hour)},
Aggregation: AggStdDev,
Interval: time.Hour,
})
if err != nil {
t.Fatalf("aggregate stddev: %v", err)
}
if len(res) != 1 {
t.Fatalf("expected 1 bucket, got %d", len(res))
}
// Population stddev of {10,20,30} = sqrt(200/3) ~= 8.16496.
if math.Abs(res[0].Value-8.16496580927726) > 1e-9 {
t.Fatalf("unexpected stddev bucket value: %v", res[0].Value)
}
}
// TestSQLAggValueExprPushdownMatrix verifies aggregation pushdown support.
//
// TestSQLAggValueExprPushdownMatrix 验证各后端支持的聚合下推矩阵。
func TestSQLAggValueExprPushdownMatrix(t *testing.T) {
cases := []struct {
driver Driver
agg Aggregation
ok bool
}{
{DriverSQLite, AggAvg, true},
{DriverSQLite, AggStdDev, false}, // no STDDEV_POP in sqlite -> memory
{DriverSQLite, AggP95, false}, // no percentile in sqlite -> memory
{DriverMySQL, AggStdDev, true}, // STDDEV_POP
{DriverMySQL, AggP95, false}, // no portable continuous percentile
{DriverPostgreSQL, AggStdDev, true},
{DriverPostgreSQL, AggP50, true}, // percentile_cont
{DriverPostgreSQL, AggP99, true},
{DriverPostgreSQL, AggFirst, false}, // needs ordered raw series
}
for _, c := range cases {
expr, ok := sqlAggValueExpr(c.driver, c.agg)
if ok != c.ok {
t.Fatalf("%s/%s: pushdown=%v want %v (expr=%q)", c.driver, c.agg, ok, c.ok, expr)
}
if ok && expr == "" {
t.Fatalf("%s/%s: pushdown ok but empty expr", c.driver, c.agg)
}
}
}
// TestSQLiteReadPoolOpens verifies SQLite read-pool creation.
//
// TestSQLiteReadPoolOpens 验证 SQLite 只读连接池会按配置打开。
func TestSQLiteReadPoolOpens(t *testing.T) {
ctx := context.Background()
dir := filepath.Join(t.TempDir(), "rp")
store, err := Open(ctx, SQLiteInDir(dir, WithSQLiteReadPool(4)))
if err != nil {
t.Fatalf("open with read pool: %v", err)
}
t.Cleanup(func() { store.Close() })
if store.readDB == nil {
t.Fatalf("expected a dedicated read pool to be opened")
}
if store.reader() != store.readDB {
t.Fatalf("reader() should return the dedicated read pool")
}
// Round-trip a write (primary) and a read (read pool).
if err := store.CreateMetric(ctx, Definition{Name: "rp", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
if err := store.Write(ctx, Point{MetricName: "rp", EntityID: "n1", Timestamp: base, Value: 7}); err != nil {
t.Fatalf("write: %v", err)
}
got, err := store.Latest(ctx, "rp", "n1", 1)
if err != nil {
t.Fatalf("latest: %v", err)
}
if len(got) != 1 || got[0].Value != 7 {
t.Fatalf("read pool round-trip failed: %#v", got)
}
}
// TestMemoryDSNSkipsReadPool verifies memory SQLite skips read pools.
//
// TestMemoryDSNSkipsReadPool 验证内存 SQLite 不会打开独立读池。
func TestMemoryDSNSkipsReadPool(t *testing.T) {
ctx := context.Background()
store, err := Open(ctx, SQLite("file:rp-mem?mode=memory&cache=shared", WithSQLiteReadPool(4)))
if err != nil {
t.Fatalf("open: %v", err)
}
t.Cleanup(func() { store.Close() })
if store.readDB != nil {
t.Fatalf("in-memory database must not open a second read pool")
}
}
// TestWriteBatchAtomicAcrossChunks verifies chunked writes are atomic.
//
// TestWriteBatchAtomicAcrossChunks 验证分块批量写入仍保持整体原子性。
func TestWriteBatchAtomicAcrossChunks(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "atom", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
// 1500 valid points, then one invalid (missing entity) to force a mid-batch
// failure on the second chunk. The whole batch must roll back.
var batch []Point
for i := 0; i < 1500; i++ {
batch = append(batch, Point{MetricName: "atom", EntityID: "n1", Timestamp: base.Add(time.Duration(i) * time.Millisecond), Value: float64(i)})
}
batch = append(batch, Point{MetricName: "atom", EntityID: "", Timestamp: base.Add(2 * time.Second), Value: 1}) // invalid
if err := s.WriteBatch(ctx, batch); err == nil {
t.Fatalf("expected WriteBatch to fail on invalid point")
}
// Nothing should have been committed.
pts, err := s.Query(ctx, Query{MetricName: "atom", EntityID: "n1", Start: base, End: base.Add(time.Hour)})
if err != nil {
t.Fatalf("query: %v", err)
}
if len(pts) != 0 {
t.Fatalf("expected 0 committed points after rolled-back batch, got %d", len(pts))
}
}
// TestAggregateBucketPagingSQLPath verifies bucket paging in SQL aggregation.
//
// TestAggregateBucketPagingSQLPath 验证 SQL 聚合路径按桶分页。
func TestAggregateBucketPagingSQLPath(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "bp", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
// 6 points across 6 one-minute buckets, values 0..5.
var batch []Point
for i := 0; i < 6; i++ {
batch = append(batch, Point{MetricName: "bp", EntityID: "n1", Timestamp: base.Add(time.Duration(i) * time.Minute), Value: float64(i)})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
// AggAvg -> SQL pushdown path. One bucket per minute. Page: offset 1, limit 2.
res, err := s.Aggregate(ctx, AggregateQuery{
Query: Query{MetricName: "bp", EntityID: "n1", Start: base, End: base.Add(time.Hour)},
Aggregation: AggAvg,
Interval: time.Minute,
BucketLimit: 2,
BucketOffset: 1,
})
if err != nil {
t.Fatalf("aggregate: %v", err)
}
if len(res) != 2 {
t.Fatalf("expected 2 paged buckets, got %d", len(res))
}
// Buckets are values 0..5; offset 1 skips the value-0 bucket, so we expect 1,2.
if res[0].Value != 1 || res[1].Value != 2 {
t.Fatalf("unexpected paged bucket values: %#v", res)
}
}
// TestAggregateBucketPagingMemoryPathMatchesSQL compares bucket paging paths.
//
// TestAggregateBucketPagingMemoryPathMatchesSQL 对比内存聚合和 SQL 聚合的桶分页语义。
func TestAggregateBucketPagingMemoryPathMatchesSQL(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "bp2", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
var batch []Point
for i := 0; i < 6; i++ {
batch = append(batch, Point{MetricName: "bp2", EntityID: "n1", Timestamp: base.Add(time.Duration(i) * time.Minute), Value: float64(i * 10)})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
q := Query{MetricName: "bp2", EntityID: "n1", Start: base, End: base.Add(time.Hour)}
// AggMin (SQL pushdown) and AggP50 (memory fallback on sqlite) with identical
// bucket paging must select the same bucket window (the same timestamps).
sqlPaged, err := s.Aggregate(ctx, AggregateQuery{Query: q, Aggregation: AggMin, Interval: time.Minute, BucketLimit: 3, BucketOffset: 2})
if err != nil {
t.Fatalf("sql aggregate: %v", err)
}
memPaged, err := s.Aggregate(ctx, AggregateQuery{Query: q, Aggregation: AggP50, Interval: time.Minute, BucketLimit: 3, BucketOffset: 2})
if err != nil {
t.Fatalf("mem aggregate: %v", err)
}
if len(sqlPaged) != 3 || len(memPaged) != 3 {
t.Fatalf("expected 3 buckets each, got sql=%d mem=%d", len(sqlPaged), len(memPaged))
}
for i := range sqlPaged {
if !sqlPaged[i].Bucket.Equal(memPaged[i].Bucket) {
t.Fatalf("bucket window mismatch at %d: sql=%v mem=%v", i, sqlPaged[i].Bucket, memPaged[i].Bucket)
}
}
// Per-bucket single point, so min == p50 == the value.
for i := range sqlPaged {
if sqlPaged[i].Value != memPaged[i].Value {
t.Fatalf("value mismatch at %d: sql=%v mem=%v", i, sqlPaged[i].Value, memPaged[i].Value)
}
}
}
// TestAggregateIgnoresRawPointLimit verifies raw limits do not affect aggregation.
//
// TestAggregateIgnoresRawPointLimit 验证原始点分页参数不会影响聚合输入。
func TestAggregateIgnoresRawPointLimit(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "ig", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
var batch []Point
for i := 0; i < 6; i++ {
batch = append(batch, Point{MetricName: "ig", EntityID: "n1", Timestamp: base.Add(time.Duration(i) * time.Minute), Value: float64(i)})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
// The embedded Query.Limit must be ignored for aggregation: a single 1-hour
// bucket should aggregate ALL 6 points (count 6), not just 2.
res, err := s.Aggregate(ctx, AggregateQuery{
Query: Query{MetricName: "ig", EntityID: "n1", Start: base, End: base.Add(time.Hour), Limit: 2},
Aggregation: AggP95, // memory path, where the bug would have surfaced
Interval: time.Hour,
})
if err != nil {
t.Fatalf("aggregate: %v", err)
}
if len(res) != 1 || res[0].Count != 6 {
t.Fatalf("embedded Limit leaked into aggregation: %#v", res)
}
}
// TestJSONTagKeyWithSpecialChars verifies JSON tag keys with special characters.
//
// TestJSONTagKeyWithSpecialChars 验证包含特殊字符的标签键可正确查询。
func TestJSONTagKeyWithSpecialChars(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "tk", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
// Keys containing a dot, a hyphen, a space, and a quote.
tags := map[string]string{
"region.zone": "ap-1",
"a-b": "yes",
"with space": "ok",
}
if err := s.Write(ctx, Point{MetricName: "tk", EntityID: "n1", Timestamp: base, Value: 1, Tags: tags}); err != nil {
t.Fatalf("write: %v", err)
}
// A second point that should NOT match the filter below.
if err := s.Write(ctx, Point{MetricName: "tk", EntityID: "n1", Timestamp: base.Add(time.Minute), Value: 2, Tags: map[string]string{"region.zone": "eu-1"}}); err != nil {
t.Fatalf("write 2: %v", err)
}
for _, tc := range []struct {
key, val string
want int
}{
{"region.zone", "ap-1", 1}, // dotted key must match the flat key, not nested
{"a-b", "yes", 1}, // hyphen
{"with space", "ok", 1}, // space
{"region.zone", "eu-1", 1}, // the other point
} {
got, err := s.Query(ctx, Query{
MetricName: "tk", EntityID: "n1",
Start: base.Add(-time.Second), End: base.Add(time.Hour),
Tags: map[string]string{tc.key: tc.val},
})
if err != nil {
t.Fatalf("query %s=%s: %v", tc.key, tc.val, err)
}
if len(got) != tc.want {
t.Fatalf("tag %q=%q: expected %d points, got %d", tc.key, tc.val, tc.want, len(got))
}
}
}
+83
View File
@@ -0,0 +1,83 @@
package metric
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
)
// encodeMap encodes a string map as JSON.
//
// encodeMap 将字符串 map 编码为 JSON 字符串,nil 会按空对象处理。
func encodeMap(m map[string]string) (string, error) {
if m == nil {
m = map[string]string{}
}
b, err := json.Marshal(m)
if err != nil {
return "", err
}
return string(b), nil
}
// decodeMap decodes a scanned JSON value into a string map.
//
// decodeMap 将数据库扫描出的 JSON 值还原为字符串 map。
func decodeMap(v any) (map[string]string, error) {
switch x := v.(type) {
case nil:
return map[string]string{}, nil
case string:
return decodeMapString(x)
case []byte:
return decodeMapString(string(x))
default:
return nil, fmt.Errorf("unsupported json value type %T", v)
}
}
// decodeMapString decodes a JSON object string into a string map.
//
// decodeMapString 将 JSON 字符串解码为字符串 map。
func decodeMapString(s string) (map[string]string, error) {
if s == "" {
return map[string]string{}, nil
}
var out map[string]string
if err := json.Unmarshal([]byte(s), &out); err != nil {
return nil, err
}
if out == nil {
out = map[string]string{}
}
return out, nil
}
// canonicalTags returns a deterministic JSON encoding of a tag map. encodeMap
// already relies on encoding/json, which sorts map keys, so equal tag maps
// always produce identical bytes — the property tagsFingerprint depends on.
//
// canonicalTags 返回标签 map 的确定性 JSON 编码;相同标签集合总会得到
// 相同字节序列,tagsFingerprint 依赖这个性质。
func canonicalTags(m map[string]string) (string, error) {
return encodeMap(m)
}
// tagsFingerprint returns a stable hex fingerprint of a tag set, used as the
// rollups table's tags_hash key column. Equal tag maps (regardless of Go map
// iteration order) hash identically, so each distinct tag combination becomes
// its own rollup series; an empty/nil tag map hashes to the fingerprint of "{}".
//
// tagsFingerprint 为标签集合生成稳定的十六进制指纹,用作 rollups 表的
// tags_hash key 列。相同标签 map(无论 Go map 迭代顺序如何)都会得到相同 hash,
// 因此每种不同标签组合都会成为自己的 rollup 序列;空或 nil 标签 map 会得到 "{}"
// 的指纹。
func tagsFingerprint(m map[string]string) (hash string, canonical string, err error) {
canonical, err = canonicalTags(m)
if err != nil {
return "", "", err
}
sum := sha256.Sum256([]byte(canonical))
return hex.EncodeToString(sum[:]), canonical, nil
}
+182
View File
@@ -0,0 +1,182 @@
package metric
import (
"context"
"fmt"
)
// Migrate creates the tables and indexes used by the metric store.
//
// Migrate 创建 metric 包需要的表和索引。
func (s *Store) Migrate(ctx context.Context) error {
if err := s.ensureOpen(); err != nil {
return err
}
d := s.dialect
jsonType := d.jsonType()
pk := d.autoIncrementPrimaryKey()
blob := d.blobType()
statements := []string{
fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (
name VARCHAR(191) PRIMARY KEY,
type VARCHAR(32) NOT NULL,
unit VARCHAR(64) NOT NULL DEFAULT '',
description TEXT NOT NULL DEFAULT '',
retention_days INTEGER NOT NULL DEFAULT 0,
metadata %s NOT NULL,
created_at BIGINT NOT NULL,
updated_at BIGINT NOT NULL
)`, s.tables.definitions, jsonType),
fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (
id %s,
metric_name VARCHAR(191) NOT NULL,
entity_id VARCHAR(191) NOT NULL,
tags_hash VARCHAR(64) NOT NULL,
ts_nano BIGINT NOT NULL,
value DOUBLE PRECISION NOT NULL,
tags %s NOT NULL,
labels %s NOT NULL,
created_at BIGINT NOT NULL,
UNIQUE(metric_name, entity_id, tags_hash, ts_nano)
)`, s.tables.points, pk, jsonType, jsonType),
fmt.Sprintf(`CREATE INDEX IF NOT EXISTS %s_points_metric_entity_time_idx ON %s (metric_name, entity_id, ts_nano)`, s.cfg.TablePrefix, s.tables.points),
fmt.Sprintf(`CREATE INDEX IF NOT EXISTS %s_points_metric_time_idx ON %s (metric_name, ts_nano)`, s.cfg.TablePrefix, s.tables.points),
fmt.Sprintf(`CREATE INDEX IF NOT EXISTS %s_points_entity_time_idx ON %s (entity_id, ts_nano)`, s.cfg.TablePrefix, s.tables.points),
fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (
id %s,
metric_name VARCHAR(191) NOT NULL,
entity_id VARCHAR(191) NOT NULL,
tags_hash VARCHAR(64) NOT NULL,
tags %s NOT NULL,
resolution_nano BIGINT NOT NULL,
bucket_nano BIGINT NOT NULL,
count BIGINT NOT NULL,
sum DOUBLE PRECISION NOT NULL,
sum_sq DOUBLE PRECISION NOT NULL,
min_val DOUBLE PRECISION NOT NULL,
max_val DOUBLE PRECISION NOT NULL,
first_val DOUBLE PRECISION NOT NULL,
first_ts BIGINT NOT NULL,
last_val DOUBLE PRECISION NOT NULL,
last_ts BIGINT NOT NULL,
digest %s,
created_at BIGINT NOT NULL,
UNIQUE(metric_name, entity_id, tags_hash, resolution_nano, bucket_nano)
)`, s.tables.rollups, pk, jsonType, blob),
fmt.Sprintf(`CREATE INDEX IF NOT EXISTS %s_rollups_lookup_idx ON %s (metric_name, entity_id, tags_hash, resolution_nano, bucket_nano)`, s.cfg.TablePrefix, s.tables.rollups),
fmt.Sprintf(`CREATE INDEX IF NOT EXISTS %s_rollups_res_time_idx ON %s (metric_name, resolution_nano, bucket_nano)`, s.cfg.TablePrefix, s.tables.rollups),
}
if s.cfg.Driver == DriverMySQL {
statements = []string{
fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (
name VARCHAR(191) PRIMARY KEY,
type VARCHAR(32) NOT NULL,
unit VARCHAR(64) NOT NULL DEFAULT '',
description TEXT NOT NULL,
retention_days INT NOT NULL DEFAULT 0,
metadata %s NOT NULL,
created_at BIGINT NOT NULL,
updated_at BIGINT NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci`, s.tables.definitions, jsonType),
fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (
id %s,
metric_name VARCHAR(191) NOT NULL,
entity_id VARCHAR(191) NOT NULL,
tags_hash VARCHAR(64) NOT NULL,
ts_nano BIGINT NOT NULL,
value DOUBLE NOT NULL,
tags %s NOT NULL,
labels %s NOT NULL,
created_at BIGINT NOT NULL,
UNIQUE KEY uq_metric_entity_time (metric_name, entity_id, tags_hash, ts_nano),
INDEX idx_metric_entity_time (metric_name, entity_id, ts_nano),
INDEX idx_metric_time (metric_name, ts_nano),
INDEX idx_entity_time (entity_id, ts_nano)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci`, s.tables.points, pk, jsonType, jsonType),
fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (
id %s,
metric_name VARCHAR(191) NOT NULL,
entity_id VARCHAR(191) NOT NULL,
tags_hash VARCHAR(64) NOT NULL,
tags %s NOT NULL,
resolution_nano BIGINT NOT NULL,
bucket_nano BIGINT NOT NULL,
count BIGINT NOT NULL,
sum DOUBLE NOT NULL,
sum_sq DOUBLE NOT NULL,
min_val DOUBLE NOT NULL,
max_val DOUBLE NOT NULL,
first_val DOUBLE NOT NULL,
first_ts BIGINT NOT NULL,
last_val DOUBLE NOT NULL,
last_ts BIGINT NOT NULL,
digest %s,
created_at BIGINT NOT NULL,
UNIQUE KEY uq_rollup (metric_name, entity_id, tags_hash, resolution_nano, bucket_nano),
INDEX idx_rollup_res_time (metric_name, resolution_nano, bucket_nano)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci`, s.tables.rollups, pk, jsonType, blob),
}
}
if s.cfg.Driver == DriverPostgreSQL {
statements = []string{
fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (
name VARCHAR(191) PRIMARY KEY,
type VARCHAR(32) NOT NULL,
unit VARCHAR(64) NOT NULL DEFAULT '',
description TEXT NOT NULL DEFAULT '',
retention_days INTEGER NOT NULL DEFAULT 0,
metadata %s NOT NULL,
created_at BIGINT NOT NULL,
updated_at BIGINT NOT NULL
)`, s.tables.definitions, jsonType),
fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (
id %s,
metric_name VARCHAR(191) NOT NULL,
entity_id VARCHAR(191) NOT NULL,
tags_hash VARCHAR(64) NOT NULL,
ts_nano BIGINT NOT NULL,
value DOUBLE PRECISION NOT NULL,
tags %s NOT NULL,
labels %s NOT NULL,
created_at BIGINT NOT NULL,
UNIQUE(metric_name, entity_id, tags_hash, ts_nano)
)`, s.tables.points, pk, jsonType, jsonType),
fmt.Sprintf(`CREATE INDEX IF NOT EXISTS %s_points_metric_entity_time_idx ON %s (metric_name, entity_id, ts_nano)`, s.cfg.TablePrefix, s.tables.points),
fmt.Sprintf(`CREATE INDEX IF NOT EXISTS %s_points_metric_time_idx ON %s (metric_name, ts_nano)`, s.cfg.TablePrefix, s.tables.points),
fmt.Sprintf(`CREATE INDEX IF NOT EXISTS %s_points_entity_time_idx ON %s (entity_id, ts_nano)`, s.cfg.TablePrefix, s.tables.points),
fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (
id %s,
metric_name VARCHAR(191) NOT NULL,
entity_id VARCHAR(191) NOT NULL,
tags_hash VARCHAR(64) NOT NULL,
tags %s NOT NULL,
resolution_nano BIGINT NOT NULL,
bucket_nano BIGINT NOT NULL,
count BIGINT NOT NULL,
sum DOUBLE PRECISION NOT NULL,
sum_sq DOUBLE PRECISION NOT NULL,
min_val DOUBLE PRECISION NOT NULL,
max_val DOUBLE PRECISION NOT NULL,
first_val DOUBLE PRECISION NOT NULL,
first_ts BIGINT NOT NULL,
last_val DOUBLE PRECISION NOT NULL,
last_ts BIGINT NOT NULL,
digest %s,
created_at BIGINT NOT NULL,
UNIQUE(metric_name, entity_id, tags_hash, resolution_nano, bucket_nano)
)`, s.tables.rollups, pk, jsonType, blob),
fmt.Sprintf(`CREATE INDEX IF NOT EXISTS %s_rollups_lookup_idx ON %s (metric_name, entity_id, tags_hash, resolution_nano, bucket_nano)`, s.cfg.TablePrefix, s.tables.rollups),
fmt.Sprintf(`CREATE INDEX IF NOT EXISTS %s_rollups_res_time_idx ON %s (metric_name, resolution_nano, bucket_nano)`, s.cfg.TablePrefix, s.tables.rollups),
}
}
for _, stmt := range statements {
if _, err := s.db.ExecContext(ctx, stmt); err != nil {
return err
}
}
return nil
}
+76
View File
@@ -0,0 +1,76 @@
package metric
import (
"strconv"
"strings"
)
// Pxx builds the Aggregation for an arbitrary percentile. The argument is a
// percentage in (0,100): Pxx(99.9) -> "p99.9", Pxx(50) -> "p50". The fixed
// AggP50/AggP95/AggP99 constants are just the common cases of this same string
// form, so they keep working unchanged.
//
// This is what turns the package from "p50/p95/p99 only" into "any percentile":
// callers can ask for p75, p90, p99.99, etc., and every path (in-memory,
// SQL pushdown, and rollup-via-t-digest) understands it.
//
// Pxx 根据任意百分位构造 Aggregation。参数是 (0,100) 内的百分比:
// Pxx(99.9) -> "p99.9",Pxx(50) -> "p50"。固定的 AggP50/AggP95/AggP99
// 常量只是同一字符串形式的常用情况,因此会保持原有行为。
//
// 这让 package 从“只支持 p50/p95/p99”变成“支持任意百分位”:调用方可以请求
// p75、p90、p99.99 等,并且每条路径(内存、SQL 下推、基于 t-digest 的 rollup)
// 都能理解它。
func Pxx(p float64) Aggregation {
// Trim trailing zeros so Pxx(95) == AggP95 ("p95"), not "p95.000000".
s := strconv.FormatFloat(p, 'f', -1, 64)
return Aggregation("p" + s)
}
// parsePercentile reports whether agg names a percentile and, if so, returns
// the corresponding fraction in [0,1]. "p99.9" -> 0.999. Out-of-range
// percentages (<=0 or >=100) are rejected so validation can reject them.
//
// parsePercentile 判断 agg 是否命名了百分位;如果是,则返回对应的 [0,1] 小数。
// 例如 "p99.9" -> 0.999。越界百分比(<=0 或 >=100)会被拒绝,以便校验逻辑
// 能拒绝它们。
func parsePercentile(agg Aggregation) (float64, bool) {
s := string(agg)
if len(s) < 2 || (s[0] != 'p' && s[0] != 'P') {
return 0, false
}
pct, err := strconv.ParseFloat(s[1:], 64)
if err != nil {
return 0, false
}
if pct <= 0 || pct >= 100 {
return 0, false
}
return pct / 100, true
}
// isPercentile reports whether agg is any percentile aggregation.
//
// isPercentile 判断聚合类型是否为任意百分位聚合。
func isPercentile(agg Aggregation) bool {
_, ok := parsePercentile(agg)
return ok
}
// percentileFractionString renders the fraction for SQL percentile_cont, e.g.
// "p99.9" -> "0.999". Trailing zeros are trimmed for stable SQL text.
//
// percentileFractionString 把百分位聚合转换为 SQL percentile_cont 需要的
// 小数字符串,并去掉尾随零以保持 SQL 文本稳定。
func percentileFractionString(agg Aggregation) (string, bool) {
f, ok := parsePercentile(agg)
if !ok {
return "", false
}
s := strconv.FormatFloat(f, 'f', -1, 64)
if !strings.Contains(s, ".") {
// f is in (0,1) so this should not happen, but guard anyway.
s = strconv.FormatFloat(f, 'f', 1, 64)
}
return s, true
}
+184
View File
@@ -0,0 +1,184 @@
package metric
import (
"context"
"fmt"
"os"
"strings"
"testing"
"time"
)
// TestPostgreSQLIntegration runs the PostgreSQL integration test when configured.
//
// TestPostgreSQLIntegration 在配置 DSN 后运行 PostgreSQL 集成测试。
func TestPostgreSQLIntegration(t *testing.T) {
dsn := os.Getenv("METRIC_POSTGRES_DSN")
if dsn == "" {
t.Skip("METRIC_POSTGRES_DSN is not set")
}
runSQLIntegration(t, "postgres", PostgreSQL(dsn), true)
}
// TestMySQLIntegration runs the MySQL integration test when configured.
//
// TestMySQLIntegration 在配置 DSN 后运行 MySQL 集成测试。
func TestMySQLIntegration(t *testing.T) {
dsn := os.Getenv("METRIC_MYSQL_DSN")
if dsn == "" {
t.Skip("METRIC_MYSQL_DSN is not set")
}
runSQLIntegration(t, "mysql", MySQL(dsn), false)
}
// runSQLIntegration exercises the SQL store against an external database.
//
// runSQLIntegration 在外部数据库上执行通用 SQL 集成测试流程。
func runSQLIntegration(t *testing.T, name string, cfg Config, expectSQLPercentile bool) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
prefix := fmt.Sprintf("it_%d_", time.Now().UnixNano())
cfg.TablePrefix = prefix
store, err := Open(ctx, cfg)
if err != nil {
t.Fatalf("open %s store: %v", name, err)
}
defer store.Close()
defer dropIntegrationTables(t, store, prefix)
if err := store.CreateMetric(ctx, Definition{
Name: "http.latency",
Type: TypeGauge,
Unit: "ms",
Description: "HTTP latency",
Metadata: map[string]string{"source": "integration"},
}); err != nil {
t.Fatalf("create metric: %v", err)
}
if err := store.CreateMetric(ctx, Definition{Name: "http.latency", Type: TypeGauge}); err == nil {
t.Fatalf("duplicate CreateMetric should fail")
}
if err := store.UpsertMetric(ctx, Definition{Name: "http.latency", Type: TypeGauge, Unit: "milliseconds"}); err != nil {
t.Fatalf("upsert metric: %v", err)
}
def, err := store.GetMetric(ctx, "http.latency")
if err != nil {
t.Fatalf("get metric: %v", err)
}
if def.Unit != "milliseconds" {
t.Fatalf("upsert did not update unit: %#v", def)
}
base := time.Date(2026, 6, 20, 1, 0, 0, 0, time.UTC)
points := []Point{
{MetricName: "http.latency", EntityID: "api-1", Timestamp: base, Value: 10, Tags: map[string]string{"region.zone": "ap-1", "route-name": "/v1/nodes"}},
{MetricName: "http.latency", EntityID: "api-1", Timestamp: base.Add(time.Minute), Value: 20, Tags: map[string]string{"region.zone": "ap-1", "route-name": "/v1/nodes"}},
{MetricName: "http.latency", EntityID: "api-1", Timestamp: base.Add(2 * time.Minute), Value: 30, Tags: map[string]string{"region.zone": "ap-1", "route-name": "/v1/nodes"}},
{MetricName: "http.latency", EntityID: "api-1", Timestamp: base.Add(3 * time.Minute), Value: 40, Tags: map[string]string{"region.zone": "eu-1", "route-name": "/v1/nodes"}},
}
if err := store.WriteBatch(ctx, points); err != nil {
t.Fatalf("write batch: %v", err)
}
got, err := store.Query(ctx, Query{
MetricName: "http.latency",
EntityID: "api-1",
Start: base.Add(-time.Second),
End: base.Add(5 * time.Minute),
Tags: map[string]string{"region.zone": "ap-1", "route-name": "/v1/nodes"},
})
if err != nil {
t.Fatalf("query with json tags: %v", err)
}
if len(got) != 3 {
t.Fatalf("expected 3 ap-1 points, got %d: %#v", len(got), got)
}
stats, err := store.Stats(ctx, Query{
MetricName: "http.latency",
EntityID: "api-1",
Start: base,
End: base.Add(5 * time.Minute),
})
if err != nil {
t.Fatalf("stats: %v", err)
}
if stats.Count != 4 || stats.Avg != 25 {
t.Fatalf("unexpected stats: %#v", stats)
}
avgBuckets, err := store.Aggregate(ctx, AggregateQuery{
Query: Query{
MetricName: "http.latency",
EntityID: "api-1",
Start: base,
End: base.Add(5 * time.Minute),
},
Aggregation: AggAvg,
Interval: 2 * time.Minute,
BucketLimit: 1,
BucketOffset: 1,
})
if err != nil {
t.Fatalf("avg aggregate: %v", err)
}
if len(avgBuckets) != 1 || avgBuckets[0].Value != 35 || avgBuckets[0].Count != 2 {
t.Fatalf("unexpected bucket-paged avg aggregate: %#v", avgBuckets)
}
p95Buckets, err := store.Aggregate(ctx, AggregateQuery{
Query: Query{
MetricName: "http.latency",
EntityID: "api-1",
Start: base,
End: base.Add(5 * time.Minute),
},
Aggregation: AggP95,
Interval: 10 * time.Minute,
})
if err != nil {
t.Fatalf("p95 aggregate: %v", err)
}
if len(p95Buckets) != 1 || p95Buckets[0].Count != 4 || p95Buckets[0].Value <= 38 {
t.Fatalf("unexpected p95 aggregate: %#v", p95Buckets)
}
if _, ok := sqlAggValueExpr(cfg.Driver, AggP95); ok != expectSQLPercentile {
t.Fatalf("%s p95 pushdown expectation mismatch: got %v want %v", name, ok, expectSQLPercentile)
}
latest, err := store.Latest(ctx, "http.latency", "api-1", 1)
if err != nil {
t.Fatalf("latest: %v", err)
}
if len(latest) != 1 || latest[0].Value != 40 {
t.Fatalf("unexpected latest point: %#v", latest)
}
deleted, err := store.DeleteBefore(ctx, "http.latency", base.Add(2*time.Minute))
if err != nil {
t.Fatalf("delete before: %v", err)
}
if deleted != 2 {
t.Fatalf("expected 2 deleted points, got %d", deleted)
}
}
// dropIntegrationTables drops integration-test tables.
//
// dropIntegrationTables 删除集成测试创建的表。
func dropIntegrationTables(t *testing.T, store *Store, prefix string) {
t.Helper()
if strings.TrimSpace(prefix) == "" {
t.Fatal("refusing to drop tables with empty prefix")
}
for _, name := range []string{prefix + "rollups", prefix + "points", prefix + "definitions"} {
if _, err := store.db.Exec("DROP TABLE IF EXISTS " + name); err != nil {
t.Fatalf("drop integration table %s: %v", name, err)
}
}
}
+320
View File
@@ -0,0 +1,320 @@
package metric
import (
"fmt"
"math"
"time"
)
// RollupTier describes one downsampled resolution: raw points (or the next
// finer tier) are aggregated into buckets Interval wide, and those buckets are
// kept for Retention. A policy lists tiers from finest to coarsest, e.g.
//
// 1m kept 7d -> 5m kept 30d -> 1h kept 1y
//
// As data ages it falls off the finer tiers and survives only in coarser ones,
// so storage shrinks with age — the "data gets sparser as it gets older"
// behavior of a downsampling TSDB.
//
// RollupTier 描述一个降采样分辨率:原始点(或下一层更细的层级)会被聚合进
// Interval 宽的桶,而这些桶会保留 Retention。策略会按从细到粗排列层级,例如:
//
// 1m 保留 7d -> 5m 保留 30d -> 1h 保留 1y
//
// 随着数据变旧,它会从更细层级中过期,只在更粗层级中保留下来,因此存储量会随
// 数据年龄下降,也就是降采样 TSDB 中“数据越旧越稀疏”的行为。
type RollupTier struct {
// Interval is the bucket width for this rollup tier.
//
// Interval 是该 rollup 层的桶宽。
Interval time.Duration `json:"interval"`
// Retention is how long buckets in this tier are kept.
//
// Retention 是该层桶的保留时间。
Retention time.Duration `json:"retention"`
}
// RollupPolicy is the full retention ladder for a store: how long raw points
// live, then a chain of progressively coarser tiers. Compact materializes the
// tiers and enforces every retention window.
//
// RollupPolicy 描述完整保留阶梯:原始点保留多久,以及后续逐级变粗的
// rollup 层;Compact 会物化这些层并执行保留窗口。
type RollupPolicy struct {
// RawRetention is how long raw points are kept before Compact deletes them
// (after they have been rolled into the finest tier). Zero means "never
// delete raw" — rollups are still built, but raw is retained.
//
// RawRetention 是 Compact 删除原始点之前保留它们的时间(在它们已进入最细
// rollup 层之后)。零值表示“永不删除原始点”;rollup 仍会构建,但原始点保留。
RawRetention time.Duration `json:"raw_retention"`
// Tiers are ordered finest-first. Each Interval must be a positive integer
// multiple of the previous tier's Interval (so a coarse bucket is composed
// of whole finer buckets), and each Retention must be >= the previous
// tier's Retention (coarse data outlives fine data).
//
// Tiers 按从细到粗排序。每个 Interval 必须是前一层 Interval 的正整数倍
// (这样粗桶由完整细桶组成),每个 Retention 必须 >= 前一层 Retention
// (粗数据比细数据活得更久)。
Tiers []RollupTier `json:"tiers"`
// Compression tunes the per-bucket t-digest (size vs. percentile accuracy).
// <=1 uses the default (100).
//
// Compression 调整每个桶的 t-digest(大小与百分位精度之间的取舍)。
// <=1 时使用默认值(100)。
Compression float64 `json:"compression"`
}
// Enabled reports whether the policy actually defines any rollup tiers.
//
// Enabled 表示策略是否实际定义了 rollup 层级。
func (p RollupPolicy) Enabled() bool { return len(p.Tiers) > 0 }
// compression returns the effective t-digest compression for a rollup policy.
//
// compression 返回 rollup 策略实际使用的 t-digest 压缩参数。
func (p RollupPolicy) compression() float64 {
if p.Compression <= 1 {
return defaultTDigestCompression
}
return p.Compression
}
func (p RollupPolicy) rawCutoff(now time.Time) time.Time {
if p.RawRetention <= 0 || len(p.Tiers) == 0 {
return time.Time{}
}
cutoff := now.UTC().Add(-p.RawRetention)
interval := p.Tiers[0].Interval.Nanoseconds()
return time.Unix(0, floorDivNano(cutoff.UnixNano(), interval)).UTC()
}
// Validate enforces the structural rules that make cascading composition and
// retention well-defined.
//
// Validate 检查策略结构是否满足级联合成和保留语义所需的约束。
func (p RollupPolicy) Validate() error {
if len(p.Tiers) == 0 {
return nil // a store with no tiers simply does no rollup work
}
if p.RawRetention < 0 {
return fmt.Errorf("%w: raw retention cannot be negative", ErrInvalidArgument)
}
var prev RollupTier
for i, tr := range p.Tiers {
if tr.Interval <= 0 {
return fmt.Errorf("%w: tier %d interval must be positive", ErrInvalidArgument, i)
}
if tr.Retention <= 0 {
return fmt.Errorf("%w: tier %d retention must be positive", ErrInvalidArgument, i)
}
if i == 0 {
// The finest tier rolls up raw points. Require raw retention (when
// set) to cover at least two of its buckets so the in-progress and
// most-recent sealed bucket still have their raw backing when Compact
// runs and deletes old raw.
if p.RawRetention > 0 && p.RawRetention < 2*tr.Interval {
return fmt.Errorf("%w: raw retention must be >= 2x the finest tier interval", ErrInvalidArgument)
}
} else {
if tr.Interval <= prev.Interval {
return fmt.Errorf("%w: tier %d interval must be larger than tier %d", ErrInvalidArgument, i, i-1)
}
if tr.Interval%prev.Interval != 0 {
return fmt.Errorf("%w: tier %d interval must be a multiple of tier %d interval", ErrInvalidArgument, i, i-1)
}
if tr.Retention < prev.Retention {
return fmt.Errorf("%w: tier %d retention must be >= tier %d retention", ErrInvalidArgument, i, i-1)
}
}
prev = tr
}
return nil
}
// rollupBucket is the in-memory accumulator for one (metric, entity,
// resolution, bucket) cell. It carries exactly the summaries that can be
// re-aggregated losslessly when composing coarser tiers — count/sum/sumSq for
// avg & population stddev, min/max, first/last by timestamp — plus a t-digest
// so arbitrary percentiles survive downsampling with bounded error.
//
// rollupBucket 是单个(指标、实体、分辨率、桶)单元的内存累加器。它只携带在
// 合成更粗层级时可无损重新聚合的摘要:用于平均值和总体标准差的 count/sum/sumSq,
// min/max,按时间记录的 first/last,外加一个 t-digest,让任意百分位能以
// 有界误差在降采样后保留下来。
type rollupBucket struct {
// count is the total number of raw points represented.
//
// count 是该桶代表的原始点总数。
count int64
// sum is the sum of represented values.
//
// sum 是该桶代表值的总和。
sum float64
// sumSq is the sum of squared values for population stddev.
//
// sumSq 是用于总体标准差的平方和。
sumSq float64
// min is the minimum represented value.
//
// min 是该桶代表的最小值。
min float64
// max is the maximum represented value.
//
// max 是该桶代表的最大值。
max float64
// firstVal is the value with the earliest timestamp.
//
// firstVal 是时间戳最早的值。
firstVal float64
// firstTS is the earliest timestamp in nanoseconds.
//
// firstTS 是最早时间戳的纳秒值。
firstTS int64
// lastVal is the value with the latest timestamp.
//
// lastVal 是时间戳最晚的值。
lastVal float64
// lastTS is the latest timestamp in nanoseconds.
//
// lastTS 是最晚时间戳的纳秒值。
lastTS int64
// digest estimates percentiles for represented values.
//
// digest 用于估算该桶代表值的百分位。
digest *TDigest
// tagsHash carries the stable tag-set fingerprint for this rollup cell.
//
// tagsHash 携带该 rollup 单元的稳定标签集合指纹。
tagsHash string
// tagsJSON carries the canonical tag map written back to the tags column.
//
// tagsJSON 携带写回 tags 列的规范标签 map。
tagsJSON string
}
// newRollupBucket creates an empty rollup accumulator.
//
// newRollupBucket 创建一个空的 rollup 累加器。
func newRollupBucket(compression float64) *rollupBucket {
return &rollupBucket{
min: 0,
max: 0,
firstTS: 0,
lastTS: 0,
digest: NewTDigest(compression),
}
}
// addPoint folds a raw observation into the bucket.
//
// addPoint 将一个原始观测值合入当前桶。
func (b *rollupBucket) addPoint(value float64, tsNano int64) {
if b.count == 0 {
b.min, b.max = value, value
b.firstVal, b.firstTS = value, tsNano
b.lastVal, b.lastTS = value, tsNano
} else {
if value < b.min {
b.min = value
}
if value > b.max {
b.max = value
}
if tsNano < b.firstTS {
b.firstVal, b.firstTS = value, tsNano
}
if tsNano > b.lastTS {
b.lastVal, b.lastTS = value, tsNano
}
}
b.count++
b.sum += value
b.sumSq += value * value
b.digest.Add(value, 1)
}
// mergeStored folds a finer rollup row (already-summarized) into this coarser
// bucket. This is the cascade step: tier i+1 buckets are built by merging the
// tier i rows they span.
//
// mergeStored 将一个已汇总的细层 rollup 行合入当前更粗桶。这是级联步骤:
// tier i+1 的桶通过合并其覆盖的 tier i 行构建。
func (b *rollupBucket) mergeStored(o *rollupBucket) {
if o.count == 0 {
return
}
if b.count == 0 {
b.min, b.max = o.min, o.max
b.firstVal, b.firstTS = o.firstVal, o.firstTS
b.lastVal, b.lastTS = o.lastVal, o.lastTS
} else {
if o.min < b.min {
b.min = o.min
}
if o.max > b.max {
b.max = o.max
}
if o.firstTS < b.firstTS {
b.firstVal, b.firstTS = o.firstVal, o.firstTS
}
if o.lastTS > b.lastTS {
b.lastVal, b.lastTS = o.lastVal, o.lastTS
}
}
b.count += o.count
b.sum += o.sum
b.sumSq += o.sumSq
if b.digest == nil {
b.digest = NewTDigest(defaultTDigestCompression)
}
b.digest.Merge(o.digest)
}
// value computes the requested aggregation from the bucket summaries. ok=false
// means the aggregation is not derivable from a rollup (only AggRate, which
// needs the ordered raw series).
//
// value 从桶摘要中计算请求的聚合值;ok=false 表示该聚合无法由 rollup 推导
// (目前主要是需要有序原始序列的 AggRate)。
func (b *rollupBucket) value(agg Aggregation) (float64, bool) {
switch agg {
case AggAvg:
if b.count == 0 {
return 0, true
}
return b.sum / float64(b.count), true
case AggMin:
return b.min, true
case AggMax:
return b.max, true
case AggSum:
return b.sum, true
case AggCount:
return float64(b.count), true
case AggFirst:
return b.firstVal, true
case AggLast:
return b.lastVal, true
case AggStdDev:
if b.count == 0 {
return 0, true
}
mean := b.sum / float64(b.count)
variance := b.sumSq/float64(b.count) - mean*mean
if variance < 0 {
variance = 0 // floating-point guard
}
return math.Sqrt(variance), true
case AggRate:
return 0, false // rate needs the ordered raw series; not in a rollup
default:
if frac, ok := parsePercentile(agg); ok {
if b.digest == nil || b.digest.Count() == 0 {
return 0, true
}
return b.digest.Quantile(frac), true
}
return 0, false
}
}
+135
View File
@@ -0,0 +1,135 @@
package metric
import (
"context"
"math"
"testing"
"time"
)
// TestSeriesHybridKeepsUncompactedRawNearCutoff verifies the hybrid read does
// not lose raw points that fall between the (finest-aligned) raw cutoff and the
// next bucket boundary, including a point written AFTER the last compaction.
//
// Reproduction: now=12:00:30, RawRetention=30m so the cutoff is 11:30:30, which
// the policy floors to 11:30:00 (finest = 1m). A point at 11:30:15 written
// before compaction is kept (>= aligned cutoff) and NOT rolled up; a point at
// 11:30:45 written after compaction is also raw. Both belong to the 11:00 1h
// output bucket. The old code aligned the split UP to 11:31:00 and read raw from
// there, dropping both boundary points (count=1, the stale rollup half only).
// Serving raw from the aligned cutoff fixes it (count=2).
//
// TestSeriesHybridKeepsUncompactedRawNearCutoff 验证混合读取不会丢失落在
// (最细对齐后的)原始 cutoff 与下一个桶边界之间的 raw 点,包括上次 compact 之后
// 才写入的点。
func TestSeriesHybridKeepsUncompactedRawNearCutoff(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: 30 * time.Minute,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "nearcut", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
now := time.Date(2026, 6, 18, 12, 0, 30, 0, time.UTC)
// 11:30:15 is >= the finest-aligned cutoff (11:30:00) -> survives compaction
// as raw and is not rolled up.
before := time.Date(2026, 6, 18, 11, 30, 15, 0, time.UTC)
if err := s.Write(ctx, Point{MetricName: "nearcut", EntityID: "n1", Timestamp: before, Value: 10}); err != nil {
t.Fatalf("write before: %v", err)
}
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact: %v", err)
}
// Written after compaction: only ever lives in raw.
after := time.Date(2026, 6, 18, 11, 30, 45, 0, time.UTC)
if err := s.Write(ctx, Point{MetricName: "nearcut", EntityID: "n1", Timestamp: after, Value: 100}); err != nil {
t.Fatalf("write after: %v", err)
}
got, err := s.Series(ctx, AggregateQuery{
Query: Query{MetricName: "nearcut", EntityID: "n1", Start: time.Date(2026, 6, 18, 11, 0, 0, 0, time.UTC), End: now},
Aggregation: AggAvg,
Interval: time.Hour,
}, now)
if err != nil {
t.Fatalf("series: %v", err)
}
bucket := alignTime(time.Date(2026, 6, 18, 11, 0, 0, 0, time.UTC), time.Hour)
var found bool
for _, p := range got {
if p.Bucket.Equal(bucket) {
found = true
if p.Count != 2 {
t.Fatalf("near-cutoff bucket count: got %d want 2 (%#v)", p.Count, p)
}
if math.Abs(p.Value-55) > 1e-9 {
t.Fatalf("near-cutoff bucket avg: got %v want 55 (%#v)", p.Value, p)
}
}
}
if !found {
t.Fatalf("expected 11:00 bucket with both raw points: %#v", got)
}
}
// TestSeriesHybridEndEqualsCutoff verifies a query whose End lands exactly on the
// raw cutoff still includes the raw point sitting at the cutoff, instead of
// returning empty.
//
// Reproduction: now=12:00:00, RawRetention=30m -> cutoff 11:30:00. A point at
// exactly 11:30:00 is kept by compaction (not < cutoff) and lives in raw. An old
// point at 11:00:10 is rolled up and its raw deleted. Querying [11:00, 11:30:00]
// (End == cutoff) must combine the rolled-up old point and the raw cutoff point.
// The old entry gate required End strictly AFTER the cutoff, so this fell through
// to the pure-rollup path and returned the cutoff point's bucket empty.
//
// TestSeriesHybridEndEqualsCutoff 验证 End 恰好等于原始 cutoff 的查询仍会包含
// 落在 cutoff 上的 raw 点,而不是返回空。
func TestSeriesHybridEndEqualsCutoff(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: 30 * time.Minute,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "endcut", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
now := time.Date(2026, 6, 18, 12, 0, 0, 0, time.UTC)
cutoff := time.Date(2026, 6, 18, 11, 30, 0, 0, time.UTC)
old := time.Date(2026, 6, 18, 11, 0, 10, 0, time.UTC) // rolled up, raw deleted
if err := s.Write(ctx, Point{MetricName: "endcut", EntityID: "n1", Timestamp: old, Value: 10}); err != nil {
t.Fatalf("write old: %v", err)
}
// Point exactly at the cutoff: kept as raw (not < cutoff) after compaction.
if err := s.Write(ctx, Point{MetricName: "endcut", EntityID: "n1", Timestamp: cutoff, Value: 100}); err != nil {
t.Fatalf("write cutoff: %v", err)
}
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact: %v", err)
}
got, err := s.Series(ctx, AggregateQuery{
Query: Query{MetricName: "endcut", EntityID: "n1", Start: time.Date(2026, 6, 18, 11, 0, 0, 0, time.UTC), End: cutoff},
Aggregation: AggSum,
Interval: time.Hour,
}, now)
if err != nil {
t.Fatalf("series: %v", err)
}
bucket := alignTime(time.Date(2026, 6, 18, 11, 0, 0, 0, time.UTC), time.Hour)
var found bool
for _, p := range got {
if p.Bucket.Equal(bucket) {
found = true
if p.Count != 2 || math.Abs(p.Value-110) > 1e-9 {
t.Fatalf("End==cutoff bucket: got count=%d value=%v want count=2 value=110", p.Count, p.Value)
}
}
}
if !found {
t.Fatalf("expected 11:00 bucket including the cutoff raw point: %#v", got)
}
}
+347
View File
@@ -0,0 +1,347 @@
package metric
import (
"context"
"math"
"testing"
"time"
)
// TestAggregateRollupExcludesPartialBucket verifies the rollup query no longer
// drags an out-of-window sample in through a partially-overlapping bucket.
//
// Two samples share one 1m rollup bucket [00:00:00, 00:01:00): 00:00:10=100 and
// 00:00:50=10. A query window of [00:00:40, 00:01:30] only overlaps part of that
// bucket. Because a rollup bucket is an indivisible summary, the old "any
// overlap" rule folded the whole bucket in and counted 00:00:10 (value 100),
// which lies OUTSIDE the window. Full containment excludes the straddling
// bucket entirely, so the out-of-window sample can no longer leak in.
//
// TestAggregateRollupExcludesPartialBucket 验证 rollup 查询不再通过部分重叠的桶
// 把窗口外的样本带进来。
func TestAggregateRollupExcludesPartialBucket(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}}}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "win", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
batch := []Point{
{MetricName: "win", EntityID: "n1", Timestamp: base.Add(10 * time.Second), Value: 100}, // out of window
{MetricName: "win", EntityID: "n1", Timestamp: base.Add(50 * time.Second), Value: 10}, // in window
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := s.Compact(ctx, base.Add(time.Hour)); err != nil {
t.Fatalf("compact: %v", err)
}
// Window [00:00:40, 00:01:30] partially overlaps the single 1m bucket.
res, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{MetricName: "win", EntityID: "n1", Start: base.Add(40 * time.Second), End: base.Add(90 * time.Second)},
Aggregation: AggSum,
Interval: time.Minute,
}, time.Minute)
if err != nil {
t.Fatalf("aggregate rollup: %v", err)
}
// The partially-overlapping bucket is excluded under full containment, so the
// out-of-window 00:00:10 sample (value 100) must not appear. Concretely: no
// returned bucket may carry the out-of-window value or a count of 2.
for _, p := range res {
if p.Count >= 2 || p.Value == 100 {
t.Fatalf("partial bucket leaked out-of-window sample: %#v", p)
}
}
}
// TestAggregateRollupEndBoundaryInclusive verifies the rollup query end boundary
// is inclusive (matching the raw query's ts <= end) and that a window aligned to
// resolution boundaries returns every whole bucket it covers.
//
// Buckets at 00:00 and 00:01 are both whole-contained in [00:00:00, 00:02:00]:
// the second bucket's window is [00:01:00, 00:02:00), whose last covered nano is
// < 00:02:00, so an inclusive end keeps it. The previous half-open (< end) rule
// could drop a bucket sitting exactly on the end edge.
//
// TestAggregateRollupEndBoundaryInclusive 验证 rollup 查询 end 边界为闭区间
// (与 raw 的 ts <= end 一致),且对齐到分辨率边界的窗口会返回它覆盖的每个完整桶。
func TestAggregateRollupEndBoundaryInclusive(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}}}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "edge", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
batch := []Point{
{MetricName: "edge", EntityID: "n1", Timestamp: base.Add(10 * time.Second), Value: 1}, // bucket 00:00
{MetricName: "edge", EntityID: "n1", Timestamp: base.Add(time.Minute + 10*time.Second), Value: 2}, // bucket 00:01
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := s.Compact(ctx, base.Add(time.Hour)); err != nil {
t.Fatalf("compact: %v", err)
}
res, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{MetricName: "edge", EntityID: "n1", Start: base, End: base.Add(2 * time.Minute)},
Aggregation: AggSum,
Interval: time.Minute,
}, time.Minute)
if err != nil {
t.Fatalf("aggregate rollup: %v", err)
}
if len(res) != 2 {
t.Fatalf("aligned window should return both whole buckets, got %d: %#v", len(res), res)
}
if res[0].Value != 1 || res[1].Value != 2 {
t.Fatalf("unexpected bucket values: %#v", res)
}
}
// TestSeriesHybridMergesStraddlingBucket verifies a query output bucket that
// straddles the raw-retention boundary aggregates BOTH its rollup half and its
// raw half, instead of letting raw replace the rollup half.
//
// With a 30m raw retention and now=12:00 the boundary is 11:30. A 1h output
// bucket [11:00, 12:00) therefore straddles it. An 11:05 sample (value 10, older
// than the boundary -> rolled up, raw deleted) and an 11:35 sample (value 100,
// within retention -> still raw) both belong to that 11:00 bucket. The correct
// answer folds both: count=2, avg=(10+100)/2=55. The old merge-by-bucket-time
// path kept only the raw half and reported count=1, avg=100.
//
// TestSeriesHybridMergesStraddlingBucket 验证跨越原始保留期边界的输出桶会同时
// 聚合 rollup 半边和 raw 半边,而不是让 raw 覆盖 rollup 半边。
func TestSeriesHybridMergesStraddlingBucket(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: 30 * time.Minute,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "straddle", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
now := time.Date(2026, 6, 18, 12, 0, 0, 0, time.UTC)
// 11:05 is older than the 11:30 cutoff (rolled up, raw deleted on compact);
// 11:35 is within retention (kept raw).
old := now.Add(-55 * time.Minute) // 11:05
recent := now.Add(-25 * time.Minute) // 11:35
batch := []Point{
{MetricName: "straddle", EntityID: "n1", Timestamp: old, Value: 10},
{MetricName: "straddle", EntityID: "n1", Timestamp: recent, Value: 100},
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact: %v", err)
}
// Confirm the old raw point was deleted and the recent one survives, so the
// two halves genuinely come from different sources.
oldRaw, err := s.Query(ctx, Query{MetricName: "straddle", EntityID: "n1", Start: old.Add(-time.Minute), End: old.Add(time.Minute)})
if err != nil {
t.Fatalf("query old raw: %v", err)
}
if len(oldRaw) != 0 {
t.Fatalf("expected old raw point deleted by retention, got %d", len(oldRaw))
}
got, err := s.Series(ctx, AggregateQuery{
Query: Query{MetricName: "straddle", EntityID: "n1", Start: now.Add(-time.Hour), End: now},
Aggregation: AggAvg,
Interval: time.Hour,
}, now)
if err != nil {
t.Fatalf("series: %v", err)
}
bucket := alignTime(now.Add(-time.Hour), time.Hour) // 11:00
var found bool
for _, p := range got {
if p.Bucket.Equal(bucket) {
found = true
if p.Count != 2 {
t.Fatalf("straddling bucket count: got %d want 2 (%#v)", p.Count, p)
}
if math.Abs(p.Value-55) > 1e-9 {
t.Fatalf("straddling bucket avg: got %v want 55 (%#v)", p.Value, p)
}
}
}
if !found {
t.Fatalf("expected a merged 11:00 bucket in result: %#v", got)
}
}
// TestSeriesHybridSumAcrossBoundary is the AggSum analogue: the straddling 1h
// bucket must sum both halves (10 + 100 = 110), proving the merge is at the
// summary level rather than a bucket-time replacement.
//
// TestSeriesHybridSumAcrossBoundary 是 AggSum 版本:跨边界的 1h 桶必须把两半求和
// (10 + 100 = 110),证明合并发生在摘要层面,而不是按桶时间替换。
func TestSeriesHybridSumAcrossBoundary(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: 30 * time.Minute,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "sumstraddle", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
now := time.Date(2026, 6, 18, 12, 0, 0, 0, time.UTC)
batch := []Point{
{MetricName: "sumstraddle", EntityID: "n1", Timestamp: now.Add(-55 * time.Minute), Value: 10},
{MetricName: "sumstraddle", EntityID: "n1", Timestamp: now.Add(-25 * time.Minute), Value: 100},
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact: %v", err)
}
got, err := s.Series(ctx, AggregateQuery{
Query: Query{MetricName: "sumstraddle", EntityID: "n1", Start: now.Add(-time.Hour), End: now},
Aggregation: AggSum,
Interval: time.Hour,
}, now)
if err != nil {
t.Fatalf("series: %v", err)
}
bucket := alignTime(now.Add(-time.Hour), time.Hour)
var found bool
for _, p := range got {
if p.Bucket.Equal(bucket) {
found = true
if p.Count != 2 || math.Abs(p.Value-110) > 1e-9 {
t.Fatalf("straddling sum bucket: got count=%d value=%v want count=2 value=110", p.Count, p.Value)
}
}
}
if !found {
t.Fatalf("expected merged sum bucket: %#v", got)
}
}
func TestSeriesHybridIncludesRawBetweenCutoffAndNextBucket(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: 30 * time.Minute,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "cutgap", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
now := time.Date(2026, 6, 18, 12, 0, 30, 0, time.UTC)
old := time.Date(2026, 6, 18, 11, 30, 10, 0, time.UTC)
if err := s.Write(ctx, Point{MetricName: "cutgap", EntityID: "n1", Timestamp: old, Value: 10}); err != nil {
t.Fatalf("write old: %v", err)
}
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact: %v", err)
}
recentUncompacted := time.Date(2026, 6, 18, 11, 30, 45, 0, time.UTC)
if err := s.Write(ctx, Point{MetricName: "cutgap", EntityID: "n1", Timestamp: recentUncompacted, Value: 100}); err != nil {
t.Fatalf("write recent: %v", err)
}
got, err := s.Series(ctx, AggregateQuery{
Query: Query{MetricName: "cutgap", EntityID: "n1", Start: time.Date(2026, 6, 18, 11, 0, 0, 0, time.UTC), End: now},
Aggregation: AggCount,
Interval: time.Hour,
}, now)
if err != nil {
t.Fatalf("series: %v", err)
}
if len(got) != 1 || got[0].Count != 2 {
t.Fatalf("series should include rollup plus raw at the aligned cutoff, got %#v", got)
}
}
func TestSeriesHybridEndAtCutoffIncludesRawBoundary(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: 30 * time.Minute,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "cutedge", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
now := time.Date(2026, 6, 18, 12, 0, 30, 0, time.UTC)
cutoff := policy.rawCutoff(now)
if err := s.Write(ctx, Point{MetricName: "cutedge", EntityID: "n1", Timestamp: cutoff, Value: 7}); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact: %v", err)
}
got, err := s.Series(ctx, AggregateQuery{
Query: Query{MetricName: "cutedge", EntityID: "n1", Start: cutoff.Add(-time.Hour), End: cutoff},
Aggregation: AggCount,
Interval: time.Hour,
}, now)
if err != nil {
t.Fatalf("series: %v", err)
}
var total int
for _, p := range got {
total += p.Count
}
if total != 1 {
t.Fatalf("series should include raw point exactly at cutoff/end, got %#v", got)
}
}
func TestCompactDoesNotDoubleCountRawBucketAfterCutoffAdvances(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: 30 * time.Minute,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "nodup", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
now := time.Date(2026, 6, 18, 12, 0, 30, 0, time.UTC)
if err := s.Write(ctx, Point{MetricName: "nodup", EntityID: "n1", Timestamp: time.Date(2026, 6, 18, 11, 30, 10, 0, time.UTC), Value: 10}); err != nil {
t.Fatalf("write old: %v", err)
}
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact 1: %v", err)
}
if err := s.Write(ctx, Point{MetricName: "nodup", EntityID: "n1", Timestamp: time.Date(2026, 6, 18, 11, 30, 45, 0, time.UTC), Value: 100}); err != nil {
t.Fatalf("write recent: %v", err)
}
now = now.Add(time.Minute)
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact 2: %v", err)
}
got, err := s.Series(ctx, AggregateQuery{
Query: Query{MetricName: "nodup", EntityID: "n1", Start: time.Date(2026, 6, 18, 11, 0, 0, 0, time.UTC), End: now},
Aggregation: AggCount,
Interval: time.Hour,
}, now)
if err != nil {
t.Fatalf("series: %v", err)
}
var total int
for _, p := range got {
total += p.Count
}
if total != 2 {
t.Fatalf("compaction should not double-count a bucket after cutoff advances, got %#v", got)
}
}
+503
View File
@@ -0,0 +1,503 @@
package metric
import (
"context"
"database/sql"
"fmt"
"sort"
"strings"
"time"
)
// AggregateRollup answers an AggregateQuery from a stored rollup tier instead of
// raw points. resolution names which tier to read (it must match a tier
// Interval that Compact has materialized). The query Interval must be a positive
// integer multiple of resolution, so each output bucket is composed of whole
// rollup buckets.
//
// query.Tags is honored: only rollup series whose stored tag set matches the
// filter are folded in, so a tag filter selects the same data it would over raw
// points (each distinct tag combination is its own rollup series). With no tag
// filter, all tag series within a bucket merge together, matching the raw path's
// cross-tag aggregation.
//
// Every aggregation works except AggRate, which needs the ordered raw series and
// is therefore raw-only. Percentiles (p50, p95, p99, and arbitrary pXX) are
// answered by merging the per-bucket t-digests, so they survive downsampling
// with bounded error.
//
// AggregateRollup 从已存储的 rollup 层回答 AggregateQuery,而不是读取原始点。
// resolution 指定要读取的层级(必须匹配 Compact 已物化的某个层级 Interval)。
// 查询的 Interval 必须是 resolution 的正整数倍,因此每个输出桶都由完整的
// rollup 桶组成。
//
// query.Tags 会被遵守:只有存储标签集合匹配过滤条件的 rollup 序列会被合入,
// 因此标签过滤会选中与原始点查询相同的数据(每个不同标签组合都是自己的
// rollup 序列)。没有标签过滤时,同一桶内的所有标签序列会合并在一起,
// 与原始路径的跨标签聚合一致。
//
// 除 AggRate 外,每种聚合都可用;AggRate 需要有序原始序列,因此只能基于原始点。
// 百分位(p50、p95、p99 和任意 pXX)通过合并每桶 t-digest 回答,因此能在
// 降采样后以有界误差保留下来。
func (s *Store) AggregateRollup(ctx context.Context, query AggregateQuery, resolution time.Duration) ([]AggregatePoint, error) {
if err := s.ensureOpen(); err != nil {
return nil, err
}
if err := query.Validate(); err != nil {
return nil, err
}
if resolution <= 0 {
return nil, fmt.Errorf("%w: rollup resolution must be positive", ErrInvalidArgument)
}
if query.Interval < resolution || query.Interval%resolution != 0 {
return nil, fmt.Errorf("%w: query interval must be a positive multiple of the rollup resolution", ErrInvalidArgument)
}
if query.Aggregation == AggRate {
return nil, fmt.Errorf("%w: rate is not derivable from rollups (raw only)", ErrInvalidArgument)
}
q := query.Query.normalized()
comp := s.cfg.RollupPolicy.compression()
// Read rollup buckets at this resolution that are FULLY contained in the
// inclusive window [Start, End] (entity and tag filters pushed into SQL),
// then fold them into query.Interval-wide output buckets. Full containment
// (rather than mere overlap) is what keeps a partially-overlapping bucket's
// out-of-window samples from leaking into the result: a rollup bucket is an
// indivisible summary, so a bucket straddling a window edge cannot be
// trimmed to the window and is therefore excluded. Callers that need every
// sample in a sub-bucket window must align the window to resolution
// boundaries (or query raw points).
rows, err := s.scanRollupRowsContained(ctx, q.MetricName, q.EntityID, q.Tags, resolution, q.Start, q.End)
if err != nil {
return nil, err
}
groups := foldRollupRows(nil, rows, query.Interval, comp)
out, err := rollupGroupsToPoints(groups, query)
if err != nil {
return nil, err
}
return pageBuckets(out, query.BucketLimit, query.BucketOffset), nil
}
// rollupGroupsToPoints turns the merged output buckets into ordered
// AggregatePoints, computing the requested aggregation from each bucket's
// summaries/digest. FillEmpty emits zero-count buckets for gaps, mirroring the
// raw AggregatePoints behavior.
//
// rollupGroupsToPoints 将合并后的输出桶转换为有序 AggregatePoint,并根据
// 每个桶的摘要或 digest 计算请求的聚合。FillEmpty 会为空洞输出零计数桶,
// 与原始 AggregatePoints 行为一致。
func rollupGroupsToPoints(groups map[int64]*rollupBucket, query AggregateQuery) ([]AggregatePoint, error) {
var bucketStarts []int64
if query.FillEmpty {
for t := alignTime(query.Start, query.Interval); !t.After(query.End); t = t.Add(query.Interval) {
bucketStarts = append(bucketStarts, t.UnixNano())
}
} else {
for k := range groups {
bucketStarts = append(bucketStarts, k)
}
sort.Slice(bucketStarts, func(i, j int) bool { return bucketStarts[i] < bucketStarts[j] })
}
out := make([]AggregatePoint, 0, len(bucketStarts))
for _, start := range bucketStarts {
b := groups[start]
if b == nil {
out = append(out, AggregatePoint{
MetricName: query.MetricName,
EntityID: query.EntityID,
Bucket: time.Unix(0, start).UTC(),
})
continue
}
v, ok := b.value(query.Aggregation)
if !ok {
return nil, fmt.Errorf("%w: aggregation %q not supported over rollups", ErrInvalidArgument, query.Aggregation)
}
out = append(out, AggregatePoint{
MetricName: query.MetricName,
EntityID: query.EntityID,
Bucket: time.Unix(0, start).UTC(),
Value: v,
Count: int(b.count),
})
}
return out, nil
}
// scanRollupRowsContained loads rollup rows for one resolution whose whole
// bucket window [bucket, bucket+resolution) lies inside the inclusive query
// window [start, end], with optional entity and tag filters pushed into SQL.
//
// Full containment (not mere overlap) is the boundary rule that keeps a rollup
// query from over-counting: a rollup bucket is an indivisible summary, so a
// bucket that only partially overlaps the window would drag its out-of-window
// samples in if it were included. The end boundary is inclusive to match the
// raw query semantics (raw uses ts <= end); a bucket whose last covered nano is
// exactly end is therefore still contained. A degenerate zero-width window
// (start == end) contains no whole bucket and yields an empty result — callers
// that need sub-bucket precision must query raw points.
//
// scanRollupRowsContained 读取某分辨率下整桶窗口 [bucket, bucket+resolution)
// 完整落在闭区间 [start, end] 内的 rollup 行,并可把实体和标签过滤下推到 SQL。
//
// 采用“完整包含”(而非仅重叠)作为边界规则,避免 rollup 查询过度计数:rollup
// 桶是不可分割的摘要,只与窗口部分重叠的桶若被纳入,会把窗口外的样本一起带入。
// end 边界为闭区间,以匹配原始查询语义(raw 使用 ts <= end);因此最后覆盖纳秒
// 恰为 end 的桶仍算被包含。零宽窗口(start == end)不包含任何完整桶,返回空结果,
// 需要亚桶精度的调用方应查询原始点。
func (s *Store) scanRollupRowsContained(ctx context.Context, metricName, entityID string, tags map[string]string, resolution time.Duration, start, end time.Time) ([]storedRollup, error) {
resNano := resolution.Nanoseconds()
startNano := start.UTC().UnixNano()
endNano := end.UTC().UnixNano()
// A fully-contained bucket has start >= startNano and end (inclusive,
// bucket+resNano-1) <= endNano, i.e. bucket in [startNano, endNano-resNano+1].
// Push that exact closed range into SQL; no post-filter is then required.
lower := startNano
upper := endNano - resNano + 1 // inclusive upper bound for bucket_nano
if upper < lower {
return nil, nil // window narrower than one bucket: nothing is contained
}
rows, err := s.scanRollupRowsBetween(ctx, metricName, entityID, tags, resNano, lower, upper)
if err != nil {
return nil, err
}
return rows, nil
}
// scanRollupRowsBetween loads rollup rows for one resolution whose bucket start
// falls in the inclusive nano range [lowerBucket, upperBucket], with optional
// entity and tag filters pushed into SQL. It is the shared SQL primitive behind
// the containment and hybrid scans; the bucket-window semantics are imposed by
// the caller through the bounds it passes.
//
// scanRollupRowsBetween 读取某分辨率下桶起点落在闭区间
// [lowerBucket, upperBucket] 内的 rollup 行,并可把实体和标签过滤下推到 SQL。
// 它是包含扫描和混合扫描共用的 SQL 原语;桶窗口语义由调用方通过传入的边界决定。
func (s *Store) scanRollupRowsBetween(ctx context.Context, metricName, entityID string, tags map[string]string, resNano, lowerBucket, upperBucket int64) ([]storedRollup, error) {
args := []any{metricName, resNano, lowerBucket, upperBucket}
parts := []string{
"metric_name = " + s.dialect.placeholder(1),
"resolution_nano = " + s.dialect.placeholder(2),
"bucket_nano >= " + s.dialect.placeholder(3),
"bucket_nano <= " + s.dialect.placeholder(4),
}
if strings.TrimSpace(entityID) != "" {
args = append(args, entityID)
parts = append(parts, "entity_id = "+s.dialect.placeholder(len(args)))
}
for _, k := range sortedKeys(tags) {
args = append(args, tags[k])
parts = append(parts, s.dialect.jsonExtractEquals("tags", k, s.dialect.placeholder(len(args))))
}
sqlText := fmt.Sprintf(
`SELECT entity_id, tags_hash, tags, bucket_nano, count, sum, sum_sq, min_val, max_val, first_val, first_ts, last_val, last_ts, digest
FROM %s WHERE %s ORDER BY bucket_nano ASC`,
s.tables.rollups, strings.Join(parts, " AND "),
)
rows, err := s.reader().QueryContext(ctx, sqlText, args...)
if err != nil {
return nil, err
}
defer rows.Close()
return scanStoredRollups(rows)
}
// foldRollupRows folds stored rollup rows into interval-wide output buckets,
// merging summaries (including the t-digest) per bucket. groups may be nil, in
// which case a fresh map is allocated; passing an existing map lets a caller
// accumulate rollup and raw contributions into the same output buckets.
//
// foldRollupRows 将存储的 rollup 行折叠进 interval 宽的输出桶,按桶合并摘要
// (包括 t-digest)。groups 可为 nil,此时会分配新 map;传入已有 map 可让调用方
// 把 rollup 和 raw 的贡献累加到同一批输出桶中。
func foldRollupRows(groups map[int64]*rollupBucket, rows []storedRollup, interval time.Duration, comp float64) map[int64]*rollupBucket {
if groups == nil {
groups = make(map[int64]*rollupBucket)
}
size := interval.Nanoseconds()
for _, r := range rows {
bkt := floorDivNano(r.bucket, size)
b := groups[bkt]
if b == nil {
b = newRollupBucket(comp)
groups[bkt] = b
}
b.mergeStored(r.bucketData)
}
return groups
}
// foldRawPoints folds raw points into interval-wide output buckets, adding each
// observation into the matching bucket's accumulator. It shares the bucket map
// with foldRollupRows so a hybrid query can combine an old rollup half and a
// recent raw half into the same output buckets and aggregate them together
// (correct count/avg/percentile across the boundary).
//
// foldRawPoints 将原始点折叠进 interval 宽的输出桶,把每个观测值加入对应桶的
// 累加器。它与 foldRollupRows 共用桶 map,因此混合查询可以把旧 rollup 半边和
// 近期 raw 半边合并到同一批输出桶中并一起聚合(跨边界的 count/avg/百分位正确)。
func foldRawPoints(groups map[int64]*rollupBucket, points []Point, interval time.Duration, comp float64) map[int64]*rollupBucket {
if groups == nil {
groups = make(map[int64]*rollupBucket)
}
size := interval.Nanoseconds()
for _, p := range points {
ts := p.Timestamp.UTC().UnixNano()
bkt := floorDivNano(ts, size)
b := groups[bkt]
if b == nil {
b = newRollupBucket(comp)
groups[bkt] = b
}
b.addPoint(p.Value, ts)
}
return groups
}
// scanStoredRollups reconstructs storedRollup rows (including tag identity and
// the decoded t-digest) from a result set whose columns are, in order:
// entity_id, tags_hash, tags, bucket_nano, count, sum, sum_sq, min_val, max_val,
// first_val, first_ts, last_val, last_ts, digest.
//
// scanStoredRollups 从结果集中还原 storedRollup 行(包括标签身份和解码后的
// t-digest)。结果集的列顺序必须是:entity_id、tags_hash、tags、bucket_nano、
// count、sum、sum_sq、min_val、max_val、first_val、first_ts、last_val、
// last_ts、digest。
func scanStoredRollups(rows *sql.Rows) ([]storedRollup, error) {
var out []storedRollup
for rows.Next() {
var (
eid string
tagsHash string
rawTags any
bucketNano int64
count int64
sum, sumSq, minV, maxV, firstV, lastV float64
firstTS, lastTS int64
digestBlob []byte
)
if err := rows.Scan(&eid, &tagsHash, &rawTags, &bucketNano, &count, &sum, &sumSq, &minV, &maxV, &firstV, &firstTS, &lastV, &lastTS, &digestBlob); err != nil {
return nil, err
}
td, err := DecodeTDigest(digestBlob)
if err != nil {
return nil, err
}
tagsJSON, err := rawTagsToJSON(rawTags)
if err != nil {
return nil, err
}
out = append(out, storedRollup{
entityID: eid,
bucket: bucketNano,
bucketData: &rollupBucket{
count: count, sum: sum, sumSq: sumSq,
min: minV, max: maxV,
firstVal: firstV, firstTS: firstTS,
lastVal: lastV, lastTS: lastTS,
digest: td,
tagsHash: tagsHash,
tagsJSON: tagsJSON,
},
})
}
return out, rows.Err()
}
// rawTagsToJSON normalizes a scanned tags column (string or []byte) into the
// canonical JSON string used when the bucket is re-written by a coarser tier.
//
// rawTagsToJSON 将扫描出的 tags 列(string 或 []byte)规范化为 JSON 字符串,
// 供更粗层级重写桶时复用。
func rawTagsToJSON(v any) (string, error) {
switch x := v.(type) {
case nil:
return "{}", nil
case string:
if x == "" {
return "{}", nil
}
return x, nil
case []byte:
if len(x) == 0 {
return "{}", nil
}
return string(x), nil
default:
return "", fmt.Errorf("unsupported tags column type %T", v)
}
}
// Series answers an AggregateQuery by transparently choosing the best data
// source for the requested window, given `now`:
//
// - If rollups are disabled, or the whole window still lies within raw
// retention, it reads raw points (Aggregate) for full fidelity.
// - Otherwise it picks the FINEST rollup tier that both (a) has an Interval
// dividing query.Interval and (b) whose retention reaches back to the start
// of the window, and serves the query from that tier.
// - If the query spans the raw retention boundary, it uses a hybrid approach:
// read rollups for the old part and raw for the recent part, then merge
// the results to avoid losing uncompacted recent data.
// - If no tier qualifies, it falls back to raw (which may be incomplete for
// data already aged out) so the call still returns its best answer.
//
// query.Tags is honored on both branches: the raw path already filters by tag,
// and the rollup path filters by the stored tag set, so a tag filter selects the
// same series regardless of which source answers. This is the "downsampling
// TSDB" read path: recent ranges answer from raw at full resolution, older
// ranges answer from progressively coarser rollups.
//
// Series 会在给定 `now` 的情况下,通过透明选择最佳数据源来回答 AggregateQuery:
//
// - 如果 rollup 已禁用,或整个窗口仍在原始保留期内,它会读取原始点
// (Aggregate)以获得完整保真度。
// - 否则它会选择最细的 rollup 层,该层必须同时满足 (a) Interval 能整除
// query.Interval,且 (b) 保留时间能覆盖到窗口起点,然后从该层服务查询。
// - 如果查询跨越原始数据保留期边界,它使用混合方式:读取旧部分的 rollup
// 和最近部分的原始点,然后合并结果,避免丢掉未 compact 的最近数据。
// - 如果没有层级符合条件,它会回退到原始点(对于已经过期的数据可能不完整),
// 让调用仍返回当前能给出的最佳答案。
//
// query.Tags 在两条分支上都会被遵守:原始路径已按标签过滤,rollup 路径会按
// 存储的标签集合过滤,因此无论由哪个数据源回答,标签过滤都会选择相同序列。
// 这是“降采样 TSDB”的读取路径:近期范围以完整分辨率从原始点回答,旧范围从
// 逐级更粗的 rollup 回答。
func (s *Store) Series(ctx context.Context, query AggregateQuery, now time.Time) ([]AggregatePoint, error) {
if err := s.ensureOpen(); err != nil {
return nil, err
}
if err := query.Validate(); err != nil {
return nil, err
}
policy := s.cfg.RollupPolicy
if !policy.Enabled() {
return s.Aggregate(ctx, query)
}
q := query.Query.normalized()
now = now.UTC()
rawCutoff := policy.rawCutoff(now)
// Whole window inside raw retention (or raw kept forever) -> raw.
if rawCutoff.IsZero() || !q.Start.Before(rawCutoff) {
return s.Aggregate(ctx, query)
}
// Rate is raw-only; the caller asked for something rollups can't provide, so
// answer from raw regardless of age.
if query.Aggregation == AggRate {
return s.Aggregate(ctx, query)
}
if q.Start.Before(rawCutoff) && !q.End.Before(rawCutoff) {
for _, tier := range policy.Tiers { // finest-first
if query.Interval < tier.Interval || query.Interval%tier.Interval != 0 {
continue
}
if now.Add(-tier.Retention).After(q.Start) {
continue
}
return s.seriesHybrid(ctx, query, rawCutoff, &tier)
}
}
// Find the finest tier that can serve the entire window from the start.
var servicingTier *RollupTier
for _, tier := range policy.Tiers { // finest-first
if query.Interval < tier.Interval || query.Interval%tier.Interval != 0 {
continue
}
if now.Add(-tier.Retention).After(q.Start) {
continue // tier doesn't reach back to the window start
}
servicingTier = &tier
break
}
if servicingTier == nil {
return s.Aggregate(ctx, query)
}
return s.AggregateRollup(ctx, query, servicingTier.Interval)
}
// seriesHybrid answers a query that spans the raw-retention boundary by reading
// rollups for the old part and raw for the recent part and folding BOTH into the
// same query.Interval-wide output buckets before reducing them.
//
// The old lossy approach reduced each half to AggregatePoints independently and
// then deduplicated by bucket time, letting raw fully replace a rollup point in
// any output bucket they shared. That dropped the rollup half of a bucket that
// straddles the boundary: e.g. with a 1h output bucket and the boundary mid-hour,
// an 11:05 rollup sample and an 11:35 raw sample both belong to the 11:00 bucket,
// yet the result kept only the raw sample (count/avg wrong). Folding both halves
// into one rollupBucket per output bucket fixes that: count, sum/avg, min/max and
// percentiles are computed over the union of the two halves.
//
// To avoid double counting or gaps, the caller passes a boundary already aligned
// to a rollup-resolution bucket. The rollup half supplies whole buckets strictly
// before the boundary and the raw half supplies points at or after it.
//
// seriesHybrid 回答跨越原始保留期边界的查询:读取旧部分的 rollup 和近期部分的
// 原始点,并在归约前把两者折叠进同一批 query.Interval 宽的输出桶。
//
// 旧的有损做法会把两半各自归约成 AggregatePoint 再按桶时间去重,让 raw 在二者
// 共有的输出桶里完全覆盖 rollup 点,从而丢掉跨边界桶的 rollup 半边(例如 1h 输出
// 桶且边界落在小时中间时,11:05 的 rollup 样本和 11:35 的 raw 样本都属于 11:00 桶,
// 结果却只剩 raw 样本,count/avg 错误)。把两半折叠进同一个 rollupBucket 即可修复:
// count、sum/avg、min/max 和百分位都在两半的并集上计算。
//
// 为避免重复计数或留空,调用方传入的边界已经对齐到 rollup 分辨率桶。rollup 半边
// 提供严格早于该边界的完整桶,raw 半边提供该边界及之后的点。
func (s *Store) seriesHybrid(ctx context.Context, query AggregateQuery, boundary time.Time, tier *RollupTier) ([]AggregatePoint, error) {
q := query.Query.normalized()
comp := s.cfg.RollupPolicy.compression()
resNano := tier.Interval.Nanoseconds()
startNano := q.Start.UnixNano()
endNano := q.End.UnixNano()
splitAt := boundary.UTC().UnixNano()
groups := make(map[int64]*rollupBucket)
// Old half: rollup buckets fully contained in [Start, min(splitAt, End]].
// "Fully before splitAt" is bucket <= splitAt-resNano (splitAt is aligned);
// also clamp to the query's own inclusive end so a bucket past End is never
// pulled in when splitAt overshoots a short window.
upperBucket := splitAt - resNano
if clamp := endNano - resNano + 1; clamp < upperBucket {
upperBucket = clamp
}
if upperBucket >= startNano {
rows, err := s.scanRollupRowsBetween(ctx, q.MetricName, q.EntityID, q.Tags, resNano, startNano, upperBucket)
if err != nil {
return nil, err
}
foldRollupRows(groups, rows, query.Interval, comp)
}
// Recent half: raw points in [splitAt, End]. Reuse the raw Query path so the
// entity and tag filters match the rollup half exactly, then fold the points
// into the SAME output buckets so a bucket straddling splitAt aggregates both
// halves. Skip when splitAt is past the window end (no recent raw to add).
if splitAt <= endNano {
rawQuery := q
rawQuery.Start = time.Unix(0, splitAt).UTC()
rawQuery.Limit = 0
rawQuery.Offset = 0
points, err := s.Query(ctx, rawQuery)
if err != nil {
return nil, err
}
foldRawPoints(groups, points, query.Interval, comp)
}
out, err := rollupGroupsToPoints(groups, query)
if err != nil {
return nil, err
}
return pageBuckets(out, query.BucketLimit, query.BucketOffset), nil
}
+740
View File
@@ -0,0 +1,740 @@
package metric
import (
"context"
"database/sql"
"fmt"
"sort"
"strings"
"time"
)
// Compact runs the store's RollupPolicy over every metric: it rebuilds the
// rollup tiers up to `now` and enforces every retention window (raw points and
// each tier). It is the explicit maintenance entry point — call it from a cron
// job or scheduler. It is safe to call repeatedly and is idempotent for
// unchanged windows.
//
// Returns the number of rollup buckets written across all tiers and metrics.
//
// Compact 会对所有指标执行 Store 的 RollupPolicy:它会重建截至 `now` 的
// rollup 层,并执行每个保留窗口(原始点和各层级)。这是显式维护入口,
// 应从 cron 任务或调度器调用。它可以重复安全调用,未变化窗口保持幂等。
//
// 返回所有指标、所有层级中写入的 rollup 桶数量。
func (s *Store) Compact(ctx context.Context, now time.Time) (int, error) {
if err := s.ensureOpen(); err != nil {
return 0, err
}
if !s.cfg.RollupPolicy.Enabled() {
return 0, nil
}
defs, err := s.ListMetrics(ctx)
if err != nil {
return 0, err
}
total := 0
for _, def := range defs {
n, err := s.CompactMetric(ctx, def.Name, now)
if err != nil {
return total, fmt.Errorf("compact metric %q: %w", def.Name, err)
}
total += n
}
return total, nil
}
// CompactMetric compacts a single metric. It builds the finest tier from raw
// points, then each coarser tier from the tier below it, upserts the resulting
// buckets, and finally deletes data that has aged out of each retention window
// (raw points first, then each tier).
//
// The entire compaction is performed within a SERIALIZABLE transaction to
// guarantee that the raw scan and the raw deletion observe the same snapshot.
// Without this, the default isolation level on PostgreSQL/MySQL lets a point
// inserted between the scan and the delete be deleted by the cutoff DELETE
// without ever being folded into a rollup: the scan never saw it (it ran on an
// earlier read), yet the DELETE (a fresh read) does. SERIALIZABLE makes the
// scan acquire predicate/range protection so a concurrent write that would
// otherwise slip into that gap either is excluded from the delete or forces a
// serialization failure that we retry on a fresh snapshot. SQLite serializes
// writes on a single connection, so its default isolation already provides this
// guarantee and needs no escalation.
//
// Rollups are keyed by tag set as well as entity, so each distinct tag
// combination (e.g. a GPU device_index) is summarized into its own series and
// can be queried independently after the raw points are gone.
//
// CompactMetric 会压缩单个指标。它先由原始点构建最细层,再由下一层之下的
// 层级逐层合成更粗层,upsert 生成的桶,最后删除每个保留窗口中过期的数据
// (先删除原始点,再删除各层级)。
//
// 整个 compaction 在一个 SERIALIZABLE 事务内执行,以保证 raw 扫描和 raw 删除
// 看到同一个快照。否则在 PostgreSQL/MySQL 的默认隔离级别下,扫描之后、删除
// 之前写入的旧时间点仍可能被 cutoff 删除却没有进入 rollup:扫描(较早的读)
// 没看到它,而删除(一次新的读)看到了。SERIALIZABLE 让扫描获得谓词/范围
// 保护,使这种并发写要么被删除排除在外,要么触发可重试的序列化失败,由我们
// 在新快照上重试。SQLite 在单连接上串行化写入,其默认隔离已提供该保证,
// 无需提升隔离级别。
func (s *Store) CompactMetric(ctx context.Context, metricName string, now time.Time) (int, error) {
if err := s.ensureOpen(); err != nil {
return 0, err
}
policy := s.cfg.RollupPolicy
if !policy.Enabled() {
return 0, nil
}
now = now.UTC()
// Retry the whole compaction on a transient serialization/deadlock failure.
// Under SERIALIZABLE, a concurrent writer touching the same raw range can
// abort our transaction; re-running on a fresh snapshot is the correct and
// expected recovery, and Compact is idempotent so a retry is safe.
const maxAttempts = 5
var lastErr error
for attempt := 0; attempt < maxAttempts; attempt++ {
written, err := s.compactMetricOnce(ctx, metricName, now)
if err == nil {
return written, nil
}
if !isRetryableSerializationError(err) {
return written, err
}
lastErr = err
}
return 0, fmt.Errorf("compact metric %q: exhausted retries after serialization failures: %w", metricName, lastErr)
}
// compactMetricOnce runs a single compaction attempt inside one transaction.
//
// compactMetricOnce 在单个事务内执行一次 compaction 尝试。
func (s *Store) compactMetricOnce(ctx context.Context, metricName string, now time.Time) (int, error) {
// Use a transaction to ensure consistency between raw scan, rollup write, and
// raw deletion. The isolation level is backend-specific (SERIALIZABLE on
// PostgreSQL/MySQL, default on SQLite) so late-arriving points cannot be
// deleted without first being rolled up.
tx, err := s.db.BeginTx(ctx, s.dialect.compactTxOptions())
if err != nil {
return 0, err
}
defer func() { _ = tx.Rollback() }()
written, err := s.compactMetricWithinTx(ctx, metricName, now, tx)
if err != nil {
return written, err
}
if err := tx.Commit(); err != nil {
return written, err
}
return written, nil
}
// isRetryableSerializationError reports whether err is a transient
// serialization failure or deadlock that should be retried on a fresh
// transaction. It matches on portable SQLSTATE codes and driver error text so
// the package stays free of driver-specific error type imports:
//
// - PostgreSQL: 40001 serialization_failure, 40P01 deadlock_detected.
// - MySQL: 1213 deadlock, 1205 lock wait timeout.
// - SQLite: SQLITE_BUSY / database is locked (only relevant with >1 conn).
//
// isRetryableSerializationError 判断 err 是否为应在新事务上重试的瞬时序列化
// 失败或死锁。它通过可移植的 SQLSTATE 码和驱动错误文本匹配,使包不需要导入
// 驱动专用错误类型。
func isRetryableSerializationError(err error) bool {
if err == nil {
return false
}
msg := strings.ToLower(err.Error())
switch {
case strings.Contains(msg, "40001"): // postgres serialization_failure
return true
case strings.Contains(msg, "40p01"): // postgres deadlock_detected
return true
case strings.Contains(msg, "serialization failure"):
return true
case strings.Contains(msg, "could not serialize"):
return true
case strings.Contains(msg, "deadlock"): // mysql 1213, postgres text
return true
case strings.Contains(msg, "lock wait timeout"): // mysql 1205
return true
case strings.Contains(msg, "database is locked"): // sqlite busy
return true
case strings.Contains(msg, "database table is locked"): // sqlite busy
return true
default:
return false
}
}
// compactMetricWithinTx is the actual compaction logic, meant to be called
// within a transaction. The tx parameter is passed explicitly to all operations
// that need transactional consistency.
func (s *Store) compactMetricWithinTx(ctx context.Context, metricName string, now time.Time, tx *sql.Tx) (int, error) {
policy := s.cfg.RollupPolicy
if !policy.Enabled() {
return 0, nil
}
comp := policy.compression()
written := 0
var prevInterval time.Duration
var prevDelta map[rollupKey]*rollupBucket
rawCutoff := policy.rawCutoff(now)
for tierIdx, tier := range policy.Tiers {
var buckets map[rollupKey]*rollupBucket
var currentDelta map[rollupKey]*rollupBucket
var err error
// Reads must run on the transaction itself, not s.reader(): with a
// single-connection pool (SQLite MaxOpenConns=1) the tx already holds
// the only connection, so a pool read here would deadlock.
if tierIdx == 0 {
buckets, err = s.buildFinestTier(ctx, tx, metricName, tier.Interval, comp)
if err == nil && !rawCutoff.IsZero() {
cutoffNano := rawCutoff.UnixNano()
for k := range buckets {
if k.bucket >= cutoffNano {
delete(buckets, k)
}
}
}
if !rawCutoff.IsZero() {
currentDelta = buckets
}
} else {
buckets, err = s.buildCoarserTier(ctx, tx, metricName, prevInterval, tier.Interval, comp)
if prevDelta != nil {
currentDelta = buildCoarserBucketsFromDelta(prevDelta, tier.Interval, comp)
}
}
if err != nil {
return written, err
}
// For the finest tier only, merge buckets older than the raw cutoff:
// those buckets may already have a retained rollup while only late raw
// samples remain in the raw table.
var n int
if tierIdx == 0 && !rawCutoff.IsZero() {
n, err = s.writeRollupBucketsWithMergePointTx(ctx, metricName, tier.Interval, buckets, rawCutoff, time.Time{}, nil, tx)
} else if tierIdx > 0 {
deltaCutoff := alignRollupRetentionCutoff(now.Add(-policy.Tiers[tierIdx-1].Retention), tier.Interval)
n, err = s.writeRollupBucketsWithMergePointTx(ctx, metricName, tier.Interval, buckets, time.Time{}, deltaCutoff, currentDelta, tx)
} else {
n, err = s.writeRollupBucketsTx(ctx, metricName, tier.Interval, buckets, tx)
}
if err != nil {
return written, err
}
written += n
prevInterval = tier.Interval
prevDelta = currentDelta
}
// Enforce retention windows after all tiers are materialized, so a coarser
// tier is always built before the finer source it depends on is trimmed.
if policy.RawRetention > 0 {
cutoff := policy.rawCutoff(now)
if _, err := s.DeleteBeforeTx(ctx, metricName, cutoff, tx); err != nil {
return written, err
}
}
for i, tier := range policy.Tiers {
cutoff := now.Add(-tier.Retention)
if i+1 < len(policy.Tiers) {
cutoff = alignRollupRetentionCutoff(cutoff, policy.Tiers[i+1].Interval)
}
if err := s.deleteRollupsBeforeTx(ctx, metricName, tier.Interval, cutoff, tx); err != nil {
return written, err
}
}
return written, nil
}
func alignRollupRetentionCutoff(cutoff time.Time, nextInterval time.Duration) time.Time {
if nextInterval <= 0 {
return cutoff.UTC()
}
return time.Unix(0, floorDivNano(cutoff.UTC().UnixNano(), nextInterval.Nanoseconds())).UTC()
}
// rollupKey identifies one rollup cell. The tag dimension (tagsHash) is part of
// the key so points carrying different tags never collapse into the same bucket.
//
// rollupKey 标识一个 rollup 单元;tagsHash 是 key 的一部分,确保不同标签
// 的点不会落入同一个桶。
type rollupKey struct {
// entityID is the entity dimension of the rollup cell.
//
// entityID 是 rollup 单元的实体维度。
entityID string
// tagsHash is the stable fingerprint of the tag set.
//
// tagsHash 是标签集合的稳定指纹。
tagsHash string
// bucket is the bucket start timestamp in nanoseconds.
//
// bucket 是桶起始时间的纳秒时间戳。
bucket int64
}
// buildFinestTier scans raw points for the metric and groups them into buckets
// of the given interval, keyed by (entity, tag set, bucket-start). Each point's
// tag map determines which series it belongs to.
//
// buildFinestTier 扫描某指标的原始点,并按给定 interval 分桶,key 为
// (实体、标签集合、桶起点)。每个点的标签 map 决定它属于哪条序列。
func (s *Store) buildFinestTier(ctx context.Context, q querier, metricName string, interval time.Duration, comp float64) (map[rollupKey]*rollupBucket, error) {
size := interval.Nanoseconds()
out := make(map[rollupKey]*rollupBucket)
sqlText := fmt.Sprintf(
`SELECT entity_id, ts_nano, value, tags FROM %s WHERE metric_name = %s ORDER BY ts_nano ASC`,
s.tables.points, s.dialect.placeholder(1),
)
rows, err := q.QueryContext(ctx, sqlText, metricName)
if err != nil {
return nil, err
}
defer rows.Close()
for rows.Next() {
var entityID string
var ts int64
var value float64
var rawTags any
if err := rows.Scan(&entityID, &ts, &value, &rawTags); err != nil {
return nil, err
}
tags, err := decodeMap(rawTags)
if err != nil {
return nil, err
}
hash, canonical, err := tagsFingerprint(tags)
if err != nil {
return nil, err
}
bucket := floorDivNano(ts, size)
k := rollupKey{entityID: entityID, tagsHash: hash, bucket: bucket}
b := out[k]
if b == nil {
b = newRollupBucket(comp)
b.tagsHash = hash
b.tagsJSON = canonical
out[k] = b
}
b.addPoint(value, ts)
}
return out, rows.Err()
}
// buildCoarserTier composes a coarser tier from the already-stored finer tier:
// it reads the finer rollup rows and merges every finer bucket into the coarse
// bucket that shares its (entity, tag set). Tag identity is preserved end to
// end, so a coarse series only ever merges finer buckets of the same tag set.
//
// buildCoarserTier 基于已存储的细层 rollup 合成更粗层:它读取细层 rollup 行,
// 并把每个细桶合并进共享相同(实体、标签集合)的粗桶。标签身份会端到端保留,
// 因此一条粗粒度序列只会合并相同标签集合的细桶。
func (s *Store) buildCoarserTier(ctx context.Context, q querier, metricName string, fineInterval, coarseInterval time.Duration, comp float64) (map[rollupKey]*rollupBucket, error) {
coarseSize := coarseInterval.Nanoseconds()
out := make(map[rollupKey]*rollupBucket)
fineRows, err := s.scanRollupRows(ctx, q, metricName, fineInterval)
if err != nil {
return nil, err
}
for _, fr := range fineRows {
bucket := floorDivNano(fr.bucket, coarseSize)
k := rollupKey{entityID: fr.entityID, tagsHash: fr.bucketData.tagsHash, bucket: bucket}
b := out[k]
if b == nil {
b = newRollupBucket(comp)
b.tagsHash = fr.bucketData.tagsHash
b.tagsJSON = fr.bucketData.tagsJSON
out[k] = b
}
b.mergeStored(fr.bucketData)
}
return out, nil
}
func buildCoarserBucketsFromDelta(delta map[rollupKey]*rollupBucket, coarseInterval time.Duration, comp float64) map[rollupKey]*rollupBucket {
out := make(map[rollupKey]*rollupBucket)
if len(delta) == 0 {
return out
}
coarseSize := coarseInterval.Nanoseconds()
for k, src := range delta {
bucket := floorDivNano(k.bucket, coarseSize)
ck := rollupKey{entityID: k.entityID, tagsHash: k.tagsHash, bucket: bucket}
b := out[ck]
if b == nil {
b = newRollupBucket(comp)
b.tagsHash = src.tagsHash
b.tagsJSON = src.tagsJSON
out[ck] = b
}
b.mergeStored(src)
}
return out
}
// storedRollup represents a rollup row reconstructed from storage.
//
// storedRollup 表示从存储中还原的一行 rollup 数据。
type storedRollup struct {
// entityID is the entity stored on the rollup row.
//
// entityID 是 rollup 行中保存的实体。
entityID string
// bucket is the stored bucket start timestamp in nanoseconds.
//
// bucket 是存储的桶起始纳秒时间戳。
bucket int64
// bucketData is the reconstructed in-memory accumulator for the row.
//
// bucketData 是该行还原出的内存累加器。
bucketData *rollupBucket
}
// scanRollupRows loads all rollup rows for a metric at a given resolution and
// reconstructs their in-memory accumulators (including tag identity and the
// decoded t-digest).
//
// scanRollupRows 读取某指标在给定分辨率下的所有 rollup 行,并还原它们的
// 内存累加器(包括标签身份和解码后的 t-digest)。
func (s *Store) scanRollupRows(ctx context.Context, q querier, metricName string, interval time.Duration) ([]storedRollup, error) {
sqlText := fmt.Sprintf(
`SELECT entity_id, tags_hash, tags, bucket_nano, count, sum, sum_sq, min_val, max_val, first_val, first_ts, last_val, last_ts, digest
FROM %s WHERE metric_name = %s AND resolution_nano = %s ORDER BY bucket_nano ASC`,
s.tables.rollups, s.dialect.placeholder(1), s.dialect.placeholder(2),
)
rows, err := q.QueryContext(ctx, sqlText, metricName, interval.Nanoseconds())
if err != nil {
return nil, err
}
defer rows.Close()
return scanStoredRollups(rows)
}
// writeRollupBuckets upserts a set of computed buckets for one resolution.
//
// writeRollupBuckets 将某个分辨率下计算出的 rollup 桶批量 upsert 到数据库。
func (s *Store) writeRollupBuckets(ctx context.Context, metricName string, interval time.Duration, buckets map[rollupKey]*rollupBucket) (int, error) {
return s.writeRollupBucketsWithMergePoint(ctx, metricName, interval, buckets, time.Time{})
}
// writeRollupBucketsTx upserts buckets within an existing transaction.
func (s *Store) writeRollupBucketsTx(ctx context.Context, metricName string, interval time.Duration, buckets map[rollupKey]*rollupBucket, tx *sql.Tx) (int, error) {
return s.writeRollupBucketsWithMergePointTx(ctx, metricName, interval, buckets, time.Time{}, time.Time{}, nil, tx)
}
// writeRollupBucketsWithMergePoint is the internal implementation that optionally
// merges buckets older than a cutoff point instead of replacing them.
func (s *Store) writeRollupBucketsWithMergePoint(ctx context.Context, metricName string, interval time.Duration, buckets map[rollupKey]*rollupBucket, mergeCutoff time.Time) (int, error) {
if len(buckets) == 0 {
return 0, nil
}
keys := make([]rollupKey, 0, len(buckets))
for k := range buckets {
keys = append(keys, k)
}
sort.Slice(keys, func(i, j int) bool {
if keys[i].entityID != keys[j].entityID {
return keys[i].entityID < keys[j].entityID
}
if keys[i].tagsHash != keys[j].tagsHash {
return keys[i].tagsHash < keys[j].tagsHash
}
return keys[i].bucket < keys[j].bucket
})
stmt := s.dialect.upsertRollupSQL(s.tables)
resNano := interval.Nanoseconds()
now := time.Now().UTC().UnixNano()
mergeCutoffNano := mergeCutoff.UnixNano()
run := func(ex execer) error {
for _, k := range keys {
b := buckets[k]
tagsJSON := b.tagsJSON
if tagsJSON == "" {
tagsJSON = "{}"
}
// If mergeCutoff is set and this bucket is older than the cutoff,
// merge with existing data instead of replacing it.
if !mergeCutoff.IsZero() && k.bucket < mergeCutoffNano {
// Read existing row if it exists
existing, err := s.readRollupBucket(ctx, metricName, k.entityID, k.tagsHash, interval, k.bucket)
if err != nil {
return err
}
if existing != nil {
// Merge the new bucket into the existing one
existing.mergeStored(b)
b = existing
}
}
// Column order must match rollupColumns in dialect_rollup.go:
// metric_name, entity_id, tags_hash, tags, resolution_nano, bucket_nano,
// count, sum, sum_sq, min_val, max_val, first_val, first_ts, last_val,
// last_ts, digest, created_at.
_, err := ex.ExecContext(ctx, stmt,
metricName, k.entityID, k.tagsHash, tagsJSON, resNano, k.bucket,
b.count, b.sum, b.sumSq, b.min, b.max,
b.firstVal, b.firstTS, b.lastVal, b.lastTS,
b.digest.Encode(), now,
)
if err != nil {
return err
}
}
return nil
}
if len(keys) == 1 {
if err := run(s.db); err != nil {
return 0, err
}
return len(keys), nil
}
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return 0, err
}
defer func() { _ = tx.Rollback() }()
if err := run(tx); err != nil {
return 0, err
}
if err := tx.Commit(); err != nil {
return 0, err
}
return len(keys), nil
}
// writeRollupBucketsWithMergePointTx is the transactional version that executes
// all operations within an existing transaction. deltaCutoff marks coarser
// buckets whose source tier may already have aged out; a smaller recomputed
// bucket in that region is treated as late-arriving delta data and merged into
// the retained coarse bucket instead of replacing it.
func (s *Store) writeRollupBucketsWithMergePointTx(ctx context.Context, metricName string, interval time.Duration, buckets map[rollupKey]*rollupBucket, mergeCutoff, protectCutoff time.Time, deltaBuckets map[rollupKey]*rollupBucket, tx *sql.Tx) (int, error) {
if len(buckets) == 0 {
return 0, nil
}
keys := make([]rollupKey, 0, len(buckets))
for k := range buckets {
keys = append(keys, k)
}
sort.Slice(keys, func(i, j int) bool {
if keys[i].entityID != keys[j].entityID {
return keys[i].entityID < keys[j].entityID
}
if keys[i].tagsHash != keys[j].tagsHash {
return keys[i].tagsHash < keys[j].tagsHash
}
return keys[i].bucket < keys[j].bucket
})
stmt := s.dialect.upsertRollupSQL(s.tables)
resNano := interval.Nanoseconds()
now := time.Now().UTC().UnixNano()
mergeCutoffNano := mergeCutoff.UnixNano()
deltaCutoffNano := protectCutoff.UnixNano()
written := 0
for _, k := range keys {
b := buckets[k]
tagsJSON := b.tagsJSON
if tagsJSON == "" {
tagsJSON = "{}"
}
// If mergeCutoff is set and this bucket is older than the cutoff,
// merge with existing data instead of replacing it.
if !mergeCutoff.IsZero() && k.bucket < mergeCutoffNano {
// Read existing row if it exists
existing, err := s.readRollupBucketTx(ctx, metricName, k.entityID, k.tagsHash, interval, k.bucket, tx)
if err != nil {
return 0, err
}
if existing != nil {
// Merge the new bucket into the existing one
existing.mergeStored(b)
b = existing
}
}
if !protectCutoff.IsZero() && k.bucket < deltaCutoffNano {
existing, err := s.readRollupBucketTx(ctx, metricName, k.entityID, k.tagsHash, interval, k.bucket, tx)
if err != nil {
return 0, err
}
if existing != nil {
if deltaBuckets == nil {
// Without delta tracking (raw is retained forever), the
// source tier is rebuilt from full raw data and can replace.
} else if delta := deltaBuckets[k]; delta != nil && delta.count > 0 {
existing.mergeStored(delta)
b = existing
} else {
continue
}
}
}
// Column order must match rollupColumns in dialect_rollup.go
_, err := tx.ExecContext(ctx, stmt,
metricName, k.entityID, k.tagsHash, tagsJSON, resNano, k.bucket,
b.count, b.sum, b.sumSq, b.min, b.max,
b.firstVal, b.firstTS, b.lastVal, b.lastTS,
b.digest.Encode(), now,
)
if err != nil {
return 0, err
}
written++
}
return written, nil
}
// readRollupBucket reads a single rollup bucket from storage, returning nil if
// the bucket doesn't exist. Used to merge late-arriving data with existing rollups.
func (s *Store) readRollupBucket(ctx context.Context, metricName, entityID, tagsHash string, interval time.Duration, bucketNano int64) (*rollupBucket, error) {
sqlText := fmt.Sprintf(
`SELECT count, sum, sum_sq, min_val, max_val, first_val, first_ts, last_val, last_ts, digest, tags
FROM %s WHERE metric_name = %s AND resolution_nano = %s AND entity_id = %s
AND tags_hash = %s AND bucket_nano = %s`,
s.tables.rollups,
s.dialect.placeholder(1), s.dialect.placeholder(2), s.dialect.placeholder(3),
s.dialect.placeholder(4), s.dialect.placeholder(5),
)
row := s.reader().QueryRowContext(ctx, sqlText, metricName, interval.Nanoseconds(), entityID, tagsHash, bucketNano)
var count int64
var sum, sumSq, minV, maxV, firstV, lastV float64
var firstTS, lastTS int64
var digestBlob []byte
var rawTags any
err := row.Scan(&count, &sum, &sumSq, &minV, &maxV, &firstV, &firstTS, &lastV, &lastTS, &digestBlob, &rawTags)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
td, err := DecodeTDigest(digestBlob)
if err != nil {
return nil, err
}
tagsJSON, err := rawTagsToJSON(rawTags)
if err != nil {
return nil, err
}
return &rollupBucket{
count: count, sum: sum, sumSq: sumSq,
min: minV, max: maxV,
firstVal: firstV, firstTS: firstTS,
lastVal: lastV, lastTS: lastTS,
digest: td,
tagsHash: tagsHash,
tagsJSON: tagsJSON,
}, nil
}
// deleteRollupsBefore deletes stored rollup rows older than a cutoff.
func (s *Store) deleteRollupsBefore(ctx context.Context, metricName string, interval time.Duration, before time.Time) error {
sqlText := fmt.Sprintf(
`DELETE FROM %s WHERE metric_name = %s AND resolution_nano = %s AND bucket_nano < %s`,
s.tables.rollups, s.dialect.placeholder(1), s.dialect.placeholder(2), s.dialect.placeholder(3),
)
_, err := s.db.ExecContext(ctx, sqlText, metricName, interval.Nanoseconds(), before.UTC().UnixNano())
return err
}
// deleteRollupsBeforeTx deletes stored rollup rows within a transaction.
func (s *Store) deleteRollupsBeforeTx(ctx context.Context, metricName string, interval time.Duration, before time.Time, tx *sql.Tx) error {
sqlText := fmt.Sprintf(
`DELETE FROM %s WHERE metric_name = %s AND resolution_nano = %s AND bucket_nano < %s`,
s.tables.rollups, s.dialect.placeholder(1), s.dialect.placeholder(2), s.dialect.placeholder(3),
)
_, err := tx.ExecContext(ctx, sqlText, metricName, interval.Nanoseconds(), before.UTC().UnixNano())
return err
}
// DeleteBeforeTx deletes raw points before a cutoff within a transaction.
func (s *Store) DeleteBeforeTx(ctx context.Context, metricName string, before time.Time, tx *sql.Tx) (int64, error) {
sqlText := fmt.Sprintf(
`DELETE FROM %s WHERE metric_name = %s AND ts_nano < %s`,
s.tables.points, s.dialect.placeholder(1), s.dialect.placeholder(2),
)
result, err := tx.ExecContext(ctx, sqlText, metricName, before.UTC().UnixNano())
if err != nil {
return 0, err
}
return result.RowsAffected()
}
// floorDivNano floors ts to the start of its size-wide bucket, handling
// negative timestamps (pre-epoch) toward negative infinity so buckets align
// consistently. Mirrors alignTime but operates on raw nanos.
//
// floorDivNano 将 ts 向下对齐到 size 宽桶的起点,并把负时间戳(Unix epoch 前)
// 朝负无穷取整,让桶保持一致对齐。它与 alignTime 逻辑一致,但直接操作纳秒值。
func floorDivNano(ts, size int64) int64 {
rem := ((ts % size) + size) % size
return ts - rem
}
// readRollupBucketTx reads a single rollup bucket from a transaction.
func (s *Store) readRollupBucketTx(ctx context.Context, metricName, entityID, tagsHash string, interval time.Duration, bucketNano int64, tx *sql.Tx) (*rollupBucket, error) {
sqlText := fmt.Sprintf(
`SELECT count, sum, sum_sq, min_val, max_val, first_val, first_ts, last_val, last_ts, digest, tags
FROM %s WHERE metric_name = %s AND resolution_nano = %s AND entity_id = %s
AND tags_hash = %s AND bucket_nano = %s`,
s.tables.rollups,
s.dialect.placeholder(1), s.dialect.placeholder(2), s.dialect.placeholder(3),
s.dialect.placeholder(4), s.dialect.placeholder(5),
)
row := tx.QueryRowContext(ctx, sqlText, metricName, interval.Nanoseconds(), entityID, tagsHash, bucketNano)
var count int64
var sum, sumSq, minV, maxV, firstV, lastV float64
var firstTS, lastTS int64
var digestBlob []byte
var rawTags any
err := row.Scan(&count, &sum, &sumSq, &minV, &maxV, &firstV, &firstTS, &lastV, &lastTS, &digestBlob, &rawTags)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
td, err := DecodeTDigest(digestBlob)
if err != nil {
return nil, err
}
tagsJSON, err := rawTagsToJSON(rawTags)
if err != nil {
return nil, err
}
return &rollupBucket{
count: count, sum: sum, sumSq: sumSq,
min: minV, max: maxV,
firstVal: firstV, firstTS: firstTS,
lastVal: lastV, lastTS: lastTS,
digest: td,
tagsHash: tagsHash,
tagsJSON: tagsJSON,
}, nil
}
+349
View File
@@ -0,0 +1,349 @@
package metric
import (
"context"
"math"
"testing"
"time"
)
// TestRollupKeepsTagSeriesSeparate is the regression test for the bug where the
// rollup dropped the tag dimension: two points in the SAME metric/entity/bucket
// but with different tags must become two independent rollup series, and a tag
// filter must return that tag's own value rather than a cross-tag merge.
//
// TestRollupKeepsTagSeriesSeparate 是针对 rollup 曾丢失标签维度这一问题的
// 回归测试:同一个指标、实体和桶内带有不同标签的两个点,必须成为两条独立的
// rollup 序列;标签过滤必须返回该标签自己的值,而不是跨标签合并结果。
func TestRollupKeepsTagSeriesSeparate(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}}}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "util", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
// Same entity, same 1-minute bucket; two GPUs distinguished only by tag.
// device 0 carries values 10..19, device 1 carries 80..89.
var batch []Point
for i := 0; i < 10; i++ {
batch = append(batch,
Point{MetricName: "util", EntityID: "host-1", Timestamp: base.Add(time.Duration(i) * time.Second), Value: float64(10 + i), Tags: map[string]string{"device_index": "0"}},
Point{MetricName: "util", EntityID: "host-1", Timestamp: base.Add(time.Duration(i) * time.Second), Value: float64(80 + i), Tags: map[string]string{"device_index": "1"}},
)
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
written, err := s.Compact(ctx, base.Add(10*time.Minute))
if err != nil {
t.Fatalf("compact: %v", err)
}
// Two distinct tag series in the one bucket => two rollup rows.
if written != 2 {
t.Fatalf("expected 2 rollup series (one per device_index), got %d", written)
}
q := func(dev string) AggregateQuery {
return AggregateQuery{
Query: Query{
MetricName: "util", EntityID: "host-1",
Start: base, End: base.Add(10 * time.Minute),
Tags: map[string]string{"device_index": dev},
},
Aggregation: AggAvg,
Interval: time.Minute,
}
}
// device 0: avg of 10..19 = 14.5
res0, err := s.AggregateRollup(ctx, q("0"), time.Minute)
if err != nil {
t.Fatalf("rollup dev0: %v", err)
}
if len(res0) != 1 || res0[0].Count != 10 || math.Abs(res0[0].Value-14.5) > 1e-9 {
t.Fatalf("device 0 should be its own series avg=14.5 count=10, got %#v", res0)
}
// device 1: avg of 80..89 = 84.5
res1, err := s.AggregateRollup(ctx, q("1"), time.Minute)
if err != nil {
t.Fatalf("rollup dev1: %v", err)
}
if len(res1) != 1 || res1[0].Count != 10 || math.Abs(res1[0].Value-84.5) > 1e-9 {
t.Fatalf("device 1 should be its own series avg=84.5 count=10, got %#v", res1)
}
// No tag filter => both series merge (count 20, avg of all 20 values).
all, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{MetricName: "util", EntityID: "host-1", Start: base, End: base.Add(10 * time.Minute)},
Aggregation: AggAvg,
Interval: time.Minute,
}, time.Minute)
if err != nil {
t.Fatalf("rollup all: %v", err)
}
wantAvg := (145.0 + 845.0) / 20.0 // sum(10..19)=145, sum(80..89)=845
if len(all) != 1 || all[0].Count != 20 || math.Abs(all[0].Value-wantAvg) > 1e-9 {
t.Fatalf("no-filter should merge both tags count=20 avg=%.2f, got %#v", wantAvg, all)
}
}
// TestRollupTagPercentileSurvivesRetention proves the headline TSDB property
// still holds per tag: after raw is deleted, a tag-filtered percentile is still
// answerable from that tag's surviving rollup series (and not contaminated by
// the other tag).
//
// TestRollupTagPercentileSurvivesRetention 验证核心 TSDB 特性在每个标签上
// 仍然成立:原始点删除后,按标签过滤的百分位仍能从该标签保留下来的 rollup 序列
// 回答,并且不会被另一个标签污染。
func TestRollupTagPercentileSurvivesRetention(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: 2 * time.Minute,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 365 * 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "lat", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
// One 1m bucket. region=ap: values 0..99; region=eu: values 1000..1099.
var batch []Point
for i := 0; i < 100; i++ {
ts := base.Add(time.Duration(i) * time.Millisecond * 500)
batch = append(batch,
Point{MetricName: "lat", EntityID: "n1", Timestamp: ts, Value: float64(i), Tags: map[string]string{"region": "ap"}},
Point{MetricName: "lat", EntityID: "n1", Timestamp: ts, Value: float64(1000 + i), Tags: map[string]string{"region": "eu"}},
)
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := s.Compact(ctx, base.Add(48*time.Hour)); err != nil {
t.Fatalf("compact: %v", err)
}
// Raw gone.
raw, err := s.Query(ctx, Query{MetricName: "lat", EntityID: "n1", Start: base, End: base.Add(time.Hour)})
if err != nil {
t.Fatalf("query raw: %v", err)
}
if len(raw) != 0 {
t.Fatalf("expected raw deleted, got %d", len(raw))
}
// p90 of region=ap must be near the ap-only exact (≈89), NOT pulled up toward
// the eu values (which would happen if tags were merged).
res, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{
MetricName: "lat", EntityID: "n1",
Start: base, End: base.Add(time.Hour),
Tags: map[string]string{"region": "ap"},
},
Aggregation: Pxx(90),
Interval: time.Minute,
}, time.Minute)
if err != nil {
t.Fatalf("rollup ap p90: %v", err)
}
if len(res) != 1 {
t.Fatalf("expected 1 ap bucket, got %d", len(res))
}
exactAP := percentileSortedRange(0, 99, 0.90)
if math.Abs(res[0].Value-exactAP)/exactAP > 0.05 {
t.Fatalf("ap p90 %v contaminated or wrong (exact ap ≈ %v)", res[0].Value, exactAP)
}
if res[0].Value > 500 {
t.Fatalf("ap p90 %v is contaminated by eu series (>500)", res[0].Value)
}
}
// TestSeriesTagFilterRoutesPerTag checks the auto-routing read path honors tags
// on both the raw branch (recent) and the rollup branch (old).
//
// TestSeriesTagFilterRoutesPerTag 检查自动路由读取路径在原始分支(近期数据)和
// rollup 分支(旧数据)上都遵守标签。
func TestSeriesTagFilterRoutesPerTag(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: time.Hour,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 30 * 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "m", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
now := time.Date(2026, 6, 18, 12, 0, 0, 0, time.UTC)
old := now.Add(-48 * time.Hour)
// Old bucket: env=prod values 0..9, env=stage values 500..509.
var batch []Point
for i := 0; i < 10; i++ {
ts := old.Add(time.Duration(i) * time.Second)
batch = append(batch,
Point{MetricName: "m", EntityID: "n1", Timestamp: ts, Value: float64(i), Tags: map[string]string{"env": "prod"}},
Point{MetricName: "m", EntityID: "n1", Timestamp: ts, Value: float64(500 + i), Tags: map[string]string{"env": "stage"}},
)
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact: %v", err)
}
// Old window, raw aged out -> Series must use rollup, filtered to env=prod
// (avg of 0..9 = 4.5), not the merged 252.
got, err := s.Series(ctx, AggregateQuery{
Query: Query{
MetricName: "m", EntityID: "n1",
Start: old.Add(-time.Minute), End: old.Add(2 * time.Minute),
Tags: map[string]string{"env": "prod"},
},
Aggregation: AggAvg,
Interval: time.Minute,
}, now)
if err != nil {
t.Fatalf("series old prod: %v", err)
}
var found bool
for _, p := range got {
if p.Count == 10 && math.Abs(p.Value-4.5) < 1e-9 {
found = true
}
if p.Count > 10 {
t.Fatalf("env=prod bucket contaminated by stage: %#v", p)
}
}
if !found {
t.Fatalf("env=prod old series not served correctly from rollup: %#v", got)
}
}
// TestDeleteSeriesRemovesTaggedTaskAcrossAgents verifies a task_id tag can be
// deleted across every agent, including stored rollups.
//
// TestDeleteSeriesRemovesTaggedTaskAcrossAgents 验证可以跨所有 agent 删除某个
// task_id 标签,包括已存储的 rollup。
func TestDeleteSeriesRemovesTaggedTaskAcrossAgents(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: 2 * time.Minute,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "ping.latency_ms", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
var batch []Point
for _, agent := range []string{"agent-001", "agent-002"} {
for i := 0; i < 60; i++ {
ts := base.Add(time.Duration(i) * time.Second)
batch = append(batch,
Point{MetricName: "ping.latency_ms", EntityID: agent, Timestamp: ts, Value: float64(20 + i), Tags: map[string]string{"task_id": "pingtask1"}},
Point{MetricName: "ping.latency_ms", EntityID: agent, Timestamp: ts, Value: float64(60 + i), Tags: map[string]string{"task_id": "pingtask2"}},
)
}
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := s.Compact(ctx, base.Add(time.Hour)); err != nil {
t.Fatalf("compact: %v", err)
}
if _, err := s.DeleteSeries(ctx, Query{
MetricName: "ping.latency_ms",
Tags: map[string]string{"task_id": "pingtask2"},
}); err != nil {
t.Fatalf("delete series: %v", err)
}
task2, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{
MetricName: "ping.latency_ms",
Start: base, End: base.Add(time.Minute),
Tags: map[string]string{"task_id": "pingtask2"},
},
Aggregation: AggAvg,
Interval: time.Minute,
}, time.Minute)
if err != nil {
t.Fatalf("rollup task2: %v", err)
}
if len(task2) != 0 {
t.Fatalf("deleted task still has rollup buckets: %#v", task2)
}
task1, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{
MetricName: "ping.latency_ms",
Start: base, End: base.Add(time.Minute),
Tags: map[string]string{"task_id": "pingtask1"},
},
Aggregation: AggAvg,
Interval: time.Minute,
}, time.Minute)
if err != nil {
t.Fatalf("rollup task1: %v", err)
}
if len(task1) != 1 || task1[0].Count != 120 {
t.Fatalf("remaining task should keep both agents, got %#v", task1)
}
}
// TestDeleteMetricRemovesRollups verifies metric deletion clears rollup rows.
//
// TestDeleteMetricRemovesRollups 验证删除指标会清理 rollup 行。
func TestDeleteMetricRemovesRollups(t *testing.T) {
ctx := context.Background()
s := newRollupStore(t, RollupPolicy{
Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}},
})
if err := s.CreateMetric(ctx, Definition{Name: "gone", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
if err := s.Write(ctx, Point{MetricName: "gone", EntityID: "agent-001", Timestamp: base, Value: 1}); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := s.Compact(ctx, base.Add(time.Hour)); err != nil {
t.Fatalf("compact: %v", err)
}
if err := s.DeleteMetric(ctx, "gone"); err != nil {
t.Fatalf("delete metric: %v", err)
}
rows, err := s.scanRollupRows(ctx, s.reader(), "gone", time.Minute)
if err != nil {
t.Fatalf("scan rollups: %v", err)
}
if len(rows) != 0 {
t.Fatalf("deleted metric still has rollups: %#v", rows)
}
}
// TestTagsFingerprintStable verifies equal tag maps fingerprint identically
// regardless of construction order, and different maps differ.
//
// TestTagsFingerprintStable 验证相同标签 map 无论构造顺序如何都会生成相同指纹,
// 而不同 map 会得到不同指纹。
func TestTagsFingerprintStable(t *testing.T) {
a := map[string]string{"region": "ap", "device_index": "3"}
b := map[string]string{"device_index": "3", "region": "ap"}
ha, _, err := tagsFingerprint(a)
if err != nil {
t.Fatalf("fp a: %v", err)
}
hb, _, err := tagsFingerprint(b)
if err != nil {
t.Fatalf("fp b: %v", err)
}
if ha != hb {
t.Fatalf("equal tag maps must fingerprint identically: %s vs %s", ha, hb)
}
hc, _, _ := tagsFingerprint(map[string]string{"region": "eu", "device_index": "3"})
if ha == hc {
t.Fatalf("different tag maps must fingerprint differently")
}
// nil and empty map agree.
hn, _, _ := tagsFingerprint(nil)
he, _, _ := tagsFingerprint(map[string]string{})
if hn != he {
t.Fatalf("nil and empty tag maps must agree: %s vs %s", hn, he)
}
}
+676
View File
@@ -0,0 +1,676 @@
package metric
import (
"context"
"errors"
"fmt"
"math"
"testing"
"time"
)
// newRollupStore opens an in-memory SQLite store with the given policy. A unique
// DSN per test keeps the shared-cache in-memory databases isolated.
//
// newRollupStore 使用给定策略打开内存 SQLite store;每个测试使用唯一 DSN,
// 避免 shared-cache 内存数据库相互影响。
func newRollupStore(t *testing.T, policy RollupPolicy) *Store {
t.Helper()
dsn := fmt.Sprintf("file:rollup-%d?mode=memory&cache=shared", time.Now().UnixNano())
store, err := Open(context.Background(), SQLite(dsn, WithRollupPolicy(policy)))
if err != nil {
t.Fatalf("open rollup store: %v", err)
}
t.Cleanup(func() { store.Close() })
return store
}
// TestArbitraryPercentileOverRaw verifies arbitrary percentile (pxx)
// aggregation end-to-end over raw points.
//
// TestArbitraryPercentileOverRaw 验证任意百分位(pxx)聚合在原始点路径上的
// 端到端行为。
func TestArbitraryPercentileOverRaw(t *testing.T) {
ctx := context.Background()
s := newMemStore(t)
if err := s.CreateMetric(ctx, Definition{Name: "lat", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
var batch []Point
for i := 1; i <= 100; i++ { // values 1..100 in one bucket
batch = append(batch, Point{MetricName: "lat", EntityID: "n1", Timestamp: base.Add(time.Duration(i) * time.Second), Value: float64(i)})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
for _, tc := range []struct {
agg Aggregation
want float64
}{
{Pxx(50), percentileSortedRange(1, 100, 0.50)},
{Pxx(90), percentileSortedRange(1, 100, 0.90)},
{Pxx(99), percentileSortedRange(1, 100, 0.99)},
{Pxx(99.9), percentileSortedRange(1, 100, 0.999)},
{Pxx(75), percentileSortedRange(1, 100, 0.75)},
} {
res, err := s.Aggregate(ctx, AggregateQuery{
Query: Query{MetricName: "lat", EntityID: "n1", Start: base, End: base.Add(time.Hour)},
Aggregation: tc.agg,
Interval: time.Hour,
})
if err != nil {
t.Fatalf("aggregate %s: %v", tc.agg, err)
}
if len(res) != 1 {
t.Fatalf("%s: expected 1 bucket, got %d", tc.agg, len(res))
}
if math.Abs(res[0].Value-tc.want) > 1e-9 {
t.Fatalf("%s: got %v want %v", tc.agg, res[0].Value, tc.want)
}
}
}
// percentileSortedRange computes an exact percentile for an integer range.
//
// percentileSortedRange 为整数范围计算精确百分位。
func percentileSortedRange(lo, hi int, q float64) float64 {
vals := make([]float64, 0, hi-lo+1)
for i := lo; i <= hi; i++ {
vals = append(vals, float64(i))
}
return percentileSorted(vals, q)
}
// TestPxxStringForms verifies arbitrary percentile aggregation names.
//
// TestPxxStringForms 验证任意百分位聚合名的字符串形式。
func TestPxxStringForms(t *testing.T) {
if Pxx(95) != AggP95 {
t.Fatalf("Pxx(95)=%q, want %q", Pxx(95), AggP95)
}
if Pxx(50) != AggP50 {
t.Fatalf("Pxx(50)=%q, want %q", Pxx(50), AggP50)
}
if Pxx(99.9) != Aggregation("p99.9") {
t.Fatalf("Pxx(99.9)=%q", Pxx(99.9))
}
// Out-of-range percentiles must be rejected by validation.
q := AggregateQuery{Query: Query{MetricName: "m", Start: time.Now(), End: time.Now().Add(time.Hour)}, Aggregation: Pxx(100), Interval: time.Minute}
if err := q.Validate(); !errors.Is(err, ErrInvalidArgument) {
t.Fatalf("p100 should be invalid, got %v", err)
}
}
// TestCompactBuildsFinestTier verifies Compact builds correct finest-tier
// rollup statistics from raw points.
//
// TestCompactBuildsFinestTier 验证 Compact 能从原始点构建正确的最细层
// rollup 统计。
func TestCompactBuildsFinestTier(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "m", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
// Two 1-minute buckets: bucket A has values 0..59, bucket B has 60..119.
var batch []Point
for i := 0; i < 120; i++ {
batch = append(batch, Point{MetricName: "m", EntityID: "n1", Timestamp: base.Add(time.Duration(i) * time.Second), Value: float64(i)})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
now := base.Add(10 * time.Minute)
written, err := s.Compact(ctx, now)
if err != nil {
t.Fatalf("compact: %v", err)
}
if written != 2 {
t.Fatalf("expected 2 rollup buckets written, got %d", written)
}
// AggregateRollup at the 1m resolution should reproduce per-bucket stats.
res, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{MetricName: "m", EntityID: "n1", Start: base, End: base.Add(10 * time.Minute)},
Aggregation: AggAvg,
Interval: time.Minute,
}, time.Minute)
if err != nil {
t.Fatalf("aggregate rollup: %v", err)
}
if len(res) != 2 {
t.Fatalf("expected 2 buckets, got %d", len(res))
}
if res[0].Value != 29.5 || res[0].Count != 60 {
t.Fatalf("bucket A avg/count wrong: %#v", res[0])
}
if res[1].Value != 89.5 || res[1].Count != 60 {
t.Fatalf("bucket B avg/count wrong: %#v", res[1])
}
// Sum/min/max derived from rollup must match raw.
for _, tc := range []struct {
agg Aggregation
wantA float64
wantB float64
}{
{AggSum, 1770, 5370}, // sum 0..59 and 60..119
{AggMin, 0, 60},
{AggMax, 59, 119},
} {
r, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{MetricName: "m", EntityID: "n1", Start: base, End: base.Add(10 * time.Minute)},
Aggregation: tc.agg,
Interval: time.Minute,
}, time.Minute)
if err != nil {
t.Fatalf("rollup %s: %v", tc.agg, err)
}
if r[0].Value != tc.wantA || r[1].Value != tc.wantB {
t.Fatalf("%s: got %v,%v want %v,%v", tc.agg, r[0].Value, r[1].Value, tc.wantA, tc.wantB)
}
}
}
// TestCompactCascadeFineToCoarse verifies a coarse tier composed from a fine
// tier matches a direct raw rollup.
//
// TestCompactCascadeFineToCoarse 验证由细层合成的粗层与直接从原始点计算的
// rollup 一致。
func TestCompactCascadeFineToCoarse(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
Tiers: []RollupTier{
{Interval: time.Minute, Retention: time.Hour},
{Interval: 5 * time.Minute, Retention: 24 * time.Hour},
},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "c", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
// 5 minutes of data, one point per second, value = second index 0..299.
var batch []Point
for i := 0; i < 300; i++ {
batch = append(batch, Point{MetricName: "c", EntityID: "n1", Timestamp: base.Add(time.Duration(i) * time.Second), Value: float64(i)})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := s.Compact(ctx, base.Add(time.Hour)); err != nil {
t.Fatalf("compact: %v", err)
}
// The single 5m coarse bucket should cover all 300 points.
res, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{MetricName: "c", EntityID: "n1", Start: base, End: base.Add(5 * time.Minute)},
Aggregation: AggAvg,
Interval: 5 * time.Minute,
}, 5*time.Minute)
if err != nil {
t.Fatalf("rollup: %v", err)
}
if len(res) != 1 || res[0].Count != 300 {
t.Fatalf("expected 1 coarse bucket covering 300 points, got %#v", res)
}
if math.Abs(res[0].Value-149.5) > 1e-9 { // avg of 0..299
t.Fatalf("coarse avg wrong: %v", res[0].Value)
}
// Percentile from the cascaded coarse digest should be close to the exact.
pres, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{MetricName: "c", EntityID: "n1", Start: base, End: base.Add(5 * time.Minute)},
Aggregation: Pxx(95),
Interval: 5 * time.Minute,
}, 5*time.Minute)
if err != nil {
t.Fatalf("rollup p95: %v", err)
}
exact := percentileSortedRange(0, 299, 0.95)
if math.Abs(pres[0].Value-exact)/exact > 0.02 {
t.Fatalf("cascaded p95 %v too far from exact %v", pres[0].Value, exact)
}
}
func TestCompactDoesNotOverwriteCoarseRollupWithPartialFineRows(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: 10 * time.Minute,
Tiers: []RollupTier{
{Interval: time.Minute, Retention: 11 * time.Minute},
{Interval: 5 * time.Minute, Retention: time.Hour},
},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "partial", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
var batch []Point
for i := 0; i < 300; i++ {
batch = append(batch, Point{MetricName: "partial", EntityID: "n1", Timestamp: base.Add(time.Duration(i) * time.Second), Value: 1})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := s.Compact(ctx, base.Add(15*time.Minute)); err != nil {
t.Fatalf("compact initial: %v", err)
}
query := AggregateQuery{
Query: Query{MetricName: "partial", EntityID: "n1", Start: base, End: base.Add(5*time.Minute - time.Nanosecond)},
Aggregation: AggCount,
Interval: 5 * time.Minute,
}
before, err := s.AggregateRollup(ctx, query, 5*time.Minute)
if err != nil {
t.Fatalf("rollup before: %v", err)
}
if len(before) != 1 || before[0].Count != 300 {
t.Fatalf("expected initial complete coarse bucket, got %#v", before)
}
if _, err := s.Compact(ctx, base.Add(18*time.Minute)); err != nil {
t.Fatalf("compact after fine retention moves: %v", err)
}
after, err := s.AggregateRollup(ctx, query, 5*time.Minute)
if err != nil {
t.Fatalf("rollup after: %v", err)
}
if len(after) != 1 || after[0].Count != 300 {
t.Fatalf("coarse bucket was overwritten from partial fine rows: %#v", after)
}
if err := s.Write(ctx, Point{MetricName: "partial", EntityID: "n1", Timestamp: base.Add(4*time.Minute + 30*time.Second), Value: 1}); err != nil {
t.Fatalf("write late: %v", err)
}
if _, err := s.Compact(ctx, base.Add(19*time.Minute)); err != nil {
t.Fatalf("compact late: %v", err)
}
late, err := s.AggregateRollup(ctx, query, 5*time.Minute)
if err != nil {
t.Fatalf("rollup after late: %v", err)
}
if len(late) != 1 || late[0].Count != 301 {
t.Fatalf("late fine delta should merge into retained coarse bucket: %#v", late)
}
if _, err := s.Compact(ctx, base.Add(20*time.Minute)); err != nil {
t.Fatalf("compact late again: %v", err)
}
again, err := s.AggregateRollup(ctx, query, 5*time.Minute)
if err != nil {
t.Fatalf("rollup after repeated compact: %v", err)
}
if len(again) != 1 || again[0].Count != 301 {
t.Fatalf("repeated compact should not merge the same late delta twice: %#v", again)
}
}
func TestCompactMergesLateFineDeltaLargerThanCoarseBucket(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: 10 * time.Minute,
Tiers: []RollupTier{
{Interval: time.Minute, Retention: 11 * time.Minute},
{Interval: 5 * time.Minute, Retention: time.Hour},
},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "latebig", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
if err := s.Write(ctx, Point{MetricName: "latebig", EntityID: "n1", Timestamp: base.Add(10 * time.Second), Value: 1}); err != nil {
t.Fatalf("write original: %v", err)
}
if _, err := s.Compact(ctx, base.Add(15*time.Minute)); err != nil {
t.Fatalf("compact initial: %v", err)
}
if _, err := s.Compact(ctx, base.Add(18*time.Minute)); err != nil {
t.Fatalf("compact expire fine: %v", err)
}
for _, ts := range []time.Time{base.Add(20 * time.Second), base.Add(30 * time.Second)} {
if err := s.Write(ctx, Point{MetricName: "latebig", EntityID: "n1", Timestamp: ts, Value: 1}); err != nil {
t.Fatalf("write late: %v", err)
}
}
if _, err := s.Compact(ctx, base.Add(19*time.Minute)); err != nil {
t.Fatalf("compact late: %v", err)
}
got, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{MetricName: "latebig", EntityID: "n1", Start: base, End: base.Add(5*time.Minute - time.Nanosecond)},
Aggregation: AggCount,
Interval: 5 * time.Minute,
}, 5*time.Minute)
if err != nil {
t.Fatalf("rollup: %v", err)
}
if len(got) != 1 || got[0].Count != 3 {
t.Fatalf("late delta should merge even when larger than existing bucket, got %#v", got)
}
}
// TestRetentionDropsRawButPercentileSurvives verifies the TSDB property that
// raw data can age out while percentiles remain answerable from rollups.
//
// TestRetentionDropsRawButPercentileSurvives 验证 TSDB 特性:原始数据可以过期,
// 但百分位仍能从 rollup 中回答。
func TestRetentionDropsRawButPercentileSurvives(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: 2 * time.Minute,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 365 * 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "old", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
var batch []Point
for i := 0; i < 60; i++ { // one minute bucket, values 0..59
batch = append(batch, Point{MetricName: "old", EntityID: "n1", Timestamp: base.Add(time.Duration(i) * time.Second), Value: float64(i)})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
// Compact far in the future: raw (older than 2m) is deleted, rollup kept.
now := base.Add(48 * time.Hour)
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact: %v", err)
}
// Raw must be gone.
raw, err := s.Query(ctx, Query{MetricName: "old", EntityID: "n1", Start: base, End: base.Add(time.Hour)})
if err != nil {
t.Fatalf("query raw: %v", err)
}
if len(raw) != 0 {
t.Fatalf("expected raw points deleted by retention, got %d", len(raw))
}
// Percentile must still be answerable from the surviving rollup.
res, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{MetricName: "old", EntityID: "n1", Start: base, End: base.Add(time.Hour)},
Aggregation: Pxx(90),
Interval: time.Minute,
}, time.Minute)
if err != nil {
t.Fatalf("rollup p90 after retention: %v", err)
}
if len(res) != 1 {
t.Fatalf("expected 1 surviving rollup bucket, got %d", len(res))
}
exact := percentileSortedRange(0, 59, 0.90)
if math.Abs(res[0].Value-exact)/exact > 0.05 {
t.Fatalf("surviving p90 %v too far from exact %v", res[0].Value, exact)
}
}
// TestCompactMergesLateRawIntoExpiredRollup verifies a late-arriving raw point
// for an already-retained rollup bucket is folded into the stored rollup instead
// of replacing the bucket with only the late sample.
//
// TestCompactMergesLateRawIntoExpiredRollup 验证已过原始保留期的 rollup 桶收到
// 迟到 raw 点时,会把迟到样本合入已有 rollup,而不是用迟到样本覆盖整桶。
func TestCompactMergesLateRawIntoExpiredRollup(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: 2 * time.Minute,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "late", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
var batch []Point
for i := 0; i < 60; i++ {
batch = append(batch, Point{MetricName: "late", EntityID: "n1", Timestamp: base.Add(time.Duration(i) * time.Second), Value: float64(i)})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := s.Compact(ctx, base.Add(time.Hour)); err != nil {
t.Fatalf("compact initial: %v", err)
}
if err := s.Write(ctx, Point{
MetricName: "late",
EntityID: "n1",
Timestamp: base.Add(30*time.Second + 500*time.Millisecond),
Value: 1000,
}); err != nil {
t.Fatalf("write late: %v", err)
}
if _, err := s.Compact(ctx, base.Add(2*time.Hour)); err != nil {
t.Fatalf("compact late: %v", err)
}
res, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{MetricName: "late", EntityID: "n1", Start: base, End: base.Add(time.Minute)},
Aggregation: AggSum,
Interval: time.Minute,
}, time.Minute)
if err != nil {
t.Fatalf("rollup sum: %v", err)
}
if len(res) != 1 {
t.Fatalf("expected 1 rollup bucket, got %d", len(res))
}
if res[0].Count != 61 || res[0].Value != 2770 {
t.Fatalf("late point should merge into existing bucket count=61 sum=2770, got %#v", res[0])
}
}
// TestSeriesRoutesByAge verifies Series auto-routes between raw and rollup data
// based on the query window's age.
//
// TestSeriesRoutesByAge 验证 Series 会根据查询窗口的新旧程度在原始数据和
// rollup 数据之间自动路由。
func TestSeriesRoutesByAge(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: time.Hour,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 30 * 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "sr", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
now := time.Date(2026, 6, 18, 12, 0, 0, 0, time.UTC)
// Recent window (10 min ago) stays raw; old window (2 days ago) uses rollup.
recent := now.Add(-10 * time.Minute)
old := now.Add(-48 * time.Hour)
var batch []Point
for i := 0; i < 60; i++ {
batch = append(batch, Point{MetricName: "sr", EntityID: "n1", Timestamp: recent.Add(time.Duration(i) * time.Second), Value: float64(i)})
batch = append(batch, Point{MetricName: "sr", EntityID: "n1", Timestamp: old.Add(time.Duration(i) * time.Second), Value: float64(100 + i)})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact: %v", err)
}
// Recent window: Series should match raw Aggregate exactly (full fidelity).
recentQ := AggregateQuery{
Query: Query{MetricName: "sr", EntityID: "n1", Start: recent.Add(-time.Minute), End: recent.Add(2 * time.Minute)},
Aggregation: AggAvg, Interval: time.Minute,
}
got, err := s.Series(ctx, recentQ, now)
if err != nil {
t.Fatalf("series recent: %v", err)
}
want, err := s.Aggregate(ctx, recentQ)
if err != nil {
t.Fatalf("aggregate recent: %v", err)
}
if len(got) != len(want) {
t.Fatalf("recent: series=%d raw=%d buckets", len(got), len(want))
}
for i := range got {
if !got[i].Bucket.Equal(want[i].Bucket) || got[i].Value != want[i].Value {
t.Fatalf("recent bucket %d differs: series=%#v raw=%#v", i, got[i], want[i])
}
}
// Old window: raw is gone, so Series must answer from the rollup tier.
oldRaw, err := s.Query(ctx, Query{MetricName: "sr", EntityID: "n1", Start: old, End: old.Add(time.Hour)})
if err != nil {
t.Fatalf("query old raw: %v", err)
}
if len(oldRaw) != 0 {
t.Fatalf("expected old raw aged out, got %d", len(oldRaw))
}
oldQ := AggregateQuery{
Query: Query{MetricName: "sr", EntityID: "n1", Start: old.Add(-time.Minute), End: old.Add(2 * time.Minute)},
Aggregation: AggAvg, Interval: time.Minute,
}
oldGot, err := s.Series(ctx, oldQ, now)
if err != nil {
t.Fatalf("series old: %v", err)
}
var found bool
for _, p := range oldGot {
if p.Count == 60 && math.Abs(p.Value-129.5) < 1e-9 { // avg of 100..159
found = true
}
}
if !found {
t.Fatalf("old window not served from rollup correctly: %#v", oldGot)
}
}
// TestSeriesAcrossRetentionIncludesUncompactedRecentRaw verifies a query that
// spans the raw-retention boundary uses rollups for old buckets and raw data for
// recent buckets that may not have been compacted yet.
//
// TestSeriesAcrossRetentionIncludesUncompactedRecentRaw 验证跨原始保留期边界的查询
// 会用 rollup 回答旧桶,并包含可能尚未 compact 的近期 raw 桶。
func TestSeriesAcrossRetentionIncludesUncompactedRecentRaw(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{
RawRetention: time.Hour,
Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}},
}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "hybrid", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
now := time.Date(2026, 6, 18, 12, 0, 0, 0, time.UTC)
old := now.Add(-2 * time.Hour)
var batch []Point
for i := 0; i < 60; i++ {
batch = append(batch, Point{MetricName: "hybrid", EntityID: "n1", Timestamp: old.Add(time.Duration(i) * time.Second), Value: float64(i)})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write old: %v", err)
}
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact old: %v", err)
}
recent := now.Add(-10 * time.Minute)
if err := s.Write(ctx, Point{MetricName: "hybrid", EntityID: "n1", Timestamp: recent, Value: 500}); err != nil {
t.Fatalf("write recent: %v", err)
}
got, err := s.Series(ctx, AggregateQuery{
Query: Query{
MetricName: "hybrid",
EntityID: "n1",
Start: old.Add(-time.Minute),
End: recent.Add(time.Minute),
},
Aggregation: AggCount,
Interval: time.Minute,
}, now)
if err != nil {
t.Fatalf("series: %v", err)
}
var oldFound, recentFound bool
for _, p := range got {
if p.Bucket.Equal(alignTime(old, time.Minute)) && p.Count == 60 {
oldFound = true
}
if p.Bucket.Equal(alignTime(recent, time.Minute)) && p.Count == 1 {
recentFound = true
}
}
if !oldFound || !recentFound {
t.Fatalf("series should include old rollup and recent raw buckets (old=%v recent=%v): %#v", oldFound, recentFound, got)
}
}
// TestRollupPolicyValidate verifies rollup policy validation rules.
//
// TestRollupPolicyValidate 验证 rollup 策略校验规则。
func TestRollupPolicyValidate(t *testing.T) {
ok := RollupPolicy{
RawRetention: 10 * time.Minute,
Tiers: []RollupTier{
{Interval: time.Minute, Retention: time.Hour},
{Interval: 5 * time.Minute, Retention: 24 * time.Hour},
},
}
if err := ok.Validate(); err != nil {
t.Fatalf("valid policy rejected: %v", err)
}
bad := []RollupPolicy{
{Tiers: []RollupTier{{Interval: 0, Retention: time.Hour}}}, // zero interval
{Tiers: []RollupTier{{Interval: time.Minute, Retention: 0}}}, // zero retention
{Tiers: []RollupTier{{Interval: time.Minute, Retention: time.Hour}, {Interval: 90 * time.Second, Retention: time.Hour}}}, // not a multiple
{Tiers: []RollupTier{{Interval: time.Minute, Retention: 2 * time.Hour}, {Interval: 5 * time.Minute, Retention: time.Hour}}}, // coarse retention < fine
{RawRetention: 30 * time.Second, Tiers: []RollupTier{{Interval: time.Minute, Retention: time.Hour}}}, // raw retention < 2x finest
}
for i, p := range bad {
if err := p.Validate(); !errors.Is(err, ErrInvalidArgument) {
t.Fatalf("bad policy %d should be invalid, got %v", i, err)
}
}
}
// TestCompactIdempotent verifies Compact is idempotent for unchanged windows.
//
// TestCompactIdempotent 验证 Compact 对未变化窗口保持幂等。
func TestCompactIdempotent(t *testing.T) {
ctx := context.Background()
policy := RollupPolicy{Tiers: []RollupTier{{Interval: time.Minute, Retention: 24 * time.Hour}}}
s := newRollupStore(t, policy)
if err := s.CreateMetric(ctx, Definition{Name: "idem", Type: TypeGauge}); err != nil {
t.Fatalf("create: %v", err)
}
base := time.Date(2026, 6, 18, 0, 0, 0, 0, time.UTC)
var batch []Point
for i := 0; i < 120; i++ {
batch = append(batch, Point{MetricName: "idem", EntityID: "n1", Timestamp: base.Add(time.Duration(i) * time.Second), Value: float64(i)})
}
if err := s.WriteBatch(ctx, batch); err != nil {
t.Fatalf("write: %v", err)
}
now := base.Add(10 * time.Minute)
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact 1: %v", err)
}
r1, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{MetricName: "idem", EntityID: "n1", Start: base, End: base.Add(10 * time.Minute)},
Aggregation: AggAvg, Interval: time.Minute,
}, time.Minute)
if err != nil {
t.Fatalf("rollup 1: %v", err)
}
if _, err := s.Compact(ctx, now); err != nil {
t.Fatalf("compact 2: %v", err)
}
r2, err := s.AggregateRollup(ctx, AggregateQuery{
Query: Query{MetricName: "idem", EntityID: "n1", Start: base, End: base.Add(10 * time.Minute)},
Aggregation: AggAvg, Interval: time.Minute,
}, time.Minute)
if err != nil {
t.Fatalf("rollup 2: %v", err)
}
if len(r1) != len(r2) {
t.Fatalf("idempotency broken: %d vs %d buckets", len(r1), len(r2))
}
for i := range r1 {
if r1[i].Value != r2[i].Value || r1[i].Count != r2[i].Count {
t.Fatalf("idempotency broken at %d: %#v vs %#v", i, r1[i], r2[i])
}
}
}
+1105
View File
File diff suppressed because it is too large. Load diff
+146
View File
@@ -0,0 +1,146 @@
package metric
import (
"context"
"errors"
"math"
"os"
"path/filepath"
"testing"
"time"
)
// TestSQLiteStoreWriteQueryAggregate verifies SQLite write, query, aggregate, and stats.
//
// TestSQLiteStoreWriteQueryAggregate 验证 SQLite 写入、查询、聚合和统计。
func TestSQLiteStoreWriteQueryAggregate(t *testing.T) {
ctx := context.Background()
store, err := Open(ctx, SQLite("file:test-metric?mode=memory&cache=shared"))
if err != nil {
t.Fatalf("open sqlite store: %v", err)
}
defer store.Close()
if err := store.CreateMetric(ctx, Definition{Name: "cpu.usage", Type: TypeGauge, Unit: "%"}); err != nil {
t.Fatalf("create metric: %v", err)
}
base := time.Date(2026, 6, 18, 10, 0, 0, 0, time.UTC)
points := []Point{
{MetricName: "cpu.usage", EntityID: "server-1", Timestamp: base, Value: 10, Tags: map[string]string{"region": "ap"}},
{MetricName: "cpu.usage", EntityID: "server-1", Timestamp: base.Add(time.Minute), Value: 20, Tags: map[string]string{"region": "ap"}},
{MetricName: "cpu.usage", EntityID: "server-1", Timestamp: base.Add(2 * time.Minute), Value: 30, Tags: map[string]string{"region": "ap"}},
{MetricName: "cpu.usage", EntityID: "server-2", Timestamp: base.Add(time.Minute), Value: 99, Tags: map[string]string{"region": "eu"}},
}
if err := store.WriteBatch(ctx, points); err != nil {
t.Fatalf("write batch: %v", err)
}
got, err := store.Query(ctx, Query{
MetricName: "cpu.usage",
EntityID: "server-1",
Start: base.Add(-time.Second),
End: base.Add(3 * time.Minute),
Tags: map[string]string{"region": "ap"},
})
if err != nil {
t.Fatalf("query: %v", err)
}
if len(got) != 3 {
t.Fatalf("expected 3 points, got %d", len(got))
}
if got[0].Value != 10 || got[2].Value != 30 {
t.Fatalf("unexpected ordered values: %#v", got)
}
agg, err := store.Aggregate(ctx, AggregateQuery{
Query: Query{
MetricName: "cpu.usage",
EntityID: "server-1",
Start: base,
End: base.Add(3 * time.Minute),
},
Aggregation: AggAvg,
Interval: 2 * time.Minute,
})
if err != nil {
t.Fatalf("aggregate: %v", err)
}
if len(agg) != 2 {
t.Fatalf("expected 2 aggregate buckets, got %d", len(agg))
}
if agg[0].Value != 15 || agg[0].Count != 2 {
t.Fatalf("unexpected first aggregate: %#v", agg[0])
}
stats, err := store.Stats(ctx, Query{
MetricName: "cpu.usage",
EntityID: "server-1",
Start: base,
End: base.Add(3 * time.Minute),
})
if err != nil {
t.Fatalf("stats: %v", err)
}
if stats.Count != 3 || stats.Avg != 20 || stats.P95 != 29 {
t.Fatalf("unexpected stats: %#v", stats)
}
}
func TestWriteRejectsNonFiniteValues(t *testing.T) {
ctx := context.Background()
store, err := Open(ctx, SQLite("file:test-non-finite?mode=memory&cache=shared"))
if err != nil {
t.Fatalf("open sqlite store: %v", err)
}
defer store.Close()
for _, value := range []float64{math.NaN(), math.Inf(1), math.Inf(-1)} {
err := store.Write(ctx, Point{
MetricName: "bad",
EntityID: "server-1",
Timestamp: time.Now(),
Value: value,
})
if !errors.Is(err, ErrInvalidArgument) {
t.Fatalf("expected ErrInvalidArgument for %v, got %v", value, err)
}
}
}
// TestSQLiteInDirCreatesDirectoryAndAppliesPragmas verifies SQLite file setup and PRAGMAs.
//
// TestSQLiteInDirCreatesDirectoryAndAppliesPragmas 验证 SQLite 文件初始化和 PRAGMA 设置。
func TestSQLiteInDirCreatesDirectoryAndAppliesPragmas(t *testing.T) {
ctx := context.Background()
dir := filepath.Join(t.TempDir(), "metrics")
store, err := Open(ctx, SQLiteInDir(
dir,
WithSQLiteProfile(SQLiteProfilePerformance),
WithSQLiteCacheSizeKB(32*1024),
))
if err != nil {
t.Fatalf("open sqlite dir store: %v", err)
}
defer store.Close()
if _, err := os.Stat(filepath.Join(dir, "metrics.db")); err != nil {
t.Fatalf("expected sqlite database file to be created: %v", err)
}
var journalMode string
if err := store.db.QueryRowContext(ctx, "PRAGMA journal_mode").Scan(&journalMode); err != nil {
t.Fatalf("query journal mode: %v", err)
}
if journalMode != "wal" {
t.Fatalf("expected WAL journal mode, got %q", journalMode)
}
var synchronous int
if err := store.db.QueryRowContext(ctx, "PRAGMA synchronous").Scan(&synchronous); err != nil {
t.Fatalf("query synchronous: %v", err)
}
if synchronous != 0 {
t.Fatalf("expected performance profile synchronous=OFF(0), got %d", synchronous)
}
}
Loaded 100 of 259 files, more files were not shown because too many files have changed in this diff. Show more