Commit graph

119 commits

Author SHA1 Message Date
4050de5b4e
fc-common: GitHub rate limit state extraction
Extract `X-RateLimit-*` headers from responses and calculate an adaptive
delay. Minimum delay is 1 seconds to prevent division by 0.

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ib35b0d0e720098e2c68ced88a8821c7b6a6a6964
2026-02-28 23:58:40 +03:00
f5c16aef83
fc-common: add unsupported_timeout for queue runner
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I76805c31bbfc11e0a596c6b3b88c52c06a6a6964
2026-02-28 23:58:39 +03:00
b43a11756a
fc-common: add unsupported_timeout for queue runner
Hydra maxUnsupportedTime compatibility. DUration is optional, defaults
to `None` and is configurable through `fc.toml` under
`queue_runner.unsupported_timeout` key. E.g., "1h".

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I299722e2a3865c873e212a7615b97b806a6a6964
2026-02-28 23:58:38 +03:00
609ac53c3f
fc-queue-server: match output names by path instead of index
Now creates a path-to-name lookup `HashMap` from the outputs JSON and
matches each output path to its correct name. Both the `build_outputs`
table insert and GC root registration loops now use the same lookup
to ensure we're consistent.

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Id9f4ce0be6131a7c1a8ce6775ab249db6a6a6964
2026-02-28 23:58:37 +03:00
ec28069b69
fc-queue-runner: store build outputs in normalized table
Replaces JSON blob storage with BuildOutput records and parse derivation
outputs after successful build, then INSERT into `build_outputs` per
output. 

Warnings are logged on storage failures, but it's not fatal.

- Parse derivation outputs after successful build
- INSERT into build_outputs table per output
- Log warnings on storage failures (non-fatal)

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I30120a5ee4aea1bdb170987b22ddc2df6a6a6964
2026-02-28 23:56:25 +03:00
959aba0933
fc-common: use btree index for build_outputs.path in db schema
Btree supports NULL values and provides better flexibility
than hash index for nullable columns.

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ida28ef6f88683c360e6f405efca435af6a6a6964
2026-02-28 23:56:24 +03:00
01cd4439aa
fc-common: add BuildOutput model; implement CRUD operations
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Iecbe7a5561caf7bf7f770a277b11f3816a6a6964
2026-02-28 23:56:23 +03:00
b1a7233a05
fc-common: add build_outputs table to base schema
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I1a9c6cb82967fe73aa403e3656c7cab96a6a6964
2026-02-28 23:56:22 +03:00
1336f998bf
fc-queue-runner: dispatch running status on build start
Send "running" commit status when builds are claimed. Also fixes
missing completion notifications for aggregate builds.

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I3b4dab8e30d6e278b7d38e2222a800a56a6a6964
2026-02-28 17:38:00 +03:00
1e28c31077
fc-evaluator: dispatch pending status on build creation
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I534a8dde536b6e0537d7c3c44c3ba0146a6a6964
2026-02-28 17:37:59 +03:00
85a43c8ca0
fc-common: add commit status lifecycle notifications
Helpers for sending forge status updates when builds are created and
claimed.

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Iddca8b33fc31a9e23968752ef112e0d26a6a6964
2026-02-28 17:37:58 +03:00
a127f3f62c
treewide: address all clippy lints
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I5cf55cc4cb558c3f9f764c71224e87176a6a6964
2026-02-28 17:37:53 +03:00
21446c6dcb
fc-queue-runner: implement persistent notification retry queue with exponential backoff
Adds a `notification_tasks` table and a background worker to (hopefully
reliably) deliver webhooks, git status updates, and e-mail notifications
with automatic retry on transient failures.

This was one of the critical gaps, finally done.

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I794967c66958658c4d8aed40793d67f96a6a6964
2026-02-28 12:18:18 +03:00
d0ffa5d9e5
fc-server: implent proper rate limiting with token bucket algorithm; fix rate_limit_rps
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I68237ff6216337eba1afa8e8606d545b6a6a6964
2026-02-28 12:18:16 +03:00
d541b7ebbf
nix: simplify tests
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I02126573ee9573fd7a1e5a12e42dd02d6a6a6964
2026-02-28 12:18:14 +03:00
e7425e0abf
fc-common: consolidate database migrations; simplify
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ia808d76241cec6e8760d87443bb0dc976a6a6964
2026-02-28 12:18:13 +03:00
23a4a8e348
fc-common: format
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I946272ee6563f5bca0844c5a25ba08f66a6a6964
2026-02-28 12:18:07 +03:00
015360ffcf
fc-queue-runner: integrate GC pinning and machine health tracking
`gc_loop` queries pinned build IDs before cleanup. `try_remote_build`
calls `record_success`/`record_failure` per builder.

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ia8e20ead3c83b78d787dba518c382f9a6a6a6964
2026-02-28 12:18:06 +03:00
5410fdc044
fc-server: add keep flag toggle and nixexprs.tar.xz channel endpoint
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I3411f76548f8061e835631a7928f899d6a6a6964
2026-02-28 12:18:05 +03:00
5b472a2f57
fc-common: add GC pinning and machine health infrastructure
Migration 017 adds `builds.keep`, `jobsets.keep_nr`, and health tracking
columns to `remote_builders`. Repo layer implements `set_keep`,
`list_pinned_ids`, `record_failure` with exponential backoff,
`record_success`, and `find_for_system` filtering of disabled builders.
GC root cleanup now skips pinned builds.

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ibba121de3dc42f71204e3a8f5776aa8b6a6a6964
2026-02-28 12:18:04 +03:00
25699e5e97
queue-runner: make fair-share assertion unconditional
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: If983d379a5a369c547dff61b57f3a9a36a6a6964
2026-02-28 12:18:03 +03:00
3716a34972
fc-queue-runner: plumb worker_count into fair-share scheduling
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I1f7f295bd7c2cbf46b64b22a3417ccc06a6a6964
2026-02-28 12:18:02 +03:00
4100ac54c2
fc-common: implement deficit-based fair-share scheduling in list_pending
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ic1345cfdf712aa6ee6f0eeae45b3e62b6a6a6964
2026-02-28 12:17:51 +03:00
0590b6c720
fc-queue-runner: add per-build cancellation tests
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: If9c4840d87615ab0d6cf81281583aa096a6a6964
2026-02-16 23:42:46 +03:00
f8586a7f3c
fc-queue-runner: implement per-build cancellation via CancellationToken
Adds an `ActiveBuild` registry (DashMap of `<Uuid, CancellationToken>`)
to `WorkerPool` and get `dispatch()` to create a per-build token to race
`run_build` against it via Tokio's `select!`.

The `cancel_checker_loop` then polls the DB every N seconds (currently 2)
for builds cancelled while running, and triggers their tokens.

Existing `kill_on_drop(true) on `nix build` processes handles
subprocess cleanup when the future is dropped. Thank you past me for
your insight.

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ic8af58e92972c7d5d104d9c717e9217d6a6a6964
2026-02-16 23:42:45 +03:00
d401177902
fc-common: add get_cancelled_among query for cancel-checker polling
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Iabc5fc932c8d7f1d19a80a27965524136a6a6964
2026-02-16 23:42:44 +03:00
9efba1bbc7
fc-queue-runner: integrate failed paths cache in build dispatch
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I418f007368916de1320dd56f425a5d8f6a6a6964
2026-02-16 23:42:42 +03:00
65a6fd853d
fc-common: add failed paths cache infrastructure
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I35f9bfb044160151cf73c43ed9ada3476a6a6964
2026-02-16 23:42:41 +03:00
4c56b192f0
fc-common: fix BuildStatus sqlx rename to snake_case
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I00434e0156b3dc1dfd26699e4b103bd46a6a6964
2026-02-16 23:42:40 +03:00
235c9834b7
fc-evaulator: allow fail-fast behaviour
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I1da41766f1c499347279c41f2316f4376a6a6964
2026-02-16 23:42:39 +03:00
49638d5d14
fc-queue-runner: use LISTEN/NOTIFY for reactive wakeups
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I3b6f0f5eff05caf7a04a9da7de8b558f6a6a6964
2026-02-16 23:42:38 +03:00
edaf4313e9
fc-evaluator: use LISTEN/NOTIFY for reactive wakeups
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ibee7506a9a5ffa008c41fae8e9758db66a6a6964
2026-02-16 23:42:37 +03:00
e274389d12
fc-common: add PostgreSQL LISTEN/NOTIFY infrastructure
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Iffdb2fa758825e8c5d5791bf4fb15c8e6a6a6964
2026-02-16 23:42:36 +03:00
541cd7832f
treewide: replace BuildStatus::Completed with BuildStatus::Succeeded
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I965dfaca211f9fde527a84a54ae972576a6a6964
2026-02-16 23:42:21 +03:00
f6fcf49aa7
tests: replace BuildStatus::Completed with BuildStatus::Succeeded
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ia12f816c203d6ce51485788d0414894f6a6a6964
2026-02-16 13:02:27 +03:00
7378d618b1
fc-server: update web UI to display extended build statuses
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I8cf767e7ed34153eedf6d1fd56a6c2016a6a6964
2026-02-16 13:02:26 +03:00
85970e249c
fc-common: extend BuildStatusFilter with all status variants
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I58c8ebbe937035c2398af4eea0eaa3cf6a6a6964
2026-02-16 13:02:25 +03:00
5b7648220c
fc-common: update notifications to handle all BuildStatus variants
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I99db557ae204f3a4ffec6be70386ecc16a6a6964
2026-02-16 13:02:24 +03:00
5f09a46d29
fc-queue-runner: map exit codes to extended BuildStatus variants
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ibaa4de9e2c789931df8c67a53528829a6a6a6964
2026-02-16 13:02:23 +03:00
f4772036ce
fc-common: add database migration for extended build status codes
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I0c13eda985d634e63189ba6907e488ae6a6a6964
2026-02-16 13:02:22 +03:00
2b763833d4
fc-common: add extended BuildStatus enum with new status code
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: If00cae8b2f7d4a7ad2d64bfe70a5c4186a6a6964
2026-02-16 13:02:21 +03:00
a5768d46eb
fc-common: allow configuring url schemes to allow for testing
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I99912d7c45f1a4664d4823ddd793b5af6a6a6964
2026-02-16 00:08:23 +03:00
b745550011
various: tiny cleanup
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I312766a6178be3898b51c2863f502bb06a6a6964
2026-02-15 23:37:53 +03:00
f7081317ee
various: reuse HTTP client; eliminate intermediate string allocations; add tests
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I18b89e1aae78a400a89c9d89423ce1da6a6a6964
2026-02-15 23:37:52 +03:00
38ed7faee2
various: replace silent error discards with logged warnings
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I465d760b5330980270b64b4a89abc09f6a6a6964
2026-02-15 23:37:51 +03:00
9bbc1754d9
fc-server: session cleanup; conditional Secure cookie; kill_on_drop on NAR processes
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ie7b9002bbdc0ad91bb041f89979881956a6a6964
2026-02-15 23:37:50 +03:00
aa4ebf2f5b
various: harden input validation; add SSRF protection; fix default API key role
Default API key role was "admin", which was something that I forgot to fix during testing. We
change it to "read-only". 

Additionally repository URLs now reject `file://` scheme (another testing artifact) localhost,
private IP ranges, and cloud metadata endpoints. Nix expressions reject path traversal (`..`)
and absolute paths. Validation is called at the evaluator endtrypoint before command construction.

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I35729c6aa9ec4ff8d1ea19bd57ea93646a6a6964
2026-02-15 23:37:49 +03:00
a2b638d4db
nix: attempt to fix VM tests; general cleanup
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I65f6909ef02ab4599f5b0bbc0930367e6a6a6964
2026-02-15 23:37:49 +03:00
83071514a3
fc-server: add metrics visualization dashboard
Adds a /metrics page with Chart.js charts, which requires an annoying
CDN fetch but until I figure out a good method of fetching things during
build it's our best bet. I've pinned the thing so it's probably good.

The page displays build counts, duration percentiles and system
distribution. Time range and project filters are included, and the
metrics page is linked from navigation.

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I99059594c29a9b35d2fd4d140628d6f46a6a6964
2026-02-14 18:08:21 +03:00
537fa823a7
fc-server: add timeseries API endpoints for visualisation
Adds:

- `build_stats_timeseries()` for build counts over time
- `duration_percentiles_timeseries()` for P50/P95/P99
- `system_distribution()` for per-system counts

and of course,  REST endpoints for `/api/v1/metrics/timeseries/*`. This
is a good start for data visualisation. My professors would be proud.

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I3c0b9d14592945a661af77b7edf338a86a6a6964
2026-02-14 18:08:20 +03:00