a127f3f62c
treewide: address all clippy lints
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I5cf55cc4cb558c3f9f764c71224e87176a6a6964
2026-02-28 17:37:53 +03:00
21446c6dcb
fc-queue-runner: implement persistent notification retry queue with exponential backoff
...
Adds a `notification_tasks` table and a background worker to (hopefully
reliably) deliver webhooks, git status updates, and e-mail notifications
with automatic retry on transient failures.
This was one of the critical gaps, finally done.
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I794967c66958658c4d8aed40793d67f96a6a6964
2026-02-28 12:18:18 +03:00
015360ffcf
fc-queue-runner: integrate GC pinning and machine health tracking
...
`gc_loop` queries pinned build IDs before cleanup. `try_remote_build`
calls `record_success`/`record_failure` per builder.
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ia8e20ead3c83b78d787dba518c382f9a6a6a6964
2026-02-28 12:18:06 +03:00
5b472a2f57
fc-common: add GC pinning and machine health infrastructure
...
Migration 017 adds `builds.keep`, `jobsets.keep_nr`, and health tracking
columns to `remote_builders`. Repo layer implements `set_keep`,
`list_pinned_ids`, `record_failure` with exponential backoff,
`record_success`, and `find_for_system` filtering of disabled builders.
GC root cleanup now skips pinned builds.
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ibba121de3dc42f71204e3a8f5776aa8b6a6a6964
2026-02-28 12:18:04 +03:00
25699e5e97
queue-runner: make fair-share assertion unconditional
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: If983d379a5a369c547dff61b57f3a9a36a6a6964
2026-02-28 12:18:03 +03:00
3716a34972
fc-queue-runner: plumb worker_count into fair-share scheduling
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I1f7f295bd7c2cbf46b64b22a3417ccc06a6a6964
2026-02-28 12:18:02 +03:00
0590b6c720
fc-queue-runner: add per-build cancellation tests
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: If9c4840d87615ab0d6cf81281583aa096a6a6964
2026-02-16 23:42:46 +03:00
f8586a7f3c
fc-queue-runner: implement per-build cancellation via CancellationToken
...
Adds an `ActiveBuild` registry (DashMap of `<Uuid, CancellationToken>`)
to `WorkerPool` and get `dispatch()` to create a per-build token to race
`run_build` against it via Tokio's `select!`.
The `cancel_checker_loop` then polls the DB every N seconds (currently 2)
for builds cancelled while running, and triggers their tokens.
Existing `kill_on_drop(true) on `nix build` processes handles
subprocess cleanup when the future is dropped. Thank you past me for
your insight.
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ic8af58e92972c7d5d104d9c717e9217d6a6a6964
2026-02-16 23:42:45 +03:00
9efba1bbc7
fc-queue-runner: integrate failed paths cache in build dispatch
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I418f007368916de1320dd56f425a5d8f6a6a6964
2026-02-16 23:42:42 +03:00
235c9834b7
fc-evaulator: allow fail-fast behaviour
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I1da41766f1c499347279c41f2316f4376a6a6964
2026-02-16 23:42:39 +03:00
49638d5d14
fc-queue-runner: use LISTEN/NOTIFY for reactive wakeups
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I3b6f0f5eff05caf7a04a9da7de8b558f6a6a6964
2026-02-16 23:42:38 +03:00
5f09a46d29
fc-queue-runner: map exit codes to extended BuildStatus variants
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ibaa4de9e2c789931df8c67a53528829a6a6a6964
2026-02-16 13:02:23 +03:00
38ed7faee2
various: replace silent error discards with logged warnings
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I465d760b5330980270b64b4a89abc09f6a6a6964
2026-02-15 23:37:51 +03:00
a2b638d4db
nix: attempt to fix VM tests; general cleanup
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I65f6909ef02ab4599f5b0bbc0930367e6a6a6964
2026-02-15 23:37:49 +03:00
75ff45fc91
various: initial support for S3 cache upload
...
Not too stable yet, but might work.
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: If134e7e45aa99ce8d18df7b78b1f881b6a6a6964
2026-02-14 18:08:19 +03:00
fcb32aa9be
fc-queue-runner: collect metrics and trigger alerts on threshold
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: If1d682abdc6a932bdf5b9bfe23737c3e6a6a6964
2026-02-14 18:08:18 +03:00
3a03cf7b3e
treewide: format with nightly rustfmt; auto-fix Clippy lints
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: If4fd0511087dbaa65afc56a34d7c2f166a6a6964
2026-02-08 22:23:28 +03:00
73919f2f9e
treewide: format with nightly rustfmt; auto-fix Clippy lints
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I15d9215ab506b37954468d99746098326a6a6964
2026-02-08 22:23:20 +03:00
10a118bd50
various: update tests for jobset state field changes
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ifdd7f216fcc27fface0cb16d56c5e37b6a6a6964
2026-02-08 22:23:19 +03:00
b791ed75f3
fc-queue-runner: one-at-a-time jobset scheduling
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Iebe127bfba39979649826dfd0d28f9db6a6a6964
2026-02-08 22:23:16 +03:00
5ee3154f8e
queue-runner: update tests
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: If15433609cb601119e77c35ff73041196a6a6964
2026-02-07 22:09:26 +03:00
1c18306822
various: eliminate redundant disk check; improve error handling
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I1f37cc60380790bc1bf11f143194ad116a6a6964
2026-02-05 23:05:02 +03:00
c306383d27
chore: format with updated rustfmt and taplo rules
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ie9ef5fc421fa20071946cf1073f7920c6a6a6964
2026-02-05 22:45:06 +03:00
c0df24c6e1
crates/queue-runner: add cache upload config and worker improvements
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I781a843b88a9b62b929a8d0407274bc86a6a6964
2026-02-02 01:49:32 +03:00
44d1ee1d6b
queue-runner: semaphore-based worker pool with atomic build claiming
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ie3a8d343c3705200f0cac566227db54f6a6a6964
2026-02-02 01:15:07 +03:00
6203ea7f52
initial commit
...
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I79a875e75937ff6b3739ca36bfb0b2836a6a6964
2026-02-02 01:14:56 +03:00
