NotAShelf/circus
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: move README to docs directory; minor cleanup

Browse source 
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I914d8ddaabc714f82c9c61dcf75de2a16a6a6964
This commit is contained in:
raf 2026-02-02 22:42:21 +03:00
commit d3908e3307
Signed by: NotAShelf
GPG key ID: 29D95B64378DB4BF
2 changed files with 525 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

34
README.md
View file

@ -1,34 +0,0 @@
# FC
FC is a modern, Rust-based continuous integration system designed to replace
Hydra for our systems. Heavily work in progress.
## Architecture
- **server**: Web API and UI (Axum)
- **evaluator**: Git polling and Nix evaluation
- **queue-runner**: Build dispatch and execution
- **common**: Shared types and utilities
## Development
```bash
nix develop
cargo build
```
## Components
### Server
Web API server providing REST endpoints for project management, jobsets, and
build status.
### Evaluator
Periodically polls Git repositories and evaluates Nix expressions to create
builds.
### Queue Runner
Processes build queue and executes Nix builds on available machines.

525
docs/README.md Normal file
View file

@ -0,0 +1,525 @@
# FC
[design document]: ./DESIGN.md
FC, also known as "feely-CI" or a CI with feelings, is a Rust-based continuous
integration system designed to replace Hydra on all of our systems, for
performant and declarative CI needs.
Heavily work in progress. See the [design document] for a higher-level overview
of this project. Documentation is still scattered, and may not reflect the
latest state of the project until it is deemed complete. Please create an issue
if you notice and obvious inaccuracy. While I not guarantee a quick response,
I'd appreciate the heads-up. PRs are also very welcome.
## Architecture
FC follows Hydra's three-daemon model with a shared PostgreSQL database:
- **server** (`fc-server`): REST API (Axum), dashboard, binary cache, metrics,
webhooks
- **evaluator** (`fc-evaluator`): Git polling and Nix evaluation via
`nix-eval-jobs`
- **queue-runner** (`fc-queue-runner`): Build dispatch with semaphore-based
worker pool
- **common** (`fc-common`): Shared types, database layer, configuration,
validation
- **migrate-cli** (`fc-migrate`): Database migration utility
```mermaid
flowchart LR
A[Git Repo] --> B[Evaluator<br/>(polls, clones, runs nix-eval-jobs)]
B --> C[Evaluation + Build Records<br/>in DB]
C --> D[Queue Runner<br/>(claims builds atomically,<br/>runs nix build)]
D --> E[BuildSteps<br/>and BuildProducts]
```
## Development
```bash
nix develop # Enter dev shell (rust-analyzer, postgresql, pkg-config, openssl)
cargo build # Build all crates
cargo test # Run all tests
cargo check # Type-check only
```
Build a specific crate:
```bash
cargo build -p fc-server
cargo build -p fc-evaluator
cargo build -p fc-queue-runner
cargo build -p fc-common
cargo build -p fc-migrate-cli
```
## Quick Start
1. Enter dev shell and start PostgreSQL:
```bash
nix develop
initdb -D /tmp/fc-pg
pg_ctl -D /tmp/fc-pg start
createuser fc_ci
createdb -O fc_ci fc_ci
```
2. Run migrations:
```bash
cargo run --bin fc-migrate -- up postgresql://fc_ci@localhost/fc_ci
```
3. Start the server:
```bash
FC_DATABASE__URL=postgresql://fc_ci@localhost/fc_ci cargo run --bin fc-server
```
4. Open <http://localhost:3000> in your browser.
## Demo VM
A self-contained NixOS VM is available for trying FC without any manual setup.
It runs `fc-server` with PostgreSQL, seeds demo API keys, and forwards port 3000
to the host.
### Running
```bash
nix build .#demo-vm
./result/bin/run-fc-demo-vm
```
The VM boots to a serial console (no graphical display). Once the boot
completes, the server is reachable from your host at <http://localhost:3000>.
### Pre-seeded Credentials
To make the testing process easier, an admin key and a read-only API key are
pre-seeded in the demo VM. This should let you test a majority of features
without having to set up an account each time you spin up your VM.
| Key | Role | Use for |
| ---------------------- | ----------- | ---------------------------- |
| `fc_demo_admin_key` | `admin` | Full access, dashboard login |
| `fc_demo_readonly_key` | `read-only` | Read-only API access |
Log in to the dashboard at <http://localhost:3000/login> using the admin key.
### Example API Calls
FC is designed as a server in mind, and the dashboard is a convenient wrapper
around the API. If you are testing with new routes you may test them with curl
without ever spinning up a browser:
<!--markdownlint-disable MD013-->
```bash
# Health check
curl -s http://localhost:3000/health | jq
# Create a project
curl -s -X POST http://localhost:3000/api/v1/projects \
-H 'Authorization: Bearer fc_demo_admin_key' \
-H 'Content-Type: application/json' \
-d '{"name": "my-project", "repository_url": "https://github.com/NixOS/nixpkgs"}' | jq
# List projects
curl -s http://localhost:3000/api/v1/projects | jq
# Try with read-only key (write should fail with 403)
curl -s -o /dev/null -w '%{http_code}' -X POST http://localhost:3000/api/v1/projects \
-H 'Authorization: Bearer fc_demo_readonly_key' \
-H 'Content-Type: application/json' \
-d '{"name": "should-fail", "repository_url": "https://example.com"}'
```
<!--markdownlint-enable MD013-->
### Inside the VM
The serial console auto-logs in as root. While in the VM, you may use the TTY
access to investigate server logs or make API calls.
```bash
# Useful commands:
$ systemctl status fc-server
$ journalctl -u fc-server -f # Live server logs
$ curl -sf localhost:3000/health | jq # Health status
$ curl -sf localhost:3000/metrics # Metics
```
Press `Ctrl-a x` to shut down QEMU.
### VM Options
The VM uses QEMU user-mode networking. If port 3000 conflicts on your host, you
can override the QEMU options:
```bash
QEMU_NET_OPTS="hostfwd=tcp::8080-:3000" ./result/bin/run-fc-demo-vm
```
This makes the dashboard available at <http://localhost:8080> instead.
## Configuration
FC reads configuration from a TOML file with environment variable overrides.
Override hierarchy (highest wins):
1. Compiled defaults
2. `fc.toml` in working directory
3. File at `FC_CONFIG_FILE` env var
4. `FC_*` env vars (`__` as nested separator, e.g. `FC_DATABASE__URL`)
See `fc.toml` in the repository root for the full schema with comments.
### Configuration Reference
A somewhat maintained list of configuration options. Might be outdated during
development.
<!--markdownlint-disable MD013 -->
| Section | Key | Default | Description |
| --------------- | ---------------------- | --------------------------------------------- | ----------------------------------------- |
| `database` | `url` | `postgresql://fc_ci:password@localhost/fc_ci` | PostgreSQL connection URL |
| `database` | `max_connections` | `20` | Maximum connection pool size |
| `database` | `min_connections` | `5` | Minimum idle connections |
| `database` | `connect_timeout` | `30` | Connection timeout (seconds) |
| `database` | `idle_timeout` | `600` | Idle connection timeout (seconds) |
| `database` | `max_lifetime` | `1800` | Maximum connection lifetime (seconds) |
| `server` | `host` | `127.0.0.1` | HTTP listen address |
| `server` | `port` | `3000` | HTTP listen port |
| `server` | `request_timeout` | `30` | Per-request timeout (seconds) |
| `server` | `max_body_size` | `10485760` | Maximum request body size (10 MB) |
| `server` | `api_key` | none | Optional legacy API key (prefer DB keys) |
| `server` | `cors_permissive` | `false` | Allow all CORS origins |
| `server` | `allowed_origins` | `[]` | Allowed CORS origins list |
| `server` | `rate_limit_rps` | none | Requests per second limit |
| `server` | `rate_limit_burst` | none | Burst size for rate limiting |
| `evaluator` | `poll_interval` | `60` | Seconds between git poll cycles |
| `evaluator` | `git_timeout` | `600` | Git operation timeout (seconds) |
| `evaluator` | `nix_timeout` | `1800` | Nix evaluation timeout (seconds) |
| `evaluator` | `max_concurrent_evals` | `4` | Maximum concurrent evaluations |
| `evaluator` | `work_dir` | `/tmp/fc-evaluator` | Working directory for clones |
| `evaluator` | `restrict_eval` | `true` | Pass `--option restrict-eval true` to Nix |
| `evaluator` | `allow_ifd` | `false` | Allow import-from-derivation |
| `queue_runner` | `workers` | `4` | Concurrent build slots |
| `queue_runner` | `poll_interval` | `5` | Seconds between build queue polls |
| `queue_runner` | `build_timeout` | `3600` | Per-build timeout (seconds) |
| `queue_runner` | `work_dir` | `/tmp/fc-queue-runner` | Working directory for builds |
| `gc` | `enabled` | `true` | Manage GC roots for build outputs |
| `gc` | `gc_roots_dir` | `/nix/var/nix/gcroots/per-user/fc/fc-roots` | GC roots directory |
| `gc` | `max_age_days` | `30` | Remove GC roots older than N days |
| `gc` | `cleanup_interval` | `3600` | GC cleanup interval (seconds) |
| `logs` | `log_dir` | `/var/lib/fc/logs` | Build log storage directory |
| `logs` | `compress` | `false` | Compress stored logs |
| `cache` | `enabled` | `true` | Serve a Nix binary cache at `/nix-cache/` |
| `cache` | `secret_key_file` | none | Signing key for binary cache |
| `signing` | `enabled` | `false` | Sign build outputs |
| `signing` | `key_file` | none | Signing key file path |
| `notifications` | `run_command` | none | Command to run on build completion |
| `notifications` | `github_token` | none | GitHub token for commit status updates |
| `notifications` | `gitea_url` | none | Gitea/Forgejo instance URL |
| `notifications` | `gitea_token` | none | Gitea/Forgejo API token |
<!--markdownlint-enable MD013 -->
## Database
FC uses PostgreSQL with sqlx fcompile-time query checking. Migrations live in
`crates/common/migrations/` and are added usually when the database schema
changes.
```bash
# Run pending migrations
$ cargo run --bin fc-migrate -- up <database_url>
# Validate schema
$ cargo run --bin fc-migrate -- validate <database_url>
# Create new migration file
$ cargo run --bin fc-migrate -- create <name>
```
Database tests gracefully skip when PostgreSQL is unavailable. To run the
database tests, make sure you build the test VMs.
## NixOS Deployment
FC ships a NixOS module at `nixosModules.default`. Minimal configuration:
```nix
{
inputs.fc.url = "github:feel-co/ci";
outputs = { self, nixpkgs, fc, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
modules = [
fc.nixosModules.default
{
services.fc = {
enable = true;
package = fc.packages.x86_64-linux.server;
migratePackage = fc.packages.x86_64-linux.migrate-cli;
server.enable = true;
# evaluator.enable = true;
# queueRunner.enable = true;
};
}
];
};
};
}
```
### Full Deployment Example
A complete production configuration with all three daemons and NGINX reverse
proxy:
```nix
{ config, pkgs, fc, ... }: {
services.fc = {
enable = true;
package = fc.packages.x86_64-linux.server;
migratePackage = fc.packages.x86_64-linux.migrate-cli;
server.enable = true;
evaluator.enable = true;
queueRunner.enable = true;
settings = {
database.url = "postgresql:///fc?host=/run/postgresql";
server.host = "127.0.0.1";
server.port = 3000;
evaluator.poll_interval = 300;
evaluator.restrict_eval = true;
queue_runner.workers = 8;
queue_runner.build_timeout = 7200;
gc.enabled = true;
gc.max_age_days = 90;
cache.enabled = true;
logs.log_dir = "/var/lib/fc/logs";
logs.compress = true;
};
};
# Reverse proxy
services.nginx = {
enable = true;
virtualHosts."ci.example.org" = {
forceSSL = true;
enableACME = true;
locations."/" = {
proxyPass = "http://127.0.0.1:3000";
proxyWebsockets = true;
extraConfig = ''
# FIXME: you might choose to harden this part further
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
client_max_body_size 50M;
'';
};
};
};
# Firewall
networking.firewall.allowedTCPPorts = [ 80 443 ];
}
```
### Multi-Machine Deployment
For larger or _distributed_ setups, you may choose to run the daemons on
different machines sharing the same database. For example:
- **Head node**: runs `fc-server` and `fc-evaluator`, has the PostgreSQL
database locally
- **Builder machines**: run `fc-queue-runner`, connect to the head node's
database via `postgresql://fc@headnode/fc`
On builder machines, set `database.createLocally = false` and provide the remote
database URL:
```nix
{
services.fc = {
enable = true;
database.createLocally = false; # <- Set this
queueRunner.enable = true;
# Now configure the database
settings.database.url = "postgresql://fc@headnode.internal/fc";
settings.queue_runner.workers = 16;
};
}
```
Ensure the PostgreSQL server on the head node allows connections from builder
machines via `pg_hba.conf` (the NixOS `services.postgresql` module handles this
with `authentication` settings).
## API Key Bootstrapping
FC uses SHA-256 hashed API keys stored in the `api_keys` table. To create the
first admin key after initial deployment:
```bash
# Generate a key and its hash
$ export FC_KEY="fc_$(openssl rand -hex 16)"
$ export FC_HASH=$(echo -n "$FC_KEY" | sha256sum | cut -d' ' -f1)
# Insert into the database
$ sudo -u fc psql -U fc -d fc -c \
"INSERT INTO api_keys (name, key_hash, role) VALUES ('admin', '$FC_HASH', 'admin')"
# Save the key (it cannot be recovered from the hash)
$ echo "Admin API key: $FC_KEY"
```
Subsequent keys can be created via the API or the admin dashboard using this
initial admin key.
### Roles
> [!NOTE]
> Roles are an experimental feature designed to bring FC on-par with
> enterprise-grade Hydra deployments. The feature is currently unstable, and
> might change at any given time. Do not rely on roles for the time being.
| Role | Permissions |
| ----------------- | ------------------------------------ |
| `admin` | Full access to all endpoints |
| `read-only` | Read-only access (GET requests only) |
| `create-projects` | Create projects and jobsets |
| `eval-jobset` | Trigger evaluations |
| `cancel-build` | Cancel builds |
| `restart-jobs` | Restart failed/completed builds |
| `bump-to-front` | Bump build priority |
## Monitoring
FC exposes a limited, Prometheus-compatible metrics endpoint at `/metrics`. The
metrics provided are detailed below:
### Available Metrics
| Metric | Type | Description |
| -------------------------- | ----- | --------------------------- |
| `fc_builds_total` | gauge | Total number of builds |
| `fc_builds_pending` | gauge | Currently pending builds |
| `fc_builds_running` | gauge | Currently running builds |
| `fc_builds_completed` | gauge | Completed builds |
| `fc_builds_failed` | gauge | Failed builds |
| `fc_projects_total` | gauge | Total number of projects |
| `fc_evaluations_total` | gauge | Total number of evaluations |
| `fc_channels_total` | gauge | Total number of channels |
| `fc_remote_builders_total` | gauge | Configured remote builders |
### Prometheus Configuration
```yaml
scrape_configs:
- job_name: "fc-ci"
static_configs:
- targets: ["ci.example.org:3000"]
metrics_path: "/metrics"
scrape_interval: 30s
```
## Backup & Restore
FC stores all state in PostgreSQL. To back up:
```bash
# Create a backup
$ pg_dump -U fc fc > fc-backup-$(date +%Y%m%d).sql
```
To restore:
```bash
# Restore a backup
$ psql -U fc fc < fc-backup-20250101.sql
```
Build logs are stored in the filesystem at the configured `logs.log_dir`
(default: `/var/lib/fc/logs`). You are generally encouraged to include this
directory in your backup strategy to ensure more seamless recoveries in the case
of a catastrophic failure. Build outputs live in the Nix store and are protected
by GC roots under `gc.gc_roots_dir`. These do not need separate backup as long
as derivation paths are retained in the database.
## API Overview
All API endpoints are under `/api/v1`. Write operations require a Bearer token
in the `Authorization` header. Read operations (GET) are public.
<!--markdownlint-disable MD013 -->
| Method | Endpoint | Auth | Description |
| ------ | -------------------------------------- | --------------------- | --------------------------------- |
| GET | `/health` | - | Health check with database status |
| GET | `/metrics` | - | Prometheus metrics |
| GET | `/api/v1/projects` | - | List projects (paginated) |
| POST | `/api/v1/projects` | admin/create-projects | Create project |
| GET | `/api/v1/projects/{id}` | - | Get project details |
| PUT | `/api/v1/projects/{id}` | admin | Update project |
| DELETE | `/api/v1/projects/{id}` | admin | Delete project (cascades) |
| GET | `/api/v1/projects/{id}/jobsets` | - | List project jobsets |
| POST | `/api/v1/projects/{id}/jobsets` | admin/create-projects | Create jobset |
| GET | `/api/v1/projects/{pid}/jobsets/{id}` | - | Get jobset |
| PUT | `/api/v1/projects/{pid}/jobsets/{id}` | admin | Update jobset |
| DELETE | `/api/v1/projects/{pid}/jobsets/{id}` | admin | Delete jobset |
| GET | `/api/v1/evaluations` | - | List evaluations (filtered) |
| GET | `/api/v1/evaluations/{id}` | - | Get evaluation details |
| POST | `/api/v1/evaluations/trigger` | admin/eval-jobset | Trigger evaluation |
| GET | `/api/v1/builds` | - | List builds (filtered) |
| GET | `/api/v1/builds/{id}` | - | Get build details |
| POST | `/api/v1/builds/{id}/cancel` | admin/cancel-build | Cancel build |
| POST | `/api/v1/builds/{id}/restart` | admin/restart-jobs | Restart build |
| POST | `/api/v1/builds/{id}/bump` | admin/bump-to-front | Bump priority |
| GET | `/api/v1/builds/stats` | - | Build statistics |
| GET | `/api/v1/builds/recent` | - | Recent builds |
| GET | `/api/v1/channels` | - | List channels |
| POST | `/api/v1/channels` | admin | Create channel |
| DELETE | `/api/v1/channels/{id}` | admin | Delete channel |
| POST | `/api/v1/channels/{id}/promote/{eval}` | admin | Promote evaluation |
| GET | `/api/v1/api-keys` | admin | List API keys |
| POST | `/api/v1/api-keys` | admin | Create API key |
| DELETE | `/api/v1/api-keys/{id}` | admin | Delete API key |
| GET | `/api/v1/admin/builders` | - | List remote builders |
| POST | `/api/v1/admin/builders` | admin | Create remote builder |
| PUT | `/api/v1/admin/builders/{id}` | admin | Update remote builder |
| DELETE | `/api/v1/admin/builders/{id}` | admin | Delete remote builder |
| GET | `/api/v1/admin/system` | admin | System status |
| GET | `/api/v1/search?q=` | - | Search projects and builds |
| POST | `/api/v1/webhooks/{pid}/github` | HMAC | GitHub webhook |
| POST | `/api/v1/webhooks/{pid}/gitea` | HMAC | Gitea/Forgejo webhook |
| GET | `/nix-cache/nix-cache-info` | - | Binary cache info |
| GET | `/nix-cache/{hash}.narinfo` | - | NAR info lookup |
| GET | `/nix-cache/nar/{hash}.nar.zst` | - | NAR download (zstd) |
<!--markdownlint-enable MD013 -->
### Dashboard
The web dashboard is available at the root URL (`/`). Pages include:
- `/` - Home: build stats, project overview, recent builds and evaluations
- `/projects` - Project listing with create form (admin)
- `/project/{id}` - Project detail with jobsets, add jobset form (admin)
- `/evaluations` - Evaluation listing with project/jobset context
- `/builds` - Build listing with status/system/job filters
- `/queue` - Current queue (pending + running builds)
- `/channels` - Channel listing
- `/admin` - System status, API key management, remote builder management
- `/login` - Cookie-based session login using API key