diff --git a/README.md b/README.md deleted file mode 100644 index fedca69..0000000 --- a/README.md +++ /dev/null @@ -1,34 +0,0 @@ -# FC - -FC is a modern, Rust-based continuous integration system designed to replace -Hydra for our systems. Heavily work in progress. - -## Architecture - -- **server**: Web API and UI (Axum) -- **evaluator**: Git polling and Nix evaluation -- **queue-runner**: Build dispatch and execution -- **common**: Shared types and utilities - -## Development - -```bash -nix develop -cargo build -``` - -## Components - -### Server - -Web API server providing REST endpoints for project management, jobsets, and -build status. - -### Evaluator - -Periodically polls Git repositories and evaluates Nix expressions to create -builds. - -### Queue Runner - -Processes build queue and executes Nix builds on available machines. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..f4eb4df --- /dev/null +++ b/docs/README.md @@ -0,0 +1,525 @@ +# FC + +[design document]: ./DESIGN.md + +FC, also known as "feely-CI" or a CI with feelings, is a Rust-based continuous +integration system designed to replace Hydra on all of our systems, for +performant and declarative CI needs. + +Heavily work in progress. See the [design document] for a higher-level overview +of this project. Documentation is still scattered, and may not reflect the +latest state of the project until it is deemed complete. Please create an issue +if you notice and obvious inaccuracy. While I not guarantee a quick response, +I'd appreciate the heads-up. PRs are also very welcome. + +## Architecture + +FC follows Hydra's three-daemon model with a shared PostgreSQL database: + +- **server** (`fc-server`): REST API (Axum), dashboard, binary cache, metrics, + webhooks +- **evaluator** (`fc-evaluator`): Git polling and Nix evaluation via + `nix-eval-jobs` +- **queue-runner** (`fc-queue-runner`): Build dispatch with semaphore-based + worker pool +- **common** (`fc-common`): Shared types, database layer, configuration, + validation +- **migrate-cli** (`fc-migrate`): Database migration utility + +```mermaid +flowchart LR + A[Git Repo] --> B[Evaluator
(polls, clones, runs nix-eval-jobs)] + B --> C[Evaluation + Build Records
in DB] + C --> D[Queue Runner
(claims builds atomically,
runs nix build)] + D --> E[BuildSteps
and BuildProducts] +``` + +## Development + +```bash +nix develop # Enter dev shell (rust-analyzer, postgresql, pkg-config, openssl) +cargo build # Build all crates +cargo test # Run all tests +cargo check # Type-check only +``` + +Build a specific crate: + +```bash +cargo build -p fc-server +cargo build -p fc-evaluator +cargo build -p fc-queue-runner +cargo build -p fc-common +cargo build -p fc-migrate-cli +``` + +## Quick Start + +1. Enter dev shell and start PostgreSQL: + + ```bash + nix develop + initdb -D /tmp/fc-pg + pg_ctl -D /tmp/fc-pg start + createuser fc_ci + createdb -O fc_ci fc_ci + ``` + +2. Run migrations: + + ```bash + cargo run --bin fc-migrate -- up postgresql://fc_ci@localhost/fc_ci + ``` + +3. Start the server: + + ```bash + FC_DATABASE__URL=postgresql://fc_ci@localhost/fc_ci cargo run --bin fc-server + ``` + +4. Open in your browser. + +## Demo VM + +A self-contained NixOS VM is available for trying FC without any manual setup. +It runs `fc-server` with PostgreSQL, seeds demo API keys, and forwards port 3000 +to the host. + +### Running + +```bash +nix build .#demo-vm +./result/bin/run-fc-demo-vm +``` + +The VM boots to a serial console (no graphical display). Once the boot +completes, the server is reachable from your host at . + +### Pre-seeded Credentials + +To make the testing process easier, an admin key and a read-only API key are +pre-seeded in the demo VM. This should let you test a majority of features +without having to set up an account each time you spin up your VM. + +| Key | Role | Use for | +| ---------------------- | ----------- | ---------------------------- | +| `fc_demo_admin_key` | `admin` | Full access, dashboard login | +| `fc_demo_readonly_key` | `read-only` | Read-only API access | + +Log in to the dashboard at using the admin key. + +### Example API Calls + +FC is designed as a server in mind, and the dashboard is a convenient wrapper +around the API. If you are testing with new routes you may test them with curl +without ever spinning up a browser: + + + +```bash +# Health check +curl -s http://localhost:3000/health | jq + +# Create a project +curl -s -X POST http://localhost:3000/api/v1/projects \ + -H 'Authorization: Bearer fc_demo_admin_key' \ + -H 'Content-Type: application/json' \ + -d '{"name": "my-project", "repository_url": "https://github.com/NixOS/nixpkgs"}' | jq + +# List projects +curl -s http://localhost:3000/api/v1/projects | jq + +# Try with read-only key (write should fail with 403) +curl -s -o /dev/null -w '%{http_code}' -X POST http://localhost:3000/api/v1/projects \ + -H 'Authorization: Bearer fc_demo_readonly_key' \ + -H 'Content-Type: application/json' \ + -d '{"name": "should-fail", "repository_url": "https://example.com"}' +``` + + + +### Inside the VM + +The serial console auto-logs in as root. While in the VM, you may use the TTY +access to investigate server logs or make API calls. + +```bash +# Useful commands: +$ systemctl status fc-server +$ journalctl -u fc-server -f # Live server logs +$ curl -sf localhost:3000/health | jq # Health status +$ curl -sf localhost:3000/metrics # Metics +``` + +Press `Ctrl-a x` to shut down QEMU. + +### VM Options + +The VM uses QEMU user-mode networking. If port 3000 conflicts on your host, you +can override the QEMU options: + +```bash +QEMU_NET_OPTS="hostfwd=tcp::8080-:3000" ./result/bin/run-fc-demo-vm +``` + +This makes the dashboard available at instead. + +## Configuration + +FC reads configuration from a TOML file with environment variable overrides. +Override hierarchy (highest wins): + +1. Compiled defaults +2. `fc.toml` in working directory +3. File at `FC_CONFIG_FILE` env var +4. `FC_*` env vars (`__` as nested separator, e.g. `FC_DATABASE__URL`) + +See `fc.toml` in the repository root for the full schema with comments. + +### Configuration Reference + +A somewhat maintained list of configuration options. Might be outdated during +development. + + + +| Section | Key | Default | Description | +| --------------- | ---------------------- | --------------------------------------------- | ----------------------------------------- | +| `database` | `url` | `postgresql://fc_ci:password@localhost/fc_ci` | PostgreSQL connection URL | +| `database` | `max_connections` | `20` | Maximum connection pool size | +| `database` | `min_connections` | `5` | Minimum idle connections | +| `database` | `connect_timeout` | `30` | Connection timeout (seconds) | +| `database` | `idle_timeout` | `600` | Idle connection timeout (seconds) | +| `database` | `max_lifetime` | `1800` | Maximum connection lifetime (seconds) | +| `server` | `host` | `127.0.0.1` | HTTP listen address | +| `server` | `port` | `3000` | HTTP listen port | +| `server` | `request_timeout` | `30` | Per-request timeout (seconds) | +| `server` | `max_body_size` | `10485760` | Maximum request body size (10 MB) | +| `server` | `api_key` | none | Optional legacy API key (prefer DB keys) | +| `server` | `cors_permissive` | `false` | Allow all CORS origins | +| `server` | `allowed_origins` | `[]` | Allowed CORS origins list | +| `server` | `rate_limit_rps` | none | Requests per second limit | +| `server` | `rate_limit_burst` | none | Burst size for rate limiting | +| `evaluator` | `poll_interval` | `60` | Seconds between git poll cycles | +| `evaluator` | `git_timeout` | `600` | Git operation timeout (seconds) | +| `evaluator` | `nix_timeout` | `1800` | Nix evaluation timeout (seconds) | +| `evaluator` | `max_concurrent_evals` | `4` | Maximum concurrent evaluations | +| `evaluator` | `work_dir` | `/tmp/fc-evaluator` | Working directory for clones | +| `evaluator` | `restrict_eval` | `true` | Pass `--option restrict-eval true` to Nix | +| `evaluator` | `allow_ifd` | `false` | Allow import-from-derivation | +| `queue_runner` | `workers` | `4` | Concurrent build slots | +| `queue_runner` | `poll_interval` | `5` | Seconds between build queue polls | +| `queue_runner` | `build_timeout` | `3600` | Per-build timeout (seconds) | +| `queue_runner` | `work_dir` | `/tmp/fc-queue-runner` | Working directory for builds | +| `gc` | `enabled` | `true` | Manage GC roots for build outputs | +| `gc` | `gc_roots_dir` | `/nix/var/nix/gcroots/per-user/fc/fc-roots` | GC roots directory | +| `gc` | `max_age_days` | `30` | Remove GC roots older than N days | +| `gc` | `cleanup_interval` | `3600` | GC cleanup interval (seconds) | +| `logs` | `log_dir` | `/var/lib/fc/logs` | Build log storage directory | +| `logs` | `compress` | `false` | Compress stored logs | +| `cache` | `enabled` | `true` | Serve a Nix binary cache at `/nix-cache/` | +| `cache` | `secret_key_file` | none | Signing key for binary cache | +| `signing` | `enabled` | `false` | Sign build outputs | +| `signing` | `key_file` | none | Signing key file path | +| `notifications` | `run_command` | none | Command to run on build completion | +| `notifications` | `github_token` | none | GitHub token for commit status updates | +| `notifications` | `gitea_url` | none | Gitea/Forgejo instance URL | +| `notifications` | `gitea_token` | none | Gitea/Forgejo API token | + + + +## Database + +FC uses PostgreSQL with sqlx fcompile-time query checking. Migrations live in +`crates/common/migrations/` and are added usually when the database schema +changes. + +```bash +# Run pending migrations +$ cargo run --bin fc-migrate -- up + +# Validate schema +$ cargo run --bin fc-migrate -- validate + +# Create new migration file +$ cargo run --bin fc-migrate -- create +``` + +Database tests gracefully skip when PostgreSQL is unavailable. To run the +database tests, make sure you build the test VMs. + +## NixOS Deployment + +FC ships a NixOS module at `nixosModules.default`. Minimal configuration: + +```nix +{ + inputs.fc.url = "github:feel-co/ci"; + + outputs = { self, nixpkgs, fc, ... }: { + nixosConfigurations.myhost = nixpkgs.lib.nixosSystem { + modules = [ + fc.nixosModules.default + { + services.fc = { + enable = true; + package = fc.packages.x86_64-linux.server; + migratePackage = fc.packages.x86_64-linux.migrate-cli; + + server.enable = true; + # evaluator.enable = true; + # queueRunner.enable = true; + }; + } + ]; + }; + }; +} +``` + +### Full Deployment Example + +A complete production configuration with all three daemons and NGINX reverse +proxy: + +```nix +{ config, pkgs, fc, ... }: { + services.fc = { + enable = true; + package = fc.packages.x86_64-linux.server; + migratePackage = fc.packages.x86_64-linux.migrate-cli; + + server.enable = true; + evaluator.enable = true; + queueRunner.enable = true; + + settings = { + database.url = "postgresql:///fc?host=/run/postgresql"; + server.host = "127.0.0.1"; + server.port = 3000; + + evaluator.poll_interval = 300; + evaluator.restrict_eval = true; + queue_runner.workers = 8; + queue_runner.build_timeout = 7200; + + gc.enabled = true; + gc.max_age_days = 90; + cache.enabled = true; + logs.log_dir = "/var/lib/fc/logs"; + logs.compress = true; + }; + }; + + # Reverse proxy + services.nginx = { + enable = true; + virtualHosts."ci.example.org" = { + forceSSL = true; + enableACME = true; + locations."/" = { + proxyPass = "http://127.0.0.1:3000"; + proxyWebsockets = true; + extraConfig = '' + # FIXME: you might choose to harden this part further + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + client_max_body_size 50M; + ''; + }; + }; + }; + + # Firewall + networking.firewall.allowedTCPPorts = [ 80 443 ]; +} +``` + +### Multi-Machine Deployment + +For larger or _distributed_ setups, you may choose to run the daemons on +different machines sharing the same database. For example: + +- **Head node**: runs `fc-server` and `fc-evaluator`, has the PostgreSQL + database locally +- **Builder machines**: run `fc-queue-runner`, connect to the head node's + database via `postgresql://fc@headnode/fc` + +On builder machines, set `database.createLocally = false` and provide the remote +database URL: + +```nix +{ + services.fc = { + enable = true; + database.createLocally = false; # <- Set this + queueRunner.enable = true; + + # Now configure the database + settings.database.url = "postgresql://fc@headnode.internal/fc"; + settings.queue_runner.workers = 16; + }; +} +``` + +Ensure the PostgreSQL server on the head node allows connections from builder +machines via `pg_hba.conf` (the NixOS `services.postgresql` module handles this +with `authentication` settings). + +## API Key Bootstrapping + +FC uses SHA-256 hashed API keys stored in the `api_keys` table. To create the +first admin key after initial deployment: + +```bash +# Generate a key and its hash +$ export FC_KEY="fc_$(openssl rand -hex 16)" +$ export FC_HASH=$(echo -n "$FC_KEY" | sha256sum | cut -d' ' -f1) + +# Insert into the database +$ sudo -u fc psql -U fc -d fc -c \ + "INSERT INTO api_keys (name, key_hash, role) VALUES ('admin', '$FC_HASH', 'admin')" + +# Save the key (it cannot be recovered from the hash) +$ echo "Admin API key: $FC_KEY" +``` + +Subsequent keys can be created via the API or the admin dashboard using this +initial admin key. + +### Roles + +> [!NOTE] +> Roles are an experimental feature designed to bring FC on-par with +> enterprise-grade Hydra deployments. The feature is currently unstable, and +> might change at any given time. Do not rely on roles for the time being. + +| Role | Permissions | +| ----------------- | ------------------------------------ | +| `admin` | Full access to all endpoints | +| `read-only` | Read-only access (GET requests only) | +| `create-projects` | Create projects and jobsets | +| `eval-jobset` | Trigger evaluations | +| `cancel-build` | Cancel builds | +| `restart-jobs` | Restart failed/completed builds | +| `bump-to-front` | Bump build priority | + +## Monitoring + +FC exposes a limited, Prometheus-compatible metrics endpoint at `/metrics`. The +metrics provided are detailed below: + +### Available Metrics + +| Metric | Type | Description | +| -------------------------- | ----- | --------------------------- | +| `fc_builds_total` | gauge | Total number of builds | +| `fc_builds_pending` | gauge | Currently pending builds | +| `fc_builds_running` | gauge | Currently running builds | +| `fc_builds_completed` | gauge | Completed builds | +| `fc_builds_failed` | gauge | Failed builds | +| `fc_projects_total` | gauge | Total number of projects | +| `fc_evaluations_total` | gauge | Total number of evaluations | +| `fc_channels_total` | gauge | Total number of channels | +| `fc_remote_builders_total` | gauge | Configured remote builders | + +### Prometheus Configuration + +```yaml +scrape_configs: + - job_name: "fc-ci" + static_configs: + - targets: ["ci.example.org:3000"] + metrics_path: "/metrics" + scrape_interval: 30s +``` + +## Backup & Restore + +FC stores all state in PostgreSQL. To back up: + +```bash +# Create a backup +$ pg_dump -U fc fc > fc-backup-$(date +%Y%m%d).sql +``` + +To restore: + +```bash +# Restore a backup +$ psql -U fc fc < fc-backup-20250101.sql +``` + +Build logs are stored in the filesystem at the configured `logs.log_dir` +(default: `/var/lib/fc/logs`). You are generally encouraged to include this +directory in your backup strategy to ensure more seamless recoveries in the case +of a catastrophic failure. Build outputs live in the Nix store and are protected +by GC roots under `gc.gc_roots_dir`. These do not need separate backup as long +as derivation paths are retained in the database. + +## API Overview + +All API endpoints are under `/api/v1`. Write operations require a Bearer token +in the `Authorization` header. Read operations (GET) are public. + + + +| Method | Endpoint | Auth | Description | +| ------ | -------------------------------------- | --------------------- | --------------------------------- | +| GET | `/health` | - | Health check with database status | +| GET | `/metrics` | - | Prometheus metrics | +| GET | `/api/v1/projects` | - | List projects (paginated) | +| POST | `/api/v1/projects` | admin/create-projects | Create project | +| GET | `/api/v1/projects/{id}` | - | Get project details | +| PUT | `/api/v1/projects/{id}` | admin | Update project | +| DELETE | `/api/v1/projects/{id}` | admin | Delete project (cascades) | +| GET | `/api/v1/projects/{id}/jobsets` | - | List project jobsets | +| POST | `/api/v1/projects/{id}/jobsets` | admin/create-projects | Create jobset | +| GET | `/api/v1/projects/{pid}/jobsets/{id}` | - | Get jobset | +| PUT | `/api/v1/projects/{pid}/jobsets/{id}` | admin | Update jobset | +| DELETE | `/api/v1/projects/{pid}/jobsets/{id}` | admin | Delete jobset | +| GET | `/api/v1/evaluations` | - | List evaluations (filtered) | +| GET | `/api/v1/evaluations/{id}` | - | Get evaluation details | +| POST | `/api/v1/evaluations/trigger` | admin/eval-jobset | Trigger evaluation | +| GET | `/api/v1/builds` | - | List builds (filtered) | +| GET | `/api/v1/builds/{id}` | - | Get build details | +| POST | `/api/v1/builds/{id}/cancel` | admin/cancel-build | Cancel build | +| POST | `/api/v1/builds/{id}/restart` | admin/restart-jobs | Restart build | +| POST | `/api/v1/builds/{id}/bump` | admin/bump-to-front | Bump priority | +| GET | `/api/v1/builds/stats` | - | Build statistics | +| GET | `/api/v1/builds/recent` | - | Recent builds | +| GET | `/api/v1/channels` | - | List channels | +| POST | `/api/v1/channels` | admin | Create channel | +| DELETE | `/api/v1/channels/{id}` | admin | Delete channel | +| POST | `/api/v1/channels/{id}/promote/{eval}` | admin | Promote evaluation | +| GET | `/api/v1/api-keys` | admin | List API keys | +| POST | `/api/v1/api-keys` | admin | Create API key | +| DELETE | `/api/v1/api-keys/{id}` | admin | Delete API key | +| GET | `/api/v1/admin/builders` | - | List remote builders | +| POST | `/api/v1/admin/builders` | admin | Create remote builder | +| PUT | `/api/v1/admin/builders/{id}` | admin | Update remote builder | +| DELETE | `/api/v1/admin/builders/{id}` | admin | Delete remote builder | +| GET | `/api/v1/admin/system` | admin | System status | +| GET | `/api/v1/search?q=` | - | Search projects and builds | +| POST | `/api/v1/webhooks/{pid}/github` | HMAC | GitHub webhook | +| POST | `/api/v1/webhooks/{pid}/gitea` | HMAC | Gitea/Forgejo webhook | +| GET | `/nix-cache/nix-cache-info` | - | Binary cache info | +| GET | `/nix-cache/{hash}.narinfo` | - | NAR info lookup | +| GET | `/nix-cache/nar/{hash}.nar.zst` | - | NAR download (zstd) | + + + +### Dashboard + +The web dashboard is available at the root URL (`/`). Pages include: + +- `/` - Home: build stats, project overview, recent builds and evaluations +- `/projects` - Project listing with create form (admin) +- `/project/{id}` - Project detail with jobsets, add jobset form (admin) +- `/evaluations` - Evaluation listing with project/jobset context +- `/builds` - Build listing with status/system/job filters +- `/queue` - Current queue (pending + running builds) +- `/channels` - Channel listing +- `/admin` - System status, API key management, remote builder management +- `/login` - Cookie-based session login using API key