FC

FC, also known as "feely-CI" or a CI with feelings, is a Rust-based continuous integration system designed to replace Hydra on all of our systems, for performant and declarative CI needs.

Heavily work in progress. Documentation is still scattered, and may not reflect the latest state of the project until it is deemed complete. Please create an issue if you notice and obvious inaccuracy. While I not guarantee a quick response, I'd appreciate the heads-up. PRs are also very welcome.

Architecture

FC follows Hydra's three-daemon model with a shared PostgreSQL database:

• server (fc-server): REST API (Axum), dashboard, binary cache, metrics, webhooks
• evaluator (fc-evaluator): Git polling and Nix evaluation via nix-eval-jobs
• queue-runner (fc-queue-runner): Build dispatch with semaphore-based worker pool
• common (fc-common): Shared types, database layer, configuration, validation
• migrate-cli (fc-migrate): Database migration utility

flowchart LR
    A[Git Repo] --> B[Evaluator<br/>(polls, clones, runs nix-eval-jobs)]
    B --> C[Evaluation + Build Records<br/>in DB]
    C --> D[Queue Runner<br/>(claims builds atomically,<br/>runs nix build)]
    D --> E[BuildSteps<br/>and BuildProducts]

See the design document for more details on the architecture, similarities and differences with Hydra. For feedback and questions, head to the issues tab.

Development

nix develop          # Enter dev shell (rust-analyzer, postgresql, pkg-config, openssl)
cargo build          # Build all crates
cargo test           # Run all tests
cargo check          # Type-check only

Build a specific crate:

cargo build -p fc-server
cargo build -p fc-evaluator
cargo build -p fc-queue-runner
cargo build -p fc-common
cargo build -p fc-migrate-cli

Quick Start

1. Enter dev shell and start PostgreSQL:

   nix develop
   initdb -D /tmp/fc-pg
   pg_ctl -D /tmp/fc-pg start
   createuser fc_ci
   createdb -O fc_ci fc_ci
   

2. Run migrations:

   cargo run --bin fc-migrate -- up postgresql://fc_ci@localhost/fc_ci
   

3. Start the server:

   FC_DATABASE__URL=postgresql://fc_ci@localhost/fc_ci cargo run --bin fc-server
   

4. Open http://localhost:3000 in your browser.

Demo VM

A self-contained NixOS VM is available for trying FC without any manual setup. It runs fc-server with PostgreSQL, seeds demo API keys, and forwards port 3000 to the host.

Running

nix build .#demo-vm
./result/bin/run-fc-demo-vm

The VM boots to a serial console (no graphical display). Once the boot completes, the server is reachable from your host at http://localhost:3000.

Pre-seeded Credentials

To make the testing process easier, an admin key and a read-only API key are pre-seeded in the demo VM. This should let you test a majority of features without having to set up an account each time you spin up your VM.

Key	Role	Use for
fc_demo_admin_key	admin	Full access, dashboard login
fc_demo_readonly_key	read-only	Read-only API access

Log in to the dashboard at http://localhost:3000/login using the admin key.

Example API Calls

FC is designed as a server in mind, and the dashboard is a convenient wrapper around the API. If you are testing with new routes you may test them with curl without ever spinning up a browser:

# Health check
curl -s http://localhost:3000/health | jq

# Create a project
curl -s -X POST http://localhost:3000/api/v1/projects \
  -H 'Authorization: Bearer fc_demo_admin_key' \
  -H 'Content-Type: application/json' \
  -d '{"name": "my-project", "repository_url": "https://github.com/NixOS/nixpkgs"}' | jq

# List projects
curl -s http://localhost:3000/api/v1/projects | jq

# Try with read-only key (write should fail with 403)
curl -s -o /dev/null -w '%{http_code}' -X POST http://localhost:3000/api/v1/projects \
  -H 'Authorization: Bearer fc_demo_readonly_key' \
  -H 'Content-Type: application/json' \
  -d '{"name": "should-fail", "repository_url": "https://example.com"}'

Inside the VM

The serial console auto-logs in as root. While in the VM, you may use the TTY access to investigate server logs or make API calls.

# Useful commands:
$ systemctl status fc-server
$ journalctl -u fc-server -f          # Live server logs
$ curl -sf localhost:3000/health | jq # Health status
$ curl -sf localhost:3000/metrics     # Metics

Press Ctrl-a x to shut down QEMU.

VM Options

The VM uses QEMU user-mode networking. If port 3000 conflicts on your host, you can override the QEMU options:

QEMU_NET_OPTS="hostfwd=tcp::8080-:3000" ./result/bin/run-fc-demo-vm

This makes the dashboard available at http://localhost:8080 instead.

Configuration

FC reads configuration from a TOML file with environment variable overrides. Override hierarchy (highest wins):

1. Compiled defaults
2. fc.toml in working directory
3. File at FC_CONFIG_FILE env var
4. FC_* env vars (__ as nested separator, e.g. FC_DATABASE__URL)

See fc.toml in the repository root for the full schema with comments.

Configuration Reference

A somewhat maintained list of configuration options. Might be outdated during development.

Section	Key	Default	Description
database	url	postgresql://fc_ci:password@localhost/fc_ci	PostgreSQL connection URL
database	max_connections	20	Maximum connection pool size
database	min_connections	5	Minimum idle connections
database	connect_timeout	30	Connection timeout (seconds)
database	idle_timeout	600	Idle connection timeout (seconds)
database	max_lifetime	1800	Maximum connection lifetime (seconds)
server	host	127.0.0.1	HTTP listen address
server	port	3000	HTTP listen port
server	request_timeout	30	Per-request timeout (seconds)
server	max_body_size	10485760	Maximum request body size (10 MB)
server	api_key	none	Optional legacy API key (prefer DB keys)
server	cors_permissive	false	Allow all CORS origins
server	allowed_origins	[]	Allowed CORS origins list
server	rate_limit_rps	none	Requests per second limit
server	rate_limit_burst	none	Burst size for rate limiting
evaluator	poll_interval	60	Seconds between git poll cycles
evaluator	git_timeout	600	Git operation timeout (seconds)
evaluator	nix_timeout	1800	Nix evaluation timeout (seconds)
evaluator	max_concurrent_evals	4	Maximum concurrent evaluations
evaluator	work_dir	/tmp/fc-evaluator	Working directory for clones
evaluator	restrict_eval	true	Pass --option restrict-eval true to Nix
evaluator	allow_ifd	false	Allow import-from-derivation
queue_runner	workers	4	Concurrent build slots
queue_runner	poll_interval	5	Seconds between build queue polls
queue_runner	build_timeout	3600	Per-build timeout (seconds)
queue_runner	work_dir	/tmp/fc-queue-runner	Working directory for builds
gc	enabled	true	Manage GC roots for build outputs
gc	gc_roots_dir	/nix/var/nix/gcroots/per-user/fc/fc-roots	GC roots directory
gc	max_age_days	30	Remove GC roots older than N days
gc	cleanup_interval	3600	GC cleanup interval (seconds)
logs	log_dir	/var/lib/fc/logs	Build log storage directory
logs	compress	false	Compress stored logs
cache	enabled	true	Serve a Nix binary cache at /nix-cache/
cache	secret_key_file	none	Signing key for binary cache
signing	enabled	false	Sign build outputs
signing	key_file	none	Signing key file path
notifications	run_command	none	Command to run on build completion
notifications	github_token	none	GitHub token for commit status updates
notifications	gitea_url	none	Gitea/Forgejo instance URL
notifications	gitea_token	none	Gitea/Forgejo API token

Database

FC uses PostgreSQL with sqlx fcompile-time query checking. Migrations live in crates/common/migrations/ and are added usually when the database schema changes.

# Run pending migrations
$ cargo run --bin fc-migrate -- up <database_url>

# Validate schema
$ cargo run --bin fc-migrate -- validate <database_url>

# Create new migration file
$ cargo run --bin fc-migrate -- create <name>

Database tests gracefully skip when PostgreSQL is unavailable. To run the database tests, make sure you build the test VMs.

NixOS Deployment

FC ships a NixOS module at nixosModules.default. Minimal configuration:

{
  inputs.fc.url = "github:feel-co/ci";

  outputs = { self, nixpkgs, fc, ... }: {
    nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
      modules = [
        fc.nixosModules.default
        {
          services.fc = {
            enable = true;
            package = fc.packages.x86_64-linux.server;
            migratePackage = fc.packages.x86_64-linux.migrate-cli;

            server.enable = true;
            # evaluator.enable = true;
            # queueRunner.enable = true;
          };
        }
      ];
    };
  };
}

Full Deployment Example

A complete production configuration with all three daemons and NGINX reverse proxy:

{ config, pkgs, fc, ... }: {
  services.fc = {
    enable = true;
    package = fc.packages.x86_64-linux.server;
    migratePackage = fc.packages.x86_64-linux.migrate-cli;

    server.enable = true;
    evaluator.enable = true;
    queueRunner.enable = true;

    settings = {
      database.url = "postgresql:///fc?host=/run/postgresql";
      server.host = "127.0.0.1";
      server.port = 3000;

      evaluator.poll_interval = 300;
      evaluator.restrict_eval = true;
      queue_runner.workers = 8;
      queue_runner.build_timeout = 7200;

      gc.enabled = true;
      gc.max_age_days = 90;
      cache.enabled = true;
      logs.log_dir = "/var/lib/fc/logs";
      logs.compress = true;
    };
  };

  # Reverse proxy
  services.nginx = {
    enable = true;
    virtualHosts."ci.example.org" = {
      forceSSL = true;
      enableACME = true;
      locations."/" = {
        proxyPass = "http://127.0.0.1:3000";
        proxyWebsockets = true;
        extraConfig = ''
          # FIXME: you might choose to harden this part further
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          client_max_body_size 50M;
        '';
      };
    };
  };

  # Firewall
  networking.firewall.allowedTCPPorts = [ 80 443 ];
}

Multi-Machine Deployment

For larger or distributed setups, you may choose to run the daemons on different machines sharing the same database. For example:

• Head node: runs fc-server and fc-evaluator, has the PostgreSQL database locally
• Builder machines: run fc-queue-runner, connect to the head node's database via postgresql://fc@headnode/fc

On builder machines, set database.createLocally = false and provide the remote database URL:

{
  services.fc = {
    enable = true;
    database.createLocally = false; # <- Set this
    queueRunner.enable = true;

    # Now configure the database
    settings.database.url = "postgresql://fc@headnode.internal/fc";
    settings.queue_runner.workers = 16;
  };
}

Ensure the PostgreSQL server on the head node allows connections from builder machines via pg_hba.conf (the NixOS services.postgresql module handles this with authentication settings).

Authentication

FC supports two authentication methods:

1. API Keys - Bearer token authentication for API access
2. User Accounts - Username/password with session cookies for dashboard access

API Key Bootstrapping

FC uses SHA-256 hashed API keys stored in the api_keys table. To create the first admin key after initial deployment:

# Generate a key and its hash
$ export FC_KEY="fc_$(openssl rand -hex 16)"
$ export FC_HASH=$(echo -n "$FC_KEY" | sha256sum | cut -d' ' -f1)

# Insert into the database
$ sudo -u fc psql -U fc -d fc -c \
  "INSERT INTO api_keys (name, key_hash, role) VALUES ('admin', '$FC_HASH', 'admin')"

# Save the key (it cannot be recovered from the hash)
$ echo "Admin API key: $FC_KEY"

Subsequent keys can be created via the API or the admin dashboard using this initial admin key.

User Management

FC supports user accounts for dashboard access. Users can authenticate with username/password and get a session cookie.

Creating Users

Users can be created via the API using an admin API key:

# Create a new user
curl -X POST http://localhost:3000/api/v1/users \
  -H 'Authorization: Bearer fc_demo_admin_key' \
  -H 'Content-Type: application/json' \
  -d '{
    "username": "developer",
    "email": "dev@example.com",
    "password": "secure-password-here",
    "role": "admin"
  }'

User Roles

Users can have roles that control their access level:

Role	Description
admin	Full access to all endpoints
read-only	Read-only access (GET requests only)
custom	See role-based permissions below

Users inherit permissions from their role. The role can be set to any string for custom permission schemes.

Dashboard Login

The dashboard supports both authentication methods:

• API Key Login: Enter your API key on the login page for a session cookie
• Username/Password: Log in with user credentials for persistent sessions

Session cookies are valid for 24 hours and allow access to admin features without re-entering credentials.

Roles

Note

Roles are an experimental feature designed to bring FC on-par with enterprise-grade Hydra deployments. The feature is currently unstable, and might change at any given time. Do not rely on roles for the time being.

Role	Permissions
admin	Full access to all endpoints
read-only	Read-only access (GET requests only)
create-projects	Create projects and jobsets
eval-jobset	Trigger evaluations
cancel-build	Cancel builds
restart-jobs	Restart failed/completed builds
bump-to-front	Bump build priority

Monitoring

FC exposes a limited, Prometheus-compatible metrics endpoint at /metrics. The metrics provided are detailed below:

Available Metrics

Metric	Type	Description
fc_builds_total	gauge	Total number of builds
fc_builds_pending	gauge	Currently pending builds
fc_builds_running	gauge	Currently running builds
fc_builds_completed	gauge	Completed builds
fc_builds_failed	gauge	Failed builds
fc_projects_total	gauge	Total number of projects
fc_evaluations_total	gauge	Total number of evaluations
fc_channels_total	gauge	Total number of channels
fc_remote_builders_total	gauge	Configured remote builders

Prometheus Configuration

scrape_configs:
  - job_name: "fc-ci"
    static_configs:
      - targets: ["ci.example.org:3000"]
    metrics_path: "/metrics"
    scrape_interval: 30s

Backup & Restore

FC stores all state in PostgreSQL. To back up:

# Create a backup
$ pg_dump -U fc fc > fc-backup-$(date +%Y%m%d).sql

To restore:

# Restore a backup
$ psql -U fc fc < fc-backup-20250101.sql

Build logs are stored in the filesystem at the configured logs.log_dir (default: /var/lib/fc/logs). You are generally encouraged to include this directory in your backup strategy to ensure more seamless recoveries in the case of a catastrophic failure. Build outputs live in the Nix store and are protected by GC roots under gc.gc_roots_dir. These do not need separate backup as long as derivation paths are retained in the database.

API Overview

All API endpoints are under /api/v1. Write operations require a Bearer token in the Authorization header. Read operations (GET) are public.

Method	Endpoint	Auth	Description
GET	/health	-	Health check with database status
GET	/metrics	-	Prometheus metrics
GET	/api/v1/projects	-	List projects (paginated)
POST	/api/v1/projects	admin/create-projects	Create project
GET	/api/v1/projects/{id}	-	Get project details
PUT	/api/v1/projects/{id}	admin	Update project
DELETE	/api/v1/projects/{id}	admin	Delete project (cascades)
GET	/api/v1/projects/{id}/jobsets	-	List project jobsets
POST	/api/v1/projects/{id}/jobsets	admin/create-projects	Create jobset
GET	/api/v1/projects/{pid}/jobsets/{id}	-	Get jobset
PUT	/api/v1/projects/{pid}/jobsets/{id}	admin	Update jobset
DELETE	/api/v1/projects/{pid}/jobsets/{id}	admin	Delete jobset
GET	/api/v1/evaluations	-	List evaluations (filtered)
GET	/api/v1/evaluations/{id}	-	Get evaluation details
POST	/api/v1/evaluations/trigger	admin/eval-jobset	Trigger evaluation
GET	/api/v1/builds	-	List builds (filtered)
GET	/api/v1/builds/{id}	-	Get build details
POST	/api/v1/builds/{id}/cancel	admin/cancel-build	Cancel build
POST	/api/v1/builds/{id}/restart	admin/restart-jobs	Restart build
POST	/api/v1/builds/{id}/bump	admin/bump-to-front	Bump priority
GET	/api/v1/builds/stats	-	Build statistics
GET	/api/v1/builds/recent	-	Recent builds
GET	/api/v1/channels	-	List channels
POST	/api/v1/channels	admin	Create channel
DELETE	/api/v1/channels/{id}	admin	Delete channel
POST	/api/v1/channels/{id}/promote/{eval}	admin	Promote evaluation
GET	/api/v1/api-keys	admin	List API keys
POST	/api/v1/api-keys	admin	Create API key
DELETE	/api/v1/api-keys/{id}	admin	Delete API key
GET	/api/v1/admin/builders	-	List remote builders
POST	/api/v1/admin/builders	admin	Create remote builder
PUT	/api/v1/admin/builders/{id}	admin	Update remote builder
DELETE	/api/v1/admin/builders/{id}	admin	Delete remote builder
GET	/api/v1/admin/system	admin	System status
GET	/api/v1/users	admin	List users
POST	/api/v1/users	admin	Create user
GET	/api/v1/users/{id}	admin	Get user details
PUT	/api/v1/users/{id}	admin	Update user
DELETE	/api/v1/users/{id}	admin	Delete user
GET	/api/v1/me	user/api key	Get current user
PUT	/api/v1/me	user	Update current user
POST	/api/v1/me/password	user	Change password
GET	/api/v1/me/starred-jobs	user	List starred jobs
POST	/api/v1/me/starred-jobs	user	Star a job
DELETE	/api/v1/me/starred-jobs/{id}	user	Unstar a job
GET	/api/v1/search?q=	-	Search projects and builds
GET	/api/v1/search/quick?q=	-	Quick search (backward compatible)
POST	/api/v1/webhooks/{pid}/github	HMAC	GitHub webhook
POST	/api/v1/webhooks/{pid}/gitea	HMAC	Gitea/Forgejo webhook
GET	/nix-cache/nix-cache-info	-	Binary cache info
GET	/nix-cache/{hash}.narinfo	-	NAR info lookup
GET	/nix-cache/nar/{hash}.nar.zst	-	NAR download (zstd)

Dashboard

The web dashboard is available at the root URL (/). Pages include:

• / - Home: build stats, project overview, recent builds and evaluations
• /projects - Project listing with create form (admin)
• /project/{id} - Project detail with jobsets, add jobset form (admin)
• /evaluations - Evaluation listing with project/jobset context
• /builds - Build listing with status/system/job filters
• /queue - Current queue (pending + running builds)
• /channels - Channel listing
• /admin - System status, API key management, remote builder management, user management
• /login - Session-based login supporting both username/password and API key authentication
• /logout - Log out and clear session