From e36693ac3f8af046a329ec62d439a84e17eaf7bd Mon Sep 17 00:00:00 2001 From: NotAShelf Date: Sat, 21 Feb 2026 22:54:48 +0300 Subject: [PATCH] docs: document project architechture Signed-off-by: NotAShelf Change-Id: Iaa99d706d61857fbd51d3b757b5066ab6a6a6964 --- README.md | 182 +++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 153 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index e99dc1e..1048d9c 100644 --- a/README.md +++ b/README.md @@ -7,8 +7,6 @@ where the plugin parses and compiles Nix code at runtime, or; **ahead-of-time** compilation where the `nix-irc` tool pre-compiles `.nix` files into `.nixir` files. -The plugin automatically chooses the fastest path based on file availability. - ## Supported Nix Constructs - Literals: integers, strings, booleans, null, paths @@ -27,6 +25,145 @@ The plugin automatically chooses the fastest path based on file availability. `||`, `->` - Unary: `-`, `!` +## Overview + +Nixir is a Nix evaluator plugin that compiles Nix expressions to a custom binary +intermediate representation (IR). Think of it like a compiler for Nix: it +translates human-readable Nix code into a compact, fast-to-execute format that +runs in a custom virtual machine. + +The plugin works in two ways: + +1. **Ahead-of-time**: Use the `nix-irc` tool to compile `.nix` files to `.nixir` + once, then load them instantly +2. **On-the-fly**: Let the plugin parse and compile Nix code at runtime when you + need it + +While Nixir _is_ designed as a toy research project, I _envision_[^1] a few +_potential_ use cases built around working with Nix. Sure, you _probably_ would +not go work with Nix willingly, science is not about why, it is about _why not_. + +Some potential use cases for Nixir _might_ include: + +- **CI/CD Acceleration**: Pre-compile stable Nix expressions to `.nixir` for + faster repeated evaluation in CI pipelines +- **Embedded Nix**: Use Nix as a configuration language in C++ applications + without bundling the full evaluator +- **Plugin Ecosystem**: Extend Nix with custom evaluation strategies via the + plugin API +- **Build Caching**: Cache compiled IR alongside source for instant startup of + Nix-based tools + +[^1]: I'm not entirely convinced either, do not ask. + +### Architecture + +```mermaid +flowchart TD + + subgraph Source["User Source"] + A[".nix Source Files"] + end + + subgraph Compiler["External Tool: nix-irc"] + B1["Parse Nix"] + B2["Static Import Resolution"] + B3["Flatten Import Graph"] + B4["Desugar + De Bruijn Conversion"] + B5["Emit Versioned IR Bundle (.nixir)"] + end + + subgraph IR["IR Bundle"] + C1["Binary IR Format"] + C2["Versioned Header"] + C3["No Names, Indexed Vars"] + end + + subgraph Plugin["nix-ir-plugin.so"] + D1["Primop Registration"] + D2["prim_loadIR"] + D3["prim_compileNix"] + D4["prim_info"] + end + + subgraph CompilePath["On-the-fly Path"] + E1["Parse Source String"] + E2["IR Generation"] + end + + subgraph LoadPath["Pre-compiled Path"] + F1["Deserialize .nixir"] + end + + subgraph VM["Custom Lazy VM"] + G1["Heap-Allocated Thunks"] + G2["Memoization"] + G3["Cycle Detection"] + G4["Closure Environments (Array-Based)"] + G5["FORCE / THUNK Execution"] + end + + A --> B1 + B1 --> B2 + B2 --> B3 + B3 --> B4 + B4 --> B5 + B5 --> C1 + + C1 --> D1 + + D2 -->|explicit| F1 + F1 --> G1 + + D3 -->|explicit| E1 + E1 --> E2 + E2 --> G1 + + G1 -.-> G2 -.-> G3 -.-> G4 -.-> G5 +``` + +The same compiler code runs both in the standalone `nix-irc` CLI tool and inside +the plugin for on-the-fly compilation. This ensures consistent behavior between +pre-compiled and runtime-compiled paths. The intermediate representation (IR) +design uses De Brujin indices instead of names for variable binding, which +eliminates string lookup and the binary format uses a versioned header +(`0x4E495258`). In addition, we make use of string interning for repeated +identifiers and type-tagged nodes for efficient dispatching. + +The runtime implements lazy evaluation using heap-allocated thunks. Each thunk +holds a delayed computation and is evaluated at most once through memoization. +Recursive definitions are handled through a blackhole mechanism that detects +cycles at runtime. Variable lookup uses array-based closure environments, +providing O(1) access by index rather than name-based lookup. + +The plugin integrates with Nix through the `RegisterPrimOp` API, exposing three +operations: `nixIR_loadIR` for loading pre-compiled `.nixir` bundles, +`nixIR_compile` for on-the-fly compilation, and `nixIR_info` for metadata. This +integration path is compatible with Nix 2.32+. + +### IR Format + +The `.nixir` files use a versioned binary format: + +```plaintext +Header: + - Magic: 0x4E495258 ("NIRX") + - Version: 1 (uint32) + - Source count: uint32 + - Import count: uint32 + - String table size: uint32 + +String Table: + - Interned strings for efficient storage + +Nodes: + - Binary encoding of IR nodes + - Each node has type tag + inline data + +Entry: + - Main expression node index +``` + ## Usage ### Building @@ -57,6 +194,8 @@ $ nix-irc --no-imports input.nix output.nixir ### Runtime Evaluation (Plugin) + + ```bash # Load the plugin and evaluate IR $ nix --plugin-files ./nix-ir-plugin.so eval --expr 'builtins.nixIR_loadIR "output.nixir"' @@ -68,6 +207,8 @@ $ nix --plugin-files ./nix-ir-plugin.so eval --expr 'builtins.nixIR_compile "1 + $ nix --plugin-files ./nix-ir-plugin.so eval --expr 'builtins.nixIR_info' ``` + + ### Running Tests ```bash @@ -80,36 +221,19 @@ done $ hexdump -C tests/simple.nixir | head -3 ``` -## IR Format - -The `.nixir` files use a versioned binary format: - -```plaintext -Header: - - Magic: 0x4E495258 ("NIRX") - - Version: 1 (uint32) - - Source count: uint32 - - Import count: uint32 - - String table size: uint32 - -String Table: - - Interned strings for efficient storage - -Nodes: - - Binary encoding of IR nodes - - Each node has type tag + inline data - -Entry: - - Main expression node index -``` - ## Contributing -This is a research/experimental project. Contributions welcome! +This is a research project (with no formal association, i.e., no thesis or +anything) that I'm working on entirely for fun and out of curiousity. Extremely +experimental, could change any time. While I do not suggest running this project +in a serious context, I am happy to receive any kind of feedback you might have. +You will notice _very_ quickly that I'm a little out of my depth, and the code +is in a rough shape. Areas where help is needed: -Areas where help is needed: - -- Expanding parser to handle more Nix syntax +- Compiler semantics - Performance optimization - Test coverage - Documentation improvements +- Expanding parser to handle more Nix syntax (module system in particular) + +Contributions _are_ welcome!