docs: document project architechture
Signed-off-by: NotAShelf <raf@notashelf.dev> Change-Id: Iaa99d706d61857fbd51d3b757b5066ab6a6a6964
This commit is contained in:
parent
2539ff7ca3
commit
e36693ac3f
GPG key ID:
29D95B64378DB4BF
1 changed files with 153 additions and 29 deletions
182
README.md
182
README.md
|
|
@ -7,8 +7,6 @@ where the plugin parses and compiles Nix code at runtime, or; **ahead-of-time**
|
|||
compilation where the `nix-irc` tool pre-compiles `.nix` files into `.nixir`
|
||||
files.
|
||||
|
||||
The plugin automatically chooses the fastest path based on file availability.
|
||||
|
||||
## Supported Nix Constructs
|
||||
|
||||
- Literals: integers, strings, booleans, null, paths
|
||||
|
|
@ -27,6 +25,145 @@ The plugin automatically chooses the fastest path based on file availability.
|
|||
`||`, `->`
|
||||
- Unary: `-`, `!`
|
||||
|
||||
## Overview
|
||||
|
||||
Nixir is a Nix evaluator plugin that compiles Nix expressions to a custom binary
|
||||
intermediate representation (IR). Think of it like a compiler for Nix: it
|
||||
translates human-readable Nix code into a compact, fast-to-execute format that
|
||||
runs in a custom virtual machine.
|
||||
|
||||
The plugin works in two ways:
|
||||
|
||||
1. **Ahead-of-time**: Use the `nix-irc` tool to compile `.nix` files to `.nixir`
|
||||
once, then load them instantly
|
||||
2. **On-the-fly**: Let the plugin parse and compile Nix code at runtime when you
|
||||
need it
|
||||
|
||||
While Nixir _is_ designed as a toy research project, I _envision_[^1] a few
|
||||
_potential_ use cases built around working with Nix. Sure, you _probably_ would
|
||||
not go work with Nix willingly, science is not about why, it is about _why not_.
|
||||
|
||||
Some potential use cases for Nixir _might_ include:
|
||||
|
||||
- **CI/CD Acceleration**: Pre-compile stable Nix expressions to `.nixir` for
|
||||
faster repeated evaluation in CI pipelines
|
||||
- **Embedded Nix**: Use Nix as a configuration language in C++ applications
|
||||
without bundling the full evaluator
|
||||
- **Plugin Ecosystem**: Extend Nix with custom evaluation strategies via the
|
||||
plugin API
|
||||
- **Build Caching**: Cache compiled IR alongside source for instant startup of
|
||||
Nix-based tools
|
||||
|
||||
[^1]: I'm not entirely convinced either, do not ask.
|
||||
|
||||
### Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
|
||||
subgraph Source["User Source"]
|
||||
A[".nix Source Files"]
|
||||
end
|
||||
|
||||
subgraph Compiler["External Tool: nix-irc"]
|
||||
B1["Parse Nix"]
|
||||
B2["Static Import Resolution"]
|
||||
B3["Flatten Import Graph"]
|
||||
B4["Desugar + De Bruijn Conversion"]
|
||||
B5["Emit Versioned IR Bundle (.nixir)"]
|
||||
end
|
||||
|
||||
subgraph IR["IR Bundle"]
|
||||
C1["Binary IR Format"]
|
||||
C2["Versioned Header"]
|
||||
C3["No Names, Indexed Vars"]
|
||||
end
|
||||
|
||||
subgraph Plugin["nix-ir-plugin.so"]
|
||||
D1["Primop Registration"]
|
||||
D2["prim_loadIR"]
|
||||
D3["prim_compileNix"]
|
||||
D4["prim_info"]
|
||||
end
|
||||
|
||||
subgraph CompilePath["On-the-fly Path"]
|
||||
E1["Parse Source String"]
|
||||
E2["IR Generation"]
|
||||
end
|
||||
|
||||
subgraph LoadPath["Pre-compiled Path"]
|
||||
F1["Deserialize .nixir"]
|
||||
end
|
||||
|
||||
subgraph VM["Custom Lazy VM"]
|
||||
G1["Heap-Allocated Thunks"]
|
||||
G2["Memoization"]
|
||||
G3["Cycle Detection"]
|
||||
G4["Closure Environments (Array-Based)"]
|
||||
G5["FORCE / THUNK Execution"]
|
||||
end
|
||||
|
||||
A --> B1
|
||||
B1 --> B2
|
||||
B2 --> B3
|
||||
B3 --> B4
|
||||
B4 --> B5
|
||||
B5 --> C1
|
||||
|
||||
C1 --> D1
|
||||
|
||||
D2 -->|explicit| F1
|
||||
F1 --> G1
|
||||
|
||||
D3 -->|explicit| E1
|
||||
E1 --> E2
|
||||
E2 --> G1
|
||||
|
||||
G1 -.-> G2 -.-> G3 -.-> G4 -.-> G5
|
||||
```
|
||||
|
||||
The same compiler code runs both in the standalone `nix-irc` CLI tool and inside
|
||||
the plugin for on-the-fly compilation. This ensures consistent behavior between
|
||||
pre-compiled and runtime-compiled paths. The intermediate representation (IR)
|
||||
design uses De Brujin indices instead of names for variable binding, which
|
||||
eliminates string lookup and the binary format uses a versioned header
|
||||
(`0x4E495258`). In addition, we make use of string interning for repeated
|
||||
identifiers and type-tagged nodes for efficient dispatching.
|
||||
|
||||
The runtime implements lazy evaluation using heap-allocated thunks. Each thunk
|
||||
holds a delayed computation and is evaluated at most once through memoization.
|
||||
Recursive definitions are handled through a blackhole mechanism that detects
|
||||
cycles at runtime. Variable lookup uses array-based closure environments,
|
||||
providing O(1) access by index rather than name-based lookup.
|
||||
|
||||
The plugin integrates with Nix through the `RegisterPrimOp` API, exposing three
|
||||
operations: `nixIR_loadIR` for loading pre-compiled `.nixir` bundles,
|
||||
`nixIR_compile` for on-the-fly compilation, and `nixIR_info` for metadata. This
|
||||
integration path is compatible with Nix 2.32+.
|
||||
|
||||
### IR Format
|
||||
|
||||
The `.nixir` files use a versioned binary format:
|
||||
|
||||
```plaintext
|
||||
Header:
|
||||
- Magic: 0x4E495258 ("NIRX")
|
||||
- Version: 1 (uint32)
|
||||
- Source count: uint32
|
||||
- Import count: uint32
|
||||
- String table size: uint32
|
||||
|
||||
String Table:
|
||||
- Interned strings for efficient storage
|
||||
|
||||
Nodes:
|
||||
- Binary encoding of IR nodes
|
||||
- Each node has type tag + inline data
|
||||
|
||||
Entry:
|
||||
- Main expression node index
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Building
|
||||
|
|
@ -57,6 +194,8 @@ $ nix-irc --no-imports input.nix output.nixir
|
|||
|
||||
### Runtime Evaluation (Plugin)
|
||||
|
||||
<!--markdownlint-disable MD013-->
|
||||
|
||||
```bash
|
||||
# Load the plugin and evaluate IR
|
||||
$ nix --plugin-files ./nix-ir-plugin.so eval --expr 'builtins.nixIR_loadIR "output.nixir"'
|
||||
|
|
@ -68,6 +207,8 @@ $ nix --plugin-files ./nix-ir-plugin.so eval --expr 'builtins.nixIR_compile "1 +
|
|||
$ nix --plugin-files ./nix-ir-plugin.so eval --expr 'builtins.nixIR_info'
|
||||
```
|
||||
|
||||
<!--markdownlint-enable MD013-->
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
|
|
@ -80,36 +221,19 @@ done
|
|||
$ hexdump -C tests/simple.nixir | head -3
|
||||
```
|
||||
|
||||
## IR Format
|
||||
|
||||
The `.nixir` files use a versioned binary format:
|
||||
|
||||
```plaintext
|
||||
Header:
|
||||
- Magic: 0x4E495258 ("NIRX")
|
||||
- Version: 1 (uint32)
|
||||
- Source count: uint32
|
||||
- Import count: uint32
|
||||
- String table size: uint32
|
||||
|
||||
String Table:
|
||||
- Interned strings for efficient storage
|
||||
|
||||
Nodes:
|
||||
- Binary encoding of IR nodes
|
||||
- Each node has type tag + inline data
|
||||
|
||||
Entry:
|
||||
- Main expression node index
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
This is a research/experimental project. Contributions welcome!
|
||||
This is a research project (with no formal association, i.e., no thesis or
|
||||
anything) that I'm working on entirely for fun and out of curiousity. Extremely
|
||||
experimental, could change any time. While I do not suggest running this project
|
||||
in a serious context, I am happy to receive any kind of feedback you might have.
|
||||
You will notice _very_ quickly that I'm a little out of my depth, and the code
|
||||
is in a rough shape. Areas where help is needed:
|
||||
|
||||
Areas where help is needed:
|
||||
|
||||
- Expanding parser to handle more Nix syntax
|
||||
- Compiler semantics
|
||||
- Performance optimization
|
||||
- Test coverage
|
||||
- Documentation improvements
|
||||
- Expanding parser to handle more Nix syntax (module system in particular)
|
||||
|
||||
Contributions _are_ welcome!
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue