Skip to content

The Road to Nix

I'm writing this as someone brand new to RedoxOS but with Nix experience, working at Flox, having done a stint on the Nix Docs team, having spoken at NixCon, and organized NixCon NA (hosted at SCALE). That is to say, I have Nix opinions, but I don't really know the norms or broader context of how the RedoxOS community operates, and I don't want to be "that guy" that joins a community and immediately tells them what they should change. For that reason, there's a ladder of Nix builds that I'm going to present here and we can choose to stop at any rung of that ladder.

I'm also writing this down so that if I disappear or burn out there's a roadmap for someone else to pick up where I've left off.

Goals

  • Anyone that wants to contribute to RedoxOS should be able to.
  • We should be able to provide easy on-ramps to contribution.
  • We should provide support for setting up a developer environment that isn't onerous.
  • We should be able to build RedoxOS and the components of its ecosystem in a reliable, reproducible way.

Nix satistfies many of these in the sense that if someone sets up the various Nix builds and development shells, someone without any Nix knowledge should be able to install Nix, clone a RedoxOS repo, run nix develop, and be good to start digging in.

Whether Nix becomes the de facto method for building RedoxOS is a different question. That is to say, you can provide Nix development shells without going all-in on Nix builds. How far in that direction we go is something we should decide.

The Nix Ladder

The Nix ladder for RedoxOS would look something like this:

  • Get a build of RedoxOS working at all for x86_64-linux hosts.
    • This would use as much of the existing build system as possible.
    • Even this step requires significant work (detailed later).
  • Automatically populate a binary cache from a Hydra-like CI server (explained later).
  • Get a build of RedoxOS working at all for other high-priority systems (aarch64-linux, aarch64-darwin).
  • Make Nix the blessed way to build RedoxOS.
  • Migrate from the existing build system to Nix-native builds.
  • Stretch goal: enable cross-compilation.

As I mentioned before, we can choose to get off that ladder at any point, but I'll now discuss what even the first rung of that ladder looks like.

Supporting Nix builds

Context: how Nix builds work

When you perform a Nix build a hash is computed from the inputs to the build. It's a common misconception that Nix builds are content addressed (where "content" refers to the contents of the build artifact), but Nix builds are actually input addressed. Nix takes that hash, the package name, and the package version to determine the location where the artifact will be placed:

/nix/store/<hash>-<package name>-<version>

This is called a "store path" as it is a path within the Nix store.

A Nix build will determine a store path for each transitive dependency in the closure for a build, all the way down to libc. Nix is fundamentally a source-based build system, so the fallback is always to build the contents of a store path from source. However, building everything from source all the time is a huge pain, so Nix makes use of distributed caching via what it calls "substituters" and "binary caches".

Rather than building the artifacts in the store path, Nix will first search a list of "substituters" for that store path. The idea is that if a store path with the same hash is present somewhere else, someone has already performed a build with the same inputs, and given the invariant that a build with all the same inputs should produce the same artifact, you can substitute the build artifact in place of performing the build yourself.

The first substituter that's checked is your local Nix store (/nix/store). If that fails, then cache.nixos.org is checked. If that fails, you continue checking any other substituters you have configured.

If the store path can't be substituted, you must build the artifact yourself (though you can farm this out to other machines using a feature called "remote builders").

A common source for needing to build something yourself is if you've applied your own patches to something in the dependency tree, for example adding support for a new operating system.

Infrastructure

Binary cache

The "standard library", so to speak, for Nix builds is called stdEnv, and it includes very little, mostly gcc, make, bash, and some other very common build necessities. This sits somewhere very near the bottom of the dependency tree for any Nix build as it's a fundamental input for basically any build.

Given that we have our own patched version of gcc, call it gccRedox, that means any artifact that has gccRedox anywhere in its dependency tree will not benefit from the existing binary cache on cache.nixos.org. With no cache at all, we'll be building everything from source all the time, and that's not practical.

Therefore, we need a binary cache so that we only have to build things ourselves once. There are a few options here.

  • Cachix provides a turnkey solution for a reasonable price and it's widely used in the Nix community. It also provides a dashboard for monitoring, and some facilities for making sure certain artifacts aren't garbage collected.
  • Self host. There's a few different ways you can do this.
    • Any machine available over SSH can act as a binary cache, but binary caches can get...large, so disk space would probably be expensive. I don't think this option is a good idea.
    • An S3-compatible storage provider. Examples:
    • A self-hosted "binary cache in a box": Attic (still requires an S3-compatible backing store)

At this stage my preference would be Cachix. The owner contributes back to the Nix ecosystem, and it's a low-overhead solution for getting Nix builds off the ground. I've asked what egress looks like if we ever graduate to managing our own infrastructure, but I haven't received a response (yet).

Package repository (redoxpkgs)

The real wealth of the Nix ecosystem is the collection of build "recipes" in the nixpkgs repository. No actual source code for the packages are stored there, just the recipes for fetching the sources and performing the build. We would need a similar setup i.e. we need a repository with build recipes for each component of the RedoxOS ecosystem, or at least the ones that we care about building with Nix. A similar repository exists today at redox-os/redox-nix, but it doesn't have the same layout as nixpkgs, and redoxpkgs better fits with Nix ecosystem conventions.

nixpkgs is a huge package repository that has grown organically over many years, which makes it inconsistent in places and thus not a great example of what a package collection would look like for an ecosystem of our size. A better example is the ngipkgs repository, which contains Nix packages for software funded by NGI. Note that ngipkgs also uses Cachix for a binary cache.

The benefit of having all of the build recipes in a single repository is that you can ensure that all of the build recipes in a single revision share a set of dependencies (the other packages in that revision), and that they all build together. The nixpkgs repository has a CI system called Hydra that on each PR determines which packages in the entire build graph need to be rebuilt and checked as a result of the changes in the PR (e.g. if the PR touches gcc, anything that transitively depends on gcc would be rebuilt and verified that it still builds). This also serves as an automatic binary cache populator. Anecdotally I've heard that Hydra can be a pain to operate, so I'm not sure if that's exactly the route we would go or if there's a better solution. Homework is required there, and a Hydra-like setup is probably not necessary for getting things off the ground.

In my mind, the major downside to how nixpkgs is structured stems from the fact that it's meant to provide a collection of build recipes for packages that may or may not themselves be built with Nix. Regardless of whether a third-party package has its own default.nix or flake.nix for building it, there will still be a build recipe in nixpkgs that serves as the canonical build recipe for the rest of the Nix ecosystem. That seems like wasted effort to me, and with an ecosystem our size it seems likely that two different build recipes would quickly fall out of date. Perhaps there's a better way to share those build recipes for projects that are already built with Nix, but I would need to think on that.

The structure of the repository would look something like this:

redoxpkgs/
    flake.nix
    pkgs/
      by-name (?)/
        kernel/
          default.nix # build recipe
        redoxfs/
          default.nix
        ...

This is the layout that nixpkgs is slowly migrating to, and is the layout that ngipkgs has followed as a result. The (?) is there because the whole reason that the by-name subdirectory exists is that nixpkgs is huge and its package organization is ad-hoc to the point that it can be hard to decide where to put a new package. The by-name subdirectory was created as a new package organization scheme to take the guesswork out of things and allow programmatic checking in CI that a new package has been placed in the right place. I'm not sure if it's really something we need even in the long run or if we can just put everything in one flat directory in perpetuity (I imagine it will be a long time before redoxpkgs hits 100k+ packages).

Supporting builds on other high priority systems

I expect a lot of this will mostly work out of the box for aarch64-linux. It's hard to predict what builds will look like for aarch64-darwin. By default stdEnv uses clang for builds on macOS, but again we have our own patched gcc, so when and where we use gccRedox vs clang will emerge as I try to get a Nix build working at all.

I don't count x86_64-darwin as a high priority system. At Flox we build for all four systems ({aarch64,x86_64}-{darwin,linux}) and our x86_64-darwin jobs take significantly longer than all of the other jobs. By the time we're even at this point those Intel Macs will be well outside the support window anyway.

That's not to say those systems should all be turned into e-waste, just that we shouldn't spend our limited resources supporting RedoxOS builds on those systems unless someone wants to step up and be the maintainer.

Make Nix the default build system

Once we know we can build on the systems that we care about, we can flip the switch and make Nix the default. At this point we'll want the infrastructure in place to automatically populate the binary cache, automatically check builds in CI, etc.

Migrating to Nix-native builds

This can be done piecemeal on whatever schedule we want. The beauty of Nix is that a build that consumes a package doesn't have to care at all about how a package is built.

In order to build something with Nix you have to write a Nix-build for it, so what do I mean by "Nix-native" build? I imagine that as part of supporting Nix builds in a world where Nix isn't the blessed way to build RedoxOS, there will be several instances of ifeq($(NIX_BUILD), 1) sprinkled throughout the Makefiles in the ecosystem. At this point we would be getting rid of those, making the Nix stuff the default, and doing any refactoring we want to better fit the Nix-first approach.

Support cross-compilation

One of Nix's super-powers is native cross-compilation support. Sometimes all you need to do in order to build something for a different system is replace a pkgs with pkgs.pkgsCross.<target system>. It's pretty magical when you get it all figured out.

I don't think this is strictly necessary, but it would make building images for systems like RISC-V easier for everyone.

Edited by Zach Mitchell