Software like nix excites me. After working in software for about ten years now, I've learned that making software reproducible is extremely valuable. In short, meaning you rarely if ever run into, "works on my machine". This is the major value proposition of nix:

Declarative builds and deployments. Nix is a tool that takes a unique approach to package management and system configuration. Learn how to make reproducible, declarative and reliable systems.

Chef and capistrano were some of the first tools I used for software building deployment, followed by ansible and terraform, and then docker and kubernetes. The stack keeps evolving and changing.

The closest I've gotten to reproducibility is with some bash scripting, compute metadata, and build rules in CI with docker and kubernetes deployments. I still run into "works on my machine" every once in a while, and I'm curious if nix is a tool that I could use to remove the need to manage all that scripting and extra servicing and eliminate the problem once and for all. Let's cover a little nix primer and see what I think.

Installing Nix

I've dug around for the best way to install nix, and from what I've found so far, I'd recommend using nix-installer. Alas it's curling and piping to sh but it works and is easy to uninstall 🤞

Run nix to make sure you've installed nix correctly. I'm using 2.17.0 at the time of writing this post.

Next, run nix doctor to make sure your system is ready to start building with nix. Note that nix will be a bit confused if you have a series of :./ in your path. You should remove that and keep going. You should see the following when nix is ready;

$ nix doctor
[PASS] PATH contains only one nix version.
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: daemon

Reproducibility

It's really easy to get started with nix. The first thing you can do is play around with nix-shell to see what it's like to automatically access scripts you don't have installed on your local machine in a nix-shell.. Cool right? It's similar to running and exec-ing into a docker container. What's comes next is interesting because the "docker in docker" scenario falls away with nested sessions.

So cool right! So how does this mean we're making reproducible software?

Just because one developer runs nix-shell -p git python3, two developers might download two different versions of git and python. Not very reproducible right?

Running something like the following ensures reproducibility;

nix-shell -p git -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/2a601aafdc5605a5133a2ca506a34a3a73377247.tar.gz

Here we provided a specific Git revision of nixpkgs, leaving no doubt about which version of the packages in that collection will be used.

So how does this differentiate from docker? My developers can still pull containers with the same versions? Let's take a look at shebang scripts to illustrate some differences.

A basic shebang script looks something like this;

#!/usr/bin/env nix-shell
#!nix-shell -p bash cacert curl jq python3Packages.xmljson

curl -s https://github.com/NixOS/nixpkgs/releases.atom | xml2json | jq .

Save that file to my-shebang.sh and run it like ./my-shebang.sh.

Cool right! With one small script, we're able to repeatably run this process anywhere we have nix installed.

In short, I think nix differentiates from docker by being a tool for environment management instead of environment capture.

When we make a container, the contents are captured. When we change the contents, we can't (to my knowledge) save those contents back into the container without rebuilding it.

Nix Language

Yup, Nix has it's own language, and it's not the prettiest. For me, I'd much rather write a Dockerfile and design it such that the layers most likely to change are last, I don't quite have the time or need to use nix such that I'll spend the time and overhead learning this language.

Explore more of the language on your own time, so for now we'll take a look at a basic example:

{ pkgs ? import <nixpkgs> {} }:
let
  message = "hello world";
in
pkgs.mkShell {
  buildInputs = with pkgs; [ cowsay ];
  shellHook = ''
    cowsay ${message}
  '';
}

Save this file as learning.nix and run it with nix-shell learning.nix.

$ nix-shell learning.nix
Restored session: Tue Sep 19 14:28:02 EDT 2023
 _____________
< hello world >
 -------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

[nix-shell:~/Desktop]$

Let's explain learning.nix.

This expression (the first line) is a function that takes an attribute set as an argument. If the argument has the attribute pkgs, it will be used in the function body. Otherwise, by default, import the Nix expression in the file found on the search path <nixpkgs> (which is a function in this case), call the function with an empty attribute set, and use the resulting value.

The name message is bound to the string value "hello world".

The attribute mkShell of the pkgs set is a function that is passed an attribute set as argument. Its return value is also the result of the outer function.

The attribute set passed to mkShell has the attributes buildInputs (set to a list with one element: the cowsay attribute from pkgs) and shellHook (set to an indented string).

The indented string contains an interpolated expression, which will expand the value of message to yield "hello world".

Here's another nix file that we can use that is similar to an earlier example:

{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell {
  buildInputs = with pkgs; [ bash cacert curl jq python3Packages.xmljson ];
  shellHook = ''
    curl -s https://github.com/NixOS/nixpkgs/releases.atom | xml2json | jq .
  '';
}

Save this file to default.nix and run nix-shell. Cool right?

Even though this is super cool, we're not pinning our nixpkgs version! 😱 This means our developers will likely use different versions of curl, jq, etc and things will only work on their machines. The simplest way to make things reproducible is to fetch the required Nixpkgs version as a tarball specified via the relevant Git commit hash.

Picking the commit can be done via https://status.nixos.org, which lists all the releases and the latest commit that has passed all tests.

So update default.nix with the following:

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/nixos-23.05.tar.gz") {} }:
pkgs.mkShell {
  buildInputs = with pkgs; [ bash cacert curl jq python3Packages.xmljson ];
  shellHook = ''
    curl -s https://github.com/NixOS/nixpkgs/releases.atom | xml2json | jq .
  '';
}

The first time you run nix-shell, nix will be really slow because you have no cached dependencies, run it a second time to see the performance boost – and alas - speed and reproducibility!

Dependency Management & Automation

niv seems to be the suggested way to go about managing dependencies. We're not going to cover this but it's important to underscore as when you're working with docker, you don't really have a consistent and reproducible way to bump dependencies in the container unless you want to completely rebuild it.

In terms of more developer environment focus, like automatic virtual environment activation in python, nix has direnv also something I would consider exploring but won't go into detail here.

Flakes & An Example

After reading about nix on the internet, I thought all .nix files were considered flakes. Which are apparently a controversial subject.

From the docs, Nix handles flake files from regular nix files:

The flake.nix file is checked for schema validity. In particular, the metadata fields cannot be arbitrary Nix > expressions. This is to prevent complex, possibly non-terminating computations while querying the metadata.

The entire flake directory is copied to Nix store before evaluation. This allows for effective evaluation caching, > which is relevant for large expressions such as Nixpkgs, but also requires copying the entire flake directory again on each > change.

No external variables, parameters, or impure language values are allowed. It means full reproducibility of a Nix expression, and, by extension, the resulting build instructions by default, but also prohibits parameterization of results by consumers.

So let's stick to the official definition: Technically, a flake is a file system tree that contains a file named flake.nix in its root directory.

This article by Mitchell Hashimoto is a fantastic example of using both nix and docker to build and ship software. I do think it's very involved and would prefer using one or the other for packaging and managing software. With that in mind, let's walk through an example of writing a small FastAPI application and create a dev environment using only nix.

All the code I've used is posted on GitHub.

Clone it and run a shell to see it working:

$ git clone https://github.com/anthonycorletti/fastapi-nix-experiment.git && cd fastapi-nix-experiment

$ nix-shell

[nix-shell:~/Desktop/fastapi-nix-experiment]$ uvicorn
Usage: uvicorn [OPTIONS] APP
Try 'uvicorn --help' for help.

Error: Missing argument 'APP'.

[nix-shell:~/Desktop/fastapi-nix-experiment]$ uvicorn app:app.app
ERROR:    Error loading ASGI app. Attribute "app.app" not found in module "app".

[nix-shell:~/Desktop/fastapi-nix-experiment]$ uvicorn app.app:app
INFO:     Started server process [53232]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     127.0.0.1:54778 - "GET / HTTP/1.1" 200 OK

Cool, right? It's not perfect. But it does work!

This was a doozie. Building a flake file is not a great of an experience as writing a Dockerfile. Just look at this thing!

{
  description = "fastapi-nix-experiment";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
    mach-nix.url = "github:davhau/mach-nix";
  };

  outputs = { self, nixpkgs, mach-nix, flake-utils, ... }:
    let
      pythonVersion = "python310";
    in
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        mach = mach-nix.lib.${system};

        pythonAppEnv = mach.mkPython rec {
          python = pythonVersion;
          requirements = builtins.readFile ./requirements.txt;
        };

        devShell = pkgs.mkShell rec {
          buildInputs = [
            pythonAppEnv
          ];

          shellHook = ''
            export PYTHONPATH="${pythonAppEnv}/bin/python"
          '';
        };

        defaultPackage = pythonAppEnv;
      in {
        inherit devShell defaultPackage;
      }
    );
}

To call out a few areas that I find confusing and will spend more time researching in the future:

Python 3.11 totally did not work
I can't use pyproject.toml the same way the app was written and went with requirements.txt 🫠
What constitutes and input and output? Why aren't my code and dependencies considered inputs?

To actually build the environment via the flake, run the following:

$ nix build

$ nix develop
Restored session: Tue Sep 19 17:19:10 EDT 2023
(nix:nix-shell-env) Anthonys-MBP:fastapi-nix-experiment anthony$ uvicorn
Usage: uvicorn [OPTIONS] APP
Try 'uvicorn --help' for help.

Error: Missing argument 'APP'.
(nix:nix-shell-env) Anthonys-MBP:fastapi-nix-experiment anthony$ uvicorn app.app:app
INFO:     Started server process [53640]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     127.0.0.1:54822 - "GET / HTTP/1.1" 200 OK

Conclusions

Nix reminds me a bit of bazel, and for now, I don't think I'm having enough pain with reproducibility issues to need tools like nix.

I'd rather use native build tools and docker/ container/ VM packaging in order to build and distribute contents in CI. It's simpler, faster, and with the right amount of process, get's you as far as you need to be so that you and your team can be wicked productive without wrestling your software into perfection.

I think I'll switch to nix once the developer experience gets a bit simpler and intuitive (for me). I think tools like flox are getting us there, and that's a post for another time.

Nix

Nix Devlog 000