NixOS and my Descent into Insanity

NixOS and my Descent into Insanity

As I tend to do, I picked a topic to write about that is much larger in scope than I could manage in a reasonable amount of time. Did I learn? Apparently not. This article started off with switching from zsh to fish. Then I thought, "Might as well manage it all with Nix!", which led me to switch to home manager to manage my dotfiles which led me to using Nix everywhere I possibly could.

As expected, using Nix where it's not supported caused some issues. Buckle up, and watch my slow descent into madness (Nix).

Want to see the fruits of my labour? Find my Nix configurations on Sourcehut.

Impulse to Distro-Hop

Our story begins in the sweet month of March. The semester at college was winding down, I was suffering through the second set of midterms this semester, and I brought it upon myself to make a Matrix client for the PS Vita.

This was, predictably, a mess. The toolchain system was a mess. I was patching CMake scripts and it was no fun. I heard about Nix first when Replit added support for it way back when. It also seems like all the cool people on Mastodon were using Nix.

The seed had been planted.

Later on, I started having issues with my few-year-old Arch Linux install. It was slowing down, had weird bugs, and was all-around unpleasant to use. It culminated in realizing that my SSD performance was miserable. Probably some combination of age, LUKS, btrfs, and who knows what else I changed. I thought to myself, "This would be a good time to reinstall Arch. I could have the BIOS do a low-level wipe of the SSD and bring it back to the original performance!"

It was not a good time to reinstall. The semester was winding down, finals were coming up, and I definitely did not have the time to do anything complicated.

It is now the 19th of April. I am sitting in my friend's dorm room backing up my files to her server so I could install NixOS. The thing I had practically no experience with. Being installed. On my only computer. That I prepare for my exams with.

Surprisingly, it went pretty well. Ignoring how it took me nearly thirty minutes to make the font size of the installer bigger because my laptop has a HiDPI screen, I had a functioning install in a few hours (after some weird issues that I don't remember anymore but put on Mastodon) and a usable one a day later.

Things were going great. I did my final exams, semester over, went home. All the while tweaking Nvidia configurations and whatnot.

Sweet Innocence

I follow one of the lead Fish devs on Mastodon and he talks about the Fish shell sometimes. It looked cool, and I was getting kinda sick of Zsh being slow in big Git repositories. Instead of fixing Zsh, I decided to give Fish a shot. Thankfully, the NixOS wiki had a page on installing Fish.

I just had to put the following code in my NixOS configuration:

programs.fish.enable = true;
users.defaultUserShell = pkgs.fish;

I'm pretty happy with the defaults. Let's load in my aliases from Zsh:

In ~/.config/fish/conf.d/aliases.fish:

alias ls="exa --color=auto -a -g"
alias cp="cp -v"
alias sl="ls"
alias clear="clear -x"
alias ip="ip -c"
alias vi="nvim"
alias vim="nvim"
alias cat="bat"
alias rm="trash-put"
alias netcat="ncat"
alias nc="ncat"
alias diff="delta"
alias wp="woodpecker-cli"

alias nvidia-on="sudo nvidia-on"
alias nvidia-off="sudo nvidia-off"

alias feh="echo imv"
alias paru="sudo sh -c 'nix-channel --update && nixos-rebuild switch'"

This is what my NixOS configuration looked like before I decided to make everything harder for myself.

The Part In Which I Make A Bad Decision

As I was first installing NixOS and setting up Sway, I saw that it could be installed with home-manager, whatever that was. When I clicked on it first, I got this wonderful warning:

Unfortunately, it is quite possible to get difficult to understand errors when working with Home Manager, such as infinite loops with no clear source reference. You should therefore be comfortable using the Nix language and the various tools in the Nix ecosystem. Reading through the Nix Pills document is a good way to familiarize yourself with them.

I tried reading the Nix Pills but it was so boring and technical so I stopped. I also literally just installed NixOS and had no idea how to use it. I thought to myself, maybe another day.

That day is upon us now. Time to install!

There were a couple of options for installing home-manager: standalone, or as a NixOS module. I decided to go with the first one and not have it be a part of NixOS for a few reasons:

• I wanted to have the same home-manager workflow on multiple machines, even the ones that don't have NixOS
• I wanted to manage my OS and my user separately
• It looked like it was easier

I started off with using home-manager to configure the Fish shell (I just copy-pasted from the wiki):

programs.fish = {
  enable = true;
  interactiveShellInit = ''
    set fish_greeting # Disable greeting
  '';
  plugins = [
    { name = "grc"; src = pkgs.fishPlugins.grc.src; }
    { name = "done"; src = pkgs.fishPlugins.done.src; }
    { name = "tide"; src = pkgs.fishPlugins.tide.src; }
    { name = "autopair"; src = pkgs.fishPlugins.autopair.src; }
    ## Manually packaging and enable a plugin
    # {
    # name = "z";
    # src = pkgs.fetchFromGitHub {
    #   owner = "jethrokuan";
    #   repo = "z";
    #   rev = "e0e1b9dfdba362f8ab1ae8c1afc7ccf62b89f7eb";
    #   sha256 = "0dbnir6jbwjpjalz14snzd3cgdysgcs3raznsijd6savad3qhijc";
    # };
    # }
  ];
};

…and it worked? Kinda. I still had to install fish globally (in my NixOS configuration along with the home-manager configuration) so I could change my shell properly (foreshadowing).

A Little (Fishy) Diversion

I wanted the done plugin to work. It's pretty simple: it'll notify you when your terminal is unfocused and a long-running command finishes. I find it pretty cool. I set it up to use notify-send, and it worked just fine!

programs.fish.shellInit = ''
  set __done_notification_command '${pkgs.libnotify}/bin/notify-send \$title \$message'
'';

Forgetting to put the /bin/ bit of the path really messed me up, and it kinda drove me insane why it wasn't working. Building the home-manager generation did not throw an error, and neither did Fish.

Fish and Nix Don't Play Nice

So Fish has this thing called "universal variables" that are set once and are persisted forever in a file called fish_variables. This is kinda antithetical to what Nix is—declarative configuration. Throwing universal variables into that mix doesn't really gel with Nix.

So, I had to do a little bit of trickery to get the variables to load in right. You see, Fish loads configuration in a specific order1: most importantly, it'll read config.fishlast. Guess what home-manager saves your configuration to? That's right. config.fish. So, we have to find a way to load in variables without universal variables that will be loaded in before the Fish plugins are loaded.

Home-manager keeps the Fish plugins in ~/.config/fish/conf.d named plugin-pluginname.fish. Naturally, we just need to save a file that'll load variables before those plugins are loaded. Home-manager can do that for you, with xdg.configFile!

xdg.configFile = {
"fish/conf.d/00-home-manager-vars.fish" = {
  enable = true;
  text = ''
    set tide_cmd_duration_threshold 3000
    set tide_prompt_add_newline_before false
    set tide_prompt_color_frame_and_connection 6C6C6C
    [...]
  '';
}

Because 00 comes before pl, the text in that file will be loaded first! Of course, I'm sure there's a better way to do this, but I'm not familiar enough with Nix to figure it out. Yet.

Everything Else

Now that I got a taste of home-manager to manage my dotfiles, of course I had to move everything else over. Over the next few days, I had the 800+ page "Appendix A: configuration reference" open in one window, the NixOS wiki for the program I was trying to configure open in another.

Day in, day out, I pieced together my configuration for Sway, Waybar, and Neovim. It took a while. I moved my custom shell scripts over to writeShellScriptBin. I read through the very-hard-to-navigate Appendix A.

And then I ran the incantation.

home-manager switch

I did not realize that it would reload Sway. That was unexpected. I thought everything that I had open would just keep going until I reloaded it to use the new config. But home-manager did that for me.

And now my desktop was broken. Joy. Swayidle went to sleep but did not wake the computer, the keybinds were ever-so-slightly off (hjkl instead of jkl;), and startup applications refused to work.

Time to debug.

Debugging Everything Else

My problems boiled down to a few main issues:

1. I forgot to put /bin/ in the Nix paths
2. I tried to adapt my shell scripts instead of using the idiomatic home-manager or Nix methods
3. The documentation was so clunky and painful that I couldn't figure out how to do the idiomatic solution

Seriously, if it weren't for sample configurations that I found online2, then I would be dead in the water.

Over the course of the next few days, I expanded my home-manager configuration to encompass everything I had dotfiles for. As part of this process, I also changed up my theming stack a little bit. I moved away from qt5ct and just used Kvantum the whole way (since that's what qt5ct was using anyway). A nice upside is that using home-manager to manage cursor themes and whatnot meant that my desktop was a lot easier to manage! I fixed all the fonts! Long-standing issues (inconsistent fonts, cursors, GTK file chooser bookmarks) that I never really bothered to fix are now fixed!

What Isn't Everything Else?

There is one notable exception to programs I am not managing with home-manager, and that's SSH. Although it's pretty easy to use home-manager to manage SSH configurations, I'd prefer the security-through-obscurity of people not knowing my SSH aliases (since, y'know, I'm making my nix configuration public).

Carving Up The Leviathan

At this point I had a home.nix that spanned a couple of thousand lines. This is a bad idea, mostly because it's hard to manage.

The main issue with splitting up the configuration is that I'm bad at the Nix programming language and I have only an inkling of what I'm doing. Eventually, I just did a "best effort" split. I split it into:

1. User-specific things (username, home directory)
2. Common data (environment variables, etc)
3. Shell configuration (Fish)
4. Wayland configuration (Sway, GTK/Qt themes, Waybar, Autostart, Foot terminal, Gammastep)
5. Neovim configuration (plugins and all)
6. Git configuration
7. Tmux configuration

Why did I split it up like this? Simple! I don't need Sway on machines I'm only going to be connecting remotely into!

In Which My Decisions Become Worse

Now that I have migrated all of my dotfiles over to being managed by home-manager, I gotta have everything managed by home-manager. I use only a couple of other machines: my server (SSH), exozyme (SSH), and my college's servers for doing school projects. I don't do development on my server (only maintainence stuff), so I don't really need to install Nix there. exozyme, however, has Nix already installed…

Home-Manager on Arch Linux

exozyme is a multi-user Arch Linux server. Only the administrator can install packages. As such, Nix was installed so users can install packages for themselves without muddying the whole OS and without going through administrator privileges. This shouldn't be too hard, right?

I clone my nix-config repository, install home-manager, create the symlink to the exozyme configuration (loads in shell, common, and Neovim), and switch. Of course, it worked perfectly. I don't know what you all expected. Sans the chsh issues (resolved by just putting exec fish into the .bash_profile), everything just worked.

It wasn't that bad, until I got nerd-sniped.

In Which The Fish Takes Revenge

Remember the done plugin we installed for Fish earlier? Yeah, that won't work over SSH, since notify-send only works if your shell has access to that FreeDesktop socket. Over SSH, I don't have that.

But I had an idea. In the course of configuring my terminal, Foot, I saw something in the configuration files:

[main]
notify=notify-send -a ${app-id} -i ${app-id} ${title} ${body}

That's weird. I've never gotten a notification from my terminal. Opening the README showed me that the terminal supports something called OSC777.

What is an OSC? An OSC is a special escape sequence that makes the terminal Do Things™, such as changing cursor colours, changing the window title, accessing the clipboard, etc etc. One of these things that an OSC can do is send a desktop notification.

Implementing OSC777 to send a notification from Fish shouldn't be too hard, right? It's just a bunch of nested escape sequences and oh no it's gonna be painful.

I'll spare you that pain.

set __done_notification_command 'echo -e "\e]777;notify;$title;$message\e\\ "'
set __done_allow_nongraphical 1

Yes, that space at the end before the first double quote is important. Yes, the order of the quotes matter. Yes, I learned the hard way that this has to be in the XDG-managed Fish configuration. No, I don't even know what backslashes are anymore.

But it works. Over SSH, I get notifications.

Was it worth the pain? Reading through the manpages realizing that the capitalization of e matters? Trial and error and maybe forty home-manager generations? Countless hours figuring this out?

Absolutely.

Oh yeah, the (relevant) terminal configuration:

programs.foot = {
  enable = true;
  settings = {
    main = {
      notify = "${pkgs.libnotify}/bin/notify-send -a \${app-id} -i utilities-terminal \${title} \${body}";
      notify-focus-inhibit = "yes";
    };
  };
};

Home-Manager Where No Home Has Been Managed Before

See, my university's machines don't have Nix installed. I originally didn't think this was going to be a problem.

So, I went on down to the NixOS wiki page. The first option that I had was use nix-user-chroot

The Climb

The first item on the list was nix-user-chroot. The university's server supported user namespaces, so I went ahead and ran the command on the wiki:

mkdir -m 0755 ~/.nix
./nix-user-chroot ~/.nix bash -c 'curl -L https://nixos.org/nix/install | sh'

And it worked! I now had Nix! Now to have it load in the chroot by default:

In .profile:

if [[ ! -e ~/.nix-profile ]]; then exec ~/nix-user-chroot ~/.nix bash; fi

This checks if symlink is not working (which means we are not in the chroot yet), then loads the chroot.

And to finish it off, we add this to the .bashrc so we can get proper Nix paths:

if [ -f ~/.nix-profile/etc/profile.d/nix.sh ]; then
    source ~/.nix-profile/etc/profile.d/nix.sh
fi

Now to move my dotfiles over! Let's first install home-manager the normal way (standalone) like we did earlier (and like how it's documented).

This is where I ran into my first issue. You see, in an attempt to curb backup costs, each user has a quota of 3072 MB (4096 MB hard). Installing home-manager used about 1.5 gigs. I do not see good things in the future.

Running the home-manager switch incantation after creating the symlinks caused my user to hit the quota. Luckily, there's a "scratch" directory every user has that has no such quota and is not backed up. I don't need my Nix store to be backed up. This is an acceptable tradeoff. One purging of the .nix folder and moving to the scratch directory later, I am no longer constrained by follies such as "storage".

But now I have another problem. As part of my Neovim configuration, I use typescript-language-server. That package refuses to install:

post-installation fixup
chmod: changing permissions of '/nix/store/6977681wxwspnc15l87zzr3zh9bavscn-typescript-language-server-3.3.2': Operation not permitted
chmod: changing permissions of '/nix/store/6977681wxwspnc15l87zzr3zh9bavscn-typescript-language-server-3.3.2/lib/node_modules/.bin': Operation not permitted
[many more]

The Stumble

Looks like a recursive chmod is being run. Let's find out what is happening.

I run home-manager switch -n --debug to get hopefully some more information about the build failure. I see auto-disabling sandboxing because the prerequisite namespaces are not available. It seems like nested namespaces aren't working, which causes chmod to fail. Disabling sandboxing in my Nix configuration did not help.

The final nail in the coffin is that the nix-user-chroot project is abandoned.

Wonderful. Let's go over to nix-portable and see if that does anything.

The Fall

After installing and getting nix-portable set up, I come across the bit in the documentation: "Missing Features: managing nix channels via nix-channel".

Home-manager needs to add a channel to install it. This will not do.

And I kept falling.

In a fit of desperation, I go back to the NixOS wiki and see if there are any other solutions that could help me on my self-inflicted journey of unifying my dotfiles.

Maybe proot can help? Turns out it can! As long as I disable Nix sandboxing. Ideally, I could keep this enabled to ensure consistent build environments and security and whatnot. It also looks like it works by intercepting syscalls and not all syscalls are supported, which causes some programs, like—oh I don't know, the Nix installer—to fail.

Can you taste the desperation?

The Pit

And now, after reading through those options and trying each one out, I come across this lovely tidbit:

You can make all nix commands use the alternate store by specifying it in ~/.config/nix/nix.conf as store = /home/USERNAME/my-nix.

Was it… was it really so easy all this time? I just needed to get a standalone Nix binary and then set the configuration?

Turns out, no. Getting a standalone Nix binary is hard enough, but also changing the Nix store path messes with everything (and also it would be a symlink).

Normally, the Nix store directory (typically /nix/store) is not allowed to contain any symlink components. This is to prevent "impure" builds. Builders sometimes "canonicalise" paths by resolving all symlink components. Thus, builds on different machines (with /nix/store resolving to different locations) could yield different results. This is generally not a problem, except when builds are deployed to machines where /nix/store resolves differently. If you are sure that you’re not going to do that, you can set NIX_IGNORE_SYMLINK_STORE to 1.

Yeah, this would be catastrophically bad.

I'm flat out of ideas.

Escaping The Pit

And then there was a glimmer of recognition in my eye. As I was reading through nix-portable's README, it mentioned bwrap. And that got the brain-gears going.

In response to the fractureiser malware, I wrote a post describing how I can isolate Minecraft. I used bwrap. This time, instead of isolating the program, I wanted to isolate as little as possible while still messing with the program-facing filesystem.

#!/usr/bin/env bash

if [ -z ${NIXDIR+x} ]; then 
    echo "NIXDIR is unset! It needs to be set in the code. Edit this shell file and read the instructions."
    echo "Executing bash without Bubblewrap…"
    exec bash
fi

if [ ! -e $NIXDIR ]; then
    echo "NIXDIR doesn't point to a valid location! Falling back to Bash"
    exec bash
fi

_bind() {
    _bind_arg=$1
    shift
    for _path in "$@"; do
        args+=("$_bind_arg" "$_path" "$_path")
    done
}

bind() {
    _bind --bind-try "$@"
}

robind() {
    _bind --ro-bind-try "$@"
}

devbind() {
    _bind --dev-bind-try "$@"
}

args=(
    --bind $NIXDIR /nix
)

bind \
    $HOME

devbind \
    /dev \
    /proc \
    /tmp \
    /run \
    /u \
    /p \
    /bin \
    /boot \
    /etc \
    /home \
    /lib \
    /lib32 \
    /lib64 \
    /libx32 \
    /media \
    /usr \
    /var

exec bwrap "${args[@]}" "$@"

And to use the Nix environment by default, I put the following into my .profile:

if [ -e $HOME/.nix-profile/etc/profile.d/nix.sh ]; then 
    source $HOME/.nix-profile/etc/profile.d/nix.sh 
else
    if [ -e $HOME/nix-configs/bwrap.sh ]; then
        exec env NIXDIR=$HOME/scratch/nix $HOME/nix-configs/bwrap.sh $HOME/.nix-profile/bin/fish
    fi
fi

The good: Everything works. Yes really.

The bad: All the other users show up as nobody. I guess that's fine?

The ugly: It's kinda slow (because the scratch directory (and thus Nix and all of the binaries and configuration files created by home-manager) are being stored on NFS). I feel like this is an acceptable tradeoff, and the slowness is only really noticeable when opening Neovim or starting a new Fish instance.

Was It Worth It?

Absolutely. I have Nix working in a rootless environment (practically) perfectly. What more could I ask for? I can finally one-up the Plan 9 folks adding their smart-bulbs to their grids and instead go use Nix where nobody right in their minds would use it.

And also it's nice having my dotfiles fun to manage again.

Now to figure out what a "flake" is…

Follow me on Mastodon to see me get mad at Nix more!

Corrections? Comments? Just wanna say hi? Get in touch!

---

CC-BY-SA 4.0 | Privacy | Search | Donate | RSS | We use no cookies