Incrementally package a Haskell program using Nix
source link: https://www.haskellforall.com/2022/08/incrementally-package-haskell-program.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Incrementally package a Haskell program using Nix
incremental-nix
This post walks through how to take a standalone Haskell file and progressively package the file using Nix. In other words, we will tour a spectrum of packaging options ranging from simple to fancy.
The running example will be the following standalone single-file Haskell program:
I won’t go into detail about what that program does, although you can study the program if you are curious. Essentially, I’m planning to deliver a talk based on that program at this year’s MuniHac and I wanted to package it up so that other people could collaborate on the program with me during the hackathon.
When I began writing this post, there was no packaging logic for this program; it’s a standalone Haskell file. However, this file has several dependencies outside of Haskell’s standard library, so up until now I needed some way to obtain those dependencies for development.
Stage 0:
ghc.withPackages
The most low-tech way that you can hack on a Haskell program using
Nix is to use nix-shell
to obtain a transient development
environment (this is what I had done up until now).
Specifically, you can do something like this:
$ nix-shell --packages 'ghc.withPackages (pkgs: [ pkgs.mtl pkgs.MemoTrie pkgs.containers pkgs.pretty-show ])'
… where pkgs.mtl
and pkgs.MemoTrie
indicate
that I want to include the mtl
and MemoTrie
packages in my Haskell development environment.
Inside of that development environment I can build and run the file
using ghc
. For example, I can use ghc -O
to
build an executable to run:
[nix-shell]$ ghc -O Spire.hs
[nix-shell]$ ./Spire
…
… or if I don’t care about optimizations I can interpret the file
using runghc
:
$ runghc Spire.hs
…
Stage 1: IDE support
Once I’m inside a Nix shell I can begin to take advantage of integrated development environment (IDE) support.
The two most common tools Haskell developers use for rapid feedback
are ghcid
and haskell-language-server
:
ghcid
provides a command-line interface for fast type-checking feedback but doesn’t provide other IDE-like featureshaskell-language-server
is more of a proper IDE that you use in conjunction with some editor
I can obtain either tool by exiting from the shell and creating a new shell that includes the desired tool.
For example, if I want to use ghcid
then I recreate the
nix-shell
using the following command:
$ nix-shell --packages ghcid 'ghc.withPackages (pkgs: [ pkgs.mtl pkgs.MemoTrie pkgs.containers pkgs.pretty-show ])'
… and then I can tell ghcid
to continuously type-check
my file using:
[nix-shell]$ ghcid Spire.hs
…
If I want to use haskell-language-server
, then I
recreate the nix-shell
using this command:
$ nix-shell --packages haskell-language-server 'ghc.withPackages (pkgs: [ pkgs.mtl pkgs.MemoTrie pkgs.containers pkgs.pretty-show ])'
… and then I can explore the code in any editor that supports the language server protocol.
Note that if you use VSCode as your editor then you may need to install some additional plugins:
… and the next section will show how to install VSCode and those plugins using Nix.
However, once you do install those plugins then you can open the file
in VSCode from within the nix-shell
using:
[nix-shell]$ code Spire.hs
… and once you trust the file the IDE features will kick in.
Stage 2: Global development environment
Sometimes I like to globally install development tools that are
commonly shared between projects. For example, if I use
ghcid
or haskell-language-server
across all my
projects then I don’t want to have to explicitly enumerate that tool in
each project’s Nix shell.
Moreover, my tool preferences might not be shared by other
developers. If I share my nix-shell
with other developers
for a project then I probably don’t want to add editors/IDEs or other
command-line tools to that environment because then they have to
download those tools regardless of whether they plan to use them.
However, I don’t want to globally install development tools like this:
$ nix-env --install --file '<nixpkgs>' --attr ghcid
$ nix-env --install --file '<nixpkgs>' --attr haskell-language-server
Part of the reason I use Nix is to avoid imperatively managing my
development environment. Fortunately, though, nix-env
supports a more declarative way of managing dependencies.
What you can do instead is save a file like this to
~/default.nix
:
let
# For VSCode
config = { allowUnfree = true; };
overlay = pkgsNew: pkgsOld: {
# Here's an example of how to use Nix to install VSCode with plugins managed
# by Nix, too
vscode-with-extensions = pkgsOld.vscode-with-extensions.override {
vscodeExtensions = [
pkgsNew.vscode-extensions.haskell.haskell
pkgsNew.vscode-extensions.justusadam.language-haskell
];
};
};
pkgs = import <nixpkgs> { inherit config; overlays = [ overlay ]; };
in
{ inherit (pkgs)
# I included some sample useful development tools for Haskell. Feel free
# to customize.
cabal-install
ghcid
haskell-language-server
stylish-haskell
vscode-with-extensions
;
}
… and once you create that file you have two options.
The first option is that you can set your global development environment to match the file by running:
$ nix-env --remove-all --install --file ~/default.nix
NOTE: At the time of this writing you may also need to add
--system x86_64-darwin
if you are trying out these examples on an M1 Macbook. For more details, see:
Carefully note the --remove-all
, which resets your
development environment to match the file, so that nothing from your old
development environment is accidentally carried over into your new
development environment. This makes our use of the nix-env
command truly declarative.
The second option is that you can change the file to create a valid shell, like this:
let
config = { allowUnfree = true; };
overlay = pkgsNew: pkgsOld: {
vscode-with-extensions = pkgsOld.vscode-with-extensions.override {
vscodeExtensions = [
pkgsNew.vscode-extensions.haskell.haskell
pkgsNew.vscode-extensions.justusadam.language-haskell
];
};
};
pkgs = import <nixpkgs> { inherit config; overlays = [ overlay ]; };
in
pkgs.mkShell {
packages = [
pkgs.ghcid
pkgs.haskell-language-server
pkgs.stylish-haskell
pkgs.vscode-with-extensions
pkgs.cabal-install
];
}
… and then run:
$ nix-shell ~/default.nix
Or, even better, you can rename the file to ~/shell.nix
and then if you’re already in your home directory (e.g. you just logged
into your system), then you can run:
$ nix-shell
… which will select ~/shell.nix
by default. This lets
you get a completely transient development environment so that you never
have to install anything development tools globally.
These nix-shell
commands stack, so you can first run
nix-shell
to obtain your global development environment and
then use nix-shell
a second time to obtain project-specific
dependencies.
My personal preference is to use the declarative nix-env
trick for installing global development tools. In my opinion it’s just
as elegant as nix-shell
and slightly less hassle.
Stage 3: Cabal
Anyway, enough about global development tools. Back to our Haskell project!
So ghc.withPackages
is a great way to just start hacking
on a standalone Haskell program when you don’t want to worry about
packaging up the program. However, at some point you might want to share
the program with the others or do a proper job of packaging if you’re
trying to productionize
the code.
That brings us to the next step, which is packaging our Haskell
program with a Cabal file (a Haskell package manifest). We’ll need the
cabal-install
command-line tool before we proceed further,
so you’ll want to add that tool to your global development environment
(see the previous section).
To create our .cabal
file we can run the following
command from the top-level directory of our Haskell project:
$ cabal init --interactive
Should I generate a simple project with sensible defaults? [default: y] n
…
… and follow the prompts to create a starting point for our
.cabal
file.
After completing those choices and trimming down the
.cabal
file (to keep the example simple), I get a file that
looks like this:
cabal-version: 2.4
name: spire
version: 1.0.0
license: BSD-3-Clause
license-file: LICENSE
executable spire
main-is: Spire.hs
build-depends: base ^>=4.14.3.0
default-language: Haskell2010
The only thing I’m going change for now is to add dependencies to the
build-depends
section and increase the upper bound on
base
::
cabal-version: 2.4
name: spire
version: 1.0.0
license: BSD-3-Clause
license-file: LICENSE
executable spire
main-is: Spire.hs
build-depends: base >=4.14.3.0 && < 5
, MemoTrie
, containers
, mtl
, pretty-show
, transformers
default-language: Haskell2010
Stage 4:
cabal2nix --shell
Adding a .cabal
file suffices to share our Haskell
package with other Haskell developers if they’re not using Nix. However,
if we want to Nix-enable package our package then we have a few
options.
The simplest option is to run the following command from the top-level of the Haskell project:
$ cabal2nix --shell . > shell.nix
That will create something similar to the following
shell.nix
file:
{ nixpkgs ? import <nixpkgs> {}, compiler ? "default", doBenchmark ? false }:
let
inherit (nixpkgs) pkgs;
f = { mkDerivation, base, containers, lib, MemoTrie, mtl
, pretty-show, transformers
}:
mkDerivation {
pname = "spire";
version = "1.0.0";
src = ./.;
isLibrary = false;
isExecutable = true;
executableHaskellDepends = [
base containers MemoTrie mtl pretty-show transformers
];
license = lib.licenses.bsd3;
};
haskellPackages = if compiler == "default"
then pkgs.haskellPackages
else pkgs.haskell.packages.${compiler};
variant = if doBenchmark then pkgs.haskell.lib.doBenchmark else pkgs.lib.id;
drv = variant (haskellPackages.callPackage f {});
in
if pkgs.lib.inNixShell then drv.env else drv
… and if you run nix-shell
within the same directory the
shell environment will have the Haskell dependencies you need to build
and run project using cabal
:
$ nix-shell
[nix-shell]$ cabal run
…
… and tools like ghcid
and
haskell-language-server
will also work within this shell,
too. The only difference is that ghcid
now takes no
arguments, since it will auto-detect the cabal project in the current
directory:
[nix-shell]$ ghcid
Note that this nix-shell
will NOT
include cabal
by default. You will need to globally install
cabal
(see the prior section on “Global development
environment”).
This cabal2nix --shell
workflow is sufficiently
lightweight that you can Nixify other people’s projects on the fly when
hacking on them locally. A common thing I do if I need to make a change
to a person’s project is to clone their repository, run:
$ cabal2nix --shell . > shell.nix
$ nix-shell
… and start hacking away. I don’t even need to upstream the
shell.nix
file I created in this way; I just keep it around
locally for my own hacking.
In fact, I typically don’t want to upstream such a
shell.nix
file (even if the upstream author were receptive
to Nix), because there are more robust Nix expressions we can upstream
instead.
Stage 5: Custom
shell.nix
file
One disadvantage of cabal2nix --shell
is that you have
to re-run the command any time your dependencies change. However, if
you’re willing to hand-write your own shell.nix
file then
you can create something more stable:
let
overlay = pkgsNew: pkgsOld: {
haskellPackages = pkgsOld.haskellPackages.override (old: {
overrides = pkgsNew.haskell.lib.packageSourceOverrides {
spire = ./.;
};
});
};
pkgs = import <nixpkgs> { overlays = [ overlay ]; };
in
pkgs.haskellPackages.spire.env
The packageSourceOverrides
is the key bit. Under the
hood, that essentially runs cabal2nix
for you any time your
project changes and then generates your development environment from the
result. You can also use packageSourceOverrides
to specify
non-default versions of dependencies, too:
let
overlay = pkgsNew: pkgsOld: {
haskellPackages = pkgsOld.haskellPackages.override (old: {
overrides = pkgsNew.haskell.lib.packageSourceOverrides {
spire = ./.;
# Example of how to pin a dependency to a non-defaul version
pretty-show = "1.9.5";
};
});
};
pkgs = import <nixpkgs> { overlays = [ overlay ]; };
in
pkgs.haskellPackages.spire.env
… although that will only work for packages that have been released prior to the version of Nixpkgs that you’re depending on.
If you want something a bit more robust, you can do something like this:
let
overlay = pkgsNew: pkgsOld: {
haskellPackages = pkgsOld.haskellPackages.override (old: {
overrides =
pkgsNew.lib.fold
pkgsNew.lib.composeExtensions
(old.overrides or (_: _: { }))
[ (pkgsNew.haskell.lib.packageSourceOverrides {
spire = ./.;
})
(pkgsNew.haskell.lib.packagesFromDirectory {
directory = ./packages;
})
];
});
};
pkgs = import <nixpkgs> { overlays = [ overlay ]; };
in
pkgs.haskellPackages.spire.env
… and then you have the option to also depend on any dependency that
cabal2nix
knows how to generate:
$ mkdir packages
$ # Add the following file to version control to preserve the directory
$ touch packages/.gitkeep
$ cabal update
$ cabal2nix cabal://${PACKAGE_NAME}-${VERSION} > ./packages/${PACKAGE_NAME}.nix
… and that works even on bleeding-edge Haskell packages that Nixpkgs hasn’t picked up, yet.
Stage 6: Pinning Nixpkgs
All of the prior examples are “impure”, meaning that they depend on
the ambient nixpkgs
channel installed on the developer’s
system. This nixpkgs
channel might vary from system to
system, meaning that each system might have different versions of
nixpkgs
installed, and then you run into issues reproducing
each other’s builds.
For example, if you have a newer version of nixpkgs
installed your Nix build for the above Haskell project might succeed,
but then another developer might attempt to build your project with an
older version of nixpkgs
, which might select an older
incompatible version of one of your Haskell dependencies.
Or, vice versa, the examples in this blog post might succeed at the
time of this writing for the current version of nixpkgs
but
then as time goes on the examples might begin to fail for future
versions of nixpkgs
.
You can fix that by pinning Nixpkgs, which this post covers:
For example, we could pin nixpkgs
for our global
~/default.nix
like this:
let
nixpkgs = builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/0ba2543f8c855d7be8e90ef6c8dc89c1617e8a08.tar.gz";
sha256 = "14ann7vz7qgfrw39ji1s19n1p0likyf2ag8h7rh8iwp3iv5lmprl";
};
config = { allowUnfree = true; };
overlay = pkgsNew: pkgsOld: {
vscode-with-extensions = pkgsOld.vscode-with-extensions.override {
vscodeExtensions = [
pkgsNew.vscode-extensions.haskell.haskell
pkgsNew.vscode-extensions.justusadam.language-haskell
];
};
};
pkgs = import nixpkgs { inherit config; overlays = [ overlay ]; };
in
{ inherit (pkgs)
cabal-install
ghcid
haskell-language-server
stylish-haskell
vscode-with-extensions
;
}
… which pins us to the tip of the release-22.05
branch
at the time of this writing.
We can likewise pin nixpkgs
for our project-local
shell.nix
like this:
let
nixpkgs = builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/0ba2543f8c855d7be8e90ef6c8dc89c1617e8a08.tar.gz";
sha256 = "14ann7vz7qgfrw39ji1s19n1p0likyf2ag8h7rh8iwp3iv5lmprl";
};
overlay = pkgsNew: pkgsOld: {
haskellPackages = pkgsOld.haskellPackages.override (old: {
overrides = pkgsNew.haskell.lib.packageSourceOverrides {
spire = ./.;
};
});
};
pkgs = import nixpkgs { overlays = [ overlay ]; };
in
pkgs.haskellPackages.spire.env
Flakes
The final improvement we can make is the most important one of all: we can convert our project into a Nix flake:
There are two main motivations for flake-enabling our project:
- To simplify managing inputs that we need to lock
(e.g.
nixpkgs
) - To speed up our shell
To flake-enable our project, we’ll save the following code to
flake.nix
:
{ inputs = {
nixpkgs.url = github:NixOS/nixpkgs/release-22.05;
utils.url = github:numtide/flake-utils;
};
outputs = { nixpkgs, utils, ... }:
utils.lib.eachDefaultSystem (system:
let
config = { };
overlay = pkgsNew: pkgsOld: {
spire =
pkgsNew.haskell.lib.justStaticExecutables
pkgsNew.haskellPackages.spire;
haskellPackages = pkgsOld.haskellPackages.override (old: {
overrides = pkgsNew.haskell.lib.packageSourceOverrides {
spire = ./.;
};
});
};
pkgs =
import nixpkgs { inherit config system; overlays = [ overlay ]; };
in
rec {
packages.default = pkgs.haskellPackages.spire;
apps.default = {
type = "app";
program = "${pkgs.spire}/bin/spire";
};
devShells.default = pkgs.haskellPackages.spire.env;
}
);
}
… and then we can delete our old shell.nix
because we
don’t need it anymore.
Now we can obtain a development environment by running:
$ nix develop
… and the above flake also makes it possible to easily build and run the program, too:
$ nix run # Run the program
$ nix build # Build the project
In fact, you can even run a flake without having to clone a repository. For example, you can run the example code from this blog post by typing:
$ nix run github:Gabriella439/spire
Moreover, we no longer have to take care of managing hashes for, say,
Nixpkgs. The flake machinery takes care of that automatically for you
and generates a flake.lock
file which you can then add to
version control. For example, the lock file I got was:
{
"nodes": {
"nixpkgs": {
"locked": {
"lastModified": 1661617163,
"narHash": "sha256-NN9Ky47j8ohgPhA9JZyfkYIbbAo6RJkGz+7h8/exVpE=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "0ba2543f8c855d7be8e90ef6c8dc89c1617e8a08",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "release-22.05",
"repo": "nixpkgs",
"type": "github"
}
},
"root": {
"inputs": {
"nixpkgs": "nixpkgs",
"utils": "utils"
}
},
"utils": {
"locked": {
"lastModified": 1659877975,
"narHash": "sha256-zllb8aq3YO3h8B/U0/J1WBgAL8EX5yWf5pMj3G0NAmc=",
"owner": "numtide",
"repo": "flake-utils",
"rev": "c0e246b9b83f637f4681389ecabcb2681b4f3af0",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "flake-utils",
"type": "github"
}
}
},
"root": "root",
"version": 7
}
… and you can easily upgrade to, say, a newer revision of Nixpkgs if you need to.
Additionally, all of the Nix commands are now faster. Specifically, the first time you run a command Nix still needs to download and/or build dependencies, but subsequent runs are faster because Nix can skip the instantiation phase. For more details, see:
Conclusion
Flakes are our final destination, so that’s as far as this post will go. There are technically some more ways that we can overengineer things, but in my experience the idioms highlighted in this post are the ones that provide the highest power-to-weight ratio.
The key thing to take away is that the Nixpkgs Haskell infrastructure lets you smoothly transition from simpler approaches to more powerful approaches, and even the final flake-enabled approach is actually not that complicated.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK