Rewrite it in Rust by ridiculousfish · Pull Request #9512 · fish-shell/fish-shel...

So, a few quick comments:

1. I basically don't know rust at all (I believe I've used it for a combined twelve minutes), so I can't speak to any of the code itself.
2. Rust 1.67 was released literally two days ago. Compared to us sticking to C++11 that's a staggering difference in version support. Is it that much easier to get a new rust on old systems? Our typical targets are old CentOS and Debian.
3. Following on from that, how easy is this to package from a distro perspective? We've expended quite a bit of effort on making that easy, and I'd hate to drop that aspect.

As of yet unknown compatibility story for Tier 2+ platforms (Cygwin, etc).

Cygwin is mostly a major PITA because of its wchar_t type. If we can get away from using libc that could make some things easier.

get away from using libc

Rust uses libc under the hood.

Rust uses libc under the hood.

Of course technically that's true, but that's not what I'm talking about.

What I mean is that we wouldn't use e.g. libc's "iswdigit" and friends, like we're forced to do now because C++ never got around to replacing them with ones that work with its strings.

And because we use those functions, we're beholden to the libc wchar_t type where we use them, which is everywhere, which means the difference between cygwin (2 byte wchar_t) and as best as I can tell any other system (4 byte wchar_t) can appear everywhere.

And we're beholden to platform differences like "does this provide a uselocale" or "does this have wcscasecmp or std::wcscasecmp", because those directly appear in libc.

In effect what I'm saying is that my hope is rust would not rely on libc for everything and abstract some of the other libc gunk away instead of throwing it in our face like C++ does. And so this historical wchar_t difference could just go away.

The progress to date is impressive, and the motivation to allow concurrent mode is compelling.

I agree that packaging is important but I'd be happy to tackle it. RHEL 7 is only supported for another 18 months, and I have managed to backport GCC to RHEL in the past so it might be possible to do the same. There's a couple of ways to generate Debian packages, which are worth exploring.

I don't know any rust, but I didn't know any C++ before I started work on fish, so I'd be keen to learn.

(Maybe a way forward to #972 etc?)

Just throwing an idea out there: Given that this is a big underlying technical change and needs some larger changes in packaging, would it make sense to:

1. Call fish-in-rust "fish 4"
2. Support the last C++ series (3.6.x?) for a bit longer with bugfixes, added completions and such

This is, of course, a change in policy and more work, but I believe it may be warranted. These releases could be source-only (just tag a 3.6.3 and send out a mail about it), for those who are unable to install rust, especially distro packagers who find themselves looking at an incompatible rustc version.

It's of course possible I'm overestimating the difficulty here, maybe it's not needed because nobody has a problem switching to 4.0 immediately.

Building on that old NetBSD VM I have lying around currently fails with

running: "ar" "s" "/home/alfa/fish-shell/build-rust/./cargo/build/x86_64-unknown-netbsd/release/build/fish-rust-6df62e1ec51440cc/out/libfish-rust.a"
  exit status: 0
  cargo:rustc-link-lib=static=fish-rust
  cargo:rustc-link-search=native=/home/alfa/fish-shell/build-rust/./cargo/build/x86_64-unknown-netbsd/release/build/fish-rust-6df62e1ec51440cc/out
  cargo:rustc-env=AUTOCXX_RS=/home/alfa/fish-shell/build-rust/fish-autocxx-gen/rs

  --- stderr

  CXX include path:
    /home/alfa/fish-shell/build-rust/./cargo/build/x86_64-unknown-netbsd/release/build/fish-rust-6df62e1ec51440cc/out/cxxbridge/include
    /home/alfa/fish-shell/build-rust/./cargo/build/x86_64-unknown-netbsd/release/build/fish-rust-6df62e1ec51440cc/out/cxxbridge/crate
  /home/alfa/fish-shell/fish-rust/../src/proc.h:103:23: error: static_assert expression is not an integral constant expression
  /home/alfa/fish-shell/fish-rust/../src/proc.h:103:23: note: cast from 'void *' is not allowed in a constant expression
  Error: 
    × the include_cpp! macro couldn't be expanded into Rust bindings to C++:
    │ Bindgen was unable to generate the initial .rs bindings for this file.
    │ This may indicate a parsing problem with the C++ headers.

--- CMakeFiles/_cargo-build_fish-rust ---
*** [CMakeFiles/_cargo-build_fish-rust] Error code 101

Removing that static_assert (which is fine) makes it go on until it eventually ends in

[ 98%] Linking CXX executable fish
ld: libfish_rust.a(nix-e8ff18d973978598.nix.35ee0c1d-cgu.15.rcgu.o): in function `nix::sys::timer::Timer::get':
nix.35ee0c1d-cgu.15:(.text._ZN3nix3sys5timer5Timer3get17h045b7bbd1ff7f87dE+0x1b): warning: warning: reference to compatibility timer_gettime(); include <time.h> to generate correct reference
ld: libfish_rust.a(nix-e8ff18d973978598.nix.35ee0c1d-cgu.15.rcgu.o): in function `nix::sys::timer::Timer::set':
nix.35ee0c1d-cgu.15:(.text._ZN3nix3sys5timer5Timer3set17h47eed997322493a6E+0x4e): warning: warning: reference to compatibility timer_settime(); include <time.h> to generate correct reference

(and a bunch more nix::sys::timer and nix::sys::aio functions)

It also absolutely requires libclang, and will fail rather late in the build process.

Rust 1.67 was released literally two days ago. Compared to us sticking to C++11 that's a staggering difference in version support. Is it that much easier to get a new rust on old systems? Our typical targets are old CentOS and Debian.

If you want to use the distro version of Rust then you'll need to use a much older rust than latest stable. You can look it up for a particular distro, but Debian's rustc (for example) is often 6 months or more behind the latest stable. If you want to ship your program in a distro you usually need to build the program using that distro's version of the compiler, so I suspect you'd need to push your "minimum supported rust verison" (MSRV) farther into the past if your project is supposed to be part of distros.

If you don't need to be shipped in a distro, it's as easy as rustup update to update to the latest compiler. It's so simple to update the compiler that it's honestly hard to convince people in the rust community to have a library support anything other than the latest stable.

You can look it up for a particular distro, but Debian's rustc (for example) is often 6 months or more behind the latest stable.

AFAIK Debian updates both rustc and other packages for a while, and then freezes all of them. So it does have the latest rustc for the duration of the development cycle, and so this should not be an issue for packages shipped by the distro itself.

Third-party repos, however, will need either their own newer rustc for the build, or have all the code written in Rust support an older compiler version, including all dependencies.

Just taking a quick look at the versions in https://packages.debian.org/sid/rustc:

• buster 1.41 (2020-01-30)
• bullseye 1.48 (2020-11-19)
• bookworm 1.63 (2022-08-11)
• sid 1.63

I'm not sure what you mean by "for the duration of the development cycle", but even sid falls behind "latest stable" on a regular basis until someone goes and updates it.

so I suspect you'd need to push your "minimum supported rust verison" (MSRV) farther into the past if your project is supposed to be part of distros.

You do not. See: BurntSushi/ripgrep#1019

Essentially, distros like Debian will just ship an older version of your program, just like they do for almost everything else.

I've been alternating between obsessed with and simply always entertaining in the back of my mind the idea of a rust-powered fish for many years now (and have discussed that with other team members), so I'm approaching this with an open mind, but also with concerns. I must say that I knew this was coming for many months now from following @ridiculousfish on GitHub (), and have been wondering how it would the topic would be broached.

I passionately believe that rust is the way forward for writing any new code, especially in a cross-platform context. Quite apart from all the memory and concurrency safety it brings to the table, the general approach of the language and the community to always put correctness first and foremost has built a strong culture of correct, well-designed, and fairly maintainable software engineering. The package management is great, the team is fantastic, and support in the broader software world has been extremely positive and welcoming.

To go further, I have written my own fish-compatible rewrite (not port) of fish's tokenizer and was making progress on the AST and parser when I last had my "burning desire" to see something like fish but in rust. I was maintaining it on my private git server and not GitHub, but I've pushed it to GH in case anyone cares to take a look: https://github.com/mqudsi/velvet

Having said all that, I worry that rushing any part of this could be a death knell for the project. A quality rewrite (or port) takes time, on the magnitude of years rather than days. One good example to take inspiration from is tectonic, a xelatex port that started off with machine-rewritten code and then manually made progress on actual, idiomatic rust conversions... but even that was a greenfield project and not a first-party effort and so not saddled with some of the concerns we would be saddled by. The recommended module-by-module approach is definitely a great suggestion for a starting place, and I think it has great potential.

Off the top of my head, some concerns or points:

• Compatibility, obviously. Fish runs on archaic hardware for fun. This would be the end of that. That's not a reason that should stand in the way of progress, but it's something we can't turn a blind eye to and would have to quite clearly signal that we're ok giving up all platforms that don't run (modern versions of) rust.
• Idiomatic rust is too high-level for fish. You don't fork in rust, using the standard library interface doesn't give you access to poll/select/epoll/kqueue/whatever natively. async io is a horrible fake cludge with tokio/async_std faking it with thread pools or whatnot and performing far worse than truly native async implementations. We'd have to go that mostly alone to keep the same code we have (thinking "port" and not "rewrite" and aiming to preserve what we have).
• Despite whether we call it a "port" or a "rewrite" there's going to be a heck of a lot of rewriting taking place. Subtle bugs will creep in. Subtle compatibility fixes that weren't properly documented or annotated will be lost.
• We will be in a state of partial limbo until this completes one way or the other. We won't want to make any major changes to the C++ codebase while we're trying to get to feature parity in the rust one. Depending on how things unfold this could greatly hinder further progress in the project.
• Not all core team members are fluent in rust. This seems unfair. (This isn't just something I'm saying in passing but rather something I've given a lot of thought to and broached with others before.)
• C++ isn't really a "legacy" language (yet), no matter how much we wish the world would start treating it like that given all its warts and shortcomings.
• I understand what @ridiculousfish meant by "Rust is what we need to turn on concurrent function execution" but respectfully disagree. A complete rewrite (or port, if you prefer) to another language is a great time to facilitate that, but it's hardly a requirement, though I guess it could functionally be considered one if you want to take into account correctness, maintainability, etc.
• (Really minor compared to everything else, but if we're ultimately going to be a rust project, then I think it makes sense for all types coming from the rust codebase to use idiomatic rust naming conventions, but that would cause churn cpp-side. I suggest using idiomatic rust naming conventions and then using statements to alias them with cpp-style names in the cpp headers.)

• Idiomatic rust is too high-level for fish.

I think it's fine if a select few modules remain non-idiomatic Rust.

Subtle bugs will creep in. Subtle compatibility fixes that weren't properly documented or annotated will be lost.

Yeah this is a real threat. We need to be extra careful.
Unfortunately, rewrite commits are not easy to check for equivalence so code review takes a lot of effort. Happily we have people we can trust.
I think we have done a great job documenting odd behavior with comments, tests and a fairly clean commit history which helps.

We won't want to make any major changes to the C++ codebase while we're trying to get to feature parity in the rust one.

I think it's unlikely that we'll want to make far-reaching changes. As a quick estimate, over the last 4 years we only had

• the new parser 4494414 (Merge branch 'parser_cleanup_3', 2020-07-04)
• 13ab9e5 (Implement new event mechanism and migrate builtin and block output, 2019-02-17)
• 6f22aad (Merge branch 'debounce', 2020-03-06)

all by @ridiculousfish who I think is also the only one maintaining long-lived branches

• Not all core team members are fluent in Rust. This seems unfair.

Sure but I don't see them complaining? The language is not easy but once we have ported a substantial chunk of code, they can pick it up by osmosis.

Not all core team members are fluent in Rust. This seems unfair.

Sure but I don't see them complaining?

So far both @zanchey and me have explicitly declined to complain.

For those of us interested in contributing to the port, is there a plan / guide for picking up components to work on? I don't want to step on toes etc. Is it likely sufficient to just pick a random C++ builtin, reimplement it, and file a PR (after this one lands)?

You don't fork in rust, using the standard library interface doesn't give you access to poll/select/epoll/kqueue/whatever natively.

If you're looking for idiomatic solutions, mio provides an abstraction on top of epoll, kqueue, and Windows equivalents, along with non-blocking read/write and socket operations. mio is best known as a core dependency of tokio, but it's also quite commonly used directly.

As a lower-level alternative, another very popular crate is nix, a thin wrapper for Unix APIs that provides more Rust-friendly type signatures but leaves the semantics alone. That includes the polling APIs as well as fork.

The standard library's I/O functionality is much more limited, but in Rust it's entirely idiomatic to bypass it.

For those of us interested in contributing to the port, is there a plan / guide for picking up components to work on?

@Lucretiel You'll probably first want to wait until this PR lands, and then I'd guess it's fine to file an issue stating what you want to work on.

If there is a need to do anything more complicated, we can figure that out when it's needed.

If you're looking for idiomatic solutions

@comex At the moment we aren't. The plan is to first do a fairly straight port and then figure out all the details on how to make it idiomatic, whether that's through libraries or specific "patterns" or whathaveyou.

(also native Windows isn't really on the table at the moment - this is a unix shell, and the unix assumptions run deep)

@faho

So my preferred MSRV would be something like the one from Debian Stable/Oldstable, maybe even down to Ubuntu LTS. Of course, me not being a big rust person I have no idea how much pain that is - hence also my question about keeping the C++ track going for a bit longer.

While I know that this port is avoiding using various crates, focusing on a direct port, I believe I saw some crates already in use and I'm assuming that will grow as the amount of Rust code grows, especially once it hits the core of fish.

The user experience around dependency management with old versions of Rust isn't quite there yet which leads to a lot of debates around what MSRV to use (e.g. the libc thread). We have enumerated some potential improvements but I'm not aware of much work being done on these yet. Until then, a few crates will work on Debian Stable but most will only have an MSRV of 3-6 months and it can be a chore to use old versions.

As a lower-level alternative, another very popular crate is nix, a thin wrapper for Unix APIs that provides more Rust-friendly type signatures but leaves the semantics alone. That includes the polling APIs as well as fork.

I also believe that this is a core issue that is being worked on by the rust team at the moment with
this blog post may not be completely relevant but there is definitely a real effort coming this year for real forking natively.

Since this PR clearly escaped our little bubble, I feel like we should add some context, because I don't think everyone caught on to the joking tone of the opening message (check https://fishshell.com/ for similar writing - we are the shell for the 90s, after all), and really got what the idea here is.

People in various internet comment section also missed the fact that this was opened by the actual maintainer of the project, @ridiculousfish. It also wasn't really a surprise to the rest of the team, so we're all good here.

I'm not going to be adding a lot of code examples, because I want to focus on the broad overview of it all and the situation that we currently have, instead of getting stuck in minutiae.

Setting the stage

Fish is a fairly old codebase. It was started in 2005 by Axel Liljencrantz, who left the project some time in 2009.
He wrote it in C, and he decided to use wchar_t, presumably because that is how one assumed Unicode would work in 2005.
(It was also kept in a darcs repo, and some of our older commit still carry a darcs id in the git commit message)

After a few years of inactivity, fish was picked back up by Peter, who goes by @ridiculousfish (the name, I understand, is a complete coincidence).
He wrote a fork initially called "fishfish", and got Axel's okay to make it the "official" fish later on, this became fish 2.0.

ridiculousfish then ported fish to C++, mainly to get some data structures to replace the typical C write-your-own cruft.

A few other notable decisions:

1. Fish should be available on servers, which run old LTS distros - this means we build our own packages for a variety of them.
2. Fish should be easy to build and run, because it should be easy to send in patches - we want completion scripts by people who know a tool but aren't otherwise "programmers" or really know build systems, and we ideally want them to test these with what our git repo has to offer so they can file a nice PR.

The pain of the current tech

C++ is an old standardized language with multiple implementations, married at the hip to C, an even older even more standardizederer language. Any changes take ages to get to users so we can actually use it. We moved to C++11 in 2016 (some quick calculation shows that's 5 years after the standard was released), and we're still on C++11 because the pain of upgrading is larger than the promises of C++14/17. We needed to backport compilers for our packages until, I believe, 2020.

Having multiple implementations means you always hit the worst of any of them, because you can't dictate which implementation your users use. So we have to deal a lot more with cmake than we would like, sometimes for things as awkward as "which header is this function in".

C++'s string handling is subpar, and it's much too easy to fall into passing raw wchar_t * around (and we don't have access to string_view and that just enables even more use-after-free bugs!). This is annoying, because a shell is almost entirely string handling and unix api wizardry.

Fish also uses threads, for things like the autosuggestion and syntax highlighting, and we would like to have more "threads"/"concurrency"/"run-thing-in-backgroundness" (please insert the correct term here, I have never been good at these things). One large project has been to run multiple fish builtins and functions "at the same time", to enable things like backgrounding functions (ideally without using "subshells" because those are an annoying environment boundary that shows up in surprising places in other shells), and to simply be able to pipe two builtins into each other and have them actually process both ends of the pipe "simultaneously".

C++ offers few guarantees on what can be accessed from which thread. @ridiculousfish has been trying to crack this for years, and hasn't been confident enough in his solution. We want a tech stack that helps us here, and C++ doesn't.

The other general issues with C++ (header files, memory safety, undefined behavior, compiler errors are terrible, templates are complicated) are well-known at this point so I'm not going to rehash them. We know them, we have them, we hate them.

I don't really have a lot to say about cmake - the language isn't great and finding dependencies is a pain. We have worked around more issues in it than I would like.

Some of this is, undoubtedly, "self-inflicted" (wchar is, on the whole, a bad choice), for various definitions of that term, and it might be possible that some C++ fairy could make it work better, but the reality is that that fairy never materialized. No C++-Navi told us to "Hey! Listen! Templates!" and I forgot my ocarina elsewhere.

The gist of it is that C++ has caused us quite some grief, and we're done with it, and so, we have decided to leave it and everything related to it behind.

Why Rust specifically?

In short: Because it seems to be up to the task, because we have people who know it on the team, because it has momentum and because it promises to help us with the threading problem.

Other languages either have some specific pain point (go, python, java) or simply aren't known to us - this includes D, Zig, Nim and waaaayyy too many to list. That means asking us to use them means getting everyone to learn them and so far none of us have been interested.

Another big point: This PR exists. We already have a start. That may be "unfair", and we haven't democratically opened this up to our users, but at the end of the day the programming language is mostly an implementation detail and so it should be decided by the implementors.

We're not making an Objective Review For All Uses For Everyone, we are choosing what works for us. And so, social questions matter a lot. On that note rust also has a bunch of buzz and mindshare that other languages lack - would we have had people queuing up to help if we had gone with D? I doubt it.

Of course, where the programming language isn't an implementation detail, we'll work on it. That's why I have been asking a bunch of questions related to packaging in this thread, and that is why we have spoken about platform support.