systemd through the eyes of a musl distribution maintainer

systemd through the eyes of a musl distribution maintainer

1. davmac

   edited 4 days ago

   | link

   An alternative implementation of systemd that is “just” the service management component, leaving out all other components, would be very appealing to distros IMO

   For what it’s worth, my own Dinit (https://davmac.org/projects/dinit/) aims to be something like that, though note that actual systemd compatibility isn’t at all a priority (Dinit service description files are a different format to Systemd’s unit files).

   Basically, Dinit is trying to be an (equal) “best in class” service manager and init, while not trying to take on responsibility for other parts of the system (eg by default it logs via syslog, there’s no journald equivalent, network management is left to other software). The idea is much more “fit in with the ecosystem” rather than subsume or replace it. Nevertheless, some things from systemd have been taken on board (or at least, coincide with Dinit’s design): declarative service descriptions, and dependency management, form the backbone; cgroups support is present though still a bit limited, and it’s a similar story with socket activation. A few things still need to be improved, but it’s already usable (and, it’s portable - it compiles and runs just fine on FreeBSD, OpenBSD and MacOS, at least). (edit: and works fine on musl-Linux systems, too).

   Two distributions offer it as an init system: it is an option in Artix, and it is the standard in Chimera. Daniel Kolesa of Chimera has been doing a lot of work towards overall system integration (including with DBus and user services; not all of it is completely specific to Dinit). Meanwhile I’m still working on the core (with occasional contributions from others and especially the support of Mobin Aydinfar).

   There are other options, too; OpenRC is actively maintained, and S6/S6-RC is a high-quality system. Both are in use in various distributions. I feel like we’re well past the point of “no viable alternatives” to systemd. What is still missing is the level of total integration that systemd provides. When you don’t opt to simply replace every package that doesn’t neatly fit in with your vision with your own re-implementation, as systemd tends to, that’s a little more complicated- but I still feel like it will be worth it in the end.

   All that said, I know a lot of people really like systemd and that’s fine too. I understand why distributions adopted it - it made their jobs a lot easier. But, I personally don’t agree with various aspects of its approach and a lot of others don’t either, so it’s good that there are still other options.

   1. I feel like we’re well past the point of “no viable alternatives” to systemd.

      My point was that I want something that is systemd without the other bits, mostly because the world seems to be rapidly standardising on unit files, systemctl, etc. I think it’d be valuable to have an implementation of systemd that didn’t require you to use all the other bits, like journald. Something like Dinit could easily be the basis of this, with a converter from unit files to service description files and the like. But that “last mile” - making it transparent to users - is what is really missing from the alternatives. That’s how you gain serious adoption, and that’s how you gain wide acceptance. I do like the overall concepts of Dinit and I’m hopeful that it will continue to grow.

      I also very much disagree that OpenRC is at all a replacement or alternative. I did all the integration work, and have personally done most of the conversion from unit files to OpenRC scripts for packages that don’t provide them, for the Adélie distribution. I loathe how difficult it is to manage and maintain as an administrator, let alone as a developer.

      I do feel like s6-rc could be a serious alternative in the future as well, but even the author admits it needs more user-facing work before it can be one.

      1. My point was that I want something that is systemd without the other bits, mostly because the world seems to be rapidly standardising on unit files, systemctl, etc

         I get what you are saying. It might be possible to use another system (like DInit) with a compatibility layer, like you say, and I think that’s more likely than an actual cut-down systemd implementation, though I sort of wonder whether a clean break would be better anyway. After all, wherever the systemd interfaces are being used, the systemd ecosystem tends to be already pretty well entrenched. I totally agree about the “last mile” - that was my point, having a cut-down-service-manager-only version of systemd is theoretically nice but if it did exist, it would have lost the main advantage that systemd really brings to the table. For that reason I’m not so convinced that systemd-lite would really be all that appealing compared to any other reasonable choice (though I guess unit files already exist for so many packages and that alone could be a deciding factor, at least until automated translation to other formats is available).

         I’m hoping the near future will see us cover at least a portion of the “last mile” of integration at least with Dinit, but I’m sure that work could also be of benefit to systems that use other service managers too.

         In honesty, I don’t know enough about the nitty-gritties of OpenRC to comment further on it; certainly there are things about it I’m not keen on. Some swear by it though. Regarding S6-RC, I’m aware of that situation (and indeed, that’s why Dinit exists; it seemed there was still a niche to fill in terms of a solid system that was also still user-friendly, including for administrators and distro maintainers, while not taking on the very broad scope that has defined systemd).

   2. q66

      3 days ago

      | link

      I’m really not happy with the integration in artix, by the way - they seem to have a goal of providing a ton of different init systems, but none of them receive the amount of effort to do proper integration, and as a result it’s undermining adoption as a whole since it’s not only not making proper use of the service manager’s potential, but also makes things harder for projects that do care, as it drives in a wedge between those and software upstreams that would potentially adopt it. As far as I know, Chimera was the first project to consider adopting it and started taking steps towards doing that, only for some people in the community at the time to snipe that idea and do their own halfassed integration, and it’s been a pain ever since.

      1. Their entire stick is just curl’ing down whatever service file from Void/Gentoo/Other distros master/main branch and jamming it into the package and calling it a day. They are really not doing any work on their own for this.

2. gf0

   4 days ago

   | link

   An alternative implementation of systemd that is “just” the service management component, leaving out all other components

   That exists, and is called: systemd! I don’t know why so many people fail (or doesn’t want) to understand that systemd is both a namespace/umbrella project, and a concrete executable running as PID 1. Is Plasma also some huge monolith, planning on eating the world by having a painting program, Krita? The same way, network management is not part of the core systemd executable, and it is quite modular.

   1. systemd is both a namespace/umbrella project, and a concrete executable running as PID 1.

      As far as I’m aware, and looking at the Meson build system for v255 right now, you cannot disable journald. It doesn’t look like nspawn is removable either, though I suppose that isn’t strictly necessary as you aren’t forced to use nspawn as a side effect of using the service manager. I also don’t see a way to remove udevd, for those who would rather use mdevd, but that is probably something you could work on using unit files alone.

      Yes, you can disable networkd, resolved, oomd, logind, hostnamed, localed, machined, portabled, homed, timedated, timesyncd, and importd. It’s great that they have made it at least somewhat modular in that way.

      Is Plasma also some huge monolith, planning on eating the world by having a painting program, Krita?

      Krita is part of KDE Applications, not Plasma. It isn’t even part of the official applications release service, last I knew. Seems a bit apples-to-oranges in my opinion.

      1. gf0

         4 days ago

         | link

         Journaling is a cross-cutting concern. In the pre-systemd world, logs from the very early init were simply discarded, no way to access them as the filesystem was not yet available. Systemd solves it and you are free to pipe its output to your log system of choice. Feel free to look at it as a necessary data structure used by systems itself.

         Udevd also seems to be similarly critical to an init system, responsive for bringing up a system by listening to all kinds of hardware devices.

         Re krita: fair enough, it was more of an over-dramatized analogy trying to show the faulty logic behind people who think that systemd incorporates the world.

         1. In the pre-systemd world, logs from the very early init were simply discarded, no way to access them as the filesystem was not yet available.

            That doesn’t match my experience. I’m using sysklogd with s6-linux-init on this computer, and /var/log/messages starts with Jun 6 22:27:07 gwyn kernel: [ 0.000000] dt-cpu-ftrs: setup for ISA 3000 which is the very first line the kernel prints, before even identifying itself. If you mean the init system itself, OpenRC has all of its early messages in its log as well:

            awilcox on gwyn ~/Code/awilfox/user-next % head /var/log/rc.log
            
            rc sysinit logging started at Sun Jun  6 22:27:05 2021
            
            
               OpenRC 0.24.1.be87f4cc is starting up Adélie Linux (ppc64)
            
             * /proc is already mounted
            

            Systemd solves it and you are free to pipe its output to your log system of choice.

            I did mention that in the article, but as I noted, that adds a lot of code in pid1 that I would rather not be there. Put another way, this should be an administrator choice, just like all the other bits of systemd. Why can I choose to manage my own network, my own DNS, my own time sync, even my own mounting, but not my own logging?

            Udevd also seems to be similarly critical to an init system

            I disagree. You could argue that a device manager is critical to have in early boot, though the kernel gives you a few static nodes by default, at least last I checked. It doesn’t have to be udevd - Alpine systems use mdev, Adélie and Artix can use mdevd. And none of these are integrated with the service manager. I think it’s possible to even not use udevd on a systemd system, if you remove all the units for it and make your own for an alternative.

            1. Why can I choose to manage my own network, my own DNS, my own time sync, even my own mounting, but not my own logging?

               You can manage your own logging? You even mentioned in your blog post that you can set up journald to just be a forwarder for syslog. If your point is that you don’t want journald code at all in pid1, well OK, but is that worth the hassle of (1) maintaining a fork of systemd, or (2) getting upstream systemd to remove it and massively break backward-compatibility with every system?

               1. FWIW journald, even configured exclusively as a forwarder, is a significant bottleneck on anything which logs at high volume. It is unable to utilize multiple cores as a “feature” and “helpfully” presents a /dev/log socket in mount space of all units meaning that you cannot avoid journald if your daemon uses the glibc (or equivalent) syslog call without doing significant hackery to provide your preferred logging daemon’s socket at /dev/log in the daemon’s mount namespace. The systemd maintainers will point at “journald namespaces” but all that does is isolate your log stream to its own journald instance which is still considerably less performant and configurable than syslog-ng or rsyslog.

                  To be clear, like the article’s author, I am a fan of many of the features and design choices of systemd, but its warts often cause me an undue level of consternation. It’d be less of an issue if Poettering et al were better at working with the community.

                  1. fanf

                     2 days ago

                     | link

                     In my limited experience, the systemd journal is also incredibly slow to read. It’s faster to use grep and less on text files. Dunno how they managed to botch the “fast” binary storage so badly to get this result.

               2. If making a component of your solution optional instead of required is “massively breaking backward-compatibility with every system” you’ve really done a terrible job at architecture.

                  And making the journal optional instead of required shouldn’t break anything, because any distro that wants to leave the journal present just leaves it enabled. But then you can have alternate solutions when the administrator wants it, or if the distro wants it.

                  1. Easy to criticize architecture that someone else designed. Difficult to understand the real-world considerations that went into it. Especially if we consider that they made pretty much everything else optional, but decided that journald had to be integrated. Applying Chesterton’s Fence is appropriate here.

                     My question is again–is it really worth the additional complexity to strip out journald from the systemd executable, maintain it as a separate binary, and tell distros to turn yet another knob? Especially when distros that don’t like systemd usually don’t like it for a host of reasons, and journald is not the deciding factor for them? Is there actually any distro that doesn’t use systemd now, but would if only journald were optional? And if that need is great enough, shouldn’t someone have done it by now?

         2. x64k

            edited 3 days ago

            | link

            In the pre-systemd world, logs from the very early init were simply discarded, no way to access them as the filesystem was not yet available.

            How “pre”-systemd are we talking here? For as long as I can remember using these things, the kernel would log to its own buffer, and whatever logging daemon you would use would then commit early dmesg history to a filesystem record as soon as a filesystem became available.

            There were a bunch of bugs early on, when Fedora and a bunch of other big distros started switching to systemd-journald. systemd-journald still lacked some features at the time and most tools relied on *syslogd (Fedora, I think, used rsyslogd?) anyway, so they shipped with both systemd-journald and syslogd. The system would log (to syslogd) via systemd-journald; but systemd-journald didn’t flush early boot messages, so syslogd never got them, which led a lot of people to believe that syslogd couldn’t log early boot messages. journalctl still printed the whole history so the easy fix on the user side was simply to cut off syslogd if you could.

            I don’t know if this was universal or maybe it needed some special flags, maybe it was a distro thing; but I could swear I remember very early boot logging working as early as… I dunno, pre-2010?

3. AFAIK you don’t have to run the ‘other bits’, they are all optional. Only systemd itself and journald are mandatory.