7

File for Divorce from LLVM · Issue #16270 · ziglang/zig · GitHub

 1 year ago
source link: https://github.com/ziglang/zig/issues/16270
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

File for Divorce from LLVM #16270

2 of 19 tasks

andrewrk opened this issue Jun 29, 2023 · 21 comments

2 of 19 tasks

File for Divorce from LLVM #16270

andrewrk opened this issue Jun 29, 2023 · 21 comments

Comments

Member

I'm sorry honey, it's just not working out. Our relationship worked when we were younger, but we're both older now and we've grown apart.

This issue is to fully eliminate LLVM, Clang, and LLD libraries from the Zig project. The remaining ties to these projects are as follows:

This will remove C++, Objective-C, and Objective-C++ compilation capabilities from Zig. In the near term, the machine code generated by Zig will become less competitive. Long-term, it may catch up or even surpass LLVM and GCC.

In the near term it would also reduce the number of targets Zig supports, since LLVM has a nice laundry list. However, LLVM supports fewer targets than you think... we are constantly running into bugs and embarrassing O(N^2) code for pretty much every target except x86, aarch64, and webassembly. Long-term, Zig plans to have Contributor-Friendly IR, or something equivalent, to optimize for low maintenance for supporting many targets that LLVM would never even dream of, for example MOS #6502.

Note that there would still be an LLVM backend for outputting .bc files (#13265), but the Zig compiler would lack the capability to compile .bc files into object files. LLVM or Clang would need to be installed and invoked separately for that use case.

In exchange, Zig gains these benefits:

  • All our bugs are belong to us.
  • The compiler becomes trivial to build from source and to bootstrap with only a C compiler on the host system.
  • We stop dealing with annoying problems introduced by Linux distributions and package managers such as Homebrew related to LLVM, Clang, and LLD. There have been and continue to be many.
  • The Zig compiler binary goes from about 150 MiB to 5 MiB.
  • Compilation speed is increased by orders of magnitude.
  • We can implement our own optimization passes that push the state of the art of computing forward.
  • We can attract research projects such as alive2
  • We can attract direct contributions from Intel, ARM, RISC-V chip manufacturers, etc., who have a vested interest in making our machine code better on their CPUs.
InKryption, alichraghi, Beyley, r00ster91, yaymalaga, Mindgibber, jtriley-eth, imlodinu, foucist, watsy0007, and 11 more reacted with thumbs up emojijvyden, viriw, and andrewschaaf reacted with laugh emojischeibo, Jarred-Sumner, pdeva, Phalangers, zekexiao, RohanKapurDEV, yihongang, and xdBronch reacted with confused emojimlugg, alichraghi, theoparis, Beyley, ianprime0509, batiati, usdogu, r00ster91, Mindgibber, jtriley-eth, and 9 more reacted with heart emojimlugg, travisstaloch, r00ster91, alichraghi, imlodinu, achille-roussel, GithubPrankster, foucist, andrewschaaf, artob, and 2 more reacted with rocket emojirodrimati1992, Congee, and 190n reacted with eyes emoji

andrewrk

added the proposal This issue suggests modifications. If it also has the "accepted" label then it is planned. label

Jun 29, 2023

Contributor

I see the 1.0.0 milestone for this, is it meant to be a blocker for 1.0? or simply the goal

Member

Author

The milestone, on an issue labeled "proposal", means that a decision must be made to accept or reject that proposal before tagging the release corresponding to that milestone.

For an issue labeled "accepted", the milestone means that it must be implemented by then. So, if this proposal is accepted, then I will evaluate at that time which milestone to move it to.

Contributor

In the near term, the machine code generated by Zig will become less competitive. Long-term, it may catch up or even surpass LLVM and GCC.

IMO, this is the biggest question. One of the most compelling reasons to use Zig is runtime performance of software written in Zig. Without LLVM's optimization passes, what will that look like?

scheibo, pdeva, and bryanhelmig reacted with heart emoji

Contributor

Long-term, it may catch up or even surpass LLVM and GCC.

We can implement our own optimization passes that push the state of the art of computing forward.
We can attract research projects such as alive2
We can attract direct contributions from Intel, ARM, RISC-V chip manufacturers, etc., who have a vested interest in making our machine code better on their CPUs.

Zig will continue to implement optimization passes of its own over time and get faster.

Sponsor

Contributor

So here the projects that depend on the ability to compile C++ that I currently developing:

And also some NDA game platform that have C++ only API that will require some C++-to-C glue code to be compiled, but obviously not implemented.

To me, the ability to seamlessly build any C, C++ and Obj-C is a big selling point of the Zig toolchain even if it is behind a optional flag to enable LLVM and Clang when compiling Zig. A part of the hype momentum around Zig is due to that fact.

If this happens, I think I will remove my donation to the Zig Software Foundation.

TL;DR: Lots of libraries in the game development world (closed or open source) require the ability to compile C++.

InKryption, feenor, foxnne, scheibo, GeffDev, yalopov, wub, directionless, nornagon, serxka, and 2 more reacted with thumbs up emojiandrewrk reacted with eyes emoji

Contributor

I think this proposal would hurt the Zig ecosystem more than it would help it, due to several reasons:

  • Adoption of compilers and languages in the embedded is basically driven by the code generation quality in terms of size. We have to be on-par or better than GCC and LLVM to be even considered tmto be adopted. 2%-5% bigger code is often not an annoyance, but a technical problem.
  • A lot of projects use C++ code and might benefit from build.zig "as is". This could lead to adoption of Zig, with the build system as a kick-start. Removing support for C++ will hurt adoption a lot, because now the build of C++ projects with some additional Zig code will be even worse than just adding more cmake. Thus, a lot of projects wouldnt benefit from Zig in terms of quality of life, as it is yet another build tool.
  • Zig is with its "batteries included" cross-compilation the best toolchain for doing native development. We would basically downgrade our toolchain to something Go where we rely on external tools to successfully build more complex projects. Even in the current state, its often so much easier to yeet zig at something to get it to build than even trying to get a cross-build running with existing tools
  • Considering the scenario i have at work, introducing Zig as a build environment for several million lines of C++ would make a lot of people happy, as the builds would be faster, easier maintainable and trivially portable to other OS. With C++ and LLVM support removed, i dont see any chance of Zig adoption at this company

Imho, this proposal strongly violates the

Together we serve end users

idea, as the current direction we're heading is a really good unified native build environment based on a single static executable that can serve projects in arbitrar, sizes, shipping compilers for several major languages, a huge ass support for targets and a build system, making work in systems programming fun, even if one doesnt use Zig as a language.

We are on a good way to replace a huge list of tools with a single executable, making contributions to projects build on Zig fun, easy and platform independent.

When this proposal is accepted, in addition to Zig one will need to have the following tools installed:

  • (GNU) Make
  • premake
  • meson
  • CMake
  • ninja
  • (arch)−(os)-$(abi)-gcc
  • llvm/clang
  • GNU autotools
  • vcpkg
  • gradle
  • conan
  • qmake
  • SCons
  • maven

We can potentially replace all of those tools wit a single, equally powerful executable, making the live easier for all native devs out there

personal projects that would be affected:

InKryption, foxnne, feenor, scheibo, GeffDev, pdeva, wub, mattnite, and teburd reacted with thumbs up emojimlarouche, xdBronch, foxnne, feenor, and wub reacted with heart emojiandrewrk reacted with eyes emoji

I'm still a beginner I believe, but if it is at all worth it for me to give my point of view, Zig's ability to replace all of the build tools mentioned above is a big reason I was interested in Zig in the first place. I struggled a lot with all the different build systems and Zig is a really refreshing breath of fresh air.

I have two personal projects that use zgui heavily and I had planned on continuing.

GeffDev and andrewvc reacted with thumbs up emoji

Sponsor

Contributor

With Mach engine we have two dependencies that would be very, very painful to remove (would set us back years):

  1. DirectXShaderCompiler (DXC) - a Microsoft fork of LLVM 3.7, which generates DXIR (LLVM IR) which is what direct3d graphics drivers consume. Microsoft is trying to upstream support for emitting DXIR (specifically LLVM v3.7 IR) to the latest LLVM/clang version.
  2. Calling into Apple's Metal shader compiler - which converts Metal's text shading language into (yet another fork) LLVM IR bytecode, which is what the Metal API consumes - only this one is proprietary and undocumented. Obj-C is the only way to invoke it, I believe.

It will be a long time before Zig's SPIRV backend is capable enough to generate non-GPGPU shaders for graphics APIs (if ever, since it would likely require major language changes) - so I don't see a way for us to escape these aside from replicating what these two projects -- LLVM forks -- do on our side.

Every other C++ dependency I believe we could safely escape from.

dustyrockpyle and serxka reacted with thumbs up emojiandrewrk reacted with eyes emoji

Sponsor

Contributor

A positive long-term effect of this change is that it would push us as a community away from wrapping C++ code and towards more pure-Zig solutions.

Many of the comments in this thread are about people using zgui, wwise, zig-gamedev, assimp, and other C++ libraries wrapped with Zig. It gives you a leg up in the short term, but I worry in the long-term that people's gamedev experiences coming to Zig will be 'initially I saw a nice language... then I encountered the guts of the libraries I was told to use were large, clunky C++ codebases'

Also a beginner, but I think being able to use Zig as a C++ build system (which is what I use for all my private C++ projects) is an invaluable feature to me and I believe many other people. The simplicity of Zig, having a compiler and a build system contained within a single executable (with the added bonus of being easily cross platform), is really cool, and it would be kinda unfortunate to see that feature be removed as an effect of removing clang and friends. However, if this does go through, the ability to fallback to generating .bc files and invoking clang is nice.

foxnne reacted with thumbs up emoji

Contributor

One of my friends pointed out that nothing stops you from invoking clang in a build.zig to compile C++ dependencies, even if Zig stops including clang. I wonder how much of a problem that would really be for these projects?

Contributor

doing that would be an additional dependency without the ease of zigs cross compilation

yujiri8 and directionless reacted with thumbs up emoji

Contributor

Hmm, the more I think about this proposal the less I like it. I'm feeling it would be better to make the LLVM backend non-default (that is, switch -fno-LLVM for -fLLVM or something) but not remove it. It seems like this would solve most of the issues: the existence of many bugs that are LLVM's fault, and the slow compile speed, aren't strong reasons to remove the LLVM backend so much as make it non-default. It wouldn't solve the issues of the binary size of the zig compiler or difficulties with building it, of course.

ethernetsellout reacted with thumbs up emoji

I think I can speak for all gamedevs by saying that removing C++ compilation would be a disaster. Too many amazing existing gamedev libraries and tools are built on C++ that disallowing easy use would strangle adoption in that field, as well as complicate existing projects. dear imgui is the obvious example but it's not the only one.

It's easy to imagine that in the long term, this change will push us towards "rewrite it in zig" with all the benefits that would entail, but the downside is that the existing corpus becomes inaccessible; limiting our options heavily even once zigs ecosystem matures

Contributor

I think llvm is needed until ecosystem of pure-zig library is very very mature and rich.

Yeah we want faster compiling speed and smaller tarball, but not at the risk of losing one-zig-to-rule-all.

yujiri8 reacted with thumbs up emoji

I love the ambition of this proposal, but to reiterate what has already been stated, losing c++ compilation would be losing one of the main selling points of zig. I was drawn to zig in part because it reduces the hellishness of depending on c/c++ projects. Zig having a c++ compiler inside it also has the benefit of there being less c++ to have to deal with, less python to have to deal with, less cmake to have to deal with.

On the other hand, the core of this proposal has too many benefits for it to be rejected entirely. I think reducing the scope would be beneficial. How about:

  • Eliminate dependencies on LLVM & LLD
  • Keep clang as an optional dependency for easy cross-compilation

Contributor

I totally get the desire to get rid of huge third-party dependencies that bring a lot of baggage. There's also something to be said for avoiding an LLVM monoculture in the programming language space.

Even so, I see this as a net negative for users. What drew me to Zig in the first place was the pragmatic approach of acknowledging that there is a world outside of Zig that needs to be interoperated with for the foreseeable future and even providing a best-in-class cross-compilation experience along with that. I built my project integrating Zig build support with the .NET/MSBuild ecosystem on that selling point.

On the whole, I think this proposal would be an unfortunate (if well-intentioned) bait-and-switch, considering the Zig website for a while has advertised this:

image

In addition, these blog posts drove a lot of attention to Zig in the past:

Just to be super clear: I don't mean to insinuate bad faith or anything of the sort here. But I think it's fair to say that you have to contend with the fact that this proposal would pull features that are not only usable today, but are also prominently advertised.


All that said... assuming this is even remotely practical, maybe there's a potential middle ground: Would it be possible for Zig to continue to use the Clang frontend to provide C-family support, but rip out the LLVM IR lowering and replace it with lowering to ZIR/AIR? (I guess this is more or less how Aro would be integrated too?)

If this could be done, the codegen dependency on LLVM could be killed, achieving at least some of the goals of this proposal. There's probably also no reason to keep LLD support around as long as zld can catch up, so that eventually goes too. And users remain happy. Some of the build and distro woes would remain, of course, but, that's compromise.

ccapitalK, axojhf, mitchellh, mlarouche, and ethernetsellout reacted with thumbs up emoji

Contributor

I’m fully in favor of making it possible to use Zig without any LLVM components, but I agree with many of the comments here that it’s important for Zig to maintain the capabilities that it currently has in terms of cross-compilation, compiling C/C++ code seamlessly, and generating maximally performant binaries.

These factors are big drivers of Zig’s adoption, and I fear that damaging them (even temporarily as in the case of code generation quality) would seriously hurt Zig’s future.

Personally, I work in the robotics space, where C++ is the dominant language for many libraries and frameworks. I think Zig has a lot of potential in this space, but being able to integrate with existing libraries is absolutely essential for adoption.

Sponsor

Contributor

Is there a story in this proposal for JIT compilation? I have a Zig project with currently relies on LLVM's Orc to JIT-compile audio DSP functions. I'm not particularly stoked about or attached to Orc itself, but it does give me in-process compilation with low-milliseconds latency, something I'd need for dynamic real-time audio applications.

WebAssembly would not be an issue here because I wrote the Wasm compiler myself, but for x86_64 and friends I would need a replacement. Passing LLVM bitcode to a separate process might work but would feel like a downgrade. Vendoring and embedding the Zig compiler source might be the best option in that case.

(Aside: I have started work on reading and writing LLVM bitcode from Zig, and if this is accepted would be happy to resume work on that.)

Sponsor

Contributor

I'll start by stating my opinion: the C language frontends are super important to me for Zig to maintain.

I agree with others on the point that Zig being a C/C++ compiler is a big point of attraction that brought me to the language. I started with loving that idea, and ultimately fell in the love with the language, and now I use both. I don't have anything more to add to that that the others above haven't.

I'll add my own personal experience. I have many personal Zig projects, but my biggest one that people tend to know about in the community is my terminal emulator. There are two important dependencies that would be impacted by this:

  • Harfbuzz - The far and away single most complete cross platform text shaping engine. Text shaping for those that don't know is the process of laying out text, processing things like multi-codepoint emoji into single glyphs, Asian language handling, etc. This is not something you want to really maintain in your own language (i.e. write natively in Zig) because Harfbuzz is so good and so well supported. This is shipped as a single large C++ file with no other dependencies. Without access to Harfbuzz, gaming and manual-GUI applications (not using a mega framework) would surely suffer.

  • Objective-C for Mac work. Admittedly, this is probably deprecated over the long term since Apple is pushing very hard into Swift. Still, I use Zig's ability to compile Objective-C files to augment my native Zig Mac applications. I've personally written Zig objective-C bindings using the C API, but I also still like to just write some ObjC sometimes which would otherwise take a big mess of Zig/ObjC-runtime code. Again, I think I could find a way around this one, but I think its worth noting for now that ObjC is still a part of [low-level] macOS GUI development.

Andrew, your dislike of C++ is well known! I don't love it either (to put it kindly). If, as an audacious goal, you wanted Zig to lower C++ usage, I think Zig being able to compile existing projects and enable iterative migration away from C++ to Zig is the way to do it. I think if Zig doesn't support C++, the C++ "people" would just avoid Zig altogether.

mlarouche, axojhf, and shepherdjerred reacted with thumbs up emoji

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Assignees

No one assigned

Labels
proposal This issue suggests modifications. If it also has the "accepted" label then it is planned.
Projects

None yet

Milestone

1.0.0

Development

No branches or pull requests

17 participants

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK