GitHub - lumen/lumen: An alternative BEAM implementation, designed for WebAssemb...
source link: https://github.com/lumen/lumen
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
README.md
Lumen - A new compiler and runtime for BEAM languages
Contributing
In order to build Lumen, or make changes to it, you'll need the following installed:
Tools
First, you will need to install rustup. Follow the instructions at that link.
Once you have installed rustup
, you will need to install the nightly version of Rust, this is
currently required due to our dependency on wasm-bindgen
when targeting WebAssembly:
rustup default nightly # install nightly toolchain
You may also want to install the following tools for editor support (rustfmt
will be required on
all pull requests!):
rustup component add rustfmt clippy
Next, you will need to install the wasm32
targets for the toolchain:
rustup target add wasm32-unknown-unknown --toolchain nightly
You will also need to install the `wasm-bindgen** command-line tools:
cargo +nightly install wasm-bindgen-cli
LLVM
Now that Rust is setup and ready to go, you will also need LLVM for building the compiler.
LLVM requires Cmake, a C/C++ compiler (i.e. GCC/Clang), and Python. It is also highly recommended that you also install Ninja and CCache to make the build significantly faster. You can find all of these dependencies in your system package manager, including Homebrew on macOS.
To install LLVM:
git clone https://github.com/lumen/llvm-project
cd llvm-project
make llvm
This will install LLVM to ~/.local/share/llvm/lumen, and assumes that Ninja and
CCache are installed, you can customize the llvm
target in the Makefile
to
use make
instead by removing -G Ninja
from the invocation of cmake
,
likewise you can change the setting to use CCache by removing that option as well.
NOTE: Building LLVM the first time will take a long time, so grab a coffee, smoke 'em if you got 'em, etc.
Once LLVM is built, you can run make build
from the root to fetch all dependencies and build the project.
Project Structure
Lumen is currently divided into a few major components:
Compiler EIR liblumen_compiler liblumen_codegen lumen Interpreter liblumen_eir_interpreter Runtime liblumen_core liblumen_alloc lumen_runtime lumen_webEIR is short for Erlang Intermediate Representation, it is managed in a separate project which can be found here. Lumen's frontend parsers and IR are found there.
At the moment, the compiler backend pieces are being worked on in another branch, and so should be ignored for now. These pieces will be merged soon, and this README will be updated at that point.
The interpreter is currently how we are testing and executing Erlang code, and it builds on top of the same compiler frontend/IR that the compiler backend will use.
The runtime is built as a static library, which is ultimately linked into compiled Erlang programs, this runtime contains the machinery necessary to interact with the underlying system, as well as provide concurrency, garbage collection, and other necessities which are not part of the compiler transformation.
Making Changes
About Lumen
Lumen is not only a compiler, but a runtime as well. It consists of two parts:
A compiler for Erlang to native code for a given target (x86, ARM, WebAssembly) An Erlang runtime, implemented in Rust, which provides the core functionality needed to implement OTPLumen is different than BEAM in the following ways:
It is an ahead-of-time compiler, rather than a virtual machine that operates on bytecode It has some additional restrictions to allow more powerful optimizations to take place, in particular hot code reloading is not supported The runtime library provided by Lumen is written in Rust, and while very similar, differs in mostly transparent ways. One of the goals is to provide a better foundation for learning how the runtime is implemented, and to take advantage of Rusts' more powerful static analysis to catch bugs early. It has support for targeting WebAssemblyGoals
Support WebAssembly as a build target Produce easy-to-deploy static executables as build artifacts Integrate with tooling provided by BEAM languages More efficient execution by removing the need for an interpreter at runtime Feature parity with mainline OTP (with exception of the non-goals listed below)Non-Goals
Support for hot upgrades/downgrades Support for dynamic code loadingArchitecture
Compiler
Internally, the compiler represents Erlang/Elixir code in a form very similar to continuation-passing style. Continuations are a powerful construct that enable straightforward implementations of non-local returns/exceptions, green threading, and more. Optimizations are primarily performed on this representation, prior to lowering to LLVM IR. See eirproject/eir for more information on the compiler frontend and EIR itself.
The compiler produces object files, and handles linking objects together into an executable.
Runtime
The runtime design is mostly the same as OTP, but we are not running an interpreter, instead the code is ahead-of-time compiled:
- The entry point sets up the environment, and starts the scheduler
- The scheduler is composed of one scheduler per thread
- Each scheduler can steal work from other schedulers if it is short on work
- Processes are spawned on the same scheduler as the process they are spawned from, but a scheduler is able to steal them away to load balance
- I/O is asynchronous, with dedicated threads and an event loop for dispatch
The initial version will be quite spartan, but this is so we can focus on getting the runtime behavior rock solid before we circle back to add in more capabilities.
NIFs
NIFs will be able to be defined in any language with C FFI, and will need to be compiled to object files and then passed via linker flags to the compiler. The compiler will then ensure that the NIFs are linked into the executable.
The design of the FFI is still up in the air - we will likely have a compatibility layer which will
mimic the existing erl_nif.h
interface, but since the runtime is different, there may be
opportunities to provide more direct hooks to parts of the system.
License
Apache 2.0
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK