Oil 0.8.pre3 - A Line Editor and a Continuous Build

This is the latest release of Oil, a Unix shell:

• /release/0.8.pre3/ : source tarballs, docs, metrics, and more.

Please try it on your shell scripts and report bugs ! To build and run it, follow the instructions inINSTALL.txt.

This release has two major themes :

1. Features and fixes toward running the Bash Line Editor (ble.sh). I learned of this fascinating program a few weeks ago.
2. Behind-the-scenes work to continuously build and test Oil. We now have travis-ci.oilshell.org , and I'm excited about offloading more work onto it.

Table of Contents

Serendipity With ble.sh

Shell: The Good Parts

Appendix A: Selected Release Metrics

Appendix B: Closed Issues

Tie-Ins

The two themes strongly relate to prior blog topics. Before diving into details, here is some higher-level context.

Project Scope

Shell is both an user interface and a programming language , and I've addressed this issue a few times.

• In the2018 FAQ, I said that Oil's UI should be written in Oil, not in C, C++, or Python. I still believe that, and I elaborated on that in this Reddit comment (cited inJanuary).
• "Fish Oil" is a good idea . More specifically,fish has a rich UI, while Oil concentrates on shell as a language.

There's no reason a shell can't do well in both respects — it's just a lot of work. For reference, bash itself is ~142K lines of code . Its companion GNUreadline, which provides a minimal experienced compared to sayfish, is ~34K lines of code.

Serendipity With ble.sh

A few weeks ago, Danilo Spinella pointed out the Bash Line Editor project, which I'll refer to asble.sh from now on.

It gives you afish-like interactive experience inbash. The astonishing thing is that it's written in bash . I'm no stranger to big shell scripts, bothbatch andinteractive, but this surprised me!

I'll write more aboutble.sh in the future, but here are some links:

• The Biggest Shell Programs in the World : I believe it's the most substantial shell program in the world by a large margin.
  • It's not just that it's large (20K-30K lines), but it also uses sophisticated data structures and algorithms.
  • If you know of other big shell programs, edit the wiki page and let me know .
• How Interactive Shells Work : Author Koichi Murase describes howble.sh works. This information is useful for API design in Oil .
  • If you have expertise on the internals of another shell, again, edit the page and let me know.

I'm not sure if Oil will be able to run ble.sh , but this release is major progress toward doing so.

If it eventually does, that will show an unexpected benefit of Oil's compatibility withbash, which has been a tremendous effort.

Shell: The Good Parts

The other major theme is the continuous build at travis-ci.oilshell.org .

Its source code lives in the new services/ directory, and I started calling it "Toil". Besides being a useful tool, its implementation demonstrates techniques that I want to write about under #shell-the-good-parts:

• Toil is a set of shell scripts that invokes many tools: Python scripts I wrote, ssh withBernstein chaining, scp , zip , etc.
  • In other words, shell is always the main() ; I reuse as many Unix tools as possible; and write my own where there are gaps.
• Travis CI and its YAML config is just another tool. I try to decouple my code from the details of code I don't control. A service is just a process on another machine .
  • Toil should eventually run on build services other than Travis, which is required for platforms other than Linux/OS X and x64 . When it does, this will be more convincing.
• Some parts of the build run in a Nix environment, but Nix is also just another tool (and it happens to be one with its own programming language). Sometimes its model is useful, and sometimes it isn't. Shell is the most general model. It's at the lowest level.

In short, I use the Unix philosophy of reusing programs written in different languages as shell "libraries".

More Shell Usage

• These release announcements are now semi-automated with a shell script . The template it generates involves agit log in HTML, as well a list of closed issues with Github's API ().
• A script called html_head makes it convenient to generate HTML from shell, as again I did for bothToil and these release notes.

New in This Release

Now that we've reviewed how Oil's infrastructure uses the Unix philosophy, here are the detail of this release.

Code Changes

This section summarizes thefull changelog, starting with external contributors:

Travis A. Everett fixedspec tests to make them more portable, e.g. to aNix environment.

• As of this release, we continuously run tests in Nix and publish results to travis-ci.oilshell.org . If you're interested in using Nix to work on Oil, let us know! Not all tests pass yet.

Koichi Murase, author ofble.sh, made many changes necessary to run it.

• Implement declare/readonly/export -p to print variables in a form that eval understands.ble.sh uses this idiom to save and restore state.
• Implement several constructs in the printf language, including %()T for timestamps and %*s and %.*s for user-defined width and precision.
• Implement most remaining file descriptor operations.
  • 3>&- to close a descriptor and 3>&1- to move a descriptor. (These features are useful for compatibility, but Avoid Directly Manipulating File Descriptors in Shell in general.)
  • {varname}>out to have the shell choose a file descriptor and store it in a variable. This needs a better syntax in Oil .
• Fixed file descriptor leaks and added tests.
• Fix the semantics of slicing the arguments array like ${@:i:j} . Surprisingly, ${@:0} gives you more items than ${@} has!
• BASH_LINENO / FUNCNAME can be referenced as scalars or arrays, like BASH_SOURCE .
• Fix a bug evaluating the ${!prefix@} construct.
• Add failing spec tests for read -d , mapfile / readarray , and dynamic RHS in arithmetic.

In addition to manyother changes, I made the following changes related toble.sh:

• Implement read -d . (It's useful to have failing spec tests before implementing a feature. I merge changes that only include failing tests because they clarify what Oil should do.)
• Implemented test -v and test-k .
• Enforced an upper limit on hard-coded file descriptors. The argument to F_DUPFD ensures that a shell's own descriptors (e.g. to implement source ) don't collide with the program's descriptors.
• Refactored the file descriptor handling code. This code may be rewritten in C++ rather than translated because it's mostly I/O.
• Added a special case for declare -A dict=() , which is now documented inQuirks.
• Added location info to the error message for $(( a )) when a doesn't look like an integer.
• Fixed eval and trap to accept -- .
• Turned on set -o emacs in interactive mode like bash does.
• Hid a strict syntax check under shopt -s parse_ignored for bash compatibility.

Other:

• Implement printf %b , which allows the argument to use backslash escapes like echo -e .
• Added failing spec tests for a word splitting bug found by GammaFn .
• Many changes to our Travis CI configuration and to "Toil".
• Changes to run spec tests in an unknown release environment as opposed to a known dev environment. For example, we want to test the Oil release binary onAlpine Linux and OS X.

Documentation

The docs are still in their early stages, but I've updated a few of them:

• Known Differences
• Quirks for compatibility. OSH has generally maintained clean semantics while staying compatible, but sometimes they can't be clean.
• I consider these two docs "missing" documentation frombash and other shells (still in draft state):
  • Interpreter State. How does "shell memory" work?
  • Process Model. When are processes started? Of course, the entire shell memory is copied on fork() .
• Architecture Notes
  • Parser Architecture was split into its own doc.

On the Wiki:

• Language Design Principles . These principles guide decisions about the evolvingOil language and help me explain it to others. I also added a section on protocol design principles, which will be important in the near future.

Zulip Threads

Reminder: There are threads on oilshell.zulipchat.com about almost every topic in this post, and more:

• Research on Interactive Shells . I'd like more people with experience to help us figure out what to do and how to implement it.
• ble.sh
• Language Design Principles
• The Unix Philosophy

What's Next?

• More work translating Oil to C++ . I want to tackle osh/cmd_exec.py next.
• Translation will also involve rewriting the Oil expression evaluator so that it doesn't reuseCPython data structures. Instead it will use the structures defined in core/runtime.asdl . This is the "model" I want to document inInterpreter State.
• More changes to runble.sh. It's been useful in flushing out the long tail ofbash language features.
• Continue to discuss and design APIs to allow users to write their own line editor / interactive interface .

If you're abash user who wants afish-like experience, please tryble.sh and let me know what you think. It took me less than 20 seconds to install and try. The instructions are at akinomyoga/ble.sh .

Appendix A: Selected Release Metrics

Let's compare this release with the previous one, version0.8.pre2, which I reported detailed metrics onearlier this month.

Tests

The features and bug fixes for ble.sh caused many more spec tests to pass. And we have more failing spec tests to guide future work.

• OSH spec tests for 0.8.pre2 : 1583 tests, 1411 passing, 66 failing
• OSH spec tests for 0.8.pre3 : 1647 tests, 1456 passing, 86 failing

A few more Oil spec tests fail because I made thetest framework stricter. If there's a Python traceback on stderr , the test fails no matter what. This surfaced a few bugs which I quickly fixed, but others remain.

• Oil spec tests for 0.8.pre2 : 247 tests, 232 passing, 15 failing
• Oil spec tests for 0.8.pre3 : 250 tests, 231 passing, 19 failing

Lines of Code

I'm happy that all the new features fit in ~400 new significant lines of code:

• cloc for 0.8.pre2 : 15,272 lines of Python and C, 293 lines of ASDL
• cloc for 0.8.pre3 : 15,633 lines of Python and C, 300 lines of ASDL

... and ~600 new physical lines of code:

• src for 0.8.pre2 : 28,407 lines of Python
• src for 0.8.pre3 : 29,009 lines of Python

Benchmarks

The translation to C++ still works, but it didn't change in this release. There was no meaningful change in parser performance:

• Parser Performance for 0.8.pre2 : 242 lines/ms and 778 lines/ms
• Parser Performance for 0.8.pre3 : 277 lines/ms and 776 lines/ms

Or the size of the binary:

• ovm-build for 0.8.pre2 : osh_eval is 663,792 bytes of native code (under GCC)
• ovm-build for 0.8.pre3 : osh_eval is 667,888 bytes of native code (under GCC)

Appendix B: Closed Issues

#674 enforce upper limit on hard-coded file descriptors #664 error message for $(( a )) doesn't have location #662 trap doesn't accept / ignore -- #661 declare -A dict=() not allowed in Oil #659 set -o vi and emacs should be have differently while interactive #650 Implement printf %*s and %.*s #637 'type' says control flow like 'return' is a builtin, but it's not in Oil #620 disallowing redirects on control flow too strict by default (Run ble.sh) #537 Implement `test -v` and `test -k` #357 implement printf %b to do backslash escaping #356 Implement read -d and other options