nREPL 0.6

nREPL 0.6 was released almost two months ago, but it never got the attention it deserved. At the time of the release I was enjoying some time off from development, and when I got back home I was swamped with work. Still, I had a created todo item to write a blog post detailing the important improvements in nREPL 0.6 and I’m finally doing this.

New Print Middleware

The most important user-visible change in this release is the replacement of the pr-values middleware with a new print middleware. pr-values , as it name implies, was the middleware that printed evaluation result values to strings, so they could be returned to the clients in an easy to consume manner.It was around since the early days of nREPL, and while it generally got the job done, it was also problematic as it had hardcoded the print function it was using and because it forced the value to be fully printed it could kill the server (by eating all its memory) or kill your client (most editors struggle with huge chunks of texts) if you happened to evaluate something huge by accident. We tried to fix the first problem in nREPL 0.5 by tweaking pr-values , but later Brandon Bloom pointed out some problems with the approach we had chosen at the time .

The new print middleware that succeeded pr-values solves all of those problems and more. This middleware is intended to provide generic printing functionality to other middlewares. By default its behaviour is entirely backwards-compatible with pr-values . Rather than being coupled to the specific key :value , the new middleware allows other ops to provide :nrepl.middleware.print/keys in responses.

Here’s a summary of the new middleware’s options are summarised below:

:nrepl.middleware.print/var
:nrepl.middleware.print/options
:nrepl.middleware.print/stream?
:nrepl.middleware.print/buffer-size
:nrepl.middleware.print/quota

Note:The options are namespaced for two reasons - to avoid conflicts with older middlewares and to highlight the fact that printing is Clojure-specific and not part of the language-agnosticnREPL protocol.

An example of non-streamed vs. streamed printing:

;; request
-> {:op :eval
    :code "(range 512)"}

;; responses
<- {:ns "user" :value "(0 1 2 3 ... 510 511)"}
<- {:status ["done"]}

-> {:op :eval
    :code "(range 512)"
    :nrepl.middleware.print/stream? 1}

<- {:value "(0 1 2 3 ... 282 2"}
<- {:value "83 284 ... 510 511)"}
<- {:ns "user"}
<- {:status ["done"]}

Clients have the choice of rendering each part of the result as they arrive, or using some buffering strategy (perhaps to prevent the output from being intermingled with that of *out* / *err* ).

An example of truncation with streaming enabled:

-> {:op :eval
    :code "(range 512)"
    :nrepl.middleware.print/stream? 1
    :nrepl.middleware.print/quota 8}

<- {:value "(0 1 2 3"}
<- {:status ["nrepl.middleware.print/truncated"]}
<- {:ns "user"}
<- {:status ["done"]}

and without streaming enabled:

-> {:op :eval
    :code "(range 512)"
    :nrepl.middleware.print/quota 8}

<- {:ns "user"
    :value "(0 1 2 3"
    :status "nrepl.middleware.print/truncated"
    :nrepl.middleware.print/truncated-keys ["value"]}
<- {:status ["done"]}

The implementation details behind this are quite interesting, but I’ll omit them here. I’ll only summarize the highlights:

• Big results start appearing much faster than before (provided you enable streaming).
• You can interrupt the printing of a big result (provided you enable streaming).
• You can set a hard limit for the size of a result.

More information about the new middleware is available in the user manual .

Special thanks to Michael Griffiths for tackling this and to Brandon Bloom, who pointed out the deficiencies in the approach we had taken in nREPL 0.5!

New Caught Middleware

nREPL 0.6 includes a caught middleware which provides a configurable hook for any java.lang.Throwable that should be conveyed interactively (generally by printing to \*err* ). Like the print middleware, any options may be provided in either requests or responses (the former taking precedence over the latter if any options are specified in both). The following options are supported:

• :nrepl.middleware.caught/caught : a fully-qualified symbol naming a var whose function to use to convey interactive errors. Must point to a function that takes a java.lang.Throwable as its sole argument. Defaults to clojure.main/repl-caught .

• :nrepl.middleware.caught/print? : if logical true, the printed value of any interactive errors will be returned in the response (otherwise they will be elided). Delegates to nrepl.middleware.print to perform the printing. Defaults to false.

{:op :eval
 :code "(/ 1 0)"
 :nrepl.middleware.caught/caught 'my.custom/print-stacktrace
 :nrepl.middleware.caught/print? true}

Note:Just like that print middleware, the caught middleware is also Clojure-specific and not considered part of nREPL’s generic protocol.

The functionality of the caught middleware is reusable by other middlewares. If a middleware descriptor’s :requires set contains #'nrepl.middleware.caught/wrap-caught , then it can expect:

• Any returned responses containing the key :nrepl.middleware.caught/throwable will have that key’s corresponding value passed to the hook.

• Any handled requests will contain the key :nrepl.middleware.caught/caught-fn , whose value is a function that can be called on a java.lang.Throwable to convey errors interactively.

That middleware is another fine piece of work by the awesome Michael Griffiths.

Single evaluation thread per session

Previously nREPL’s evaluation mechanism was based on a homemade agents implementation, which worked well, but would change the evaluation thread used by the REPL unexpectedly. We’ve simplified the evaluation logic by simply associating a permanent evaluation thread to each nREPL session. The thread never changes now unless you interrupt an evaluation in progress, which kills the thread and spawns a new one. The new implementation is much simpler and its behaviour is completely predictable. See the discussion here for more details.

Technically this change is breaking , as it touched a few public vars, but in practice it seemed that cider-nrepl was the only project making use of them, so no one really noticed any of this.

Special thanks to Christophe Grand for working on this!

Stop Evaluation After the First Read Error

Check out this:

user=> (comment {:a} (println "BOOM!"))
Syntax error reading source at (REPL:1:14).
Map literal must contain an even number of forms
BOOM!
nil
Syntax error reading source at (REPL:1:33).
Unmatched delimiter: )

Or this:

user=> (defn foo [] (println {:a}) (println "BOOM!"))
Syntax error reading source at (REPL:8:27).
Map literal must contain an even number of forms
Syntax error reading source at (REPL:8:28).
Unmatched delimiter: )
BOOM!
nil
Syntax error reading source at (REPL:8:47).
Unmatched delimiter: )

That’s pretty weird, right?

Observe how because of a syntactic error (odd number of elements in a map) the println form gets executed, even though it shouldn’t (since it’s in comment and defn form respectively).

Besides this potentially dangerous but rare behavior, there is also an much more common inconvenience when you have to debug such errors. What should be the result of evaluating a single form triggers multiple exceptions, all but one of which don’t make any sense. They obscure the original problem and make life harder, especially to the beginners. The only thing to do is to remember that Unmatched delimiter: ) means you probably have an odd-arg map somewhere.

That was the behavior in the standard Clojure REPL and nREPL until now. Starting with nREPL 0.6 you’d get only:

user=> (comment {:a} (println "BOOM!"))
Syntax error reading source at (REPL:1:14).
Map literal must contain an even number of forms
BOOM!

Notice the lack of the “fake” result and the second read error.

All the credit here goes to Alex Yakushev who implemented this improvement!

Grab Bag

As with every release there were also other small bug fixes and improvements here and there. I think there are two worth sharing here:

• nREPL has 0 runtime dependencies (again). I’ve reconsidered the decision to extract nrepl.bencode into a separate library and brought it back to nREPL itself. The separate library will continue to exist and will be kept in sync with the nrepl.bencode namespace in nREPL.
• We’ve exposed a public CLI API in the nrepl.cmdline namespace. The API is experimental and subject to changes, but I think some people might find it useful.

Check out the release notes for a complete list of changes in nREPL 0.6.

nREPL’s user manual was fully updated to reflect the changes in the latest version.

(next nREPL)

The plan for the next nREPL release is fairly simple:

• Provide a native EDN transport, as an alternative to the default bencode transport.
• Provide an injection mechanism for clients, so they can supply nREPL with resources and classes directly. See this ticket for details.

I’d love for us to also make some headway with respect to being able to upgrade a plain socket REPL to nREPL and with porting unrepl’s rich data printer .

Once those features are completed I think we’d be ready for long-awaited 1.0 release. More importantly - nREPL will reach an amazing level of flexibility and would serve you even better!

Epilogue

By the time I got to write this post nREPL 0.6 is already widely available, as it quickly made it’s way to Leiningen 2.9. It’s also merged in Boot’s master, but there’s no stable Boot release shipping nREPL 0.6 yet. I’m also quite pleased to report that 2 months after the release we got 0 reported bugs/regressions from the previous version. Good work, everyone!

CIDER users can already take full advantage of all the improvements if they upgrade to CIDER 0.21 (New York). I hope the other clients will quickly follow in its steps (if they haven’t done so already).

nREPL 0.6 is a massive step forward for nREPL and I couldn’t be more proud of it. That’s certainly the most important release we’ve had since the project left Clojure Contrib. I’ve had very few direct contributions to nREPL 0.6 (other than acting as project coordinator) and I’d like to thank once again all the great people who contributed their time and knowledge! Extra special thanks to Christophe Grand and Michael Griffiths. You rock! We wouldn’t have made it so far without you! This release is truly a testament to the power of open-source that’s all about you!

That’s all from me for now. I hope this long overdue post was useful to you. Keep hacking!