Why Racket? Why Lisp?
source link: https://beautifulracket.com/appendix/why-racket-why-lisp.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Beautiful Racket / appendix
Why Racket? Why Lisp?
Beautiful Racket and Practical Typography were made possible by a publishing system called Pollen. I created Pollen with the Racket programming language. Racket is a descendant of Scheme, which in turn is a descendant of Lisp.
So while Racket is not Lisp (in the specific Common Lisp sense), it is a Lisp (in the familial sense). Its core ideas—and core virtues—are shared with Lisp. So talking about Racket means talking about Lisp.
In practical programming projects, Lisps are rare. Racket especially so. Thus, before I originally embarked on my Lisp adventure, I wanted to understand the costs & benefits of using a Lisp. Why do Lisps have such a great reputation, yet few users? Was I seeing something everyone else missed? Or did they know something I didn’t? To find out, I read whatever I could find about Lisps, including Paul Graham’s Hackers & Painters and Peter Seibel’s Practical Common Lisp. (OK, parts. It’s a big book.)
What I found was plenty of Lisp flattery from expert Lisp programmers. (Also plenty of Lisp kvetchery from its detractors.) What I didn’t find were simple, persuasive arguments in its favor. So here’s why Racket was the right tool for me, and what I see as the practical virtues of Lisps in general.
I didn’t study computer science in college (though I was a math major for two years, before switching to design). I’ve never held an official job as a programmer. Rather, programming has been a secondary skill I’ve used in my work as a web designer, type designer, and writer.
But in the last few years, I’ve spent an increasing amount of my time programming. This programming generates income. So by the simplest definition—does the skill make you money?—I suppose I qualify as a professional programmer. And since most of my programming efforts are in Racket, I qualify as a professional Racket programmer.
Mind you, I’m not claiming that I’m an expert programmer. Among the Racket community, which is laden with computer-science PhDs & professors, I (have no choice but to) embrace my relative inexperience. Hence the title of my talk at RacketCon 2014: Like a Blind Squirrel in a Ferrari.
Yet despite my limitations as a programmer, with Racket I’ve been able to render bigger ideas into programs more quickly, and with fewer bugs, than any language I’ve used before (and there have been many—BASIC, C, C++, Perl, Java, JavaScript, Python, and others). Since I haven’t gotten a brain transplant recently, there must be something special about Racket as a language.
Lisp is a language most programmers have heard of, for two reasons. First, it’s one of the oldest computer languages, in use since 1958. Second, it’s accrued a reputation as a language for brainiacs. Originally this reputation arose from its association with the field of artificial intelligence. Since then, this reputation has been maintained by periodic endorsements from respected programmers (latterly, Eric Raymond and Paul Graham) and the enduring fame of the textbook used in introductory computer-science courses at MIT, Structure and Interpretation of Computer Programs (which uses Scheme, and that one I did read start to finish).
But as mainstream programming tools, Lisp and its descendants have been largely ignored. Popularity of programming languages is tricky to measure, but here’s a simple proxy—let’s count the number of projects currently hosted on GitHub. One could quibble about the accuracy of this method, except that the results aren’t even close:
| Language | GitHub projects |
|---|---|
| JavaScript | 7,488,087 |
| Java | 6,570,575 |
| Python | 4,017,958 |
| PHP | 2,123,489 |
| Ruby | 1,699,590 |
| Clojure | 64,239 |
| Lisp | 17,989 |
| Scheme | 12,412 |
| Racket | 11,725 |
The last four languages are Lisps, and together account for only 106,365 projects. Racket itself only accounts for a small fraction of this small fraction.
Popular programming languages aren’t necessarily good—look what’s at the top of that list. + [JavaScript] has a lot of stupid in it … The good parts of [JavaScript] go back to Scheme and Self.
—Brendan Eich, here and here But unpopular languages often have fatal flaws that prevent wider adoption. As I was considering languages, Racket had a lot to recommend it. But was there a fatal flaw I was overlooking? And by committing to a Lisp, would I be painting myself into a corner? I wanted to understand the risks and benefits.
I said above that Lisp flattery is easy to find. The problem with Lisp flattery is that it makes sense only to experienced Lisp programmers. To others—especially those who are trying to decide whether to learn and use a Lisp—it just comes across as unsubstantiated hoodoo.
For example, in his essay How to Become a Hacker, Eric Raymond says “Lisp is worth learning for … the profound enlightenment experience you will have when you finally get it. That experience will make you a better programmer for the rest of your days, even if you never actually use Lisp itself a lot.” Unfortunately Raymond doesn’t follow up this claim by describing the “enlightenment experience”, nor why it’s “profound”, nor how it will improve your programming skills generally.
To be fair, Raymond’s essay is not focused on Lisp. But compare Beating the Averages, by Paul Graham, which is. Graham starts off by citing Raymond’s compliment to Lisp and seems ready to make the claim concrete.
Instead, he breaks it into smaller chunks of flattery. “We knew Lisp was a really good language for writing software quickly.” Because of what characteristics? He doesn’t say, but then describes Lisp as his “secret weapon”. OK, so what’s the secret? He says “programming languages vary in power”. Fine, but what exactly makes Lisp more powerful?
Graham offers one concrete example: Lisp’s macro facility, which he describes as its ability to make “programs that write programs”. After four years using a Lisp language, I’d agree with Graham that macros are great when you need them. But for someone new to Lisp languages, they’re not necessarily a bread-and-butter benefit.
I was hopeful when I opened Peter Seibel’s Practical Common Lisp and saw that the introduction was subtitled “Why Lisp?” Yes, tell me! Seibel echoes Graham’s claim: “You’ll get more done, faster, using [Lisp] than you would using pretty much any other language.” OK, but how? Seibel wonders whether “I like Lisp because of some quirk in the way my brain is wired. It could even be genetic, since my dad has it too.” That’s not encouraging to those of us outside your family. Ultimately, he sums up the appeal of Lisp by describing it as “the programmable programming language”. But I’ve never used a programmable programming language. Why should I start?
And by the way, when do I get the speed and power you keep promising?
In short—what’s in it for me, now?
This is the fundamental question that Lisp advocates have to answer for new users. But more often, it’s sidestepped. I’m not picking on Raymond or Graham or Seibel. They’re excellent writers. As programmers, they’re way out of my league. As I learn more about Lisps, I return to these articles and they make more sense.
But these articles are also emblematic of a general weakness of messaging about Lisp. I say that not as a ranking member of the Lisp community, but rather as someone who spent a lot of time seeking an answer to that fundamental question. I never got it.
Seibel is passing the buck when he says that to understand the benefits of Lisp, “you’re going to have to learn some Lisp and see for yourself”. Sure, this method works—using Racket for a few months finally made the benefits of Lisp clear to me. But it also required an investment of about 100–200 hours. + For more on the perils of taxing reader patience, see why does typography matter in Practical Typography.
That’s asking too much. If Lisp languages are so great, then it should be possible to summarize their benefits in concise, practical terms. It should be possible to demonstrate the power of Lisp in one hour, not 100. If Lisp advocates refuse to do this, then we shouldn’t be surprised when these languages remain stuck near the bottom of the charts.
In a word, expressiveness: the measure of how easy it is to put your ideas into code. For instance, an expressive language like Racket lets you write the “Hello world” program like this:
"Hello world"
Whereas a less expressive language—I won’t name names—requires this:
public class HelloWorld {
public static void main(String[] args) {
System.out.println("Hello world");
}
}
Concision is valuable, but expressiveness also embodies other qualities: precision, readability, flexibility, potential for generalization.
Compared to other languages, Lisps are tremendously expressive. Like the overpowered Japanese motorcycle I once owned, they go where you want, very quickly, with a minimum of input. If you’ve ridden a motorcycle, then you know what I mean. If you haven’t, good news—Lisps are cheaper and safer.
Here’s my ranking of the language features that offered the most immediate value to me, when I was a programmer new to the Lisp world. For each, I’ve noted whether it’s a feature of Racket specifically, or Lisps generally.
Everything is an expression. [Lisps] Most programming languages are a combination of two distinct ingredients: expressions (things that are evaluated to produce a value) and statements (things that denote an action). For instance, in Python, x = 42 is a statement, and x + 42 is an expression.
Statements and expressions are distinct because while expressions can be naturally nested with each other, statements and expressions cannot. For instance, in Python, this is a valid expression:
42 + 101And so is this, substituting another expression for the right-hand value:
42 + (100 + 100)But this is not valid:
42 + if 1 < 0: 100 else: 200Why? Because in Python, a standard if conditional is a statement, and can only be used in certain positions.
By making everything an expression, however, Lisps remove this limitation. Since expressions are nestable, anything in the language can be combined with nearly anything else. For instance, because an if conditional is an expression, you can use it in place of a value:
“But wait! Python has a ternary conditional expression!” It doesn’t change the essential point, but OK—you can indeed write this:
42 + (100 if 1 < 0 else 200)But now suppose we want to use a conditional in place of the operator, not the right-hand value. In a Lisp, because everything is an expression—including the operator itself—this is easy:
But if you try the same thing in Python, it will raise a syntax error:
42 (+ if 1 < 0 else *) 100Why? Because Python operators are not expressions.
This is a synthetic example. The point is not that you’d necessarily want to do this, but that Lisps permit it. You don’t run into the syntactic guardrails that are lurking in other languages. As a programmer, this simplifies your work, because everything snaps together easily. It also expands your possibilities, because you can combine parts of the language in unusual ways if you feel like it.
It’s similar to the basic idea behind Legos. Other building sets offer specialized pieces that can only fit together certain ways. But by sharing uniform measurements, Lego bricks offer maximum possibilities for combinations. This ends up being more flexible & more fun.
So it is with an expression-based language. If you find this idea exciting, congratulations—you might be a Lisp programmer. (If you find this idea weird and scary, this is a good moment to bail out.)
Every expression is either a single value or a list. [Lisps] Single values are things like numbers and strings and hash tables. (In Lisps, they’re sometimes called atoms.) That part is no big deal.
The list part, however, is a big deal. In a language like Python, the list is one data type within the language. But in Lisps, the list is more like an organizing principle for everything that happens. + The name “Lisp” is an abbreviation for “list processing”. So yes, you can use the list as a data type. But a function call is also a list. In fact, the source code for the function is a list. Actually, the rest of the program is too. Lists are everywhere. + The fancy CS term for this property is homoiconicity.
The benefits of lists are similar to that of expressions. By bringing more of the language into a consistent form, more possibilities arise for how pieces can be combined and manipulated.
Seibel describes Lisp as a tool for getting “more done, faster”. Here, you can start to see why this is so. Lisp languages are immensely flexible and permissive in how their pieces can be connected. This means that the way you think about a programming problem can be quite close to the way you actually program it. (This is also why Lisps have traditionally excelled for prototypes and exploratory work.)
To be fair, getting the most out of a Lisp means learning to think more in the Lisp idiom of lists and expressions. For that reason, I agree with Seibel—trying it yourself is the best way to be convinced of the benefits. As you get a feel for lists and expressions, it does pay increasing dividends throughout the language. You see how tiny lines of code can produce epic amounts of work. You also start to appreciate that even in a well-designed language like Python, you’re spending a lot of time shaping your ideas to fit its limitations, like shaving an invisible yak.
Functional programming. [Lisps] Yes, I know that other languages offer functional-programming features, and that Lisps aren’t considered pure functional languages. But many programmers haven’t been exposed to this idiom, and thus tend to underrate its benefits. I know I was in that category.
Functional programming doesn’t mean programming with functions. Everybody does that. Functional programming refers to a stricter style where functions receive certain data as input, process only that data, and return a result. In functional programming, functions avoid two habits common in other languages: mutation (= changing data in-place rather than returning a value) and relying on state (= extra context that’s not provided as input, for instance global variables).
“Wait—I love state and mutation. Why would you take them away?” Because they’re false friends. They contradict the essential concept of a function, which is to encapsulate data and algorithms. When a function relies on state or mutation, it’s operating outside those boundaries. Therefore, you either take on an increasing housekeeping burden to keep track of how functions affect each other, or watch your program sink into a swamp of mysterious, complicated bugs.
Programming in a functional style takes more effort at the outset. But it encourages you to structure the program in a clean, compartmentalized way. This pays off immediately in programs that are easier to test and debug. It’s also more likely to lead to reusable components, since functions are truly independent.
This bite-the-bullet aspect of functional programming is another reason why you can get “more done, faster” with a Lisp. The difference between prototype and production code often ends up being small, because you don’t take as many shortcuts at the start. The program grows and evolves more smoothly because it’s easy to change one part without causing ripple effects elsewhere.
Macros. [Racket] Some Racketeers quibble with this term, preferring syntax transformers, because a Racket macro can be more sophisticated than the usual Common Lisp macro.
A macro in Common Lisp is a function that runs at compile time, accepting symbols as input and injecting them into a template to produce new code.
Macros in Racket, on the other hand, rely on the concept of hygiene. They can handle Common Lisp-style macros, but also more elaborate syntax rearrangements.
But forget that—what’s in it for you? As a programmer, you end up getting two bites at the apple every time you run a file: Racket runs the macros (which alter the source code), and then the source code itself.
Unlike something like the C preprocessor, which is basically a separate mini-language, Racket macros are themselves Racket functions that give you access to everything in Racket. Like lists and expressions, macros add another layer of expressive possibilities.
Create new programming languages. [Racket] When I first read that Racket could be used to create new languages, I had two thoughts—are they serious? and would I really want to do that? The answers were yes and oh hell yes.
Between expressions, lists, and macros, Racket gives you a huge amount of semantic flexibility. But on top of that, it also adds syntactic flexibility, in that you can define a reader that converts surface syntax into standard Racket S-expressions, and an expander that determines the meaning of these S-expressions. + Paul Graham’s programming language Arc, a dialect of Lisp, was built on top of Racket.
You can use this facility to make specialized dialects of Racket. Or implement earlier languages. Or create entirely new languages with their own rules. You can use any of these languages within DrRacket to code new projects. (These special languages are sometimes called domain-specific languages, or DSLs.) Scribble is a DSL based on Racket; Pollen is a set of DSLs based on Scribble.
If you’re like most programmers, you’ve never had a tool for making a new language, so you’ve not considered it a realistic approach to a problem. And you won’t need it all the time. But when you do, it is awesome, in both the new and old senses of that word.
(The sequel to this piece—Why language-oriented programming? Why Racket?—is devoted to this topic.)
Libraries & documentation. [Racket] This might not look like a competitive differentiator—doesn’t every programming language have libraries & documentation?
Yes, but probably not like this. As a consequence of being used in research settings for many years—Racket’s core development team is made of computer-science professors—Racket’s libraries & docs are more like a transmission from a highly evolved alien intelligence.
I won’t pretend to know what all this shit does. A lot of it is over my head. But I like that. Each week I use Racket, I end up exploring a new part of the library, and learning something new. As opposed to other languages that seem to kill brain cells on contact (= pretty much anything named *Script, I find).
This learning is only possible because of Racket’s truly outstanding documentation. It’s vast, thorough, precise, and approachable. See for yourself. + If you don’t like the typography and layout of the docs, blame me.
DrRacket. [Racket] Yes, I know how to use a command line. But Racket includes a cross-platform graphical IDE called DrRacket that’s pretty great. DrRacket lets you edit, run, and debug Racket source files (or any other language based on Racket—see item #9 on this list.)
No, it doesn’t have the Ginsu-level search-and-replace facilities of something like Sublime Text. But it does have helpful editing features optimized for Racket code (for instance, you can right-click on a symbol name and rename it throughout the file, or jump from a function to its documentation).
Moreover, the command line within DrRacket doesn’t just show plain text—it can show stacked fractions, drawings, math plots, and other unexpected guests. If your command line does all that, by all means keep using it.
X-expressions. [Racket] This choice is somewhat biased by my work with Racket, which mostly involves document processing and typesetting. But related topics arise in most web programming. An X-expression is a special native data structure that Lisps use to represent HTML and other XML-ish data.
Well, not “special” in a Lispy sense—keeping with the usual policy, an X-expression is just another list—but special in the sense that other programming languages don’t have it. Usually your choice is to represent HTML either as a string or as a full XML tree. A string is wrong because it doesn’t capture the structure of the HTML, as defined by its tags and attributes. An XML tree shows this structure, but conceals the sequential nature of the data elements, and is unwieldy to work with.
An X-expression ends up being an ideal hybrid between a string and a tree. Moreover, because it’s just another list-based expression in the language, you have a lot of options for processing it. Translating an X-expression to or from a text representation using angle brackets is trivial and fast. (Details.)
Given the close kinship between XML-ish data structures and Lisp languages, I have no explanation why, during the Internet era, they’ve not been paired more often. They’re like peanut butter and jelly.
Scribble. [Racket] Pollen wouldn’t have been possible without Scribble, so for me, this has been the stone-cold killer feature of Racket. But that won’t be true for everyone, so I’m moving it down the list. + Scribble was originally created to serve as Racket’s documentation language (a job it does well).
Scribble is a dialect of Racket that inverts the ordinary relationship of plain text and code: rather than embedding text strings within source, a Scribble document consists of code expressions embedded within plain text.
“So it’s like an HTML template language.” Yes, in the sense that a template language allows code to be embedded in text. But also no, because a template language is usually a pidgin version of a real programming language. Scribble, by contrast, lets you invoke any Racket code simply by adding a command character to the front. In keeping with the theme already established, this approach is both simpler (because there’s almost nothing new to learn) and more powerful (because you can invoke anything in Racket).
In its combination of text and code, Scribble has more kinship with LaTeX. While it doesn’t have the typesetting facilities of LaTeX, the programming facilities are much better.
Opportunities to participate. [Racket] In theory, open-source software projects create the opportunity for groups of developers to join together and make better things in collaboration than they could separately.
In practice, I’ve found that they sort into a bimodal distribution: over here, the underdocumented solo projects that sputter along fitfully (if at all); over there, the mature, popular projects that can be intimidating for new contributors.
As an open-source project, Racket is positioned at a happy medium. The core development team has been working together for years, and the commits remain fast & furious. But they’re friendly scientists, not Shire-dwelling egotists, and remain receptive to improvements across the whole system. If you have a better idea, they’ll listen; if you code it up to their standards and make a pull request, they’ll take it.
[2021 update: I no longer contribute to Racket due to abuse & bullying by the project leadership. Everyone in the broader Racket community, however, has always been helpful and kind.]
The point of this list has been to tell you about the positives. That doesn’t mean there aren’t negatives. The small pool of Racket programmers means that when you hit a pothole, it’s possible no one’s ever seen your problem (= the inverse of Linus’s Law). If I wanted to hire a Racket programmer, the options would be few.
Still, why shouldn’t I be enthusiastic? What I’ve been able to accomplish so far with Racket has been tremendously useful, educational, and fun—the most fun I’ve had in 25+ years of programming.
If you think I sound like a fanboy or cult member, I can live with that. But those are people whose enthusiasm is disproportionate to reality. Here, I’ve tried to stay out of the clouds (and the weeds) and explain the concrete, practical features that have made Racket such a pleasure in my own work.
As always, your mileage may vary. But if I persuade a few people to download Racket and try it, I’ll be happy. In fact, if you try it and don’t like it, I invite you to contact me, because I’m always curious to hear dissenting opinions.
I will end by taking on the big kahuna—
I won’t claim I’ve reached the top of the mountain. But I can tell you what the view looks like so far.
There’s a sense in which Lisp and its descendants are more than programming languages. They’re tools in the broader intellectual inquiry into the theory of computation. Lisp’s inventor, John McCarthy, originally considered Lisp a “way of describing computable functions much neater than the Turing machines”, adapting the notation of lambda calculus to do so. Racket, likewise, has grown out of scientific research and exploration.
The theory of computation is just one of many great scientific discoveries in the last 100 years. But I don’t get to use quantum mechanics or relativity or DNA sequencing in my daily work. When I’m programming, however, I’m using computation.
Racket, as a Lisp dialect, has many practical benefits. But it also opens a window onto a vast theoretical world that underlies everything we can do with programs. I’m not a brainiac computer scientist. But some days, through that window, I can start to see a bit of what they see—some math, some science, a lot of truth, and more than a little beauty and mystery.
Paul Graham calls Lisp a “secret weapon”. I would clarify: Lisp itself isn’t the secret weapon. Rather, you are—because a Lisp language offers you the chance to discover your potential as a programmer and a thinker, and thereby raise your expectations for what you can accomplish.
If that’s not a step toward enlightenment, I don’t know what is.
Racket’s killer app is language-oriented programming, which I discuss in the sequel to this piece—Why language-oriented programming? Why Racket?
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK