Why Language-Oriented Programming? Why Racket?



If you’re inter­ested in learn­ing more about lan­guage-ori­ented pro­gram­ming and Racket, con­sider Racket School 2019 , hap­pen­ing July 8–12 in Salt Lake City. Racket School offers two LOP classes: the three-day Beau­ti­ful Racket Work­shop (taught by me) and the five-day How to Design Lan­guages (taught by the Racket archi­tects).



This is a sequel to my 2014 piece Why Racket? Why Lisp? , which I wrote a year or so after I’d start­ing using Racket. As some­one new to Lisp lan­guages, I had trudged through bogs of Lisp flat­tery, never sure what to make of the often hand­wavy claims. For instance, that Lisp even­tu­ally induces “ pro­found enlight­en­ment ”. Sure—what­ever you say, bro.



I had a sim­pler ques­tion: what’s in it for me, now? My ear­lier piece tried to answer that ques­tion. I sum­ma­rized why some­one who hadn’t looked into a Lisp lan­guage—or Racket in par­tic­u­lar—might want to.



I included a list of the nine lan­guage fea­tures that were most valu­able to me as a Racket noob. Fea­ture #5 was “cre­ate new pro­gram­ming lan­guages”. This tech­nique also goes by the name lan­guage-ori­ented pro­gram­ming , or LOP .



In the years since, lan­guage-ori­ented pro­gram­ming has risen to become my favorite part of Racket. Along the way, I con­verted my enthu­si­asm into action. In addi­tion to mak­ing lan­guages, I wrote this online book— Beau­ti­ful Racket —that teaches LOP as a tech­nique, and Racket as a tool.



In my own work, one exam­ple is Pollen , a text-based lan­guage I wrote to make my online books Prac­ti­cal Typog­ra­phy and Beau­ti­ful Racket pos­si­ble.  +  Pollen is based on Racket’s doc­u­men­ta­tion lan­guage Scrib­ble , which han­dles most of the heavy lift­ing. In Pollen, the pre­vi­ous para­graph is pro­grammed like so:

#lang pollen

I included a list of the ◊link["why-racket-why-lisp.html#so-really-whats-in-it-for-me-now"]{nine language features} that were most valuable to me as a Racket noob. Feature #5 was "create new programming languages". This technique also goes by the name ◊em{language-oriented programming}, or ◊em{LOP}.



Another exam­ple is brag , a parser gen­er­a­tor (in the lex/yacc style) that takes a BNF gram­mar as its source code. A sim­ple exam­ple for the bf lan­guage:

#lang brag

bf-program : (bf-op | bf-loop)*
bf-op      : ">" | "<" | "+" | "-" | "." | ","
bf-loop    : "[" (bf-op | bf-loop)* "]"



Both these lan­guages are imple­mented in Racket, and can be run with the usual Racket inter­preter, or inside the Racket IDE (called Dr­Racket).





And yet. Though this book has started thou­sands of peo­ple on their LOP jour­ney, I some­times fear that I’ve fallen into the same quick­sand as the Lisp advo­cates I once crit­i­cized.



If LOP is so great, then you shouldn’t need to spend a few days work­ing through the tuto­ri­als in this book. Right? I should be able to explain it con­cisely, with min­i­mum hand­wav­ing. I should be able to answer two sim­ple ques­tions:

1. 

   What prob­lems are best suited to lan­guage-ori­ented pro­gram­ming?

2. 

   Why is Racket best suited for the task of mak­ing lan­guages?



The sec­ond ques­tion is easy. The first ques­tion isn’t. I’ve been asked it many times. I’ve often resorted to the answer made famous by Jus­tice Pot­ter Stew­art: you’ll know it when you see it. That answer is good enough for those already LOP-curi­ous. But not for those still on the side­lines who want more of a prac­ti­cal jus­ti­fi­ca­tion.



So here’s a bet­ter attempt. Bear in mind that I’m not a com­puter-sci­ence pro­fes­sor. I’m not going to be talk­ing about pro­gram­ming lan­guages the way they might. Rather, I use Racket and DSLs for prac­ti­cal pur­poses—my daily work depends on them. My goal is to frame the issue in a way that will be use­ful for other prac­ti­cal users. (If I haven’t, you’re wel­come to click in the left mar­gin and send me a com­ment.)



1. 

   Lan­guage-ori­ented pro­gram­ming is really an inter­face-design tech­nique. It’s unbeat­able for tasks that demand min­i­mum nota­tion while pre­serv­ing max­i­mum pre­ci­sion . Min­i­mum nota­tion = the only nota­tion is what you per­mit. There is noth­ing extra­ne­ous. Max­i­mum pre­ci­sion = the mean­ing of that nota­tion is exactly what you say. There is no scaf­fold­ing or boil­er­plate. LOP gets to the point like noth­ing else.

   

   (The impa­tient can jump ahead to thesecat­e­gories of tasksthat are likely to ben­e­fit from LOP.)

2. 

   Racket is ideal for LOP because of its macro sys­tem . Macros are indis­pens­able for mak­ing lan­guages because they make com­piler-style code trans­for­ma­tions easy. Racket’s macro sys­tem is bet­ter than any other.



About half the read­ers of this piece are now depart­ing to post anony­mous inter­net com­ments dis­put­ing the above claims. Before you leave, please know: I’m hedged either way. LOP and Racket have been an incred­i­ble force mul­ti­plier on my pro­gram­ming pro­duc­tiv­ity. I’m happy to share this knowl­edge with you, so that you too may ben­e­fit. But I’m equally happy for these tools to remain my secret weapons, so I can keep pro­duc­ing work that is more ambi­tious, more impres­sive, and more prof­itable than the other 99.9%.



The choice, how­ever, is yours.





I finally started to unravel the Big Ques­tions by con­sid­er­ing a metaque­s­tion: why does it seem hard to explain the ben­e­fits of lan­guage-ori­ented pro­gram­ming?



Per­haps because when we speak of pro­gram­ming lan­guages —or just lan­guages , short­hand that I’ll use inter­change­ably—the term is freighted with expec­ta­tions about what a lan­guage is and what it does. As long as we’re stand­ing inside that box, it’s more dif­fi­cult to see the value of lan­guage-ori­ented pro­gram­ming.



But if we zoom out and look at lan­guages as part of the larger cat­e­gory of human–com­puter inter­faces, then it becomes eas­ier to see the spe­cial ben­e­fits of LOP.



So let’s do that.



First, some ter­mi­nol­ogy. Lan­guage-ori­ented pro­gram­ming (aka LOP ) is the idea of solv­ing a pro­gram­ming prob­lem by mak­ing a new pro­gram­ming lan­guage, and then writ­ing a pro­gram with the new lan­guage. Often, these “lit­tle lan­guages” are known as domain-spe­cific lan­guages or DSLs .



As the name implies, a domain-spe­cific lan­guage is one tai­lored to the needs of a par­tic­u­lar set of prob­lems. For instance, Post­Script, SQL, make , reg­u­lar expres­sions, .htaccess , and HTML qual­ify as domain-spe­cific lan­guages. They don’t try to do every­thing. Rather, they focus on doing one thing well.



At the other end of the lan­guage spec­trum are what we’ll call gen­eral-pur­pose lan­guages . Out here we find the usual sus­pects—C, Pas­cal, Perl, Java, Python, Ruby, Racket, etc. Why aren’t these lan­guages DSLs? Because they hold them­selves out as being suit­able for a wide range of com­put­ing tasks.



In prac­tice, gen­eral-pur­pose lan­guages often have cer­tain jobs that they do bet­ter than oth­ers, depend­ing on the con­di­tions of their birth. For instance, C excels at sys­tems pro­gram­ming. Perl excels as a sysad­min script­ing lan­guage. Python excels as a lan­guage for begin­ners. Racket excels at LOP. In each case, because that’s what the lan­guage was designed to do.



The dis­tinc­tion between domain-spe­cific and gen­eral-pur­pose lan­guages is per­me­able. For instance, Ruby started as a gen­eral-pur­pose lan­guage, but became pop­u­lar largely as a web-appli­ca­tion DSL through its asso­ci­a­tion with Ruby on Rails. Going the other direc­tion, Java­Script was orig­i­nally a DSL for web-browser script­ing. Like a mutat­ing virus, it has since grown far beyond.





If all these things on the spec­trum between domain-spe­cific lan­guages and gen­eral-pur­pose lan­guages deserve to be called lan­guages , then what are the defin­ing fea­tures of a lan­guage?



I know what you’re think­ing: “Well, that’s where you’re wrong. HTML isn’t a lan­guage. It’s just markup. It can’t express algo­rithms.” Or maybe: “Reg­u­lar expres­sions aren’t a lan­guage. They can’t stand on their own. They’re just syn­tax within another lan­guage.”



I once felt that way too. But the closer I looked, the more these dis­tinc­tions seemed vaporous. Thus, my first major claim (of three): a pro­gram­ming lan­guage is inher­ently a medium of exchange—a nota­tion sys­tem mutu­ally intel­li­gi­ble to humans and com­put­ers .



“Nota­tion” con­notes that a lan­guage has syn­tax ; “intel­li­gi­ble” con­notes that a lan­guage has mean­ing (or seman­tics , to use the fancier word) attached to the syn­tax. This def­i­n­i­tion cov­ers all gen­eral-pur­pose pro­gram­ming lan­guages. And all DSLs. (But not every data stream—about which.)



(By the way, even though “pro­gram­ming” and “lan­guage” are words idiomat­i­cally used together, these lan­guages are not used strictly by humans to pro­gram com­put­ers. Some­times they’re used by com­put­ers to com­mu­ni­cate with us; +  For instance: S-expres­sions. some­times with each other.  +  For instance: XML, JSON, HTTP. Def­i­n­i­tion­ally, it seems wrong to exclude these pos­si­bil­i­ties. But in prac­tice, yes—what we’re usu­ally doing with a pro­gram­ming lan­guage is, you know, pro­gram­ming.)



Con­sider HTML. It’s a way of telling a com­puter—in par­tic­u­lar, a web browser—how to draw a web page. It’s a nota­tion sys­tem (= angle brack­ets, tags, attrib­utes, and so on) intel­li­gi­ble to the human and com­puter (= the charset meta defines the char­ac­ter encod­ing, p tag con­tains a para­graph, and so on).



Here’s a lit­tle HTML page:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>My web page</title>
  </head>
  <body>
    <p>Hello <strong>world</strong></p>
  </body>
<html>



Sup­pose you dis­pute that HTML is a pro­gram­ming lan­guage. Fine. We’ll out­put our page with Python. That’s a real pro­gram­ming lan­guage, right?

print "<!DOCTYPE html>"
print "<html>"
print "<head>"
print "<meta charset=\"UTF-8\">"
print "<title>My web page</title>"
print "</head>"
print "<body>"
print "<p>Hello <strong>world</strong></p>"
print "</body>"
print "<html>"



If Python is a pro­gram­ming lan­guage and HTML is not, then it must be true that this Python sam­ple is a pro­gram, and the HTML sam­ple is not.



Plainly, this dis­tinc­tion is tor­tured. Here, the Pythoniza­tion adds noth­ing but com­plex­ity and boil­er­plate. More piquantly, the only inter­est­ing seman­tic con­tent in the Python pro­gram—in terms of con­trol­ling the web browser—is what’s embed­ded in the HTML. +  Arguably, HTML tags—like DOCTYPE and meta and strong —are noth­ing more than func­tions that take argu­ments. My DSL Pollen builds on this cor­re­spon­dence. Con­sis­tency com­pels us to con­clude that HTML, though sim­pler and less flex­i­ble than Python, is nev­er­the­less a pro­gram­ming lan­guage too.





Our HTML exam­ple was con­trived. But this pat­tern—a DSL nested within another lan­guage—is per­va­sive. Lan­guages used in this way are called embed­ded lan­guages . They rep­re­sent the most com­mon form of lan­guage-ori­ented pro­gram­ming. As a pro­gram­mer, you’ve relied on LOP for years, even if you didn’t know its name.



For instance, reg­u­lar expres­sions. +  Other exam­ples: printf for­mat­ting strings, CLDR date/time pat­terns, SQL. We may not think of reg­u­lar-expres­sion nota­tion as an inde­pen­dent lan­guage. But every pro­gram­mer knows what this means:

^fo+(bar)*$



More­over, you can prob­a­bly type this reg­u­lar expres­sion into your favorite pro­gram­ming lan­guage and it will just work. This con­sis­tent behav­ior is only pos­si­ble because reg­u­lar-expres­sion nota­tion is an embed­ded lan­guage, exter­nally defined by POSIX .



As with HTML, we could write the equiv­a­lent string-match­ing com­pu­ta­tion in the nota­tion of the host lan­guage. For instance, Racket sup­ports Scheme Reg­u­lar Expres­sions (SREs), which are reg­u­lar expres­sions that use S-expres­sion nota­tion. The above pat­tern would be writ­ten like so:

(seq bos "f" (+ "o") (* (submatch "bar")) eos)



But Racket pro­gram­mers rarely use SREs. They’re too long and too hard to remem­ber.



Another ubiq­ui­tous exam­ple of an embed­ded DSL: math expres­sions. Every pro­gram­mer knows what this means:

(1 + 2) * (3 / 4) - 5



On their own, math expres­sions don’t make inter­est­ing pro­grams. We need to com­bine them with other lan­guage con­structs. But as with reg­u­lar expres­sions, that’s an ergonomic and prac­ti­cal con­sid­er­a­tion. Math expres­sions have their own nota­tion and mean­ing, intel­li­gi­ble by both humans and com­put­ers, and thus qual­ify as a sep­a­rate embed­ded lan­guage.



Indeed I am. I’m claim­ing that HTML (and reg­u­lar expres­sions and math expres­sions) qual­ify as rudi­men­tary pro­gram­ming lan­guages. This nec­es­sar­ily implies that writ­ing HTML (or reg­u­lar expres­sions or math expres­sions) qual­i­fies as rudi­men­tary pro­gram­ming.



Please—don’t panic. We can agree that some­one who billed them­selves as a “pro­gram­mer” on Linked­In while only know­ing HTML and arith­metic would be con­sid­ered delu­sional. +  And next week, will prob­a­bly have a $180K soft­ware-engi­neer­ing job in Menlo Park. But that spills into a sep­a­rate issue about what the des­ig­na­tion “pro­gram­mer” typ­i­cally denotes in the job mar­ket. That’s not our con­cern here.



If this def­i­n­i­tion of pro­gram­ming lan­guages still ran­kles, maybe it’s because you think a real pro­gram­ming lan­guage needs to be capa­ble of express­ing all pos­si­ble algo­rithms—that is, it must be Tur­ing com­plete .



I see why that might be part of the intu­ition. Every gen­eral-pur­pose pro­gram­ming lan­guage is Tur­ing com­plete.



The prob­lem? Tur­ing com­plete­ness is a low bar. It’s a tech­ni­cal mea­sure­ment that doesn’t tell us any­thing inter­est­ing about the lan­guage in the real world. +  “Beware of the Tur­ing tarpit in which every­thing is pos­si­ble but noth­ing of inter­est is easy.”—Alan Perlis For instance, reg­u­lar expres­sions aren’t Tur­ing com­plete, but are valu­able because they express a lot of com­pu­ta­tion with min­i­mal nota­tion. HTML is also not Tur­ing com­plete, but it’s a use­ful way to con­trol a browser. By con­trast, the bf lan­guage is Tur­ing com­plete, but even the most banal tasks require acres of impen­e­tra­ble code.



Does my def­i­n­i­tion of a pro­gram­ming lan­guage include every­thing? No.

• 

  Binary data for­mats don’t qual­ify as lan­guages. For instance, a jpeg file. Though a com­puter can under­stand it, a human can­not. Or a PDF: if you crack one open, you’ll find some parts that seem human read­able. But that’s inci­den­tal to how PDF works. There’s no sense in which a human is intended to notate ideas using PDF con­structs.

• 

  Plain text files are not lan­guages. Sup­pose we have a file con­tain­ing Homer’s Iliad . We humans can read and under­stand this file. Though a com­puter can triv­ially process the file—say, by print­ing its con­tents—the text within the file is unin­tel­li­gi­ble to the com­puter.

• 

  Graph­i­cal user inter­faces are not lan­guages. Yes, they’re nota­tion sys­tems (that rely on text and image). But they’re only intel­li­gi­ble to humans. Com­put­ers draw GUIs, but they’re not intel­li­gi­ble to the com­puter.  +  But see Racket’s 2d lan­guage , an embed­ded lan­guage that makes ASCII-art boxes intel­li­gi­ble.



Above, I described a pro­gram­ming lan­guage as a “medium of exchange” between humans and com­put­ers. In that way, lan­guages fit among the broader cat­e­gory of things we call inter­faces .



This brings me to my sec­ond major claim (of three): that lan­guage-ori­ented pro­gram­ming is fun­da­men­tally an inter­face-design tech­nique . If you like think­ing about inter­faces, you’ll love LOP. If you don’t, you may still love LOP, because it enables cer­tain inter­faces that are oth­er­wise unat­tain­able.



One of my favorite exam­ples of lan­guage as inter­face is brag , a parser-gen­er­a­tor lan­guage made with Racket. If you’ve ever used the lex/yacc tool­chain, you know the goal is often gen­er­ate a parser from a BNF gram­mar . For instance, the BNF gram­mar for the bf lan­guage looks like this:

bf-program : (bf-op | bf-loop)*
bf-op      : ">" | "<" | "+" | "-" | "." | ","
bf-loop    : "[" (bf-op | bf-loop)* "]"



To make a parser in a gen­eral-pur­pose lan­guage, we’d have to trans­late this gram­mar into a pile of native code. Even with the help of a parser gen­er­a­tor, it’s a tedious job. And point­less—haven’t we already notated the gram­mar? Why do it again?



With brag , how­ever, our wish comes true. To make a parser, we sim­ply add #lang brag to the file, which mag­i­cally con­verts our BNF gram­mar into brag source code:

#lang brag
bf-program : (bf-op | bf-loop)*
bf-op      : ">" | "<" | "+" | "-" | "." | ","
bf-loop    : "[" (bf-op | bf-loop)* "]"



That’s it! When com­piled, this file will export a func­tion called parse that imple­ments this BNF gram­mar.



This is one of my favorite exam­ples of LOP because it’s unas­sail­ably supe­rior to the long-winded alter­na­tive. More­over, with a gen­eral-pur­pose lan­guage, this kind of inter­face is essen­tially impos­si­ble.



But the lan­guage-ori­ented pro­gram­mer does it all the time.



This brings me to my third and final major claim, which is that among inter­faces, lan­guages have unique strengths . The cat­e­gories below are not exhaus­tive or exclu­sive, of course. But I’ve found that LOP has a lot to offer:

1. 

   When you want to cre­ate an inter­face usable by less skilled pro­gram­mers, or non­pro­gram­mers, or lazy pro­gram­mers (don’t under­es­ti­mate the size of that last cat­e­gory).

   

   For instance, Racket has an elab­o­rate web-appli­ca­tion library . But it’s also pos­si­ble to spin up a sim­ple web server quickly with the web-server/insta lan­guage:

   

   #lang web-server/insta

   (define (start request)

   (response/xexpr

   '(html (body "Hello LOP World"))))

   #lang web-server/insta
   (define (start request)
     (response/xexpr
      '(html (body "Hello LOP World"))))

   

   Matthew Flatt’s arti­cle Cre­at­ing Lan­guages in Racket demon­strates a lan­guage that gen­er­ates playable text adven­tures. As with brag , it looks more like a spec­i­fi­ca­tion than a pro­gram, but it works:

   

   #lang txtadv

   ===VERBS===

   north, n

   "go north"

   south, s

   "go south"

   get _, grab _, take _

   "get"

   ===THINGS===

   ---cactus---

   get

   "Ouch!"

   ===PLACES===

   ---desert---

   "You're in a desert. There is nothing for miles around."

   [cactus, key]

   north

   meadow

   south

   desert

   #lang txtadv
   
   ===VERBS===
   
   north, n
    "go north"
   
   south, s
    "go south"
   
   get _, grab _, take _
    "get"
   
   ===THINGS===
   
   ---cactus---
   get
     "Ouch!"
   
   
   ===PLACES===
   
   ---desert---
   "You're in a desert. There is nothing for miles around."
   [cactus, key]
   
   north
     meadow
   
   south
     desert

2. 

   When you want to sim­plify the nota­tion. Reg­u­lar expres­sions are one exam­ple. Another exam­ple is my DSL Pollen , a lan­guage for mak­ing online books (includ­ing this one). Pollen is like Racket, except that you start in plain-text mode, and use a spe­cial char­ac­ter to denote Racket com­mands, which are eval­u­ated and spliced into the con­tent.  +  Pollen is based on Racket’s doc­u­men­ta­tion lan­guage Scrib­ble , which han­dles most of the heavy lift­ing. So the first part of this para­graph is pro­grammed like so:

   

   #lang pollen

   When you want to simplify the notation. Regular expressions are one example.

   My DSL ◊link["https://pollenpub.com"]{Pollen} is a language for making online books.

   #lang pollen
   
   When you want to simplify the notation. Regular expressions are one example.
   My DSL ◊link["https://pollenpub.com"]{Pollen} is a language for making online books.

   

   Pollen takes care of insert­ing all the nec­es­sary tags and con­vert­ing it to infal­li­ble HTML. I get the ben­e­fits of man­ual markup (I still have total con­trol of what ends up in the page) but none of the costs (I never, say, omit a clos­ing tag).

   

   Another exam­ple of sim­pli­fied nota­tion is lindenmayer , a lan­guage for gen­er­at­ing and draw­ing frac­tals called Lin­den­mayer sys­tems , like this one:

   

   

   

   In ordi­nary Racket, a Lin­den­mayer pro­gram might look like this:

   

   #lang racket/base

   (require lindenmayer/simple/compile)

   (define (finish val) (newline))

   (define (A value) (display 'A))

   (define (B value) (display 'B))

   (lindenmayer-system

   (void)

   finish

   3

   (A)

   (A -> A B)

   (B -> A))

   #lang racket/base
   (require lindenmayer/simple/compile)
   (define (finish val) (newline))
   (define (A value) (display 'A))
   (define (B value) (display 'B))
   (lindenmayer-system
    (void)
    finish
    3
    (A)
    (A -> A B)
    (B -> A))

   

   But one can use the sim­pli­fied nota­tion merely by chang­ing the #lang des­ig­na­tion at the top of the file:

   

   #lang lindenmayer/simple

   ## axiom ##

   A

   ## rules ##

   A -> AB

   B -> A

   ## variables ##

   n=3

   #lang lindenmayer/simple
   ## axiom ##
   A
   ## rules ##
   A -> AB
   B -> A
   ## variables ##
   n=3

   

   The lan­guage pre­sumes you already know some­thing about Lin­den­mayer sys­tems. But the sim­pli­fied nota­tion makes it easy to trans­late what you know into a pro­gram that does what you want.

3. 

   When you want to build on exist­ing nota­tion. Above, we saw brag , a DSL that uses a BNF gram­mar as source code.

   

   #lang brag

   bf-program : (bf-op | bf-loop)*

   bf-op : ">" | "<" | "+" | "-" | "." | ","

   bf-loop : "[" (bf-op | bf-loop)* "]"

   #lang brag
   bf-program : (bf-op | bf-loop)*
   bf-op      : ">" | "<" | "+" | "-" | "." | ","
   bf-loop    : "[" (bf-op | bf-loop)* "]"

   

   Another exam­ple. Early on, peo­ple who tried Pollen said “that’s cool dude, but I pre­fer Mark­down.” Wish granted— pollen/markdown is a Pollen dialect that offers the seman­tics of Pollen but accepts ordi­nary Mark­down nota­tion:

   

   #lang pollen/markdown

   When you want to simplify the notation. Regular expressions are one example.

   My DSL (Pollen)["https://pollenpub.com"] is a language for making online books .

   #lang pollen/markdown
   
   When you want to simplify the notation. Regular expressions are one example.
   My DSL (Pollen)["https://pollenpub.com"] is a language for making online books .

   

   The nicest part? It took me about one hour to cre­ate this dialect by com­bin­ing a Mark­down parser with my exist­ing code.

4. 

   When you want to cre­ate an inter­me­di­ate tar­get for other lan­guages. JSON and YAML and S-expres­sions and XML are all exam­ples of data for­mats meant to be machine-writable and -read­able.

   

   In Beau­ti­ful Racket , one tuto­r­ial lan­guage is called jsonic . It lets us embed Racket expres­sions in JSON, thereby mak­ing JSON pro­gram­ma­ble. So a source file like this:

   

   #lang jsonic

   // a line comment

   [

   @$ 'null $@,

   @$ (* 6 7) $@,

   @$ (= 2 (+ 1 1)) $@,

   @$ (list "array" "of" "strings") $@,

   @$ (hash 'key-1 'null

   'key-2 (even? 3)

   'key-3 (hash 'subkey 21)) $@

   ]

   #lang jsonic
   // a line comment
   [
     @$ 'null $@,
     @$ (* 6 7) $@,
     @$ (= 2 (+ 1 1)) $@,
     @$ (list "array" "of" "strings") $@,
     @$ (hash 'key-1 'null
              'key-2 (even? 3)
              'key-3 (hash 'subkey 21)) $@
   ]

   

   Com­piles to an ordi­nary JSON result:

   

   [

   null,

   42,

   true,

   ["array","of","strings"],

   {"key-1":null,"key-3":{"subkey":21},"key-2":false}

   ]

   [
     null,
     42,
     true,
     ["array","of","strings"],
     {"key-1":null,"key-3":{"subkey":21},"key-2":false}
   ]

5. 

   When the bulk of the pro­gram is con­fig­u­ra­tional. Dot­files, for instance, can be char­ac­ter­ized as DSLs. A more sophis­ti­cated exam­ple in Racket is Riposte by Jesse Alama, a lan­guage for test­ing JSON-based HTTP APIs:

   

   #lang riposte

   $productId := 41966

   $qty := 5

   $campaignId := 1

   $payload := {

   "product_id": $productId,

   "campaign_id": $campaignId,

   "qty": $qty

   }

   POST $payload cart/{uuid}/items responds with 200

   $itemId := /items/0/cart_item_id

   GET cart responds with 200

   #lang riposte
   
   $productId := 41966
   $qty := 5
   $campaignId := 1
   
   $payload := {
     "product_id": $productId,
     "campaign_id": $campaignId,
     "qty": $qty
   }
   
   POST $payload cart/{uuid}/items responds with 200
   
   $itemId := /items/0/cart_item_id
   
   GET cart responds with 200

   

   As a minia­ture script­ing lan­guage, Riposte is a lot smarter than the aver­age dot­file. It hides all the inter­me­di­ate code required for HTTP trans­ac­tions, and lets the lan­guage user focus on writ­ing tests. It’s still house­keep­ing. But at least you can focus on the house­keep­ing that you care about.

6. 

   When you’re teach­ing pro­gram­ming. For Beau­ti­ful Racket , I have read­ers use a lan­guage called br/quicklang that lets them make work­ing lan­guages in a few lines of code, and defers cer­tain details till later.

   

   But I got the idea from How to Design Pro­grams , a long­stand­ing project of the CS pro­fes­sors who lead Racket. The book comes with a set of teach­ing lan­guages that pro­gres­sively reveal more fea­tures of Racket. This forces stu­dents to solve exer­cises with more lim­ited tools at the out­set.





A com­mon cri­tique of LOP is “why make a domain-spe­cific lan­guage? Isn’t that more work than writ­ing a native library?”



Not if you have the right tool. Racket has been designed from the ground up to sup­port LOP. Imple­ment­ing a DSL in Racket is faster, cheaper, and eas­ier than the alter­na­tives. For instance, in the first tuto­r­ial in this book, I show how you can make a lan­guage in one hour —even if you’ve never used Racket.



Under the hood, every DSL in Racket is actu­ally a source-to-source com­piler that con­verts the nota­tion and seman­tics of the DSL into the equiv­a­lent Racket pro­gram. For this rea­son, a Racket DSL isn’t going to run as fast as one writ­ten in hand-tuned C. But it also makes all of Racket’s tool­ing and libraries acces­si­ble in every Racket DSL. So what you lose in per­for­mance, you get back many times over in con­ve­nience. And when DSLs are con­ve­nient and cheap, they become an option for a wider range of prob­lems.



Thus, to answer the cri­tique—no, a DSL is not nec­es­sar­ily more work than a native library. More­over, as we’ve already seen, as an inter­face, a lan­guage can do things a native library can­not.





Because Racket DSLs com­pile to Racket, a lan­guage-ori­ented pro­gram­mer using Racket needs to spend a cer­tain amount of time writ­ing syn­tax trans­form­ers that con­vert the DSL syn­tax into Racket. These syn­tax trans­form­ers are known as macros . Indeed, macros can be char­ac­ter­ized as exten­sions to the Racket com­piler.



Racket’s macro sys­tem is vast, ele­gant, and unde­ni­ably its crown jewel. But a lot of this book is about the joy of Racketmacros. That mate­r­ial is ready when you are. For now, the two stand­out fea­tures:

1. 

   Racket has a spe­cial­ized data struc­ture called a syn­tax object that is the medium of exchange for macros. Unlike a raw string con­tain­ing only code, a Racket syn­tax object pack­ages the code with meta­data like lex­i­cal con­text and source loca­tions , or arbi­trary fields called syn­tax prop­er­ties . This meta­data stays attached to the code dur­ing its var­i­ous trans­for­ma­tions. (Seesyn­tax objects for the details.)

2. 

   Racket macros are hygienic , which means that by default, the code pro­duced by a macro retains the lex­i­cal con­text from where the macro was defined. In prac­tice, this elim­i­nates a huge amount of the house­keep­ing that would ordi­nar­ily be required to make DSLs work. (Seehygiene for the details.)



Is it pos­si­ble to imple­ment a DSL in, say, Python? Sure. In fact, I wrote my first DSL in Python—one that I still use to help with my type-design work . But once was enough. Ever since, I’ve used Racket.



At this point, you may be hav­ing one of two reac­tions:

1. 

   “LOP seems inter­est­ing, but I don’t know what I’d do with it.” Sure—I wasn’t nec­es­sar­ily expect­ing to imme­di­ately recruit you into the LOP army. Rather, my goal has been to torque your think­ing in a pro­duc­tive way. You’ve now learned about a pre­vi­ously unfa­mil­iar tool. Today, you might not see a use for it. But some­day, you’ll con­front a prob­lem that exceeds the lim­its of your cur­rent favorite lan­guage. On that day, LOP will have its open­ing.

2. 

   “OK, you’ve con­vinced me, but there’s no way I can get LOP or Racket into my work­place.” Jesse Alama’s story of how he intro­duced his DSL Riposte is a great exam­ple of win­ning over col­leagues with LOP (empha­sis mine below):

   

   [One can try] to get per­mis­sion upfront to do some­thing in Racket. That’s less likely to suc­ceed, I think, than just mak­ing some­thing great and explain­ing its ben­e­fits ... In my case, that meant talk­ing with co-work­ers about their work and ask­ing: “How can we model the pro­posed change in the API and be sure that we’ve really suc­ceeded?” The implied answer being: “Write a Riposte script.” That’s the moment where it becomes clear that [the DSL] I made has real ben­e­fits. I don’t even “push” Racket. I just intro­duce the DSL and show them how it helps them .



At the end of Why Racket? Why Lisp? , I said that a Lisp lan­guage “offers you the chance to dis­cover your poten­tial as a pro­gram­mer and a thinker, and thereby raise your expec­ta­tions for what you can accom­plish.”



LOP offers a sim­i­lar oppor­tu­nity: to raise our expec­ta­tions for what pro­gram­ming lan­guages can do for us. Lan­guages are not black boxes. They are inter­faces that we can design. In so doing, we open up new pos­si­bil­i­ties for what we can accom­plish with pro­grams.



If you can find a bet­ter pro­gram­ming tech­nique, use it. Now that I have LOP & Racket, I’m never going back.



• 

  Beau­ti­ful Racket by me, Matthew But­t­er­ick. The online book that you’re cur­rently vis­it­ing. But you’re in the appen­dix. The main part of the book is a pro­gres­sive series of LOP tuto­ri­als, all rely­ing on Racket and its fan­tas­tic macro sys­tem.

• 

  Racket School 2019 . This sum­mer, Racket School offers two LOP classes: the three-day Beau­ti­ful Racket Work­shop (taught by me) and the five-day How to Design Lan­guages (taught by the Racket archi­tects).

• 

  Lan­guage-Ori­ented Pro­gram­ming in Racket: A Cul­tural Anthro­pol­ogy by Jesse Alama. An impos­ing title, but inside is a friendly and read­able set of inter­views that Jesse has con­ducted with numer­ous Rack­e­teers (me included), with con­trast­ing per­spec­tives on how LOP fits into our work.

• 

  Cre­at­ing Lan­guages in Racket by Matthew Flatt. Matthew is the lead archi­tect of Racket (and wrote thefore­word for this book). This brief arti­cle nicely demon­strates the increas­ing lev­els of sophis­ti­ca­tion in DSL design, using a clever game-cre­ation DSL.