This Old Lisp

This is the keynote talk that I gave at the 2018 European Lisp Symposium in Marbella, Spain.

Abstract

Lisp was invented 60 years ago. Coral Common Lisp, the ancestor of today's Clozure Common Lisp, was released over 30 years ago. Over this time, processor architectures and operating systems have come and gone, but Clozure CL (under various names and forms) has survived and is still with us today. Clozure CL, Common Lisp, and Lisp itself are the product of many intelligent and clever people. Indeed, we find ourselves saying, with Newton, “If I have seen a little farther than others, it is because I have stood on the shoulders of giants.” I will say a few words, looking down from the giant's shoulders, on the subject of Clozure CL, that old Lisp, including where it stands today, and how it might evolve in the future.

Well, good morning everyone. ⊕ I used an informal style of speech in the talk, and that is reproduced here in this transcription. I’m really pleased to be here with you today. I know sometimes people say that sort of thing just as a way of being polite, but I sincerely am extremely grateful to be here with you and it’s a real pleasure you be here with you at the 2018 European Lisp Symposium.

So my official name, like you can see up there, is R. Matthew Emerson, but in real life you can call me Matt or Matthew; I just use that for branding purposes.

Just as a little bit about me so you can decide whether I’m worth listening to or whether my opinions are worth anything, I have worked on the Lisp implementation Clozure CL (it was called OpenMCL before that) since about 2007. I used to be an employee of Clozure Associates, but now I work on it independently.

I was just thinking the other day when was my first exposure to Common Lisp and I think it must have been on the NeXT machine using the bundled Allegro CL with that, you know way back in the day. So that’s just sort of an interesting little side note.

So, the title of my talk is This Old Lisp, and you know, usually when you have to explain your joke, it’s sort of a sign that joke isn’t too good, so maybe my joke isn’t too good here. But…so I’ll explain it to you.

There’s an American TV show that’s been around for a pretty long time called This Old House. And, the point of This Old House is it was this home improvement show, right? So, what they would do is they would show how you would fix a boiler, or, you know, fix an old-style wall in an old house. And you have to understand also that, you know, in the United States, if something’s a hundred years, right, that’s really old. And if something’s 150 years old, why, it practically belongs in a museum.

But anyway, the deal with This Old House was that the show recognized that a lot of people liked these old houses because the old houses were good. They were sometimes constructed using techniques that we wouldn’t use today to construct a house. For instance, you know, the way they would make walls would be different, and the way they would wire it up would be different, or maybe the way they would install heating would be different. But the house was fundamentally good. You had good bones.

And so, my thought was, well, you know, Lisp is kind of like that, I think. You know, and Clozure CL is kind of like that. You know, it’s something that is a little strange maybe from the perspective of somebody coming into computing today. Some of the choices that Lisp and Common Lisp make may seem to be pretty peculiar. For instance, you know some of the calling conventions for some of the sequence functions are sort of weird sometimes, some of the names are weird, like unwind-protect, what the heck’s that, but even so, I think that Lisp has really solid bones. And like an old house, I think it’s worth preserving and using. So there we go. So I had to explain my joke. Sorry about that. Anyway.

⊕ From a tweet

They always say that you should use humor in your talk, so one day I was wasting time, you know, looking at Twitter, and I ran across this. And it sort of tickles me. I think it’s sort of funny.

Years ago, I used to be involved in literary studies. You had literature, and criticsm, and all that, so I can’t help but kind of put on my literary studies hat, and look at this and ask, well, what do they mean by this Lisp developer rising from the grave as an undead zombie? What’s that supposed to represent? Is it that nowadays Lisp is experiencing some sort of renaissance, so that you have to get Lisp developers out of the ground and out of the graveyard in order to meet the needs of the growing market? But I sort of think also, that another way to look at it, at least in the Lisp developer case, that it represents the language itself.

So, it’s hard to keep Lisp down. You try to say, well, you know, can’t it stay dead and buried in the ground where it kind of belongs, maybe? Why does this shambling, zombie horror keep rising from the tomb? And I think that there’s—and I’m sort of joking, and I don’t really think that—I don’t think that Lisp belongs in the ground. So don’t get hostile.

I really think that Lisp has some genius, some genius to it, that hasn’t been rendered obsolete yet. So, I think that our little zombie there recognizes that. You can’t keep it down; it’s too good of an idea.

So, just a few words about how Clozure CL came to be, and about its background.

We already mentioned that Lisp is 60 years old this year, which is sort of remarkable. I think that if you were to look at the Lisp of 1958 and the Common Lisp of today you could see clearly that they are related, and that those same ideas that we had in 1958 are still present in the Lisp of today.

By contrast, consider Fortran. Fortran’s a little bit older than Lisp; the first Fortran compiler came out in about 1957, I believe. But if you looked at modern Fortran—what are they up to now? ⊕ In fact, there is a Fortran 2008, and another revision called Fortran 2018 is in the works. Fortran 90, do they have Fortran 2000? Well, whatever, but if you looked at modern Fortran code and the Fortran code of 1957 they would be almost, well, they would be almost nothing alike.

So, Lisp started out on a really good foot, because the essential ideas have endured.

Well. Lisp spawned many descendants. One particularly prominent descendant, of course, is Common Lisp, which first was specified by Steele’s book in 1984. And around that same time, an intrepid band of hackers formed a little company called Coral Software. And Coral Common Lisp was their product that they managed to put together, and it came out in 1987, and if you can possibly believe it, they had a Common Lisp (it didn’t have CLOS, so it was CLTL1), but if you can possibly believe it, it ran on a 1MB Macintosh Plus, this incredibly weak hardware. So that was a real accomplishment. Hats off to those guys, I think they wrote a lot of 68000 assembly code.

I remember talking to my colleague Gary Byers one time about current CCL, and I said, “Hey, look, we waste a lot of space sometimes, in certain data representations, and if we made a little bit of effort we could make things a lot smaller.” And he said, “Listen, when I was working on old Coral Common Lisp and MCL after that, we worried about how many symbols were left over, maybe useless symbols in the heap image because we didn’t want the distribution to span extra floppies. I’ve done the space optimization game, got the T-shirt, not in a big rush to do it again.”

Well anyway, after that some stuff happened, and Apple ended up acquiring Coral Software because Apple at the time was using Lisp and Smalltalk and other dynamic languages to do their research. So that happened.

And then, as you might remember, Apple decided that the 68000 series was too slow, so they decided, hey, we’re going to switch to the PowerPC processor. So this was the cause of a lot of big upheaval, and during that time Apple decided that they didn’t want to be in the Lisp vendor business any more, so they transferred their Lisp to this company called Digitool.

At Digitool they ported MCL to the PowerPC and they released it as a product. That was around 1995 or so. And if anybody has ever had the chance to use MCL on a PowerPC Mac, it’s a really nice implementation. Really, really a good piece of work. A lot of clever design in there to get acceptable performance on fairly modest hardware.

Well, then some stuff happened, and I’m not really sure what it was because I wasn’t there, but Gary Byers ended up working at JPL for Ron Garret (who was known as Erann Gat at the time). Ron was interested in having Lisp on little embedded PowerPC thingies that were on robots or rovers or flight hardware. In order to do that, they got a source license for MCL and then Gary Byers worked and ported it to this VxWorks operating system, which is a realtime operating system that runs on these little embedded guys, and since Linux/PPC shared that same ABI, they kind of got that for free, too.

Now, obviously there wasn’t going to be any window system or graphics or editor on this. Iit was meant to run on these little embedded things, so it’s all like non-graphical bits of MCL.

Well word about this got out somehow, and people were interested in it, and somehow permission was obtained from Digitool to redistribute the results of that port under an open source license, this LLGPL, which is the LGPL with this little Lisp-specific preface. So, that was in about 2001, and that’s how OpenMCL came to be.

Since that time, there’s been a lot of hacking and a lot of work on [Open]MCL. A software consultancy was formed, called Clozure Associates (that’s Clozure spelled with a “z”) and sometimes there were Clozure clients who funded these improvements, and sometimes Clozure itself funded these improvements. They took [Open]MCL and ported it to a bunch of processor architectures, ported it to a whole bunch of different operating systems, just a whole lot of stuff.

Well, eventually, MCL kind of ran out of gas, and I’m going to kind of skip over talking about that a.) because it’s probably not that interesting and it’s sort of sad, and b.) we kind of got a little late start.

But Alice Hartley, who had been carrying the MCL torch almost single-handedly for many years, finally, basically threw in the towel, because it was just too hard to keep up with all of Apple’s changes. Apple said, “oh, well, this PowerPC, it hasn’t really got what it takes any more, we’re going to use Intel chips now.” So, MCL was looking at, “well, you know, what do we do? Do we retarget our compiler to x86? What do we do about the user interface? We’re using this old Carbon library, and now Apple hates that, and they want us to use this other thing. What do we do?”

So, I think they just decided to, they just couldn’t keep up. Apple just kind of got them by changing so much.

We had OpenMCL, which was the thing that was originally done at JPL. Then we had MCL, which was becoming open source, So we thought, well, you know, we can’t have this, because it’s going to cause tons of confusion. So, to make it less confusing, we’ll call OpenMCL “Clozure CL”, right, because nobody uses the name “Clozure” for anything, do they?

How many different ways can you possibly spell [the word] “Clozure”? ⊕ All these names are punning on the programming language term closure. So, “closure” with an “s”, “Clozure” with a “z”, and Rich Hickey’s Clojure with a “j” thing, how are you going to spell the next “closure” thing? I don’t know, it’s kind of crazy.

I remember being at a conference, I think it might have been in about 2008, and Rich Hickey was there and I went up to him and I said, “Hey, man, you know, Clozure CL with a ‘z’ was around a lot longer than your Lisp. You ought to change your name.” And, he didn’t really want to.

Let me talk a little bit about where Clozure CL stands today.

It supports a couple of processor architectures. Right now it’s the x86, 64- and 32-bit versions. It supports the 32-bit ARM. Support for PowerPC kind of fell away because the hardware that we had to keep the port going just sort of died after a while. And although I really liked having the port around because it kept us honest in multiple ways, it was just too much of a pain.

And it runs on a bunch of operating systems here.

It’s old. It’s one of these old Lisps. If you look in the source code, sometimes you’ll see comments like this from slisp reader2.lisp and not touched in 20 years. So, here’s parse-integer from the reader. Along with other implementations, Clozure CL borrows a lot of code from Spice Lisp (that’s what slisp is, Spice Lisp). And Spice Lisp was the ancestor of CMUCL.

And if you look at the fancy loop macro, or the implementation of the pretty-printer or of the format function and you compare that to code in say CMUCL you can see there’s a lot of commonality there. The old Lisp guys borrowed a lot of code from Spice Lisp. So, let’s pour one out for Spice Lisp. Thank you, Spice Lisp.

Another benefit of being old, is that when you’re making a Common Lisp implementation, there are just a ton of problems, a ton of subproblems, that you have to solve. If I had to make a Common Lisp implementation from scratch, I don’t know what I would do. I would be at it for a pretty long time. There’s things like printing and reading floating-point numbers which are an interesting problem in themselves. There’s papers about this problem. ⊕ Such as: Guy L. Steele, Jr. and Jon L. White. 1990. How to print floating-point numbers accurately. SIGPLAN Not. 25, 6 (June 1990), 112-126. Implementing this in a Common Lisp from scratch would be kind of a pain, so it’s nice to have the old Lisp behind you to do that.

There’s other things that you need, and even simple things like random number generation, how hard can it be? There’s all sort of trade-offs in that that need to be looked at. Do you want it to have good statistical properties, do you not care about that, do you just want it to be really fast, do you want to use it for cryptography, all sorts of different things.

One thing that I’ve always sort of wondered, and always had a hard time learning. I don’t know if I could have done anything better to know this, is, who uses the Lisp?

We distribute it on an FTP server or via a web server and people just download it. We’ve got a version of it on the Mac App Store. I don’t know who gets it; Apple doesn’t tell you.

So, just for jollies, I had this survey that I put out on Twitter and the mailing list and the Freenode channels just to see who might pipe up and tell me what they use to run CCL on. And since the survey kind of confirmed my preconceived expectations, I put a lot of stock in it. So I believe that it must be true since it confirmed what I already thought.

Just to give you an idea of the respondents, most used it on Linux and on the Mac, and the only thing that really surprised me was quite a number of people ran the Lisp on Windows. So, that’s great, I’m glad it’s working for you. The Windows port is probably the weakest port because the CCL hackers don’t really use Windows as an every-day system. That’s sort of unfortunate, so I think Windows is kind of getting the shaft here. In some sense it’s the CCL red-headed step-child. But that’s just how it is.

But, it seems to work pretty well, modulo a couple of annoying bugs, like sometimes when you try to quit the Lisp it hangs. But, you never quit the Lisp, right? Why quit Lisp?

And as far as processor architectures go, as you would expect, 64-bit x86, by far, the most commonly-used processor architecture. There’s a couple of people, a quarter of the respondents use the ARM, and then some still, holdouts on 32-bit x86.

The reason I asked that question is because I was thinking about giving 32-bit x86 the axe, and dropping support for it. But, as it’s still working, I don’t think it makes sense to shoot it in the head, so to speak. But, on the other hand, since the backends for the x86 platforms are kind of combined, it probably is going to make sense to separate those. Whether the work involved in doing that going to be worthwhile I don’t know. You don’t want continued evolution of the 64-bit backend to be hindered by, I guess the “legacy” 32-bit backend. I like the “legacy” 32-bit backend because I wrote it.

I think CCL has a number of different constituencies and their preferences and their requirements don’t necessarily align. I think there’s a certain constituency that uses CCL in this batch mode where they’ll run this big job, and submit it to this supercomputer clustery thing and then they’ll crank on stuff. So there’s sort of this batch-style processing.

And then I think there’s some just normal hackers who use it with Emacs and SLIME in the way you would expect. And I think there’s a substantial and interesting contingent who are interested in Lisp, but they’re not too interested in investing time learning Emacs and SLIME. I know, I know, some people are shocked, but it’s true. That’s one of the reasons I made the Mac App Store version of CCL where we have this kind of IDE that grew from example code into something much more than that now.

I think maybe MCL refugees or people who would have used MCL would like something like this.

Briefly I’m just going to talk about some technologies in CCL. I kind of avoided, I hope not incorrectly, getting too deep into the technical weeds about any particular thing. I just kind of wanted to give you an overview of some of this stuff. Normally I get a big charge out of talking about all these little bits, and data representation, and all this trickery, but I thought maybe for this talk that wouldn’t be so appropriate.

More with the cartoon thing. One of CCL’s hallmark features has always been that its compiler is pretty speedy. And that’s nice. If you’ve got a big project in, say, Xcode and you’re trying to compile Apple Swift, maybe the language is swift, or maybe something else good about it is swift, it’s swift in some other ways, but it’s not too swift when it comes to compiling.

And I think that’s really a surprisingly nice feature. With Lisp, it’s not so important maybe because you can compile things incrementally and save your image and so forth. You don’t maybe so often build the whole system. I guess that’s what I want to say about it.

The compiler doesn’t have a lot of brains; it doesn’t work all that hard. But its goal is to produce something reasonable pretty quickly, and as often as not, it’s able to produce something halfway decent, even good, pretty quickly.

One historical weakness has always been if you’re a number cruncher, CCL tends to be kind of bad about that. It would cons double-floats like crazy and just be a big mess. You can make some effort with some declarations and you can get some floating-point code that is halfway decent if not good. So, the compiler could definitely stand to work a little harder and it would still be fast.

One nice thing about it is that you can build the Lisp in under a minute. On my iMac at home it takes 30 seconds to rebuild the whole Lisp from scratch. That’s nice, so I guess in some ways, CCL is optimized for CCL hackers. The CCL hackers are often rebuilding the Lisp, so being able to do that in 30 seconds is pretty nice.

I think this happens with all sorts of software, but when new people come, and they start using CCL in a new software system, new issues arise. New bugs turn up, including performance bugs. More often than not, those performance bugs tend to be not endemic so much, as is, “well, nobody ever really complained about it before.” And we can often resolve those performance bugs without too much trouble. But it is true that CCL’s goal is to produce something reasonable quickly. It doesn’t do anything heroic.

We have native threads in CCL. This isn’t even very interesting; this is like table stakes for a general-purpose Lisp implementation. It has a couple of…some funny things happen, because Lisp wasn’t designed with multiple threads in mind, so for instance, with hash tables, what happens when you’re accessing a hash table from multiple threads? Do you have to do some synchronization somehow, or what? In CCL, hash tables happen to be thread-safe and they use this fancy-pants lock-free algorithm that does all this complicated stuff and tends to work pretty well for read-mostly hash tables. There’s keyword arguments to make different kinds of hash tables that might work better in certain circumstances.

What happens when you try to write to a stream from multiple threads? Something bad, maybe? So, one sort of weird thing that CCL does, is when you make a stream, it only works from the thread you made it in, by default. This is totally a performance hack, it’s semantically really gross, but the performance was such that they decided to do this. And again, there’s keyword arguments to the open function where you can make streams that will work across threads but then locking becomes involved and performance becomes an issue.

And then, what do you do with special variables with threads? We’ve had papers about this before, and CCL just uses the pretty much standard scheme of these thread-local shallow bindings. So, there’s a small number of instructions you have to execute to get whatever the current special binding is for the thread in question. Sorry if I’m getting down into the little technical details that are boring.

The other thing about CCL is it’s got this sort of interesting garbage collector. I don’t claim that any of it is really novel, it’s all in the literature, but the way CCL works is that it manages one contiguous range of address space; this single space. And when CCL starts up, it reserves this large address space from the operating system, like 512 GB large, so I’m talking large. And then it commits a certain amount of pages to be used by the Lisp. And the old objects, they get allocated on the left, and as allocation takes place, the objects to go the right. And when it’s time to do some garbage collection, the garbage collector scans the heap, marks all the live objects, and then it squishes them all together. So it just compacts them together. So all the objects on the right get scooted over towards the objects on the left and squeeze out all the little gaps.

So, the heap is temporally ordered, so the old objects are always on the left, and the young objects are always on the right. We do some tricks to exploit what’s called the generational hypothesis; you might have heard of this. The generational hypothesis is “most objects die young.” That’s basically what it is. So, what we do then, is we have some generations here, and we treat those just as though they were the big heap. So, we treat it as this was the base of the heap. You start gc-ing from here, you mark, and you squish. But you only use this teeny, teeny, teeny, little fraction of the heap, and that’s really fast. So, a millisecond or so, and boom, you’re done. And there some complications with that, as you may know. You have to keep track of the case when old objects are mutated to have references to new objects and we use a software write barrier to do that, but it works out pretty well.

So there it is. I’ve heard that the .NET gc is think mark/compact type; I don’t know. There’s this guy named Patrick Dussud who used to be a Lisp guy at TI, and he ended up on that at Microsoft. So, maybe that’s where he got that idea.

Some implications of this GC are that objects move around on you. So, since objects move around on you, you can’t really pass Lisp objects to foreign code, to C code, because C doesn’t expect that you’re going to move some object out from under it, and it’s going to get very unhappy. So, whenever you have to operate on Lisp data from foreign code, with very few exceptions, you’re looking at copying.

And another limitation is that the GC runs single-threaded. That’s just a question of difficult engineering. Either you can make the marking and the collection, you can maybe use multiple threads to do that. Or if you’re really ambitious, you can say, well, let’s invent a collector where the mutator threads and the collector threads can run at the same time. And that’s doable, but that takes some work.

The final CCL feature that I’d like to talk about is that it has this sort of cool foreign-function interface to call C code. I know that probably a lot of you prefer to use a library like CFFI to handle your foreign function needs, but I think CCL’s native FFI has a couple of cool features that I’d just like to mention here.

So, if you know a C function by name you can call it, using this sort of middle-level external-call function. So, basically, you can just dlsym some symbol, and you call it, specifying the return type in this case.

There’s also some notation for describing and accessing some foreign data. So, what I do here is there’s this form called rletz, and rlet stands for record-let, and again this is sort of the Macintosh heritage of CCL that shows through, because back in the Macintosh day, you used to use Pascal to program the applications, and Pascal would call its aggregates—it didn’t call them structures—it called them records. So sometimes you’ll see in CCL, there’ll be something like make-record, or rlet (for record-let). So, rletz just, we allocate an array of 10 double-floats on the stack, on the C stack, zero those guys, we use this paref operator to initialize some of our foreign data, and then we can just use external-call to call this BLAS routine to perform this operation.

And notice that we have to specify the types of the foreign arguments so that the foreign function interface code can do the appropriate marshaling and un-marshaling and conversion of the types.

That’s sort of a drag, so we use this thing called the interface translator, which parses the C header file. So here’s an example from cURL, curl.h, so here’s this declaration from curl.h that’s for this function curl_global_init. And then this program, which has been called ffigen, and it’s based on a patched gcc, though nowadays were working on a new one that’s based on libclang, and I want to give a shout-out here to a volunteer contributor, his name’s Bo Yao, and he worked on figuring out how libclang works, so thanks to him (if you’re out there, thank you very much). And it produces this s-expression representation of the C declaration.

So, with that in hand, there’s Lisp code that then reads that, and writes this little database that tells the Lisp, hey, this is the sort of types and sizes that these foreign functions need. And in that case we can use this little convenient reader macro #_ to call this foreign code. So we say #_curl_global_init and we can also look up the names of the constants. So there’s that header, constant from C, that we can refer to from Lisp.

So that is sort of convenient and cool. Maybe you don’t care that much, maybe you say, “bleagh, foreign code, who wants foreign code, we want code in Common Lisp”. And that’s certainly true; if you can have your code in Common Lisp you’re a lot better off usually, but sometimes you just have to.

A couple related projects: we use a version of Dietz’s ANSI test suite, which we made a copy of and we’ve since added CCL-specific tests to; that’s on GitHub.

We’ve got this documentation system that is sort of this homebrew thing; it’s called CCLDoc. It actually runs on SBCL too thanks to some patches that people have sent in.

The interesting thing about this, is the manual for CCL, I think it used to be written in texinfo. Nobody liked texinfo. So, somebody volunteered and said, hey, you know texinfo, that really blows, let’s use something better, so they picked DocBook.

Well, DocBook’s not better than texinfo, it’s horrible. And even though DocBook might be really flexible, and you can do all sorts of things with it, writing DocBook documentation totally stinks. And the CCL hackers didn’t do it. So, we’d add new features to CCL or do things, and say, you know, we really ought to add this to the manual, but that would involve editing DocBook so let’s just not.

So, we went round and round about this, and said, maybe we should just use LaTeX. No, LaTeX is almost as horrible as DocBook. And then, well, how about this program LyX, you know, LyX is all right, that produces pretty good output. That didn’t get much traction, so Gail Zacharias of Clozure finally got sick and tired of all of us whining, and refusing to do anything, so she cooked up this homebrew documentation system called CCLDoc.

And the nice thing about CCLDoc is it’s not too horrible, it uses this s-expression based representation of the document, and writing renderers for it is pretty easy.

So, all you have to do is, you’ve got this data structure in memory, and then you just follow this little protocol and then you can write an HTML renderer for it pretty easy, a TeX renderer for it pretty easy, Markdown renderer for it, whatever. So that’s sort of a neat thing.

And like I said, we’re working on this updated ffigen based on libclang. Because gcc-4.0, that’s pretty old, and it’s hard to maintain a set of patches for something that your system needs. The nice thing about libclang is they say “libclang is stable”, because otherwise if you’ve ever worked with LLVM, they break things all over the place, and every release changes and you don’t want to get on that treadmill if you can avoid it. So that’s why libclang comes into play.

So some people have wondered hey, you know, what’s the future of CCL?

One item is, well, we just kind of need to keep up. For instance, an example of keeping up: the upcoming FreeBSD 12 has some ABI breakage in it; it changes the ABI. For instance, the size of struct stat changes because they decided, well, you know, a 32-bit quantity that's too small for inode numbers; we need to have a 64-bit quantity for inode numbers.

So now, you know, we’ve got that database, right, that I talked to you about foreign-function interface so when we want to do #_stat from CCL it thinks, well, here’s the layout of struct stat, well in FreeBSD 12, it’s different. So you have to make those modifications and just keep up with that sort of stuff.

One other thing that we'd like to do is we recognize that the compiler hasn’t got a lot of brains, so there’s this experimental new register allocator that we’ve been working on that’s sort of opt-in on the development branch that tries a lot harder to keep values in registers rather than treating the stack frame locations as the canonical locations for variables.

We’re probably going to need to do a port to ARM64 pretty soon; 64-bit ARM. I like the Macintosh platform, so that’s an important platform for me, and it won’t surprise me at all if Apple comes out with an ARM64-based MacBook in the next couple of years, and I’d like to have CCL there.

And there’s just the usual bugs that come up.

One thing that I think would be really great is a modern macOS IDE. I guess it could be on another system too, but like I say, I like the Macintosh. Even though Apple is kind of a slave driver; they really make you keep up, and that’s sort of hard.

I’ve said for years that I think Emacs and SLIME are kind of a local optimum when it comes to Lisp IDEs and I’m always sort of nonplussed when I hear people who are really enthusiastic, and they say, “Emacs and SLIME, that’s the most wonderful IDE ever invented by man.” Well, it’s nice, I guess, but I think you’re overstating the case.

I use Emacs, and I’m sort of a SLIME caveman myself; don’t get me wrong, but I think that there’s opportunities to do better.

One thing that I’ve always been bothered by, although maybe it’s never bothered anybody else, is that due to the fact that the Lisp is named “Clozure CL”, I’ve always worried that people might have the impression that it’s Clozure’s (Clozure Associates, the consulting company); I’ve always worried that they think that it’s their product. Or that they “own” it, or that it’s their private playground somehow and that they let you play in it but they don’t really want you messing around with it.

That’s really never been the case. Even in the days when we ran our own Subversion server and used Trac, we were more than happy to give out commit access to anybody who demonstrated the least amount of competence.

All you have to is, “do you know up from down? OK, go ahead”. We got, really, very few takers.

One thing that we were able to do and that I was able to do while I was at Clozure is they were able to obtain the copyright for the sources from Digitool before Digitool disappeared.

And having done that, I suggested that we change the license of CCL from the LLGPL, which some people find a little bit maybe hard to understand or a little bit peculiar, or the cause of some worry, to Apache 2.0.

I thought Apache 2.0 was pretty easy to understand; it made it clear that contributions you made to CCL weren’t going to be exploited; you weren’t going to be working for free; nobody was going to exploit you; nobody was going to take your changes and give you the shaft in any way whatsoever, so I thought that might be useful.

As far as who is hacking on CCL, you might know (or you might not) that Gary Byers, who has worked on CCL since the old days, recently had to retire for health reasons.

And this really sucks because he is a great programmer, a great hacker, and if ever there was a 10x hacker—have you ever heard the term 10x hacker?

There are some programmers who are just 10 times better than just your average programmer. They’re the good ones, and you want those people, and if ever there was a 10x programmer, it’s Gary.

So, not having him in the project is a real loss; that is a total drag. So the project is left with me, and I’m not as good a hacker as Gary, that’s for sure. If he’s Batman at best I’m going to be Robin.

You can help CCL. It’s on GitHub now. I moved it to GitHub from Subversion. I thought maybe that I could benefit from GitHub’s network effects that way and maybe make it easier for people to contribute.

There’s a channel on Freenode that I hang out on; you’re welcome to come to; you’re welcome to participate on the mailing list; and if you just want to do cool stuff with CCL, that helps too.

When, again, I was wasting time on Twitter one day, as I sometimes do, and when the news broke that Gary was going to have to retire because of his health, I was at a full-time job where I was writing Clojure (with a “j”) of all things. And while I was there, that was taking up a lot of my energy.

You’re working full-time, you’re working on this other dialect, and I was really worried, because perhaps neurotically, I feel like I have a responsibility not only to Clozure CL, which I love, a software system which I really like, and I felt that I had a responsibility to its users.

So when I ran across this tweet, that sort of, you know, I said, “Yeah, I know how that feels. I’m going to have to call this guy up and find out who his therapist is, because maybe I’m going to have to go to him sometime.”

One nice thing about Clozure CL is that you can get help for it. If you’re using it in your business, Clozure Associates—Clozure Associates is the name of the company, Clozure CL is the name of the Lisp—don’t call the Lisp “Clozure” or “Clozure Lisp,” OK? It’s Clozure Common Lisp.

I said that five years ago, by the way, so that’s just a reminder.

So, Clozure can offer paid support for it, or you can hire me to do whatever you want to with Clozure CL. So, you can get help.

For some reason, I felt that I just really needed to say this: you know what I mean? I like Common Lisp, and I like Clozure CL and so many times I’ve been at Lisp conferences, and there’s so much hand-wringing and we have these panels and say “oh, what are we going to do about Lisp, it’s dying, we’re all, woe, it’s all weird and nobody likes it…”

What the heck? Let’s have a little optimism here. Common Lisp is a great programming language, Clozure Common Lisp is a great implementation.

So, the standard is old, yes, it is old; it’s probably never going to be revised. But, it still provides an awesome baseline level of functionality that you can rely on and that works across multiple implementations. And we have multiple Common Lisp implementations.

Even for a programming language like Common Lisp, which is relatively obscure, we must admit, there are multiple implementations to choose from. From specialized to general-purpose, there’s got to be 8, easy, right? A dozen? That’s sort of amazing. And out of all these implementations, I happen to like Clozure CL. Maybe you like some other implementation. But I think we can still be friends.

This is an excerpt from Peter Norvig’s book Paradigms of AI Programming, and he gave a talk at Lisp NYC. ⊕ Recording of that talk: As We May Program And I was watching that talk, and he said, You know what? Here are all these features that Lisp has, and all these features, except for the underlined features, have been adopted by other languages. So, even though Lisp hasn’t taken over the world yet, even so, he was saying, since its ideas had been adopted by other programming languages, Lisp had ultimately won in some way.

And I was watching that, and I was sort of bemused, and I said, this is sort of strange, that doesn’t really seem right to me. And especially given the importance of some of the features that have been left out. For instance, the whole Lisp macro situation is extremely powerful, and that’s the reason a lot of people, or at least some people use Lisp, because they exploit that. And, I just want to remind you that there’s a programming language you can use that has all these features: it’s called Common Lisp!

[Comment from an audience member] Peter actually made the point that they actually implemented before and after methods using decorators in Python and what he said was we’re still doing Lisp, we just call it Python, and there’s actually a lot of truth to that.

That may be.

One other thing that I like about Lisp very much is that interactivity is kind of built into the design of the language. The language assumes that you are going to be using it in an interactive way.

There’s these standardized functions and macros like trace, and break, and they only make sense if you’re using the language in an interactive way. And this is just sort of implied, and I think that the people who worked on the standard thought, “well, duh, of course you’re going to use it in an interactive way.”

You even have sort of crazy things, like this update-instance-for-redefined-class method, because, of course, you’re going create a whole bunch of CLOS instances, and, of course, they’re going to be floating around in your image, and of course you’re going to want to change them, and of course you’re going to want to be able to update them, and that sort of attitude I think is really awesome, and I just totally like that.

And for me, when I interact with Lisp, when I’m at the top level, and I’m typing at Lisp and I’m doing things and I’m recompiling my functions, I really feel like I’m talking to this spirit that’s inside the computer. I’m having this conversation.

And that was one of the coolest things about the early micros, right? You turned them on, you turned on your Atari 800 or whatever, and bing, you know, it said “Ready”. And you could type stuff, like “plot x, y” and you could do all sorts of stuff. You weren’t having to compile anything, you just did stuff.

I think that’s so wonderful. You talk to the spirit inside the computer.

For me, that works for me for some reason. Maybe there’s some problem that I’m trying to solve, and I interact with the Lisp, incrementally, and I do stuff. My one-time colleague Mikel Evins, he talked about this. ⊕ Mikel's insightful Reddit comment. You teach the computer how to do stuff. And that just works for me; I don’t know why. That just works for me.

It’s been said, there’s this sort of a joke running around that Lisp is a great programming language to use when you don’t know what you’re doing. And since I hardly ever know what I’m doing, Lisp is a great programming language for me.

On the other hand, there’s something to be said for knowing how to solve the problem. So, sometimes it’s easy to get into the, to be tempted to peck away, and to mess around with stuff, when really, you might want to think about what you’re doing a little while first. That’s kind of a good idea.

So, what do we say, then, to the notion that we’re stuck in the past? This is Edna Mode, she’s from the film The Incredibles. Mr. Incredible had come to visit her because he’d torn a hole in his super-suit. He says, “Edna, fix my super-suit”, and she says, “This is trash, I don’t want to do this any more.”

So, what do we say to that? Here’s a quotation from Hamming that I like. The essence of the quotation was that Newton would say, I stood on the shoulders of giants, and that’s what enabled me to see a little bit farther when it comes to him inventing the calculus and so forth. And Hamming’s point is that too often in the computing field we re-implement stuff and we don’t build on each other’s discoveries too much. We stand on each other’s toes.

I think there’s some truth to that. And I don’t think anybody really escapes from that.

Let’s laugh at the JavaScript programmers, OK? That’s always fun. Here’s this comic called Commit Strip, which I like to waste time on. Here’s our intrepid front-end JavaScript hacker, and she’s wondering, “do I pick one JavaScript framework and master it, or do I pick a new one every week just so I can keep up to date?” Because there are so many!

So, ha-ha, we all laugh. You JavaScript programmers are all so silly.

But, this is not a technical problem. This is a social problem. And back in the time before Common Lisp was standardized, ARPA called a meeting one time of all the Lisp hackers that it was supporting; all the research environments it was supporting, and it was getting concerned about this proliferation of Lisp dialects.

And they said, “Hey, what about this? It looks like you’re in a state of chaos.” And I think it was Scott Fahlman who came out and said, “Look, the MacLisp community (which was the MIT Lisp that didn’t have anything do to with the Macintosh), the MacLisp community is not in a state of chaos at all. We’re four well-defined groups going in four well-defined directions!”

Same thing.

Sometimes people ask me what I do, what I do for work. And what I try to do, I try to get away with something simple. “Oh, well, you know, I write software.” And hardly anybody is satisfied with that. So then, it’s pretty hard to say, “Oh, yes, I work on an implementation and runtime system and compiler for the programming language Common Lisp.” At which point they sort of just back away. So I’ve what I taken to saying is that I build tools. Programmers need tools to build their software, and I help build tools for those programmers. So, just like artists need brushes and canvases and paint, and sculptors need chisels and stone and hammers, I help make those things.

Here’s another quotation from Knuth, where he talks about how liking your tools is such an important component of enjoying your work. And I think this is really true.

There’s another proverb, “it’s a poor workman who blames his tools”. Have you ever heard of that? Well, I think my response to that is that if you’ve got crappy tools, you need to use those tools to make some good tools that you like.

So, finally, one of the coolest things about working on a tool, about working on Clozure CL, is seeing what people have done with it. That really is awesome, to see what people do.

This is a screenshot from the program Opusmodus which I helped work on. I made the user interface. What it really is, is that it’s a Lisp development environment specialized for musicians is pretty much what it is. But there’s a whole raft of functions that do all sorts of operations on music. You can process notes and generate stuff in weird ways.

There’s a couple of screenshots from that. I’m very pleased to have worked on that, and very proud of that.

It’s always nice to get a testimonial. There’s some people who use CCL to run the software called ACL2, ⊕ The ACL2 web page has anybody heard of ACL2? It’s a theorem-proving software. And because the ACL2 guys and the CCL guys talk to teach other, we’ve been able to benefit from each other’s expertise. We were able, in CCL, to add a feature called static conses. What a static cons is, you can cons up something, you make a cons cell, and it doesn’t ever change its address. And this is really helpful for them, because ACL2 stands for Applicative Common Lisp. So it’s all purely functional. So when they do that, that really helped them; it really made their software perform a lot better. That’s in the sort of the hardware verification space.

Well, let me wrap up here. Again, years ago, I used to work in the theatre. And one of my jobs that I had was, I was in a video that was going to be shown at a sales conference for rice salesmen, of all people. Rice, the grain.

In the video, I was a swashbuckling swordsman and I represented the forces of mediocrity. And in the video we were doing this sword fight and I was ultimately vanquished by another swordsman who represented some virtue: excellence or whatever it was, you know, diligence.

And they showed this video to the rice salesmen in an attempt to get them to think about how important and great it would be if they would meet their sales quotas for selling rice. Maybe it worked.

I wish, I wish, that I had a fancy hat with feathers, and a sword that I could use to vanquish the Lisp naysayers. You know what I mean? But I left all that stuff at home.

But what I can say is, I really like Common Lisp. I really like Lisp, a lot. And I want to tell you that it’s OK to like Lisp, and that it’s OK to write programs in Common Lisp. And I’m really looking forward here in the rest of the symposium to hearing about what you’ve done with Lisp, what you’re doing now with Lisp, what you’re going to do with Lisp in the future, and I’m interested to hear about what your dreams are for doing new stuff with Lisp.

Because Lisp is great.

Thanks very much. I appreciate your attention. Let’s go out there and hack some more Lisp. ❦