Fragments: Posts tagged 'lisp'

Making CLOS slot access less slow

2026-05-01T15:43:22Z

Access to slots in CLOS instances is often very slow. It’s probably not possible for it ever to be really fast, but the AMOP MOP does provide a way of making it, at least, less slow.

How slow is it?

Here are some benchmarks for accessing fields in objects of various kinds, using SBCL. All of these tests do something equivalent to

(defclass a ()
  ((i :initform 0 :type fixnum)))

(defclass a/no-fixnum ()
  ((i :initform 0)))

(defmethod svn ((a a) n)
  (declare (type fixnum n)
           (optimize speed (safety 0)))
  (dotimes (i n)
    (incf (the fixnum (slot-value a 'i)))))

(defmethod svn ((a a/no-fixnum) n)
  (declare (type fixnum n)
           (optimize speed (safety 0)))
  (dotimes (i n)
    (incf (the fixnum (slot-value a 'i)))))

They then call svn (or equivalent) with a large value of $n$, do that a number of times $m$ and then divide by $2 \times n \times m$ to get an average time per access (incf accesses the slot twice).

For SBCL 2.6.3.178-a190d9710 on ARM64 Apple M1, seconds per access:

raw fixnum increment $1.58\times 10^{-10}$, ratio $1.0$;
slot access with slot-value (slot type fixnum) $1.20\times 10^{-8}$, ratio $76$;
slot access with slot-value (no slot type) $1.22\times 10^{-8}$, ratio $77$;
slot access with slot-value (single slot-value-using-class method) $1.69\times 10^{-8}$, ratio $107$;
slot access using standard-instance-access $1.00\times 10^{-9}$, ratio $6.4$;
slot access, struct (slot type fixnum) $1.57\times 10^{-10}$, ratio $1.0$;
slot access, struct (no type) $1.58\times 10^{-10}$, ratio $1.0$;
slot access, cons (car) $1.59\times 10^{-10}$, ratio $1.0$.

These numbers vary slightly, but this gives a good picture of what is going on. In particular you can see that slot-value within a method specialised on the class is more than 70 times slower than access for a structure slot, but if you can use standard-instance-access it is only about 6 times slower: standard-instance-access speeds things up by a factor of about 10, which changes CLOS slot access performance from laughably slow to merely pretty slow.

A macro

I’ve written a macro, called with-sia-slots which is like with-slots but uses standard-instance-access. It therefore has all the constraints imposed by that, but it is significantly faster than with-slots or slot-value. It has some overhead, as it has to dynamically compute the slot locations: this is better done outside any inner loop. This means that, for instance, you probably want to write code that looks like

(with-sia-slots (x) o
  (dotimes (i many)
    (setf x (... x ...))))

which will mean you only pay the overhead once.

The above tests don’t use with-sia-slots, as I wrote them partly to see if something like this was worth writing. However on a current (at the time of writing) SBCL with-sia-slots is asymptotically about 10 times faster than with-slots as demonstrated by these tests.

Up to package names it should be portable to any CL with an AMOP-compatible MOP. It can be found in my implementation-specific hacks, linked from here.

Structures of arrays

2026-04-16T11:01:13Z

Or, second system.

A while ago, I decided that I’d like to test my intuition that Lisp (specifically implementations of Common Lisp) was not, in fact, bad at floating-point code and that the ease of designing languages in Lisp could make traditional Fortran-style array-bashing numerical code pretty pleasant to write.

I used an intentionally naïve numerical solution to a gravitating many-body system as a benchmark, so I could easily compare Lisp & C versions. The brief result is that the Lisp code is a little slower than C, but not much: Lisp is not, in fact, slow. Who knew?

The point here though, is that I wanted to dress up the array-bashing code so it looked a lot more structured. To do this I wrote a macro which hid what was in fact an array of (for instance) double floats behind a bunch of syntax which made it look like an array of structures. That macro took a couple of hours.

This was fine and pretty simple, but it only dealt with a single type for each conceptual array of objects, there was no inheritance and it was restricted in various other ways. In particular it really was syntactic sugar on a vector: there was no distinct implementational type at all. So I thought well, I could make it more general and nicer.

Big mistake.

The second system

Here is an example of what I wanted to be able to do (this is in fact the current syntax):

(define-soa-class example ()
  ((x :array t :type double-float)
   (y :array t :type double-float)
   (p :array t :type double-float :group pq)
   (q :array t :type double-float :group pq)
   (r :array t :type fixnum)
   (s)))

This defines a class, instances of which have five array slots and one scalar slot. Of the array slots:

x and y share an array and will be neighbouring elements;
p and q share a different array, because the group option says they must not share with x and y;
r will be in its own array, unless the upgraded element type of fixnum is the same as that of double-float;
s is just a slot.

The implementation will tell you this:

> (describe (make-instance 'example :dimensions '(2 2)))
#<example 8010059EEB> is an example
[...]
dimensions      (2 2)
total-size      4
rank            2
tick            1
its class example has a valid layout
it has 3 arrays:
 index 0, element type double-float, 2 slots
 index 1, element type (signed-byte 64), 1 slot
 index 2, element type double-float, 2 slots
it has 5 array slots:
 name x, index 0 offset 0
 name y, index 0 offset 1
 name r, index 1 offset 0
 name p, index 2 offset 0
 name q, index 2 offset 1

This is already too complicated: the ability to control sharing via groups is almost certainly never going to be useful: it’s only even there because I thought of it quite early on and never removed it.

The class definition macro then needs to arrange life so that enough information is available so that a macro can be written which turns indexed slot access into indexed array access of the underlying arrays which are secretly stored in instances, inserting declarations to make this as fast as possible: anything slower than explicit array access is not acceptable. This might (and does) look like this, for example:

(with-array-slots (x y) (thing example)
  (for* ((i ...) (j ...))
    (setf (x i j) (- (y i j) (y j i)))))

As you can see from this, the resulting objects should be allowed to have rank other than 1. Inheritance should also work, including for array slots. Redefinition should be supported and obsolete macro expansions and instances at least detected.

In other words there are exactly two things I should have aimed at achieving: the ability to define fields of various types and have them grouped into (generally fewer) underlying arrays, and an implementational type to hold these things. Everything else was just unnecessary baggage which made the implementation much more complicated than it needed to be.

I had not finished making mistakes. The system needs to store some metadata about how slots map onto the underlying arrays, element types and so on, so the macro can use this to compile efficient code. There are two obvious ways to do this: use the property list of the class name, or subclass standard-class and store the metadata in the class. The first approach is simple, portable, has clear semantics, but it’s ‘hacky’; the second is more complicated, not portable, has unclear semantics¹, but it’s The Right Thing². Another wrong decision I made without even trying.

The only thing that saved me was that the nature of software is that you can only make a finite number of bad decisions in a finite time.

More bad decisions

I was not done. Early on, I thought that, well, I could make this whole thing be a shim around defstruct: single inheritance was more than enough, and obviously I could store metadata on the property list of the type name as described above. And there’s no nausea with multiple accessors or any of that nonsense.

But, somehow, I found writing a thing which would process the (structure-name ...) case of defstruct too painful, so I decided to go for the shim-around-defclass version instead. I even have a partly-complete version of the defstructy code which I abandoned. Another mistake.

I also decided that The Right Thing was to have the system support objects of rank 0. That constrains the underlying array representation (it needs to use rank $n+1$ arrays for an object of rank $n$) in a way which I thought for a long time might limit performance.

Things I already knew

At any point during the implementation of this I could have told you that it was too general and the implementation was going to be too complicated for no real gain. I don’t know why I made so many bad choices.

The whole process took weeks and I nearly just gave up several times.

The light at the end of the tunnel

Or: all-up testing.

Eventually, I had a thing I thought might work. The macro syntax was a bit ugly (that macro still exists, with a different name) but it seemed to work. But since the whole purpose of the thing was performance, that needed to be checked. I wasn’t optimistic.

What I did was to write a version of my naïve gravitational many-body system using the new code, based closely on the previous one. The function that updates the state of the particles looks like this:

(defun/quickly step-pvs (source destination from below dt G &aux
                                (n (particle-vector-length source)))
  ;; Step a source particle vector into a destination one.
  ;;
  ;; Operation count:
  ;;  3
  ;;  + (below - from) * (n - 1) * (3 + 8 + 9)
  ;;  + (below - from) * (12 + 6)
  ;;  = (below - from) * (20 * (n - 1) + 18) + 3
  (declare (type particle-vector source destination)
           (type vector-index from)
           (type vector-dimension below)
           (type fpv dt G)
           (type vector-dimension n))
  (when (eq source destination)
    (error "botch"))
  (let*/fpv ((Gdt (* G dt))
             (Gdt^2/2 (/ (* Gdt dt) (fpv 2.0))))
    (binding-array-slots (((source particle-vector :check nil :rank 1 :suffix _s)
                           m x y z vx vy vz)
                          ((destination particle-vector :check nil :rank 1 :suffix _d)
                           m x y z vx vy vz))
      (for ((i1 (in-naturals :initially from :bound below :fixnum t)))
        (let/fpv ((ax/G zero.fpv)
                  (ay/G zero.fpv)
                  (az/G zero.fpv)
                  (x1 (x_s i1))
                  (y1 (y_s i1))
                  (z1 (z_s i1))
                  (vx1 (vx_s i1))
                  (vy1 (vy_s i1))
                  (vz1 (vz_s i1)))
          (for ((i2 (in-naturals n t)))
            (when (= i1 i2) (next))
            (let/fpv ((m2 (m_s i2))
                      (x2 (x_s i2))
                      (y2 (y_s i2))
                      (z2 (z_s i2)))
              (let/fpv ((rx (- x2 x1))
                        (ry (- y2 y1))
                        (rz (- z2 z1)))
                (let/fpv ((r^3 (let* ((r^2 (+ (* rx rx) (* ry ry) (* rz rz)))
                                      (r (sqrt r^2)))
                                 (declare (type nonnegative-fpv r^2 r))
                                 (* r r r))))
                  (incf ax/G (/ (* rx m2) r^3))
                  (incf ay/G (/ (* ry m2) r^3))
                  (incf az/G (/ (* rz m2) r^3))))))
          (setf (x_d i1) (+ x1 (* vx1 dt) (* ax/G Gdt^2/2))
                (y_d i1) (+ y1 (* vy1 dt) (* ay/G Gdt^2/2))
                (z_d i1) (+ z1 (* vz1 dt) (* az/G Gdt^2/2)))
          (setf (vx_d i1) (+ vx1 (* ax/G Gdt))
                (vy_d i1) (+ vy1 (* ay/G Gdt))
                (vz_d i1) (+ vz1 (* az/G Gdt)))))))
  destination)

And it not only worked, the performance was very close to the previous version, straight out of the gate. The syntax is not as nice as that of the initial, quick-and-dirty version, but it is much more general, so I think that’s worth it on the whole.

There have been problems since then: in particular the dependency on when classes get defined. It will never be as portable as I’d like because of the unnecessary MOP dependencies³, but it is usable and quick⁴.

Was it worth it? May be, but it should have been simpler.

When exactly do classes get defined? Right. ↩
Nothing that uses the AMOP MOP is ever The Right Thing, because the whole thing was designed by people who were extremely smart, but still not as smart as they needed to be and thought they were. It’s unclear if any MOP for CLOS can ever be satisfactory, in part because CLOS itself suffers from the same smart-but-not-smart-enough problem to a large extent not helped by bring dropped wholesale into CL at the last minute: by the time CL was standardised people had written large systems in it, but almost nobody had written anything significant using CLOS, let alone the AMOP MOP. ↩
A mistake I somehow managed to avoid was using the whole slot-definition mechanism the MOP wants you to use. ↩
I will make it available at some point. ↩

Rules for Lisp programs

2026-04-08T10:48:26Z

Some very serious rules. Very serious.

The essential rule. If you are not building languages in Lisp why are you even here?

The lesser rules.

If you write a program which uses defclass you are probably making a mistake.
If you write a program which uses the CLOS MOP you are making a mistake.
If you write a program which uses LOOP for any purpose other than creating a better iteration construct you are making a mistake.
If you write a program which uses LOOP only to create a better iteration construct you are probably making a mistake.
If you write a program which uses explicit package-qualified names more than very infrequently you will be cast into the outer darkness along with your program.

I will not be taking questions.

Literals and constants in Common Lisp

2025-12-04T16:23:45Z

Or, constantp is not enough.

Because I do a lot of things with Štar, and for other reasons, I spend a fair amount of time writing various compile-time optimizers for things which have the semantics of function calls. You can think of iterator optimizers in Štar as being a bit like compiler macros: the aim is to take a function call form and to turn it, in good cases, into something quicker¹. One important way of doing this is to be able to detect things which are known at compile-time: constants and literals, for instance.

One of the things this has made clear to me is that, like John Peel, constantp is not enough. Here’s an example.

(in-row-major-array a :simple t :element-type 'fixnum) is a function call whose values Štar can use to tell it how to iterate (via row-major-aref) over an array. When used in a for form, its optimizer would like to be able to expand into something involving (declare (type (simple-array fixnum *) ...), so that the details of the array are known to the compiler, which can then generate fast code for row-major-aref. This makes a great deal of difference to performance: array access to simple arrays of known element types is usually much faster than to general arrays.

In order to do this it needs to know two things:

that the values of the simple and element-type keyword arguments are compile-time constants;
what their values are.

You might say, well, that’s what constantp is for². It’s not: constantp tells you only the first of these, and you need both.

Consider this code, in a file to be compiled:

(defconstant et 'fixnum)

(defun ... ...
  (for ((e (in-array a :element-type et)))
    ...)
  ...)

Now, constantpwill tell you that et is indeed a compile-time constant. But it won’t tell you its value, and in particular nothing says it needs to be bound at compile-time at all: (symbol-value 'et) may well be an error at compile-time.

constantp is not enough³! instead you need a function that tells you ‘yes, this thing is a compile-time constant, and its value is …’. This is what literal does⁴: it conservatively answers the question, and tells you the value if so. In particular, an expression like (literal '(quote fixnum)) will return fixnum, the value, and t to say yes, it is a compile-time constant. It can’t do this for things defined with defconstant, and it may miss other cases, but when it says something is a compile-time constant, it is. In particular it works for actual literals (hence its name), and for forms whose macroexpansion is a literal.

That is enough in practice.

Śtar’s iterator optimizers are not compiler macros, because the code they write is inserted in various places in the iteration construct, but they’re doing a similar job: turning a construct involving many function calls into one requiring fewer or no function calls. ↩
And you may ask yourself, “How do I work this?” / And you may ask yourself, “Where is that large automobile?” / And you may tell yourself, “This is not my beautiful house” / And you may tell yourself, “This is not my beautiful wife” ↩
Here’s something that staryed as a mail message which tries to explain this in some more detail. In the case of variables defconstant is required to tell constantp that a variable is a constant at compile-time but is not required (and should not be required) to evaluate the initform, let alone actually establish a binding at that time. In SBCL it does both (SBCL doesn’t really have a compilation environment). In LW, say, it at least does not establish a binding, because LW does have a compilation environment. That means that in LW compiling a file has fewer compile-time side-effects than it does in SBCL. Outside of variables, it’s easily possible that a compiler might be smart enough to know that, given (defun c (n) (+ n 15)), then (constantp '(c 1) <compilation environment>) is true. But you can’t evaluate (c 1) at compile-time at all. constantp tells you that you don’t need to bind variables to prevent multiple evaluation, it doesn’t, and can’t, tell you what their values will be. ↩
Part of the org.tfeb.star/utilities package. ↩

A timing macro for Common Lisp

2025-11-27T11:50:16Z

For a long time I’ve used a little macro to time chunks of code to avoid an endless succession of boilerplate functions to do this. I’ve finally published the wretched thing.

If you’re writing programs where you care about performance, you often want to be able to make programatic comparisons of performance. time doesn’t do this, since it just reports things. Instead you want something that runs a bit of code a bunch of times and then returns the average time, with ‘a bunch of times’ being controllable. timing is that macro. Here is a simple example:

(defun dotimes/in-naturals-ratio (&key (iters 10000000) (tries 1000))
  (declare (type fixnum iters)
           (optimize speed))
  (/
   (timing (:n tries)
     (let ((s 0))                       ;avoid optimizing loop away
       (declare (type fixnum s))
       (dotimes (i iters s)
         (incf s))))
   (timing (:n tries)
     (let ((s 0))
       (declare (type fixnum s))
       (for ((_ (in-naturals iters t)))
         (incf s))))))

and then, for instance

> (dotimes/in-naturals-ratio)
1.0073159

All timing does is to wrap up its body into a function and then call a function which calls this function the number of times you specify and averages the time, returning that average as a float.

There are some options which let it print a progress note every given number of calls, wrap a call to time around things so you get, for instance, GC reporting, and subtract away the same number of calls to an empty function to try and account for overhead (in practice this is not very useful).

That’s all it is. It’s available in version 10 of my Lisp tools:

documentation here;
git repo here;
tarball here.

The lost cause of the Lisp machines

2025-11-18T08:52:44Z

I am just really bored by Lisp Machine romantics at this point: they should go away. I expect they never will.

History

Symbolics went bankrupt in early 1993. In the way of these things various remnants of the company lingered on for, in this case, decades. But 1983 was when the Lisp Machines died.

The death was not unexpected: by the time I started using mainstream Lisps in 1989¹ everyone knew that special hardware for Lisp was a dead idea. The common idea was that the arrival of RISC machines had killed it, but in fact machines like the Sun 3/260 in its ‘AI’ configuration² were already hammering nails in its coffin. In 1987 I read a report showing the Lisp performance of an early RISC machine, using Kyoto Common Lisp, not a famously fast implementation of CL, beating a Symbolics on the Gabriel benchmarks [PDF link].

1993 is 32 years ago. The Symbolics 3600, probably the first Lisp machine that sold in more than tiny numbers, was introduced in 1983, ten years earlier. People who used Lisp machines other than as historical artefacts are old today³.

Lisp machines were both widely available and offered the best performance for Lisp for a period of about five years which ended nearly forty years ago. They were probably never competitive in terms of performance for the money.

It is time, and long past time, to let them go.

But still the romantics — some of them even old enough to remember the Lisp machines — repeat their myths.

‘It was the development environment’

No, it wasn’t.

The development environments offered by both families of Lisp machines were seriously cool, at least for the 1980s. I mean, they really were very cool indeed. Some of the ways they were cool matter today, but some don’t. For instance in the 1980s and early 1990s Lisp images were very large compared to available memory, and machines were also extremely slow in general. So good Lisp development environents did a lot of work to hide this slowness, and in general making sure you only very seldom had to restart everthing, which took significant fractions of an hour, if not more. None of that matters today, because machines are so quick and Lisps so relatively small.

But that’s not the only way they were cool. They really were just lovely things to use in many ways. But, despite what people might believe: this did not depend on the hardware: there is no reason at all why a development environent that cool could not be built on stock hardware. Perhaps, (perhaps) that was not true in 1990: it is certainly true today.

So if a really cool Lisp development environment doesn’t exist today, it is nothing to do with Lisp machines not existing. In fact, as someone who used Lisp machines, I find the LispWorks development environment at least as comfortable and productive as they were. But, oh no, the full-fat version is not free, and no version is open source. Neither, I remind you, were they.

‘They were much faster than anything else’

No, they weren’t. Please, stop with that.

‘The hardware was user-microcodable, you see’

Please, stop telling me things about machines I used: believe it or not, I know those things.

Many machines were user-microcodable before about 1990. That meant that, technically, a user of the machine could implement their own instruction set. I am sure there are cases where people even did that, and a much smaller number of cases where doing that was not just a waste of time.

But in almost all cases the only people who wrote microcode were the people who built the machine. And the reason they wrote microcode was because it is the easiest way of implementing a very complex instruction set, especially when you can’t use vast numbers of transistors. For instance if you’re going to provide an ‘add’ instruction which will add numbers of any type, trapping back into user code for some cases, then by far the easiest way of doing that is going to be by writing code, not building hardware. And that’s what the Lisp machines did.

Of course, the compiler could have generated that code for hardware without that instruction. But with the special instruction the compiler’s job is much easier, and code is smaller. A small, quick compiler and small compiled code were very important with slow machines which had tiny amounts of memory. Of course a compiler not made of wet string could have used type information to avoid generating the full dispatch case, but wet string was all that was available.

What microcodable machines almost never meant was that users of the machines would write microcode.

At the time, the tradeoffs made by Lisp machines might even have been reasonable. CISC machines in general were probably good compromises given the expense of memory and how rudimentary compilers were: I can remember being horrified at the size of compiled code for RISC machines. But I was horrified because I wasn’t thinking about it properly. Moore’s law was very much in effect in about 1990 and, among other things, it meant that the amount of memory you could afford was rising exponentially with time: the RISC people understood that.

‘They were Lisp all the way down’

This, finally, maybe, is a good point. They were, and you could dig around and change things on the fly, and this was pretty cool. Sometimes you could even replicate the things you’d done later. I remember playing with sound on a 3645 which was really only possible because you could get low-level access to the disk from Lisp, as the disk could just marginally provide data fast enough to stream sound.

On the other hand they had no isolation and thus no security at all: people didn’t care about that in 1985, but if I was using a Lisp-based machine today I would certainly be unhappy if my web browser could modify my device drivers on the fly, or poke and peek at network buffers. A machine that was Lisp all the way down today would need to ensure that things like that couldn’t happen.

So may be it would be Lisp all the way down, but you absolutely would not have the kind of ability to poke around in and redefine parts of the guts you had on Lisp machines. Maybe that’s still worth it.

Not to mention that I’m just not very interested in spending a huge amount of time grovelling around in the guts of something like an SSL implementation: those things exist already, and I’d rather do something new and cool. I’d rather do something that Lisp is uniquely suited for, not reinvent wheels. Well, may be that’s just me.

Machines which were Lisp all the way down might, indeed, be interesting, although they could not look like 1980s Lisp machines if they were to be safe. But that does not mean they would need special hardware for Lisp: they wouldn’t. If you want something like this, hardware is not holding you back: there’s no need to endlessly mourn the lost age of Lisp machines, you can start making one now. Shut up and code.

And now we come to the really strange arguments, the arguments that we need special Lisp machines either for reasons which turn out to be straightforwardly false, or because we need something that Lisp machines never were.

‘Good Lisp compilers are too hard to write for stock hardware’

This mantra is getting old.

The most important thing is that we have good stock-hardware Lisp compilers today. As an example, today’s CL compilers are not far from CLANG/LLVM for floating-point code. I tested SBCL and LispWorks: it would be interesting to know how many times more work has gone into LLVM than them for such a relatively small improvement. I can’t imagine a world where these two CL compilers would not be at least comparable to LLVM if similar effort was spent on them⁴.

These things are so much better than the wet-cardboard-and-string compilers that the LispMs had it’s not funny. In particular, if some mythical ‘dedicated Lisp hardware’ made it possible to write a Lisp compiler which generated significantly faster code, then code from Lisp compilers would comprehensively outperform C and Fortran compilers: does that seem plausible? I thought not.

A large amount of work is also going into compilation for other dynamically-typed, interactive languages which aim at high performance. That means on-the-fly compilation and recompilation of code where both the compilation and the resulting code must be quick. Example: Julia. Any of that development could be reused by Lisp compiler writers if they needed to or wanted to (I don’t know if they do, or should).

Ah, but then it turns out that that’s not what is meant by a ‘good compiler’ after all. It turns out that ‘good’ means ‘compillation is fast’.

All these compilers are pretty quick: the computational resources used by even a pretty hairy compiler have not scaled anything like as fast as those needed for the problems we want to solve (that’s why Julia can use LLVM on the fly). Compilation is also not an Amdahl bottleneck as it can happen on the node that needs the compiled code.

Compilers are so quick that a widely-used CL implementation exists where EVAL uses the compiler, unless you ask it not to.

Compilation options are also a thing: you can ask compilers to be quick, fussy, sloppy, safe, produce fast code and so on. Some radically modern languages also allow this to be done in a standardised (but extensible) way at the language level, so you can say ‘make this inner loop really quick, and I have checked all the bounds so don’t bother with that’.

The tradeoff between a fast Lisp compiler and a really good Lisp compiler is imaginary, at this point.

‘They had wonderful keyboards’

Well, if you didn’t mind the weird layouts: yes, they did⁵. And has exactly nothing to do with Lisp.

And so it goes on.

Bored now

There’s a well-known syndrome amongst photographers and musicians called GAS: gear acquisition syndrome. Sufferers from this⁶ pursue an endless stream of purchases of gear — cameras, guitars, FX pedals, the last long-expired batch of a legendary printing paper — in the strange hope that the next camera, the next pedal, that paper, will bring out the Don McCullin, Jimmy Page or Chris Killip in them. Because, of course, Don McCullin & Chris Killip only took the pictures they did because he had the right cameras: it was nothing to do with talent, practice or courage, no.

GAS is a lie we tell ourselves to avoid the awkward reality that what we actually need to do is practice, a lot, and that even if we did that we might not actually be very talented.

Lisp machine romanticism is the same thing: a wall we build ourself so that, somehow unable to climb over it or knock it down, we never have to face the fact that the only thing stopping us is us.

There is no purpose to arguing with Lisp machine romantics because they will never accept that the person building the endless barriers in their way is the same person they see in the mirror every morning. They’re too busy building the walls.

As a footnote, I went to a talk by an HPC person in the early 90s (so: after the end of the cold war⁷ and when the HPC money had gone) where they said that HPC people needed to be aiming at machines based on what big commercial systems looked like as nobody was going to fund dedicated HPC designs any more. At the time that meant big cache-coherent SMP systems. Those hit their limits and have really died out now: the bank I worked for had dozens of fully-populated big SMP systems in 2007, it perhaps still has one or two they can’t get rid of because of some legacy application. So HPC people now run on enormous shared-nothing farms of close-to-commodity processors with very fat interconnect and are wondering about / using GPUs. That’s similar to what happened to Lisp systems, of course: perhaps, in the HPC world, there are romantics who mourn the lost glories of the Cray–3. Well, if I was giving a talk to people interested in the possibilities of hardware today I’d be saying that in a few years there are going to be a lot of huge farms of GPUs going very cheap if you can afford the power. People could be looking at whether those can be used for anything more interesting than the huge neural networks they were designed for. I don’t know if they can.

Before that I had read about Common Lisp but actually written programs in Cambridge Lisp and Standard Lisp. ↩
This had a lot of memory and a higher-resolution screen, I think, and probably was bundled with a rebadged Lucid Common Lisp. ↩
I am at the younger end of people who used these machines in anger: I was not there for the early part of the history described here, and I was also not in the right part of the world at a time when that mattered more. But I wrote Lisp from about 1985 and used Lisp machines of both families from 1989 until the mid to late 1990s. I know from first-hand experience what these machines were like. ↩
If anyone has good knowledge of Arm64 (specifically Apple M1) assembler and performance, and the patience to pore over a couple of assembler listings and work out performance differences, please get in touch. I have written most of a document exploring the difference in performance, but I lost the will to live at the point where it came down to understanding just what details made the LLVM code faster. All the compilers seem to do a good job of the actual float code, but perhaps things like array access or loop overhead are a little slower in Lisp. The difference between SBCL & LLVM is a factor of under 1.2. ↩
The Sun type 3 keyboard was both wonderful and did not have a weird layout, so there’s that. ↩
I am one: I know what I’m talking about here. ↩
The cold war did not end in 1991. America did not win. ↩

Disentangling iteration from value accumulation

2025-10-31T12:40:33Z

Iteration forms and forms which accumulate values don’t have to be the same thing. I think that it turns out that separating them works rather well.

There’s no one true way to write programs, especially in Lisp¹: a language whose defining feature is that it supports and encourages the seamless construction of new programming languages². In particular there are plenty of different approaches to iteration, and to accumulating values during iteration. In CL there are at least three approaches in the base language:

constructs which map a function over some ‘iterable’ object, often a list or a sequence of some other kind, to build another object with the results, as by mapcar for instance;
constructs which just iterate, as by dotimes;
iteration constructs which combine iteration with possible value accumulation, such as do and of course loop.

What CL doesn’t have is any constructs which simply accumulate values. So, for instance, if you wanted to acquire the even numbers from a list with dolist you might write

(let ((evens '()))
  (dolist (e l (nreverse evens))
    (when (and (realp e) (evenp e))
      (push e evens))))

Of course you could do this with loop:

(loop for e in l
      when (and (realp e) (evenp e)) collect e)

but loop is a construct which combines iteration and value collection.

It’s tempting to say that, well, can’t you turn all iteration into mapping? Python sort of does this: objects can be ‘iterable’, and you can iterate over anything iterable, and then comprehensions let you accumulate values. But in general this doesn’t work very well: consider a file which you want to iterate over. But how? Do you want to iterate over its characters, its bytes, its lines, its words, over some other construct in the file? You can’t just say ‘a file is iterable’: it is, but you have to specify the intent before iterating over it³. You also have the problem that you very often only want to return some values, so the notion of ‘mapping’ is not very helpful. If you try and make everything be mapping you end up with ugly things like mapcan.

You do need general iteration constructs, I think: constructs which say ‘is there more? if there is give me the next thing’. In CL both the standard general iteration constructs combine, or can combine, iteration with accumulation: there is no pure general iteration construct. And there are no pure value accumulation constructs at all.

From Maclisp to CL

An interesting thing happened in the transition from Maclisp to CL.

Maclisp had prog, which was a special operator (it would have called it a special form), and which combined the ability to use go and to say return. This is a construct which dates back to the very early days of Lisp.

Common Lisp also has prog, but now it’s a macro, not a special operator. The reason its a macro is that CL has split the functionality of prog into three parts (four parts if you include variable binding):

progn is a special operator which evaluates the forms in its body in order;
tagbody is a special operator whch allows tags and go in its body;
block is a special operator which supports return and return-from
and of course let provides binding of variables.

Maclisp had let and progn: what it didn’t have was tagbody and block.

These can be combined (you don’t in fact need progn in this case) to form prog, which is something like

(defmacro prog ((&rest bindings)
                &body tags/forms)
  `(block nil
     (let ,@bindings
       (tagbody
        ,@tags/forms)
       nil)))

So what CL has done is to divide prog into its component parts, which then can be used individually in other ways: it has provided the components of prog as individual constructs. You can build prog from these, but you can build other things as well (defun expands to something involving block, for instance), including things which don’t exist in base CL.

A linguistic separation of concerns

What CL has achieved is a separation of concerns at the language level: it has reduced the number of concerns addressed by each construct. It hasn’t done this completely: progn is not the only special operator which sequences the forms in its body, for instance, and let is not a macro defined in terms of lambda. But it’s taken steps in this direction compared to Maclisp.

This approach is really only viable for languages which have powerful macro systems where macros are not syntactically distinguished. Without a macro system then separating concerns at the language level would make almost all programs more verbose since constructs which combine lower-level ones can’t be created. With a macro system where macros are syntactically distinguished, such as Julia’s, then such constructs are always second-class citizens. With a macro system like CL’s this is no longer a problem: CL has prog, for instance, but it’s now a macro.

It seems to me that the only reason not to take this process as far as it can go in Lisps is if it makes the compiler’s job unduly hard. It makes no difference to users of the language, so long as it provides, as CL does the old, unseparated, convenient constructs.

From CL to here knows when

I can’t redesign CL and don’t want to do that. But I can experiment with building a language I’d like to use on top of it.

In particular CL has already provided the separated constructs you need to build your own iteration constructs, and no CL iteration constructs are special operators. Just as do is constructed from (perhaps) let, block and tagbody, and loop is constructed from some horrid soup if the same things, you can build your own iteration constructs this way. And the same is true for value accumulation constructs. And you can reasonably expect these to perform as well as the ones in the base language.

This is what I’ve done, several times in fact.

The first thing I built, long ago, was a list accumulation construct called collecting: within its body there is a local function, collect, which will accumulate a value onto the list returned from collecting. It secretly maintains a tail-pointer to the list so accumulation is constant-time. This was originally built to make it simpler to accumulate values when traversing tree or graph structures, to avoid the horrid and, in those days, slow explicit push … nreverse idiom.

So, for instance

(collecting
  (labels ((walk (node)
             ...
             (when ... (collect thing))
             ...
             (dolist (...) (walk ...))))
    (walk ...)))

might walk over some structure, collecting interesting things, and returning a list of them.

collecting was originally based on some ideas in Interlisp-D, and has since metastasized into a, well, collection of related constructs: multiple named collectors (collecting itself is now defined in terms of this construct), explicit collector objects, general accumulators and most recently a construct which accumulates values into vectors. It works pretty well.

The second part of the story is high-performance iteration constructs which just iterate, which are general, which are pleasant to use and have semantics which are easy to understand. Both loopand do fail the first three of these conditions for me, and loop fails the fourth as well.

Well, I’ve written a number of iteration constructs and constructs related to iteration. Finally, last year, my friend Zyni & I (the ideas are largely hers, I wrote most of the code I think) came up with Štar which we’ve described as ‘a simple and extensible iteration construct’. Lots of other people have written iteration constructs for CL: Štar occupies a position which tries to be as extreme as possible while remaining pleasant to use. There are no special keywords, the syntax is pretty much that of let and there is no value accumulation: all it does is iterate. The core of Štar exports six names, of which the three that support nested iteration are arguably unneeded in the same way that let* is. Teaching it how to iterate over things is simple, teaching it how to optimize such iterations is usually simple enough to do when it’s worth it. And it’s within $\varepsilon$ of anything in terms of performance.

It’s simple (at least in interface) and quick because it hardly does anything, of course: it relies entirely on iterators to do anything at all and iterator optimizers to do anything quickly. Even then all it does is, well, iterate.

These two components are thus attempts at separating the two parts of something like loop, Iterate or For, or other constructs which combine iteration and value accumulation: they are to these constructs what tagbody and block are to prog.

Reinventing the wheel

I used to ride bicycles a lot. And I got interested in the surprisingly non-obvious way that bicycle wheels work. After reading The bicycle wheel I decided that I could make wheels, and I did do that.

And a strange thing happened: although I rationally understood that the wheels I had made were as good or better than any other wheel, for the first little while after building them I was terrified that they would bend or, worse, collapse. There was no rational reason for this: it was just that for some reason I trusted my own workmanship less than I trusted whoever had made the off-the-shelf wheels they’d replaced (and, indeed, some of whose parts I had cannibalised to make them).

Of course they didn’t bend or collapse, and I still rode on one of them until quite recently.

The same thing happened with Štar: for quite a while after finishing it I had to work hard to force myself to use it: even though I knew it was fast and robust. It wasn’t helped that one of the basic early iterators was overcomplex and had somewhat fragile performance. It wasn’t until I gave up on it and replaced it by a much simpler and more limited one, while also making a much more general iterator fast enough to use for the complicated cases that it felt comfortable.

This didn’t happen with collecting: I think that’s because it did something CL didn’t already have versions of, while it’s very often possible to replace a construct using Štar with some nasty thing involving do or some other iteration construct. Also Štar is much bigger than collecting and it’s hard to remember that I’m not using a machine with a few MB of memory any more. Perhaps it’s also because I first wrote collecting a very long time ago.

But I got over this, and now almost the only times I’d use any other iteration construct are either when mapcar &c are obviously right, or when I’m writing code for someone else to look at.

And writing iterators is easy, especially given that you very often do not need optimizers for them: if you’re iterating over the lines in a file two function calls per line is not hurting much. Iterators, of course, can also iterate over recursively-defined structures such as trees or DAGs: it’s easy to say (for ((leaf (in-graph ... :only-leaves t))) ...).

Would it help?

In my biased experience, yes, quite a lot. I now much prefer writing and reading code that uses for to code that uses almost any of the standard iteration constructs, and collecting, together with its friends, simply does not have a standard equivalent at all: if you don’t have it, you need either to write it, or implement it explicitly each time.

But my experience is very biased: I have hated loop almost since it arrived in CL, and I find using do for anything non-trivial clumsy enough that I’ve previously written versions of it which require less repetition. And of course I was quite involved in the design and implementation of Štar, so it’s not surprising that I like it.

I’m also very comfortable with the idea that Lisp is about language design — in 2025 I don’t see any compelling advantage of Lisp other than constructing languages — and that people who write Lisp end up writing in their own idiolects. The argument against doing this seems to be that every Lisp project ends up being its own language and this means that it is hard to recruit people. I can only assume that the people who say that have never worked on any large system written in languages other than Lisp⁴: Greenspun’s tenth rule very much applies to these systems.

In summary: yes, it would help.

An example

In the examples directory for Štar there is an iterator called in-graph which can iterate over any graph, if it knows how to find the neighbours of a node. For instance:

> (for ((n (in-graph (list '(a b (c b) d))
                     (lambda (n)
                       (if (atom n) '() (cdr n))))))
    (print n))

(a b (c b) d) 
b 
(c b) 
b 
d 
nil

> (for ((n (in-graph (list '(a b (c b) d))
                     (lambda (n)
                       (if (atom n) '() (cdr n)))
                     :unique t)))
    (print n))

(a b (c b) d) 
b 
(c b) 
d 
nil

> (for ((n (in-graph (list '(a b (c b) d))
                     (lambda (n)
                       (if (atom n) '() (cdr n)))
                     :order :breadth-first)))
    (print n))

(a b (c b) d) 
b 
(c b) 
d 
b 
nil

> (collecting (for ((n (in-graph (list '(a b (c b) d))
                                 (lambda (n)
                                   (if (atom n) '() (cdr n)))
                                 :unique t
                                 :only-leaves t)))
                (collect n)))
(b d)

> (setf *print-circle* t)
t

> (for ((n (in-graph (list '#1=(a #2=(b c #1#) d #2#))
                     (lambda (n)
                       (if (atom n) '() (cdr n)))
                     :unique t)))
    (print n))

#1=(a #2=(b c #1#) d #2#) 
#1=(b c (a #1# d #1#)) 
c 
d 
nil

> (for ((p (in-graph (list *package*) #'package-use-list
                     :unique t :order :breadth-first)))
    (format t "~&~A~%" (package-name p)))
COMMON-LISP-USER
ORG.TFEB.DSM
ORG.TFEB.HAX.ITERATE
ORG.TFEB.HAX.COLLECTING
ORG.TFEB.STAR
ORG.TFEB.TOOLS.REQUIRE-MODULE
COMMON-LISP
HARLEQUIN-COMMON-LISP
LISPWORKS
ORG.TFEB.HAX.UTILITIES
ORG.TFEB.HAX.SIMPLE-LOOPS
ORG.TFEB.HAX.SPAM
ORG.TFEB.DSM/IMPL
nil

in-graph is fairly simple, and uses both collectors and Štar in its own implementation:

(defun in-graph (roots node-neighbours &key
                       (only-leaves nil)
                       (order ':depth-first)
                       (unique nil)
                       (test #'eql)
                       (key #'identity))
  ;; Preorder / postorder would be nice to have
  "Iterate over a graph

- ROOTS are the nodes to start from.
- NODE-NEIGHBOURS is a function which, given a node, returns its
  neighbours if any.
- ORDER may be :DEPTH-FIRST (default) or :BREADTH-FIRST.
- UNIQUE, if given, will iterate nodes uniquely.
- TEST is the comparison test for nodes: it must be something
  acceptable to MAKE-HASH-TABLE.  Default is #'EQL.
- KEY, if given, extracts a key from a node for comparison in the
  usual way.

There is no optimizer.

If the graph is cyclic an iteration using this will not terminate
unless UNIQUE is true, unless some other clause stops it.  If the
graph is not directed you also need to use UNIQUE."
  (check-type order (member :depth-first :breadth-first))
  (let ((agenda (make-collector :initial-contents roots))
        (duplicate-table (if unique (make-hash-table :test test) nil))
        (this nil))
    (values
     (thunk                             ;predicate does all the work
       (if (collector-empty-p agenda)
           nil
         (for ((it (stepping (it :as (pop-collector agenda)))))
           (let ((neighbours (funcall node-neighbours it))
                 (k (and unique (funcall key it))))
             (cond
              ((and unique (gethash k duplicate-table))
               ;; It's a duplicate: skip
               (if (collector-empty-p agenda)
                   (final nil)
                 (next)))
              ((null neighbours)
               ;; Leaf, add it to the duplicate table if need be and say we found something
               (when unique
                 (setf (gethash k duplicate-table) t))
               (setf this it)
               (final t))
              (t
               ;; Not a leaf: update the agenda ...
               (setf agenda
                     (case order
                       (:depth-first
                        (nconc-collectors (make-collector :initial-contents neighbours) agenda))
                       (:breadth-first
                        (nconc-collectors agenda (make-collector :initial-contents neighbours)))))
               ;; .. add it to the duplicate table if need be so it's
               ;; skipped next time ...
               (when unique               
                 (setf (gethash k duplicate-table) t))
               ;; ... and decide if we found something
               (cond
                (only-leaves
                 (if (collector-empty-p agenda)
                     (final nil)
                   (next)))
                 (t
                  (setf this it)
                  (final t)))))))))
     (thunk this))))

‘Lisp’ here will usually mean ‘Common Lisp’. ↩
Although if you use loop you must accept that you will certainly suffer eternal damnation. Perhaps that’s worth it: Robert Johnson thought so, anyway. ↩
This is the same argument that explains why a universal equality predicate is nonsensical: equality of objects depends on what they are equal as and that is often not implicit in the objects. ↩
Or in Lisp, more than likely. ↩

Tracing the expansion of external macros

2025-10-10T10:39:18Z

I have improved my trace-macroexpand system so you can say, in effect ‘trace the expansion of only the macros in the interface to a given package’. This is a fairly useful thing.

Tracing macroexpansion in Common Lisp is a pretty useful thing to be able to do, in my experience. It is completely possible to do this in portable CL via *macroexpand-hook*: you simply put your tracing function on this hook, making sure it actually does expand the macro. trace-macroexpand does just this, and lets you specify which macros you want to be traced.

It has always allowed you to say ‘trace all macros whose home package is this package’. That’s less useful than you might think:

it means that not only macros whose names are exported from the packqge are traced, but any macros in its guts are also traced, which generally a user of the package should not be interested in;
it doesn’t trace macros which are exported from a package but whose home package is not that package.

Very often the second thing is exactly what you want: you want to be able to say ‘let me see the expansion of macros in the public interface to this package, but I don’t care about the internal details of it’.

It can now do exactly that.

trace-macro-package now takes a list of package specifiers. If a package specifier is a list of one or more other package specifiers, then it changes their meaning to be ‘trace the exports of these packages only’.

Here is an example:

> (find-symbol "FOR" :org.tfeb.star)
for
:external

> (symbol-package *)
#<The ORG.TFEB.STAR/IMPL package, 188/512 internal, 6/16 external>

> (trace-macroexpand t)
nil

> (setf *trace-macroexpand-per-line-prefix* "| ")
"| "

> (trace-macro-package :org.tfeb.star)
("ORG.TFEB.STAR")

> (for ((_ (in-naturals 10))))
nil

> (untrace-macro-package :org.tfeb.star)
nil

> (trace-macro-package '(:org.tfeb.star))
(("ORG.TFEB.STAR"))

> (for ((_ (in-naturals 10))))
| (for (#))
|  -> (multiple-value-bind (#:<v>) 0 ...)
nil

As well as this, both trace-macro-package and untrace-macro-package now canonicalise the specifiers they are given, which means, for instance that (trace-macro-package '("FOO" "BAR")) is exactly the same as (trace-macro-package '("FOO") '("BAR")): this means that things like

> (trace-macro-package '("FOO" "BAR"))

[...]

> (untrace-macro-package '("FOO"))

will work properly.

This change is in version 10.9.0 of the TFEB.ORG Lisp hax, git repo.

Optional per-line prefix for trace-macroexpand

2025-10-08T16:52:23Z

My macroexpansion tracer can now print per-line prefixes when tracing, which can make things more readable.

I find trace-macroexpand pretty useful: if you write Lisp with lots of nontrivial macros¹ then it can be fairly hard to understand what’s going on when something is not working properly. trace-macroexpand lets you see this either for individual macros or many of them. It can, portably, trace any macro including ones defined by CL, which is even nicer.

However it’s not always easy to distinguish its output from other output. So, I realised I could add a per-line prefix which can help distinguish its output. Here is an example.

> (defvar *a* (cons nil nil))
*a*

> (trace-macroexpand t)
nil

> (trace-macro setf)
(setf)

> (setf (car *a*) 10)
(setf (car *a*) 10)
 -> (system::%rplaca *a* 10)
10

> (setf *trace-macroexpand-per-line-prefix* "| ")
(setf *trace-macroexpand-per-line-prefix* "| ")
 -> (let* (#) (setq *trace-macroexpand-per-line-prefix* #:|Store-Var-1692|))
"| "

> (setf (car *a*) 11)
| (setf (car *a*) 11)
|  -> (system::%rplaca *a* 11)
11

This is in version 10.8.1² of the TFEB.ORG Lisp hax, git repo.

If you aren’t doing that, why are you writing Lisp at all? ↩
10.8.0 had a stupid bug. ↩

Defaulting places in Common Lisp

2025-10-07T11:50:22Z

Or: less boilerplate.

Common Lisp (CL) has a general notion of a place, which is a form which has a value or values and into which a value or values can be stored. Variables are places, but so are forms like (car c): (setf (car c) 2) will store 2 into the car of the cons bound to c. Places can even store multiple values:

(let ((a 1) (b 3))
  (setf (values a b) (values 3 4))
  (values a b))

for instance. Here the place is (values a b) which is a place composed of two other places.

This is a really useful notion, not only because places mean the language no longer needs all sorts of special-purpose mutation functions — rplaca still exists for compatibility but there is no sethash or aset — but because you can implement your own places which behave just like the ones the language provides.

Here’s an example of a place called a ‘wrapped alist’: it’s just a cons whose cdr is an alist. It’s done like this so storing works in general (think about empty alists).

(defun make-wrapped-alist (&optional (for-alist '()))
  (cons nil for-alist))

(defun wrapped-alist-alist (wa)
  (cdr wa))

(defun wav (item wrapped-alist &key (test nil testp) (default nil))
  (let ((found (if testp
                   (assoc item (cdr wrapped-alist) :test test)
                 (assoc item (cdr wrapped-alist)))))
    (if found
        (values (cdr found) t)
      (values default nil))))

(defun (setf wav) (new item wrapped-alist &key (test nil testp) default)
  (declare (ignore default))
  (let ((found (if testp
                   (assoc item (cdr wrapped-alist) :test test)
                 (assoc item (cdr wrapped-alist)))))
    (if found
        (setf (cdr found) new)
      (progn
        (push (cons item new) (cdr wrapped-alist))
        new))))

I will use these wrapped alist places in the examples below.

Defaulting places

Quite often, a place has a default value or a way of indicating that there is no value in it, and you want to be able to say ‘if this place has not been stored into, then store this into it’. In the case of hash tables, the indicator is that gethash returns a second value of nil, and that is the same for av and my wrapped alists.

Sometimes this is not a problem, especially when the accessor for a place lets you provide a default:

(defun symbol-counts (l)
  (let ((table (make-hash-table)))
    (dolist (e l)
      (when (symbolp e)
        (incf (gethash e table 0))))
    (collecting
      (maphash (lambda (k v)
                 (collect (cons k v)))
               table))))

(defun symbol-counts/probably-slower (l)
  (let ((wa (make-wrapped-alist)))
    (dolist (e l)
      (when (symbolp e)
        (incf (av e wa :default 0))))
    (wrapped-alist-alist wa)))

But sometimes it is a problem. Consider the case where the fallback thing you want to store is expensive, or has side-effects. Now you need to write some boilerplate code:

(unless (nth-value 1 (wav item wa)
    (setf (wav item wa) (compute-complicated-thing))))

The wrong way

Well, boilerplate is bad. So you might want to replace this by a macro:

(defmacro defaulting/wrong (place-form value-form)
  ;; This just assumes that PLACE-FORM returns NIL if it has no value:
  ;; in real life you need to be cleverer.
  `(or ,place-form
       (setf ,place-form ,value-form)))

This is not only limited, but incorrect. It’s incorrect because it multiply evaluates subforms to place-form. Consider this:

(let ((i 0) (table (make-hash-table)))
  (defaulting/wrong (gethash (incf i) table) 3))

Well, using wrapped alists it’s easy to see what this is doing wrong:

> (let ((i 0) (wa (make-wrapped-alist)))
    (defaulting/wrong (wav (incf i) wa) 3)
    (wrapped-alist-alist wa))
((2 . 3))

So, not great. The boilerplate you’d need to write is:

> (let ((i 0) (wa (make-wrapped-alist)))
    (let ((k (incf i)))
      (unless (wav k wa)
        (setf (wav k wa) 3)))
    (wrapped-alist-alist wa))
((1 . 3))

The right way

The problem is that any such defaulting macro doesn’t know anything about the place it’s defaulting. So it can’t know which subforms of the place it needs to stash values for.

Well, it turns out that the designers of CL thought of this, and they provided the tool you need, which is get-setf-expansion. Given a place and optionally an environment, this will tell you exactly what you need to know to both read from that place and write to it, and to do so multiple times if need be.

get-setf-expansion is what you need to be able to write your own setf:

(defmacro assign (&rest pairs &environment e)
  ;; This should be SETF give or take
  `(progn
     ,@(collecting
         (for ((tail (on-list pairs :by #'cddr)))
           (destructuring-bind (place-form value-form . _) tail
             (declare (ignore _))
             (multiple-value-bind (vars vals store-vars writer-form reader-form)
                 (get-setf-expansion place-form e)
               (declare (ignore reader-form))
               (collect
                `(let* ,(mapcar #'list vars vals)
                   (multiple-value-bind ,store-vars ,value-form
                     ,writer-form)))))))))

But you can also use it to write defaulting properly. Here is a much fancier version of it, which is now correct (I hope):

(defmacro defaulting (place value-form
                            &body options
                            &key test default-value nth-value &environment e)
  (declare (ignore options))            ;just for indent
  (multiple-value-bind (tvars tforms store-variables storing-form accessing-form)
      (get-setf-expansion place e)
    `(let* ,(mapcar #'list tvars tforms)
         (when ,(cond
                 ((and test nth-value)
                  `(not (funcall ,test ,default-value (nth-value ,nth-value ,accessing-form))))
                 (test
                  `(not (multiple-value-call ,test ,default-value ,accessing-form)))
                 ((and default-value nth-value)
                  `(eql ,default-value (nth-value ,nth-value ,accessing-form)))
                 (default-value
                  `(eql ,default-value ,accessing-form))
                 (nth-value
                  `(not (nth-value ,nth-value ,accessing-form)))
                 (t
                  `(not ,accessing-form)))
           (multiple-value-bind ,store-variables ,value-form
             ,storing-form))
         ,accessing-form)))

So now:

> (let ((i 0) (wa (make-wrapped-alist)))
    (defaulting (wav (incf i) wa) 3)
    (wrapped-alist-alist wa))

Or, using options to this defaulting to tell it the value to be checked:

> (let ((i 0) (wa (make-wrapped-alist)))
    (defaulting (wav (incf i) wa) 3 :nth-value 1)
    (wrapped-alist-alist wa))
((1 . 3))

Finally, you can see the expansion using trace-macroexpand:

> (let ((a (make-wrapped-alist)))
    (defaulting (wav 'k a) 3 :nth-value 1))
(defaulting (wav 'k a)
    3
  :nth-value 1)
 -> (let* ((#:a1 a))
      (when (not (nth-value 1 (wav 'k #:a1)))
        (multiple-value-bind (#:new0) 3 (funcall #'(setf wav) #:new0 'k #:a1)))
      (wav 'k #:a1))
3
t

and this is obviously correct.

This macro exists in org.tfeb.hax.utilities, the git repo for which is tfeb.org/computer/repos/tfeb-lisp-hax.git. Note it is not in the archived GitHub repo.

This is version 10.7.0 of the TFEB.ORG Lisp hax.

Minimum clown

2025-08-26T14:29:31Z

Hosting code on GitHub now seems like an invitation to have it turned into AI slop. Here’s what I did to move.

Isn’t it rich?
Are we a pair?
Me here at last on the ground,
You in mid-air.
Send in the clowns.

I think the number of people who care about, let alone rely on, my Lisp code who I don’t personally know is somewhere between few and none by now. Nevertheless, it makes me feel honest to publish some of what I write, in the hope someone might find it useful.

Long ago, I did that by simply putting files on my vanity site (in the days when people had vanity sites). At some point it became clear that this was not the right answer, and the thing to do was to keep things on one of the newfangled source-hosting sites. I have never been and will never be comfortable with the idea of the canonical version of anything I make not sitting on storage that I control — people who do that are not thinking hard enough in my opinion — which means that no source-hosting site based on a centralised source control system was at all interesting.

Send in the clowns

And then git arrived, and sometime after it, GitHub. Git meant that I could both keep the canonical versions of things somewhere safe while making them publicly available. At about the same time I was realising that vanity sites were, well, vanity, and the antique documentation written in raw HTML with its origins in the mid 1990s really needed to be turned into something better, anyway.

Well, it was more complicated, of course. I have a source tree which contains history that’s not public and some of which is very old¹. So after initially pushing a frozen version of what used to be on my vanity site, I organised things into various ‘publication repos’ which get populated (by make) from the source tree. These have simplified histories, often really only being a series of release versions. And this worked fine.

I never really used much of the mechanism provided by GitHub: I didn’t need it, and doing so would have broken my rule of not depending on things I don’t have control over.

In 2018 Microsoft bought GitHub, and

THIS IS WHAT HAPPENS WHEN YOU STORE YOUR DATA IN THE CLOWN.

The Clown is just someone else’s computer and they can and will fuck you. If it’s not on your computer, it’s not under your control. Why do you all keep doing this to yourselves??

Stop hitting yourself. Seriously, stop it.
— jwz

But I wasn’t worried, because I was not storing my data in the clown: I was just publishing copies of it there. I felt kind of smug about that. I hadn’t worked out, in 2018, how bad things could get and how fast. I had worked out that if you were not paying the clown, then the clown was feeding on your soul, but I didn’t think it would get any useful nutrition from the parts of my soul it could reach. And other things were going on which distracted me.

Fast forward to mid–2025, and GitHub is a subsidiary of Microsoft’s ‘weaponized bullshit’ division: it is time, and past time, to go.

Where? Well there are other clowns. But no: they may be good clowns, for now, but they’re still clowns. Clowns are, well, clowns: you don’t want to be trusting them. You want a minimum clown policy.

Where are the clowns?

So here’s what I am doing.

If you can live with git’s dumb HTTP protocol then any web server which can host static files can host read-only git repos. To push to them you need some smarter protocol, and you need to have a hook which does git update-server-info after any change: this should be in the post-update hook, really. All these repos will be read-only and public, so I don’t need access control or authentication.

To actually make them exist you need more. Fortunately the people who host my vanity domain provide SSH access and git exists on the server. So that’s enough to create a repo, add the hook and then push everything to it as and when I need to.

For documentation: I am already using Frog for, for instance, this post. And Frog has the nice feature that you can just add Markdown files in a directory (or any subdirectory) it knows about and it will turn them into HTML. So I wrote a makefile to copy README.md files from publication repos and suitably munged the original GitHub Pages index page.

Then there were a bunch of small changes such as telling rsync not to clobber the repos when updating things, gluing makefiles together so suitable things happen in subdirectories and make publish does everything needed.

What this needs is a web server that can host static files with enough storage (which is not a great deal), and a way of syncing local content to it. SSH access, git on the server and rsync make things quite a bit easier but I could maintain the public repos locally and sync them to the web server however it allowed that.

I’m pretty confident that any other hosting provider I might one day switch to² will provide that.

Well, maybe next year

If I was a big player I think I would simply delete my GitHub repos and put up a message saying where people could now find things. I’m not a big player, so what I did is to make the repos read-only (GitHub calls this ‘archiving’) and add suitable links. I’ve done this for all my main Lisp repos: I have some more to do. Archiving means people may never notice: so be it. I plan to tell, for instance, Quicklisp in due course. I also have some non-public (clones of) repos on GitHub which I will just host outside the web server tree, accessing them only via SSH.

There are, inevitably, some changes to the Markdown parsing, which I need to chase down and fix in the source. There already were incompatibilities amongst GitHub’s various parsers.

When I’ve finished all this, I probably will replace my GitHub Pages site by something which just points at the canonical place.

There will be no releases: I’ll just post things announcing changes with tag names. There is no big tracking: just mail me.

Where things are

tfeb.org/computer/ is the list of things;
tfeb.org/computer/repos/ is where the repos are;
tfeb.org/fragments/documentation/ is the list of documentation.

And that is all.

Isn’t it rich?
Isn’t it queer,
Losing my timing this late
In my career?
And where are the clowns?
There ought to be clowns.
Well, maybe next year.

It turns out I never did import all the CVS versions of things, although I meant to. ↩
Pair, my current provider, are fine. But they are also American, and I’d rather use a provider based in a democracy. ↩

The modern real programmer

2025-01-31T19:17:18Z

This is adapted from an email from my friend Zyni, used with her permission. Don’t take it too seriously.

Real programmers do not write programs like this. If a real programmer has to deal with a collection of particles, they do not have some silly object which represents a particle, perhaps made up of other objects representing physical vectors, and then some array of pointers to these particle objects. That is a bourgeois fantasy and the people who do that will not long survive the revolution. They will die due to excessive pointer-chasing; many of them have already died of quiche.

Real programmers do today as they have always done: if they have some particles to simulate a galaxy they make an array of floating point numbers, in which the particles live.

This is how it has always been done, and how it always will be done, by people who care about performance.

And this is why Lisp is so superb. Because you can write this:

(for* ((i1 (in-particle-vector-indices pv))
       (i2 (in-particle-vector-indices pv i1)))
  (declare (type particle-vector-index i1 i2))
  (with-particle-at (i1 pv :name p1)
    (with-particle-at (i2 pv :name p2)
      (let/fpv ((rx (- p2-x p1-x))
                (ry ...)
                ...)
        ... compute interactions ...))))

And this is:

very fast¹, because it all turns into optimized loops over suitable (simple-array double-float (*)) with no silly objects or consing;
relatively easy for a human to read, since you can see, for instance what (for ((i (in-particle-vector-indices v))) ...) is doing and do not have to second-guess some idiot loop form which will be full of obscure bugs;
quiche-compatible: you can easily write a function particle-at which will construct a particle object from a particle vector entry (such a function will later be excised as it has no callers, of course);
perhaps most important it is possible for a program to take this code and to look at it and to say, ‘OK, this is an iteration over a particle vector – it is not some stupid hard-to-parse (loop for ... oh I have no idea what this is ...) as used by the quiche people, it is (for ((i (in-particle-vector-indices v))) ...) and it is very easy to see what this is – and there are things I can do with that’ and generate Fortran which can be easily (or, less difficultly — is ‘difficultly’ a word? English is so hard) be made to run well on proper machines with sensible numbers of processors.

And this is the thing they still do not see. You write your program which uses the only useful data structure, but you also write your program in a language you have built designed so that both a human and another program can understand it, and do useful things with it, because your program says what it means. Every construct in your program should be designed so that this other program can get semantic information from that construct to turn it into something else.

And this is why Lisp is so uniquely useful for real orogrammers. Lisp has only one interesting feature today: it is a language not for writing programs, but for writing languages.

That is what real programmers do: they build languages to solve their problems. The real programmer understands only two things:

the only data structure worth knowing about is the array;
her job as a programmer is to write languages which will make writing programs to manipulate arrays easy for a human to understand;
and her other job is to write other programs which will take these programs and turn them into Fortran;
and when that is done she can go and ride her lovely cob to the fair.

Real programmers also can count only to two.

I (Tim, not Zyni, who would use a cleverer integrator) wrote a mindless program to integrate systems of gravitating particles to test some of the things we’ve written that are mentioned in this email. On an Apple M1 it sustains well over 1 double precision GFLOP. Without using the GPU I think this is about what the processor can do. ↩

Wild pathnames in Common Lisp

2024-08-18T17:00:02Z

Common Lisp’s pathname system has many problems. Here is proposal to make the situation a little better in one respect. This is not a general fix: it’s just trying to solve one problem.

The problem

The underlying problem is that on many platforms pathnames which ‘look like’ they contain wildcards are perfectly legal pathnames to the filesystem. So, on Unix & related systems [foo].* is a legal filename. On these platforms wildcard handling is generally implemented in a library, or often in multiple semi-compatible libraries¹.

CL then has two problems:

there is no portable way to construct pathnames which look wild but are not;
there is no portable way to parse a string which looks like a wild pathname but in fact should not be interpreted as one, for instance a string coming from some other application or library, or a filename stored in some file, such as an archive.

(1) happens because 19.2.2.3 says, in part

When examining wildcard components of a wildcard pathname, conforming programs must be prepared to encounter any of the following additional values in any component or any element of a list that is the directory component: […] A string containing implementation-dependent special wildcard characters. […]

That means that implementations are allowed to represent wildcard components of pathnames as strings, and that means that you can’t portably construct a non-wildcard pathname.

(2) happens because there’s no way to tell parse-namestring or pathname that the string you’ve handed to them is not wild, even though it looks like it is. That in turn means that to deal with this case you need to either write or find a pathname-parsing library which doesn’t have this problem.

These problems arise in practice: for instance some programs create filenames which look like [foo].xml: SBCL at least parses strings like this as wild, as it is allowed to do. This then breaks programs which want to, for instance, process zip files, tar files or other archive formats.

A proposed solution

For (1) change 19.2.2.3 to say that wildcard components are never strings. Change the description of make-pathname to say that if the corresponding components to it are strings (or suitably-constrained lists for the directory component) then the pathname is not wild, except if the default provides a component which is wild.

For (2) add an extra argument to both parse-namestring and pathname named wild with a default of true. If given as nil this will force string parsing to construct a non-wild pathname. If that is not possible, such as when pathname is handed a pathname which is already wild, then an error will be signalled.

Notes

This is the smallest change I can think of which will solve the problem. Some implementations, SBCL for instance, already solve (1) in the suggested way. None, I think, solve (2).

For added value, it might be useful to specify that wildcard components can be given either as symbols or as lists whose first element is a symbol, and encourage implementations to return them as such if possible. So, for instance (:sequence "foo-" (:alternation "bar" "zap")) might represent a wild name which matches "foo-bar" and "foo-zap". I am not suggesting this particular notation however.

Let me introduce you to the joys of Unix. ↩

The abominable shadow

2024-08-01T10:55:11Z

Most uses of shadow and shadowing-import in Common Lisp packages point to design problems.

Let’s assume you are designing a language which is going to be a variant CL: most of it will be just CL, but perhaps some things will be different. For example, let’s imagine that you want if to have a mandatory else clause. You might start by designing your package like this:

(defpackage :my-language
  (:use :cl)
  (:shadow #:if)
  (:export #:if))

(in-package :my-language)

...

(defmacro if (test then else)
  `(cl:if ,test ,then ,else))

...

That all seems fine, right? Well, not so much. Consider for a minute people who want to use your language. They need to write something like this:

(defpackage :my-language-user-package
  (:use :cl :my-language)
  (:shadowing-import-from :my-language #:if))

(in-package :my-language-user-package)

...

‘Oh well’, you say, ‘that’s not so bad’. Well, now let’s say you want to add a version of cond to your language which understands else and otherwise. So:

(defpackage :my-language
  (:use :cl)
  (:shadow #:if #:cond)
  (:export #:if #:cond #:else))

(in-package :my-language)

...

(defmacro if (test then else)
  `(cl:if test then else))

(defmacro cond (&body clauses)
  `(cl:cond
    ,@(mapcar (lambda (clause)
                (if (and (consp clause)
                         (member (first clause) '(else otherwise)))
                    `(t ,@(rest clause))
                  clause))
              clauses)))

...

And now every user of your language has to modify their package definitions:

(defpackage :my-language-user-package
  (:use :cl :my-language)
  (:shadowing-import-from :my-language #:if #:cond))

(in-package :my-language-user-package)

...

I’ll say that again: every user of your language has to modify their package definitions every time you enhance it in a way which is not compatible with CL.

That … sucks. It’s an absolutely terrible design. Wouldn’t it be nice if it could be avoided?

It can. Rather than shadowing symbols, you can instead construct the packages you actually would like to exist. In the example above what you probably want people to be able to do is to say

(defpackage :my-language-user-package
  (:use :my-language))

and have that work, even when your language changes. So, you need the MY-LANGUAGE package to export most of the symbols from CL as well as a few of its own. You can do this by hand:

(defpackage :my-language
  (:use)
  (:export #:if #:cond #:else)
  (:import-from :cl
   cl:&allow-other-keys ...)
  (:export
   cl:&allow-other-keys ...))

Where the :import-from and the second :export clause specify all the symbols from CL except those which are replaced by ones defined by your language.

Note the empty :use clause: this avoids symbol clashes and therefore the need to shadow things.

You can then either define your language in this package or in an implementation package which uses it: the package has imported all of the external symbols from CL other than the ones it overrides, so it doesn’t need to use the CL package at all.

The benefit of doing things this way is that it means that every user of this system doesn’t have to care about the details of it and isn’t forced to change their code because of implementation changes. That’s worth it, even though writing the defpackage forms is laborious: you should do the work, not every user of your systm.

Of course, in real life you would not have to remember the names of all the symbols you are reexporting: you’d write a program to do it for you. You’d write, in fact, a macro.

Well other people have already done that for you, in particular I did this in 1998 when I decided that this idea was interesting. Other people have since done similar things I think and may have done so before me, but I will describe my version: conduit packages. In particular I’ll mostly describe the functionality exported from the ORG.TFEB.CONDUIT-PACKAGES/DEFINE-PACKAGE package, which doesn’t replace macros like defpackage and functions like export, but rather provides functionality under different names.

The basic notion is that packages can be conduits for one or more other packages: they serve to gather together and reexport subsets of the exported names from the packages for which they are conduits. define-package lets you define conduit packages easily, and define-conduit-package is even more specialised to the task.

Here is how you would define the package above

(define-package :my-language
  (:use)
  (:export #:if #:cond #:else)
  (:extends/excluding :cl
   #:if #:cond))

or with define-conduit-package:

(define-conduit-package :my-language
  (:export #:if #:cond #:else)
  (:extends/excluding :cl
   #:if #:cond))

Now you can quite happily define your language as before.

This works, of course, even if your package wants to extend other packages whose exports might change in a way that CL’s are unlikely to do any time soon: the symbols to import & reexport are computed based on the state of the package system at the time the form is evaluated. In some cases — if the package you are extending is itself known to the system — the packages will be dynamically recomputed:

> (define-package :foo
    (:export #:one))
#<The FOO package, 0/16 internal, 1/16 external>

> (define-conduit-package :bar
    (:extends :foo))
#<The BAR package, 0/16 internal, 1/16 external>

> (do-external-symbols (s :bar (values)) (print (symbol-name s)))

"ONE"

> (define-package :foo
    (:export #:one #:foo))
Warning: Using DEFPACKAGE to modify #<The FOO package, 0/16 internal, 1/16 external>.
#<The FOO package, 0/16 internal, 2/16 external>

> (do-external-symbols (s :bar (values)) (print (symbol-name s)))

"FOO"
"ONE"

And thus was the abominable shadow cast into the outer darkness.

A remaining question is: are there good uses for shadowing? Well, conduit packages itself uses them in its implementation package, mostly because I was too lazy to write the code which would explicitly map over CL. And there must, I suppose, be other good uses, but it’s very hard to think of them. The other common case, where you want to use two packages which export the same names, is dealt with by simply using a conduit of course.

I think it’s worth remembering that when the CL package system was initially defined, people didn’t really understand how such a thing should work. MACLISP didn’t have a package system, Lisp Machine Lisp probably did (certainly Zetalisp did), but there was no great experience with what a package system should be like. Indeed the first CL version didn’t have defpackage: instead you had to construct packages by hand, and there were all sorts of weirdnesses in the way the compiler handled make-package and other package functions (or you had to use eval-when all over the place).

Finally, when I wrote conduit packages I was still thinking that packages were big expensive objects, because in the late 1980s they were, and I hadn’t yet realised that this was no longer true. In the late 1980s a big workstation on which you ran CL might have had 16MB of memory. Today laptops have perhaps a thousand times as much memory: data structures which ate a lot of precious memory in 1990 don’t any more. So I think, today, it’s appropriate to use packages in a fairly fine-grained way: having a few extra packages really is not hurting you very much.

So here is another way to define the little language above.

First, define a conduit for CL which exports just the symbols you want:

(define-conduit-package :my-language/cl
  (:extends/excluding :cl
   #:if #:cond))

Now define the implementation package for the language: this exports the new symbols:

(define-package :my-language/impl
  (:use :my-language/cl)
  (:export
   #:if #:cond #:else))

Now, finally, define the public package, which is a conduit for both MY-LANGUAGE/CL and MY-LANGUAGE/IMPL:

(define-conduit-package :my-language
  (:extends :my-language/cl :my-language/impl))

This is absurd overkill in this tiny example, but for real examples, where there might be several implementation packages, it lets you split things up in a nice way, while not burdening your users with lots of tiny packages.

Monochrome

2024-05-20T08:58:05Z

Or, why limitations matter.

After we released Štar, my friend Zyni had a rather distressing and inconclusive exchange with someone on reddit. Apart from making me feel even better about walking away from reddit a decade and more ago, I realised an interesting thing when talking to her about her experience.

Limitations

Here are some curious things that people do.

Why, in 2024, would anyone use a digital camera which can only make monochrome pictures?
Why, in 2024, would anyone use a film camera?
Why, in 2024, would anyone record music in a studio which uses tape?
Why, in 2024, would any guitarist use a collection of flaky old FX pedals connected by noisy and unreliable cables, and a valve amplifier which occasionally catches fire¹?
Why, in 2024, would anyone shoot a movie on film?

Yet people do all these things: there are at least two manufacturers of dedicated monochrome cameras, and you can pay to have your colour digital camera converted to monochrome by removing the filter array; many people use film; studios which use tape still exist and probably are becoming more common; too many guitarists to count use elderly FX pedals, a nest of cables and a valve amplifier; many very famous recent movies have been made on film.

There is, of course, a lot of myth and lore about the special magic properties of these things: people go on endlessly about how monochrome digital sensors have more effective resolution than Bayer sensors, as if anyone needs more resolution today, or how the effective sensitivity is higher, as if anyone needs more sensitivity today. The same for film (no, it’s not magic, no, it’s not somehow better than digital in any objective way), analogue recording, old FX pedals and amplifiers, and movies made on film. If you really think movies made on film are somehow better or ‘more natural’, watch The Holdovers, which was shot digitally but is a really beautiful simulacrum of what movies looked like in about 1972.

The awful truth is that none of these technologies get you anything objectively better, even if what you want to do is reproduce what people did when those technologies were all there was. You can simulate film so well it is impossible to tell the difference, you can simulate analogue tape just as well. Modern digital FX/amplification systems for guitar are a wonder. You can produce really beautiful monochrome images from colour files.

So why, really, do people do these things?

The clue, for me, is in the first one: why would anyone use a dedicated monochrome digital camera? Why would I use one if I could afford to? The answer, for me and others, is because it restricts what you can do. If your camera will only do monochrome, then that’s all you can do, and somehow that matters. I can’t afford a monochrome-only digital camera, but I do use a digital camera which has a monochrome-only workflow (this is why I bought it, in fact): the camera sets a ‘monochrome’ flag in its files which the raw-conversion tool understands, and unless you work hard you never see a colour version of the photographs you’ve taken. And this matters to me: it needs to be difficult to overcome the limitation, so I can think in monochrome².

And of course I use film as well, and mostly black and white film with some recent excursions into colour reversal (slide) film. Partly I do this because I enjoy working in the darkroom and the tactile quality of film cameras from the 1980s and before, but more important, I think, is how limiting film is. Once you’ve taken a picture, it’s there: you can’t chimp and decide to do it again. You have 36 (or 10, or 1) exposures before you have to change film: every frame matters. Film can’t see in the dark. Mostly it can’t even see in colour (reversal film is now absurdly expensive). 35mm film is always going to have grain. Making photographs using film is a festival of limitations.

It turns out that, for many people, limitations matter. If you take them away, then suddenly you can do anything. So you spend your time fiddling with the parameters of what you can do, and doing nothing as a result, rather than being forced to pull your finger out and do something.

That’s why many people make photographs with monochrome cameras, or on film. That’s why people still make movies on film. That’s why people paint or draw, rather than using some vast digital image-creation system. That’s why people write books on typewriters, or with a pen. That’s why Colin Chapman designed cars the way he did, and why more than 160 companies have made (and still do make) replicas of one of his designs.

Not everyone: not even most people. But some people. Enough people.

And it’s not enough to simply say ‘I won’t use all the features I do not want’: those features need not to be there. Using a camera which can only make monochrome images is not like using a camera which can make colour images but choosing not to. Choosing not to chimp is not like being unable to chimp. Using a film camera is not like using a digital camera and film simulation. Drawing with a pencil on paper is not like using a drawing program but choosing only to use the pencil tool, however good that tool is. Limitations have to be, well, limitations. I don’t know why this is, but they do.

Štar

And that’s why Štar exists: because it’s limited, because it does one thing. Because there is as little syntax as we could make there be. In Štar everything is an iterator³ so any expression like

(for ((<var/s> <iterator>))
  ...)

can be turned into

(multiple-value-bind (v c) <iterator>
  (for ((<var/s> (values v c)))
    ...)

And it will mean exactly the same thing, although often it will be slower⁴. And all clauses are like that. There is only one case: the form on the right-hand side of a clause is a perfectly general expression evaluated, once, in the way you think it will be. There is no magic syntax, at all. And because Štar does only one thing — iterate — it forces you to use other tools to do other things, and to make sure all these tools compose well with each other. There was once a famous operating system whose designers played the same trick.

Richard Gabriel’s famous essay, usually known as Worse is better, is celebrated for describing the difference between right thing systems and worse is better systems⁵. It is less celebrated for another distinction it made amongst the right thing systems. That distinction is between big complex systems and diamond-like jewel systems:

The big complex system scenario goes like this:

First, the right thing needs to be designed. Then its implementation needs to be designed. Finally it is implemented. Because it is the right thing, it has nearly 100% of desired functionality, and implementation simplicity was never a concern so it takes a long time to implement. It is large and complex. It requires complex tools to use properly. The last 20% takes 80% of the effort, and so the right thing takes a long time to get out, and it only runs satisfactorily on the most sophisticated hardware.

The diamond-like jewel scenario goes like this:

The right thing takes forever to design, but it is quite small at every point along the way. To implement it to run fast is either impossible or beyond the capabilities of most implementors.

The two scenarios correspond to Common Lisp and Scheme.

Štar aspires to be the diamond-like jewel of iteration frameworks. Its interface is tiny: org.tfeb.* exports six symbols, four of which could be removed with no loss of functionality⁶. But it aims to be general and to be able to turn this clean, minimal syntax into fast code: apart from the catalogue of built-in iterators, almost all the rest of the interface is to the tools that let you do this for your own iterators.

It may seem odd that Štar is written in Common Lisp, not Scheme, but CL is what we actually use and so producing tools which turn CL into the system we’d like to have is what we care about. In my case, I also remember the movement to reexpress CL as a small core language combined with libraries.

Štar is not for everybody. If you like loop you probably will hate it (but you have no taste, so we don’t care). If you are an adherent of the big complex system school you probably won’t like it either (and we respect your opinion). That’s not who Štar is for: Štar is for the people who appreciate the diamond-like jewel, who believe that limitations matter, in a deep way.

If that’s not for you, that is completely fine: not everybody is the same: not everyone wants a monochrome camera, and not everyone knew, or liked if they did know, that famous operating system⁷. Some people even like loop, apparently. But understand that it is for some people, and try to avoid sneering at them, if you can.

Well-maintained valve amplifiers hardly ever catch fire. Often they are also electrically safe. ↩
Why a photographer or a movie maker would want to work in monochrome is another, related question: why, in 2024, intentionally throw away all the colour information in a scene? ↩
In this way, Štar is like some Scheme compilers which turn everything into $\lambda$ expressions and then try to make those expressions quick. ↩
One of the things Štar’s tests do is exactly this transformation as a way of testing that iterator optimizers do not change the semantics of the program. ↩
Entertainingly, that famous operating system is often disparaged as an example of worse is better. In terms of its implementation that may be justified: its ideas are certainly not examples of worse is better. ↩
We have argued, at length, about this. We decided that, just as Scheme includes let and, even more extravagantly, let*, Štar should include final, for* and final*. In the language of the Scheme reports, they are library syntax (for*) and library procedures (final, final*, next* which serves no purpose without for*). ↩
There’s a book about that. ↩

Štar: an iteration construct for Common Lisp

2024-05-15T06:37:18Z

Štar is a concise and extensible iteration construct for Common Lisp which aims to be pleasant to use, easy to understand, fast if needed, general, and not to look like Fortran.

Common Lisp has multiple iteration constructs: mapping functions such as mapcar, special-purpose constructs such as dotimes and dolist, the general but somewhat clumsy construct which is do and do*, and finally the extended loop macro which aims to embed a ‘more friendly’ iteration language into CL and succeeds in being so complex that it is often hard to know whether a given form is legal or not without poring over loop’s grammar.

None of these constructs manage to be all three of pleasant to use, easy to understand and general. loop somehow fails to be any of these things in many cases. None are extensible¹.

But Common Lisp is a Lisp, and Lisp’s huge advantage is that it is a programming language in which it is easy to write programming languages, or parts of them, like iteration constructs. That is, after all, how most or all of the existing constructs started life.

Lots of these have been written, of course. Štar tries to distinguish itself by being as simple as possible: it has as little special syntax as I could work out how to give it – there is no special little language you need to learn. It also has no inherent knowledge about how to iterate over any particular structure: it doesn’t know how to iterate over lists, or ranges of numbers. Rather it knows that iterating has to answer two questions:

is there more?
what is the next thing?

In addition it knows how to ask another question:

is there any information I can use to make asking the first two questions faster?

What it looks like

(for ((e (in-list l)))
  (print e))

(for (((k v) (in-hash-table h)))
  ...)

(for* ((entry (in-list entries))
       (element (in-list entry)))
  ...)

(defun in-alist (alist)
  (values
    (lambda () (not (null alist)))
    (lambda ()
      (destructuring-bind ((k . v) . more) alist
        (setf alist more)
        (values k v)))))

(for (((k v) (in-alist ...)))
  ...)

These are some simple examples: the last shows how easy it is to teach Štar to iterate over new things, or over existing things in new ways. Not shown here is that it’s also pretty easy to teach it how to optimize new iterators and to make various declarations about things.

What Štar is not

Štar is an iteration construct: what it does is to iterate. It has nothing to do with collecting values. For Štar, iteration and value accumulation are orthogonal problems which should be solved by orthogonal constructs. In particular if you wanted to make a list of the even numbers from a list, you might to this by using Štar together with a value-collection macro:

(collecting
  (for ((e (in-list ...)))
    (when (evenp e)
      (collect e))))

This is yet another way in which Štar differs from loop and many other constructs. In particular Štar’s point of view is that mixing together iteration and value accumulation results in a system that is not very good at either.

Similarly, Štar doesn’t contain a mass of syntax letting you select only certain values, or allowing you to terminate iteration early: you don’t write

(loop for x in l while (numberp x) do ...)

Instead you write

(for ((x (in-list l)))
  (unless (numberp x) (final))
  ...)

The body of an iteration is exactly like the body of defun, except there are some local functions which you can call to skip to the next iteration or finish the iteration.

Štar, of course, doesn’t bundle some destructuring system which will inevitably be subtly incompatible with other destructuring systems while also not being usable independently. If you want destructuring, use a full-fat system of your choice.

You probably get the idea: Štar is a tool whose job is to iterate: not some leaking bag of broken abstractions.

Multiple values

One thing that Štar does do is to take multiple values seriously. A clause which specifies a list of variables will bind them to the multiple values returned by the iterator. Multiple values, unlike destructuring, are something you really have to have in the iteration construct itself.

The thing that doesn’t really matter but everyone cares about

So, OK, Štar is an extensible, general, iteration construct. Obviously it will have traded performance for all this. I mean, it’s the old Lisp story, the one Gabriel told us long ago. Right?

* Running benchmarks
** lists of length 2000, nesting 3
what                                                  seconds      ratio
star                                                   30.312      1.000
loop                                                   30.071      0.992
dolist                                                 21.594      0.712
** range 100000, nesting 2
what                                                  seconds      ratio
star/with-step                                          9.414      1.000
star/no-step                                            9.406      0.999
loop                                                   18.412      1.956
dotimes                                                 9.469      1.006

A sketch of Štar

Štar has three parts, four if you count the iterators:

the iteration constructs themselves and bindings they make;
a protocol for defining new iterators;
a protocol for defining optimizers for iterators;
a collection of predefined iterators and optimizers for them.

The first three parts are much more finished than the fourth: most of the existing iterators were written as proofs of concept and may well change, get better, or go away.

Iterators

These are forms (usually, named function calls) which return two values, both functions of no arguments:

the first value is called and should return true if there is more to do;
the second value is called to return the next value or values of the iterator.

These functions answer the first two questions Štar needs to ask: is there more, and if there is, what is it? These two functions obviously share state in general.

To answer the third question – how do I make things faster? – named iterator functions can have optimizers: these are functions called by Štar at macroexpansion time which tell it how to make things faster. It’s up to an optimizer to ensure the semantics are the same for the optimized and unoptimized versions. Optimizers can specify a set of bindings to make, declarations for them, how to iterate, and some other things.

It’s possible to install and remove optimizers, and to dynamically bind sets of them. This might be useful, for instance, to compile a file where some particular assumptions (‘all vectors are vectors of floats’) are true.

Iteration constructs

There are two:

for iterates in parallel;
for* defines nested loops: (for* ((...) (...)) ...) is like (for ((...)) (for ((...)) ...)).

Note that because of the way iterators work, sequential binding within one loop makes no sense.

The first argument of for or for* specifies a number of clauses: each clause is of the form <var/s> <iterator>). Multiple values are supported, and it is possible to make various declarations about variables: this matters for for*, where there is no room for declarations for other than the last (innermost) clause. Variables whose name is "_" are ignored by default.

Within the body of an iteration there are four local functions:

next skips to the next iteration;
next* skips to the next outer-level iteration for for* and is the same as next for for;
final returns zero or more values from the current iteration
final* returns zero or more values for the outer-level iteration for for* and is the same as final for for.

Provided iterators

There are iterators which work for lists, vectors, general sequences, hash tables, ranges of reals, as well as some more interesting ones. Not all iterators currently have optimizers. You only need to care about writing optimizers if you need to make iterators very fast: they can be significantly fiddly to write.

The iterator over ranges of reals has an optimizer which tries hard to make things pretty fast.

As well as this there are some more interesting iterators. An example is sequentially:

(for ((a (sequentially 1 2)))
  ...)

will bind a sequentually to 1 and 2. But this iterator is a macro which has the behaviour of a FEXPR, so it evaluates its arguments only when needed (and as many times as needed). A variant of sequentially is sequentially* which ‘sticks’ on its last argument. So for instance:

> (let ((v 0))
    (for ((a (sequentially* (incf v)))
          (_ (in-range 3)))
      (print a)))

1 
2 
3

Another way to write this is:

> (for ((a (let ((v 0)) (sequentially* (incf v))))
        (_ (in-range 4)))
    (print a))

1 
2 
3 
4

And of course there is a meta-iterator (also implemented as a macro), in-iterators:

> (for ((a (in-iterators
            (in-list '(1 2))
            (in-list '(3 4)))))
    (print a))

1 
2 
3 
4

It’s possible to construct very general iterators with tools like this.

As I said above, Štar’s current iterators are in a fairly rough state: a lot of this might change.

Notes

Declarations

A form like

(for* ((a (in-range 10))
       (b (in-range 10)))
  ...)

corresponds roughly to

(for ((a (in-range 10)))
  (for ((b (in-range 10)))
    ...)

The problem is now how to make declarations which apply to a. This is hard because CL doesn’t provide the tools you need to know whether a declaration refers to a variable or not: to know whether (declare (foo a)) refers to a or not you need to know, at least, whether foo names a type, which you can’t do. You often can guess, but not always.

So, rather than trying to solve an intractable problem, Štar lets you specify some properties of a variable in the clause that binds it: you can say

(for* (((a :type fixnum) (in-range 10))
       (b (in-range 10)))
  ...)

for instance. This is a bit ugly, but it solves the problem. It is only useful for for*.

Binding

Štar binds, rather than assigns:

(collecting
  (for ((a (in-list 1 2)))
    (collect
      (lambda (&optional (v vp))
        (if vp (setf a v) a)))))

will return two closures over independent bindings.

Epilogue

Štar’s source code is here. The manual is included with the source code but also here.

Štar is pronounced roughly ‘shtar’.

Much of the inspiration for Štar came from my friend Zyni: thanks to her for the inspiration behind it, actually making me write it and for many other things.

Štar is dedicated to her, and to Ian Anderson.

Some implementations have mechanisms for extending loop. ↩

Symbol nicknames: a broken toy

2023-10-12T14:08:27Z

Symbol nicknames allows multiple names to refer to the same symbol in supported implementations of Common Lisp. That may or may not be useful.

People often say the Common Lisp package system is deficient. But a lot of the same people write code which is absolutely full of explicit package prefixes in what I can only suppose is an attempt to make programs harder to read. Somehow this is meant to be made better by using package-local nicknames for packages. And let’s not mention the unspeakable idiocy that is thinking that a package name like, say, XML is suitable for any kind of general use at all. So forgive me if I don’t take their concerns too seriously.

The CL package system can’t do all the things something like the Racket module system can do. But it’s not clear that, given its job of collecting symbols into, well, packages, it could do that much more than it currently does. Probably some kind of ‘package universe’ notion such as Symbolics Genera had would be useful. But the namespace has to be anchored somewhere, and if you’re willing to give packages domain-structured names in the obvious way and spend time actually constructing a namespace for the language you want to use, it’s perfectly pleasant in my experience.

One thing that might be useful is to allow multiple names to refer to the same symbol. So for instance you might want to have eq? be the same symbol as eq:

> (setf (nickname-symbol "EQ?") 'eq)
eq

> (eq 'eq? 'eq)
t

> (eq? 'eq 'eq?)
t

This allows you to construct languages which have different names for things, but where the names are translated to the underlying name efficiently. As another example, let’s say you wanted to call eql equivalent-p:

> (setf (nickname-symbol "EQUIVALENT-P") 'eql)
eql

> (eql 'eql 'equivalent-p)
t

Well, now you can use equivalent-p as a synonym for eql wherever it occurs:

> (defmethod foo ((x (equivalent-p 1)))
    "x is 1")
#<standard-method foo nil ((eql 1)) 801005BD23>

> (foo 1)
"x is 1"

Symbol nicknames is not completely portable as it requires hooking string-to-symbol lookup. It is supported in LispWorks and SBCL currently: it will load in other Lisps but will complain that it can’t infect them.

Symbol nicknames is also not completely compatible with CL. In CL you can assume that (find-symbol "FOO") either returns a symbol whose name is "FOO" or nil and nil: with symbol nicknames you can’t. In the case where a nickname link has been followed the second value of find-symbol will be :nickname.

Symbol nicknames is a toy. I am not convinced that the idea is even useful, and if it is it probably needs to be thought about more than I have.

But it exists.

A horrible solution

2023-05-04T11:33:41Z

Yesterday I wrote an article describing one of the ways traditional Lisp macros can be unhygienic even when they appear to be hygienic. Here’s a horrible solution to that.

The problem I described is that the expansion of a macro can refer to the values (usually the function values) of names, which the user of the macro can bind, causing the macro to fail. So, given a function

(defun call-with-foo (thunk)
  ...
  (funcall thunk))

Then the macro layer on top of it

(defmacro with-foo (&body forms)
  `(call-with-foo (lambda () ,@forms)))

is not hygienic so long as local functions named call-with-foo are allowed:

(flet ((call-with-foo (...) ...))
  (with-foo ...))

The sensible solution to this is to say, just as the standard does about symbols in the CL package that you are not allowed to do that.

Here’s another solution:

(defmacro with-foo (&body forms)
  `(funcall (symbol-function 'call-with-foo) (lambda () ,@forms)))

This is robust against anything short of top-level redefinition of call-with-foo. And you can be mostly robust even against that:

(defmacro with-foo (&body forms)
  `(funcall (load-time-value (symbol-function 'call-with-foo))
            (lambda () ,@forms)))

This still isn’t safe against really malignant users, since the load time of the macro’s definition and its uses are not generally the same. But it’s probably fairly good.

I hope I never feel I have to use techniques like this.

Two sides to hygiene

2023-05-03T11:28:09Z

It’s tempting to think that by being sufficiently careful about names bound by traditional Lisp macros you can write macros which are hygienic. This is not true: it’s much harder than that.

Hygienic macros

I do not fully understand all the problems which Scheme-style hygienic macros try to solve, and the implementation of the solutions is usually sufficiently difficult to understand that I have always been put off doing so, especially as the details of the implementation in Racket, the Scheme-related language I use most, seems to change every few years. I’m happy enough that I am mostly competent to write the macros I need in Racket, without understanding the details of the implementation.

Traditional Lisp macros are, to me, far more appealing because they work in such an explicit and simple way: you could pretty easily write a macroexpander which did most of what the Common Lisp macroexpander does, for instance. I have written several toy versions of such a thing: I’m sure most Lisp people have. Traditional Lisp macros are just functions between bits of language expressed explicitly as s-expressions: what could be simpler?

In fact I am reasonably confident that, if I had to choose one, I’d choose CL’s macros over Racket’s: writing macros in raw CL is a bit annoying because you need explicit gensyms and you need to do pattern matching yourself. But you can write, and I have written tools to make most of this go away. With these, writing macros in CL can often be very pleasant. And it’s easy to understand what is going on.

What is far harder though, is to make it completely hygienic. Here’s one reason why.

Several versions of a macro in Common Lisp

Let’s imagine I want a macro which allows you to select actions based on the interval a real number is in. It might look like this:

(interval-case x
  ((0 1) ...)
  ((1) 2) ...)
  (otherwise ...))

Here intervals are specified the way they are in type specifiers for reals:

(a b) where a and b are reals means $[a,b]$;
((a) b) where a and b are reals means $(a,b]$;
and so on.

There can be only one interval per clause, for simplicity.

I will write several versions of this macro. For all of them I will use dsm and, later, metatronic macros to make things better.

First of all here’s a function¹ which, given an interval specification, returns a form which will match numbers in that interval:

(defun compute-interval-form (v iv)
  (destructuring-match iv
    (((l) (h))
     (:when (and (realp l) (realp h)))
     `(< ,l ,v ,h))
    ((l (h))
     (:when (and (realp l) (realp h)))
     `(and (<= ,l ,v) (< ,v ,h)))
    (((l) h)
     (:when (and (realp l) (realp h)))
     `(and (< ,l ,v) (<= v ,h)))
    ((l h)
     (:when (and (realp l) (realp h)))
     `(<= ,l ,v ,h))
    (default
     (:when (member default '(t otherwise)))
     t)
    (otherwise
     (error "~S is not an interval designator" iv))))

A hopeless version

Here is a version of this macro which is entirely hopeless:

(defmacro interval-case (n &body clauses)
  ;; Hopeless
  `(cond
    ,@(mapcar (lambda (clause)
                (destructuring-bind (iv &body forms) clause
                  `(,(compute-interval-form n iv) ,@forms)))
              clauses)))

It’s hopeless because of this:

> (let ((x 1))
    (interval-case (incf x)
      ((1 (2)) '(1 (2)))
      ((2 (3)) '(2 (3)))))

So (incf x) where x is initially 1 is apparently neither in $[1,2)$ nor $[2,3)$ which is strange. This is happening, of course, because the macro is multiply-evaluating its argument, which it should not do.

An obviously unhygienic repair

So let’s try to fix that:

(defmacro interval-case (n &body clauses)
  ;; Unhygenic
  `(let ((v ,n))
     (cond
      ,@(mapcar (lambda (clause)
                  (destructuring-bind (iv &body forms) clause
                    `(,(compute-interval-form 'v iv) ,@forms)))
                clauses))))

Well, this is better:

> (let ((x 1))
    (interval-case (incf x)
      ((1 (2)) '(1 (2)))
      ((2 (3)) '(2 (3)))))
((2) (3))

but … not much better:

> (let ((x 1) (v 10))
    (interval-case (incf x)
      ((1 (2)) nil)
      ((2 (3)) v)))
2

The macro binds v, which shadows the outer binding of v and breaks everything.

A repair which might be hygienic

Here is the normal way to fix that:

(defmacro interval-case (n &body clauses)
  ;; OK
  (let ((vn (make-symbol "V")))
    `(let ((,vn ,n))
       (cond
        ,@(mapcar (lambda (clause)
                    (destructuring-bind (iv &body forms) clause
                      `(,(compute-interval-form vn iv) ,@forms)))
                  clauses)))))

And now

> (let ((x 1) (v 10))
    (interval-case (incf x)
      ((1 (2)) nil)
      ((2 (3)) v)))

Good. I think it is possible to argue that this version of the macro is hygienic, at least in terms of names.

A simpler repair using metatronic macros

Here is the previous macro written using metatronic macros:

(defmacro/m interval-case (n &body clauses)
  ;; OK, easier
  `(let ((<v> ,n))
       (cond
        ,@(mapcar (lambda (clause)
                    (destructuring-bind (iv &body forms) clause
                      `(,(compute-interval-form '<v> iv) ,@forms)))
                  clauses))))

This is simpler to read and should be as good.

An alternative approach …

Although it is not entirely natural in the case of this macro, many macros can be written by having the macro expand into a call to a function, passing another function whose body is the body of the macro as an argument. These things often exist as pairs of with-* (the macro) and call-with-* (the function).

We can persuade interval-case to work like that: it’s not a natural macro to write that way and writing it that way will end up with something almost certainly less efficient as (at least the way I’ve written it) as it needs to interpret the interval specifications at runtime rather than compile them². But I wanted to have just one example.

Here is call/intervals, the function layer:

(defun call/intervals (n ivs/thunks)
  ;; Given a real n and a list of (interval-spec thunk ...), find the
  ;; first spec that n matches and call its thunk.
  (if (null ivs/thunks)
      nil
    (destructuring-bind (iv thunk . more) ivs/thunks
      (if (destructuring-match iv
            (((l) (h))
             (:when (and (realp l) (realp h)))
             (< l n h))
            ((l (h))
             (:when (and (realp l) (realp h)))
             (and (<= l n) (< n h)))
            (((l) h)
             (:when (and (realp l) (realp h)))
             (and (< l n) (<= n h)))
            ((l h)
             (:when (and (realp l) (realp h)))
             (<= l n h))
            (default
             (:when (member default '(t otherwise)))
             t)
            (otherwise
             (error "~S is not an interval designator" iv)))
          (funcall thunk)
        (call/intervals n more)))))

As well, here is a ‘nospread’ variation on call/intervals which serves as an impedence matcher:

(defun call/intervals* (n &rest ivs/thunks)
  ;; Impedence matcher
  (declare (dynamic-extent ivs/thunks))
  (call/intervals n ivs/thunks))

Now here’s the macro layer:

(defmacro interval-case (n &body clauses)
  ;; Purports to be hygienic
  `(call/intervals*
    ,n
    ,@(mapcan (lambda (clause)
                `(',(car clause)
                  (lambda () ,@(cdr clause))))
              clauses)))

So we can test this:

> (let ((x 1) (v 10))
    (interval-case (incf x)
      ((1 (2)) nil)
      ((2 (3)) v)))
10

So, OK, that’s good, right? This is another hygienic macro. Not so fast.

… which is not hygienic

> (flet ((call/intervals* (&rest junk)
           (declare (ignore junk))
           86))
    (interval-case 2
      ((1 2) 'two)))
86

Not so hygienic, then.

The alternative approach in Racket

Here is a similar alternative approach implemented in Racket:

(define (call/intervals n ivs/thunks)
  ;; Here ivs/thunks is a list of (iv thunk) pairs, which is not the same
  ;; as the CL version: that's because I can't work out how to do the
  ;; syntax rule otherwise.
  (match ivs/thunks
    ['() #f]
    [(list (list iv thunk) more ...)
     (if
      (match iv
        [(list (list (? real? l))
               (list (? real? h)))
         (< l n h)]
        [(list (? real? l)
               (list (? real? h)))
         (and (<= l n) (< n h))]
        [(list (list (? real? l))
               (? real? h))
         (and (< l n) (<= n h))]
        [(list (? real? l) (? real? h))
         (<= l n h)]
        [(or 'otherwise #t)
         #t]
        [_
         (error 'call/intervals "~S is not an interval designator" iv)])
      (thunk)
      (call/intervals n more))]))

(define (call/intervals* n  . ivs/thunks)
  ;; impedence matcher (not so useful here)
  (call/intervals n ivs/thunks))

(define-syntax-rule (interval-case n (key body ...) ...)
  (call/intervals* n (list 'key (thunk body ...)) ...))

And now:

> (call/intervals* 1 (list '(0 1) (thunk 3)))
3
> (interval-case 2
    ((1 2) 'two))
'two
> (let ([call/intervals* (thunk* 86)])
    (interval-case 2
      ((1 2) 'two)))
'two
> (let ([call/intervals* (thunk* 86)])
    (call/intervals* 2))
86

In Racket this macro is hygienic.

Two sides to hygiene

So the problem here is that there are at least two sides to hygiene for macros:

names they use, usually by binding variables but also in other ways, must not interfere with names used in the program where the macro is used;
the program where the macro is used must not be able to alter what names the macro refers to mean.

In both cases, of course, there need to be exceptions which are part of the macro’s contract with its users: with-standard-io-syntax is allowed (and indeed required) to bind *print-case* and many other variables.

I think almost everyone understands the first of these problems, but the second is much less often thought about.

Dealing with this problem in Common Lisp

I think a full solution to this problem in CL would be very difficult: macros would have to refer to the names they rely on by names which were somewhow unutterable by the programs that used them. Short of actually writing a fully-fledged hygienic macro system for CL this sounds impractical.

In practice the solution is to essentially extend what CL already does. For symbols (so, names) in the CL package there are strong restrictions on what conforming programs may do. This program is not legal CL³ for instance:

(flet ((car (x) x))
  ... (car ...))

So the best answer is then, I think, to:

use packages with well-defined interfaces in the form of exported symbols;
disallow or strongly discourage the use of internal symbols of packages by programs which are not part of the implementation of the package;
and finally place restrictions similar to those placed on the CL package on exported symbols of your packages.

Note that package locks don’t answer this problem: they usually forbid the modification of various attributes of symbols and the creation or deletion of symbols, but what is needed is considerably stronger than that: it needs to be the case that you can’t establish any kind of binding, even a lexical one, for symbols in the package.

Is this a problem in practice? Probably not often. Do I still prefer traditional Lisp macros? Yes, I think so.

This function what you would want to make more complicated to allow multiple intervals per clause. ↩
This interpretation could be avoided by havig the compiler turn the interval specifications into one-argument functions. I think it’s still not a natural way to write this macro. ↩
Assuming that car means ‘the symbol whose name is "CAR" in the "COMMON-LISP" package’. ↩

Nirvana

2023-05-02T13:16:58Z

An article constructed from several emails from my friend Zyni, reproduced with her permission. Note that Zyni’s first language is not English.

Many people have tried to answer what is so special about Lisp by talking about many things.

Such as interactive development, a thing common now to many languages of course, and if you use Racket with DrRacket not in fact how development usually works there at all. Are we to cast Racket into the outer darkness?¹

Such as CLOS, a thing specific to Common Lisp: can you not achieve Lisp enlightenment unless you program in Common Lisp? Was Lisp enlightmenent impossible before CLOS existed? What stupid ideas. Could you implement CLOS in a language which was not Lisp? Certainly you could.

Such as the CL condition system: a thing also specific to Common Lisp. Something also which could be implemented in any sufficiently dynamic language. Something almost nobody who writes in Common Lisp understands I think.

And so it goes on.

None of this is the answer. None of this is close to the answer. To find the answer ask why did these things arise in Lisp first? What is the property of Lisp which is in fact unique to Lisp and which defines Lisp in strict sense that if any other language had this property it would be a Lisp? To see answer to this you must understand Bradshaw’s law and my corollary to it:

Bradshaw’s law. All sufficiently large software systems end up being programming languages.

Zyni’s corollary. At whatever size you think Bradshaw’s law applies, it applies sooner than that.

This means that all programming is language construction.² When you write a program you are writing a language in which to express the problem you wish to solve.

Now you can begin understand what is so interesting about Lisp. In almost all programming languages when you solve a problem you define a lot of new words for the language you have, and perhaps you define elaborate classifications of the nouns of the language you will allow. But you can do nothing with the structure of the language you must use because the language will not allow that: it has a fixed grammar handed down by the great and good who designed it who are sometimes not fools. And indeed you are fiercely discouraged from even understanding what it is you are doing: discouraged from understanding that you are building a new language.

And quite soon (sooner than you think and in fact immediately) you find you must actually have new structure, new grammar. But you cannot do this easily both because the language you use does not allow it and also because you do not know what it is you are doing – you do not realise that you are making a language. So probably you use a templating system or something and build an awful horror. Often this horror will have nested languages where inner languages appear in strings in outer languages. Often it will have evaluation rules so obscure and inconsistent that it is impossible for humans to write safe large programs in this language (Unix shells: I look at you). We have all seen these things.

And so you live out your life crawling in the dirt, never understanding what thing it is of which you are making a very bad, very unsafe, very ugly version. Because you have been taught there is only mud so all you do is pile up structures out of mud, to be washed away by the next rain. A little way over is a tribe who knows only straw and they build structures from straw which blow away in the first wind. You hate them; they hate you. Sometimes you have little wars.

What, on the other hand, do you do in Lisp? Well, few days ago I needed a way to express the idea of searching some (very) large structure and being able to fail in a structured way. So after ten minutes work, my program now says things like this:

(defun big-serch-thing (thing)
  (attempting
    (quick-and-dirty thing)
    (try-harder thing)))

(defun try-harder (thing)
  (walking-thing (node thing :level 0)
    (attempting
      (first-pass thing)
      (desparate-fallback thing))))

(defun first-pass (thing)
  ...
  (when doom (fail))
  ...)

Well it does not matter what this does and this is not what my program is actually like, but what is clear just by looking is that this language is not Common Lisp. Instead it is Common Lisp extended with at least two new grammatical constructs: attempting with its friend fail which looks like a verb but in fact is a control construct really, and walking-thing which is some kind of new iteration construct perhaps.

And there is more: when you look at attempting you will find it is implemented (by a function which) uses a construct called looping which is another extension to Common Lisp. And similarly for walking-thing (which is not really called that) which uses I think four separate new grammatical constructs I do not remember.

And there is more: when I started this essay these constructs were mostly as I showed above, but we have decided this was wrong, so the new language is now somewhat different and somewhat richer. A few more tens of minutes of work, most of it altering the existing programs in the old language to use the new language. The new language is even defined using a language-extending construct which itself is an extension to CL’s provided ones.

And this is how you program in Lisp. In Lisp, writing programs is building languages: in Lisp to solve a problem is to first build a language in which the problem may be solved. And because doing this is so easy in Lisp, this is what you do even for very small problems: you incrementally extend the grammar of the language — not just its lexicon — to create a language in which to describe the problem.

Well, this is not surprising, is it? This is what the laws imply: programming is constructing languages, and this applies even for very small programs. What is surprising is that so few languages encourage this. And because they do not we end up with the horror we all know. Perhaps even this is not surprising: any language which supports this well will have all the characteristics of Lisp, will in fact be a Lisp. So no other languages do this because to do it requires being Lisp. So why is Lisp not more popular? Well, answer is fairly easy but this is discussion for another day, I think.

And now we see why Lisp got features first: because it could. Let us say you wish to explore an object system in Lisp. Well, perhaps you will want a class-defining construct, so you write a macro, define-class or something. And you wish to be able to send messages, so you write a send function and then you modify the readtable so [o message ...] is (send o message ...). And perhaps you wish some new binding construct for fields so you write with-fields and so, and so.

And now you have a new language. If you were careful you may even have constructed that new language inside a single running Lisp image. And this took, perhaps, some hours. And later, you decide that no, you wish your new language to be different, so you change it. Another few hours. Eventually, in a different world, you call this part of the language ZLOS and there is a standard.

And this is why these linguistic innovations happen in Lisp: because Lisp is a machine for linguistic innovation. It is that feature of Lisp which makes it interesting, and it is only that feature: both because all other features derive from that one and because to have that feature is to be Lisp.

That is all.

Do not answer this or I will kill you with a stale loaf of bread. ↩
This is exaggeration: if you define no names in your program you are, perhaps, not constructing a language. ↩

Something unclear in the Common Lisp standard

2023-04-18T09:53:46Z

There is what I think is a confusion as to bound declarations in the Common Lisp standard. I may be wrong about this, but I think I’m correct.

Bound and free declarations

Declarations in Common Lisp can be either bound or free:

a bound declaration appears at the head of a binding form and applies to a variable or function binding made by that form;
a free declaration is any declaration which is not bound.

There are declarations which do not apply to bindings, such as optimize: these are always free.

Examples of bound and free declarations

In the form

(let ((x 1))
  (declare (type integer x))
  ...)

the declaration is bound and applies to the binding of x. In the form

(let ((/x/ 1))
  (declare (special /x/)
           (optimize (speed 3)))
  ...)

the special declaration is bound and applies to the binding of /x/, while the optimize declaration is free.

In the form

(let ((x 1))
  (locally
    (declare (type integer x)
             (optimize speed))
    ...)
  ...)

Both declarations are free and apply only to the body of the locally form.

Declarations which may not be ignored

Most declarations may be ignored by the implementation: this is the case for all type declarations, for instance. Two may not be:

notinline forbids inline compilation of the functions it names;
special requires dynamic bindings to be made when it is bound, and requires references to be to dynamic, not lexical bindigns when it is free.

I’m going to exploit the non-ignorability of special declarations to show a case where the confusion arises.

The confusion

Forms like let* bind sequentially:

(let* ((x 1) (y x))
  ...)

first binds x and then binds y to the value of x. Now, I am not sure of the standard ever says this, but all implementations I have tried take this to mean that the same name can be bound several times by let*:

(let* ((x 1) (x x))
  ...)

is legal, if stylistically awful. That’s because the obvious transformation of let* into nested lets turns this into:

(let ((x 1))
  (let ((x x))
    ...))

which is clearly fine.

So now we come to the problem: what should this mean?

(let* ((x 1) (x x))
  (declare (type fixnum x))
  ...

Which binding of x does the declaration apply to? The standard does not say. In this case it might not matter, because this declaration can be ignored, but here is a case where it does matter:

(let (c)
  (let* ((/x/ 1)
         (/x/ (progn
                (setf c (lambda () /x/))
                2)))
    (declare (special /x/))
    (values c (lambda () /x/))))

This expression returns two values, both of which are functions:

if the first /x/ is special then calling the first function will result in an error;
if the second /x/ is special then calling the second function will result in an error.

So using this trick you can know whether the first binding, second binding, or both bindings are affected by the special declaration.

And, again, the standard does not say which binding is affected, or whether both should be. And implementations differ. Given the following file

(in-package :cl-user)  

(defun call-ok-p (f)
  (multiple-value-bind (v c)
      (ignore-errors
       (funcall f)
       t)
    (declare (ignore c))
    v))

(defun ts ()
  (multiple-value-bind (one two)
      (let (c)
        (let* ((/x/ 1)
               (/x/ (progn
                      (setf c (lambda () /x/))
                      2)))
          (declare (special /x/))
          (values c (lambda () /x/))))
    (values (call-ok-p one)
            (call-ok-p two))))

(multiple-value-bind (first-lexical second-lexical) (ts)
  (format t "~&first  ~:[special~;lexical~]~%~
               second ~:[special~;lexical~]~%"
          first-lexical second-lexical))

SBCL

first  lexical
second special

CCL

first  special
second special

LispWorks

first  special
second special

What should the answer be?

I think that the interpretation taken by CCL and LispWorks is better: in forms like this declarations should apply to all the bindings made by the form. An alternative answer is that the declarations should apply to the visible bindings at the point of the declaration, which is the approach taken by SBCL.

It’s tempting to say that the obvious rewrite of let* as nested lets gives you the SBCL answer, but it does not. In a form like

(let* ((x 3) (y x))
  (declare (type integer x)
           (type (integer 0) y))
  ...)

This must be rewritten as

(let ((x 3))
  (declare (type integer x))
  (let ((y x))
    (declare (type (integer 0) y))
    ...))

So the declaration for x must be raised out of the inner let so it remains bound: the implementation already has to do work to get declarations in the right place and can’t just naïvely rewrite the form.

I prefer the first interpretation both because I think it represents what people are likely to want more closely, but also because I think the standard could be interpreted as meaning that without being rewritten.

Does this matter?

Probably only in very obscure cases! I just thought it was interesting.

Thanks to vrious people on the Lisp-HUG mailing list for coming up with this.

Measuring some tree-traversing functions

2023-03-26T09:25:50Z

In a previous article my friend Zyni wrote some variations on a list-flattening function, some of which were ‘recursive’ and some of which ‘iterative’, managing the stack explicitly. We thought it would be interesting to see what the performance differences were, both for this function and a more useful variant which searches a tree rather than flattening it.

What we measured

The code we used is here¹. We measured four variations of each of two functions.

List flattening

All these functions use collecting to build their results forwards. They live in flatten-variants.lisp.

flatten/implicit-stack works in the obvious recursive way, with an implicit stack. This uses iterate to express the local recursive function.
flatten/explicit-stack uses an explicit stack (called agenda in the code) represented as a vector, and uses looping to express iteration.
flatten/explicit-stack/adja is like the previous function but it is willing to extend the explicit stack, which it does by using adjust-array and assignment.
flatten/explicit-stack/adjb is like flatten/explicit-stack/adja but uses a local tail-recursive function to bind the extended stack rather than assignment.
Finally flatten/consy-stack is very close to Zyni’s original iterative solution: it represents the stack as a list. This version necessarily conses fairly copiously.

Searching cons trees

These functions, in treesearch-variants.lisp, correspond to the flattening variants, except they are searching for some atomic value in the tree of conses:

search/implicit-stack uses an implicit stack;
search/explicit-stack uses a vector;
search/explicit-stack/adja uses a vector and adjusts by assignment;
search/explicit-stack/adjb uses a vector and adjusts by binding;
search/consy-stack uses a consy stack.

Notes on the code

The functions all have (declare (optimize (speed 3))) but specifically don’t turn off safety or use implementation-specific settings: we wanted to test code we felt we’d be happy running, and that means code compiled with reasonable settings for safety: if you turn safety off you’re brave, foolish, or both.

We did not compare looping with do or loop: we probably should. However the expansion of looping is pretty straightforward:

(looping ((this o) (depth 0))
  (declare ...)
  ...)

Turns into

(let ((this o) (depth 0))
  (declare ...)
  (block nil
    (tagbody
      #:start
      (multiple-value-setq (this depth) ...)
      (go #:start))))

The only real question here, we think is whether multiple-value-setq is compiled well: brief inspection implies it is. We should probably still compare the current version with more ‘native CL’ variants.

The variants which use a vector as a stack maintain the current element themselves: that’s because we tested using a fill pointer and vector-push / vector-pop and it was really significantly slower in both implementations.

What we did

The Lisp implementations we used

We used LispWorks 8.0 and very recent SBCL builds, compiled from the master branch no more than a few days before we ran the tests in mid March 2023.

In the case of SBCL we paid attention to notes and warnings during compilation. The significant one we did not address was that it complained vociferously about not being able to optimize calls to eql: that’s because we don’t know the type of the thing we are searching for: it needs to do the work it is trying to avoid. Apart from this the only warnings were about the computation of the new length of the agenda, which never actually happens in the tests we ran.

The machines we benchmarked on

We both have M1-based Macbook Airs so this is what we used. In particular we have not run any benchmarks on x64.

What we ran

make-car-cdr, in common.lisp, makes a list where each element is a chain linked by cars, finally terminating in a specified element. Controlling the length of the list and the depth of the chains gives the functions more iterative or more recursive work to do respectively. The benchmarking code then made a series of suitable structures of increasing size and timed many iterations of each function on the same structure, computing the time per call. We then wrote a program in Racket to plot the results on axes of ‘breadth’ (length of the list) and ‘depth’ (depth of the car-linked chain). For the search functions the element being searched for was not in the tree so they had to do as much work as possible.

Life was usually arranged so that the initial agenda was big enough for the functions which used a vector as the agenda, so none of that aspect of them was teated, except for one case below. Apart from that case, the ‘vector stack’ timings refer to flatten/explicit-stack and treesearch/explicit-stack, not the adjustable-stack variants.

Some results

We timed 1,000 iterations of each call, for list lengths (breadth in the plots and below) from 30 to 1,000 in steps of 10 and depths (depth in the plots and below) from 10 to 300 in steps of 10, computing times in μs per iteration. Neither of us knows anything about how data like this should be best presented but simply plotting the performance surfaces seemed reasonable. We used bilinear interpolation to make the surface from the points².

LispWorks

Treesearch: implicit compared with vector stack

This is nicely linear in both breadth and depth, and so quadratic in breadth $\times$ depth. And it’s easy to see that for LW using the implicit stack is faster than the manually-managed stack.

Treesearch: vector stack compared with consy stack

This compares the vector stack with the consy stack, for treesearch. The consy stack is slightly faster which surprised us. This conses a list as long as the depth of the tree for each ‘leftward’ branch, and then immediately unwinds that and throws the whole list away. So it creates significant garbage, but the allocation and garbage collection overhead together is still faster than using a vector. Consing really is (almost) free.

Treesearch compared with flatten, both with implicit stacks

Here is more evidence that consing is very cheap: the difference between treesearch (which does not cons) and flatten (which does) is tiny.

SBCL

Treesearch: implicit compared with vector stack

So here is SBCL. For SBCL explicitly managing the stack as a vector is significantly faster than the implicit stack. Something that is also apparent here is how variable SBCL’s timings are compared with LW’s: we don’t know why that is although we suspect it might be because SBCL’s garbage collector is more intrusive than LW’s. We also don’t know whether this variation is repeatable, or whether it’s due to a single very slow run or something like that.

Treesearch: vector stack compared with consy stack

For SBCL the consy stack is significantly slower than the vector stack, so for SBCL the vector stack is the fastest.

Treesearch compared with flatten, both with implicit stacks

SBCL has a slightly larger difference between treesearch and flatten, with flatten being slower. There are also curious ‘waves’ in the plot as depth increases.

LispWorks compared with SBCL

Treesearch: SBCL compared with Lispworks, implicit stacks

LW is significantly faster than SBCL for implicit stacks except for very small depths.

Treesearch: SBCL compared with Lispworks, best stacks

This compares LW using an implicit stack with SBCL using an explicit vector stack. The difference is pretty small now.

Flatten: SBCL compared with Lispworks, consy stacks

This was meant to be the worst-case for both: flattening and a consy stack. But it’s not particularly informative, I think.

The outer reaches: LispWorks with a deep tree

We did one run with the maximum depth set to 10,000 with a step of 500, and maximum breadth set to 1,000 with a step of 100, averaged over 100 iterations instead of 1,000. This is too deep for LW’s stack, but LW allows stack extension, and we wrote what later became this to extend the stack as required. Note that this happens only during the first recursion into the left-hand branch of the tree so has minimal effect on performance. This also used search/explicit-stack/adjb for the vector stack.

Treesearch: implicit compared with consy stack, deep tree

As before the implicit stack is much better for LW. This is much more bumpy than LW was for smaller depths, this might have been because the machine did other things while it was running but we don’t think so.

Some conclusions

None of the differences were really large. In particular there’s no enormous advantage from managing the stack yourself.

Consing and the resulting garbage-collection does really seem to be very cheap, especially in LispWorks: the days of long GC pauses are long gone.

We were surprised that LispWorks was fairly reliably faster than SBCL: surprised enough that we ran everything several times to be sure. It’s also interesting how much smoother LW’s performance surface is in most cases.

It is possible that our implementations just suck, of course.

Mostly it’s just some pretty pictures.

All of the functions should be portable CL. Some of the mechanism for expressing dependencies and loading things is not. However it should be easy for anyone to run this if they wish to. ↩
Getting the bilinear interpolation right took longer than anything else, and perhaps longer than everything else put together. ↩

The absurdity of stacks

2023-03-25T10:57:19Z

Very often people regard the stack as a scarce, expensive resource, while the heap is plentiful and very cheap. This is absurd: the stack is memory, the heap is also memory. Deforming programs so they are ‘iterative’ in order that they do not run out of the stack we imagine to be so costly is ridiculous: if you have a program which is inherently recursive, let it be recursive.

In a previous article my friend Zyni wrote some variations on a list-flattening function¹, some of which were ‘recursive’ and some of which ‘iterative’. Of course, the ones which claim to be iterative are, in fact, recursive: any procedure which traverses a recursively-defined data structure such as a tree of conses is necessarily recursive. The ‘iterative’ versions just use an explicitly-maintained stack rather than the implicit stack provided by the language. That makes sense only if stack space is very small compared to the heap and must therefore be conserved. And, well, for many systems that’s true. But it is small only because we have administratively decided it should be small: the stack is just memory. If there is plenty of memory for the heap, there is plenty for the stack.

There are, or may be, arguments for why stacks needed to be small on ancient machines. The history is fascinating, but it is not relevant to today’s systems, other than tiny embedded ones. The persistent view of modern machines as giant PDP–11s has been a blight for well over two decades now: it needs to stop.

The argument that the stack should be small often seems to be that, if it’s not, people will write programs which run away. That’s spurious: if such a program is, in fact, iterative, then good compilers will eliminate the tail calls and it will not use stack: a small limit on the stack will not help. If it’s really recursive then why should it run out of storage before its conversion to a program which manages the stack explicitly does? Of course that’s exactly what compilers which do CPS conversion already do: programs written using compilers which do that won’t have these weird stack limits in the first place. But it should not be necessary to rely on a CPS-converting compiler, or to write in continuation-passing style manually to avoid stack usage: it should be used for other reasons, because the stack is not, in fact, expensive.

Still less should people feel the need to write programs which explicitly manage a stack except in extraordinary cases.

There need to be some limits on stack size, just as there need to be some limits on heap size, but making the limit on stack size far smaller than the limit on heap size simply encourages people to believe things which aren’t true, and to live in fear of recursive programs.

I still want to know how often functions like this are used in real life. ↩

Variations on a theme

2023-03-13T12:36:33Z

My friend Zyni wrote a comment to a thread on reddit with some variations on a list-flattening function. We’ve since spent some time thinking about things related to this, which is written up in a following article. Here is her comment so the following article can refer to it. Other than notes at the end the following text is Zyni’s, not mine.

The reddit comment by Zyni

First of all we all know that CL does not promise to optimize tail recursion: means that tail recursive program may generate recursive not iterative process. So recursive program in CL even if tail recursive is not safe on data of unknown size, assuming stack is limited.

But let us assume as good implementations do that tail recursion is optimized in implementation (no need for general tail calls here but is obvious nice thing if implementations do this). Certainly if we are deploying code in space we know what implementation we use and can check this.

So we look at this supposed wonder of code, which I rewrite slightly to use iterate macro which is simply Scheme’s named-let to be compatible with later examples:

(defun flatten (o)
  ;; original terrible one
  (iterate ftn ((x o) (accumulator '()))
    (typecase x
      (null accumulator)
      (cons (ftn (car x) (ftn (cdr x) accumulator)))
      (t (cons x accumulator)))))

This … is really bad program. It makes an essential mistake that it wishes to build result forwards but lists wish to be built backwards, so it must therefore recurse (not tail) on cdr of structure first. But most list-based structures have little weight in car but much in cdr, so this will fail even on list which is already flat: (flatten (make-list 100000 :initial-element 1)) will fail if your example fails.

Any person presenting this code as good example should be ashamed of self.

So first change: we accept that we must build lists backwards but we change program so that tail call is on cdr not car, and reverse result:

(defun flatten (o)
  ;; not TR but better on usual assumptions
  (nreverse
   (iterate ftn ((x o) (accumulator '()))
     (typecase x
       (null accumulator)
       (cons (ftn (cdr x) (ftn (car x) accumulator)))
       (t (cons x accumulator))))))

This function will be fine on assumption of structures which have most weight in their cdrs, which often is true.

Well, you say, ugly reverse. OK this is easy: we simply add in a collecting macro which allows construction of list forwards, implementation is obvious (tail pointer). Now we have done this we can also reorder calls to be more obvious (car call, not TR, is now first):

(defun flatten (o)
  ;; not TR, better on usual assumptions, no reverse
  (collecting
    (iterate ftn ((x o))
      (typecase x
        (cons
         (ftn (car x))
         (ftn (cdr x)))
        (null)
        (t (collect x))))))

This is still not fully TR, so will fail on structures which have much weight in car.

Well, of course, we can deal with this as well: we use explicit agenda to move stack onto heap and turn into pure tail recursive version. First one which builds list backwards in obvious way, therefore needs reverse again:

(defun flatten (o)
  ;; pure TR
  (iterate ftn ((agenda (list o))
                (accumulator '()))
    (if (null agenda)
        ;; can write own reverse as tail recursive of course if wish
        ;; to be pure of heart
        (nreverse accumulator)
      (destructuring-bind (this . more) agenda
        (typecase this
          (null
           (ftn more accumulator))
          (cons
           (ftn (list* (car this) (cdr this) more) accumulator))
          (t
           (ftn more (cons this accumulator))))))))

Assuming implementation optimizes tail recursion this will flatten completely arbitrary structure limited only by memory.

We can avoid this reversery of course:

(defun flatten (o)
  ;; pure TR, no reverse
  (collecting
    (iterate ftn ((agenda (list o)))
      (when (not (null agenda))
        (destructuring-bind (this . more) agenda
          (typecase this
            (null
             (ftn more))
            (cons
             (ftn (list* (car this) (cdr this) more)))
            (t
             (collect this)
             (ftn more))))))))

As before this is limited only by memory assuming implementation optimizes tail calls.

Well, I have written Lisp for only couple of years really (but have maths background). But even I can see that this idea of having to put scary label on recursive function is very bad. Instead people using such code should perhaps read it and understand it to see what its problems and advantages are. Radical idea, I know.

Finally idea that stack space is scarce may or may not be true. Example, if we rewrite original version in Racket (first Lisp I used before being lured to dark side):

(define (flatten o)
  (let ftn ([x o] [accumulator '()])
    (cond
      [(null? x) accumulator]
      [(cons? x) (ftn (car x) (ftn (cdr x) accumulator))]
      [else (cons x accumulator)])))

This will happily ‘flatten’ 100,000 element list and is only limited by memory available because Racket does not treat stack same way.

Finally here is variant of final version using looping macro which does applicative iteration: this is iterative, on any implementation:

(defun flatten (o)
  ;; Iterative
  (collecting
    (looping ((agenda (list o)))
      (when (null agenda)
        (return))
      (destructuring-bind (this . more) agenda
        (typecase this
          (null more)
          (cons (list* (car this) (cdr this) more))
          (t (collect this) more))))))

looping part of this turns into:

(let ((agenda (list o)))
  (block nil
    (tagbody
     #:start (setq agenda
                   (progn
                     (when (null agenda) (return))
                     (destructuring-bind (this . more) agenda
                       (typecase this
                         (null more)
                         (cons (list* (car this) (cdr this) more))
                         (t (collect this) more)))))
             (go #:start))))

which is iterative.

I think iterate one is nicer.

Notes from Tim

English is Zyni’s third language: she wanted me to fix up the above but I refused as I find the way she writes so charming.

Both of us would like to know how often flatten is actually used: everyone seems to be very keen on it, but we can’t think of any cases where we’ve ever wanted it or anything very much like it.

All of the macros referenced are ‘mine’ in a somewhat loose sense: They’re all published by me, and some of them are mine, some of them were mine but have been made much better by Zyni, some of them are really hers. There are generally comments in the code. Zyni refuses to have anything but a very minimal internet presence for reasons I used to think were absurd but no longer do: you can’t be too careful when your parents and by extension you might be on the wrong side of Putin.

Zyni is not her real name, obviously.

Two tiny Lisp evaluators

2023-02-27T14:19:38Z

Everyone who has written Lisp has written tiny Lisp evaluators in Lisp: here are two more.

Following two recent articles I wrote on scope and extent in Common Lisp, I thought I would finish with two very tiny evaluators for dynamically and lexically bound variants on a tiny Lisp.

The language

The tiny Lisp these evaluators interpret is not minimal: it has constructs other than lambda, and even has assignment. But it is pretty small. Other than the binding rules the languages are identical.

λ & lambda are synonyms and construct procedures, which can take any number of arguments;
quote quotes its argument;
if is conditional expression (the else part is optional);
set! is assignment and mutates a binding.

That is all that exists.

Both evaluators understand primitives, which are usually just functions in the underlying Lisp: since the languages are Lisp–1s, you could also expose other sorts of things of course (for instance true and false values). You can provide a list of initial bindings to them to define useful primitives.

Requirements

Both evaluators rely on my iterate and spam hacks: they could easily be rewritten not to do so.

The dynamic evaluator

A procedure is represented by a structure which has a list of formals and a body of one or more forms.

(defstruct (procedure
            (:print-function
             (lambda (p s d)
               (declare (ignore d))
               (print-unreadable-object (p s)
                 (format s "λ ~S" (procedure-formals p))))))
  (formals '())
  (body '()))

The evaluator simply dispatches on the type of thing and then on the operator for compound forms.

(defun evaluate (thing bindings)
  (typecase thing
    (symbol
     (let ((found (assoc thing bindings)))
       (unless found
         (error "~S unbound" thing))
       (cdr found)))
    (list
     (destructuring-bind (op . arguments) thing
       (case op
         ((lambda λ)
          (matching arguments
            ((head-matches (list-of #'symbolp))
             (make-procedure :formals (first arguments)
                             :body (rest arguments)))
            (otherwise
             (error "bad lambda form ~S" thing))))
         ((quote)
          (matching arguments
            ((list-matches (any))
             (first arguments))
            (otherwise
             (error "bad quote form ~S" thing))))
         ((if)
          (matching arguments
            ((list-matches (any) (any))
             (if (evaluate (first arguments) bindings)
                 (evaluate (second arguments) bindings)))
            ((list-matches (any) (any) (any))
             (if (evaluate (first arguments) bindings)
                 (evaluate (second arguments) bindings)
               (evaluate (third arguments) bindings)))
            (otherwise
             (error "bad if form ~S" thing))))
         ((set!)
          (matching arguments
            ((list-matches #'symbolp (any))
             (let ((found (assoc (first arguments) bindings)))
               (unless found
                 (error "~S unbound" (first arguments)))
               (setf (cdr found) (evaluate (second arguments) bindings))))
            (otherwise
             (error "bad set! form ~S" thing))))
         (t
          (applicate (evaluate (first thing) bindings)
                     (mapcar (lambda (form)
                               (evaluate form bindings))
                             (rest thing))
                     bindings)))))
    (t thing)))

The interesting thing here is that applicate needs to know the current set of bindings so it can extend them dynamically.

Here is applicate which has a case for primitives and procedures

(defun applicate (thing arguments bindings)
  (etypecase thing
    (function
     ;; a primitive
     (apply thing arguments))
    (procedure
     (iterate bind ((vtail (procedure-formals thing))
                    (atail arguments)
                    (extended-bindings bindings))
       (cond
        ((and (null vtail) (null atail))
         (iterate eval-body ((btail (procedure-body thing)))
           (if (null (rest btail))
               (evaluate (first btail) extended-bindings)
             (progn
               (evaluate (first btail) extended-bindings)
               (eval-body (rest btail))))))
        ((null vtail)
         (error "too many arguments"))
        ((null atail)
         (error "not enough arguments"))
        (t
         (bind (rest vtail)
               (rest atail)
               (acons (first vtail) (first atail)
                      extended-bindings))))))))

The thing that makes this evaluator dynamic is that the bindings that applicate extends are those it was given: procedures do not remember bindings.

The lexical evaluator

A procedure is represented by a structure as before, but this time it has a set of bindings associated with it: the bindings in place when it was created.

(defstruct (procedure
            (:print-function
             (lambda (p s d)
               (declare (ignore d))
               (print-unreadable-object (p s)
                 (format s "λ ~S" (procedure-formals p))))))
  (formals '())
  (body '())
  (bindings '()))

The evaluator is almost identical:

(defun evaluate (thing bindings)
  (typecase thing
    (symbol
     (let ((found (assoc thing bindings)))
       (unless found
         (error "~S unbound" thing))
       (cdr found)))
    (list
     (destructuring-bind (op . arguments) thing
       (case op
         ((lambda λ)
          (matching arguments
            ((head-matches (list-of #'symbolp))
             (make-procedure :formals (first arguments)
                             :body (rest arguments)
                             :bindings bindings))
            (otherwise
             (error "bad lambda form ~S" thing))))
         ((quote)
          (matching arguments
            ((list-matches (any))
             (first arguments))
            (otherwise
             (error "bad quote form ~S" thing))))
         ((if)
          (matching arguments
            ((list-matches (any) (any))
             (if (evaluate (first arguments) bindings)
                 (evaluate (second arguments) bindings)))
            ((list-matches (any) (any) (any))
             (if (evaluate (first arguments) bindings)
                 (evaluate (second arguments) bindings)
               (evaluate (third arguments) bindings)))
            (otherwise
             (error "bad if form ~S" thing))))
         ((set!)
          (matching arguments
            ((list-matches #'symbolp (any))
             (let ((found (assoc (first arguments) bindings)))
               (unless found
                 (error "~S unbound" (first arguments)))
               (setf (cdr found) (evaluate (second arguments) bindings))))
            (otherwise
             (error "bad set! form ~S" thing))))
         (t
          (applicate (evaluate (first thing) bindings)
                     (mapcar (lambda (form)
                               (evaluate form bindings))
                             (rest thing)))))))
    (t thing)))

The differences are that when constructing a procedure the current bindings are recorded in the procedure, and it is no longer necessary to pass bindings to applicate.

applicate is also almost identical:

(defun applicate (thing arguments)
  (etypecase thing
    (function
     ;; a primitive
     (apply thing arguments))
    (procedure
     (iterate bind ((vtail (procedure-formals thing))
                    (atail arguments)
                    (extended-bindings (procedure-bindings thing)))
       (cond
        ((and (null vtail) (null atail))
         (iterate eval-body ((btail (procedure-body thing)))
           (if (null (rest btail))
               (evaluate (first btail) extended-bindings)
             (progn
               (evaluate (first btail) extended-bindings)
               (eval-body (rest btail))))))
        ((null vtail)
         (error "too many arguments"))
        ((null atail)
         (error "not enough arguments"))
        (t
         (bind (rest vtail)
               (rest atail)
               (acons (first vtail) (first atail)
                      extended-bindings))))))))

The difference is that the bindings it extends when binding arguments are the bindings which the procedure remembered, not the dynamically-current bindings, which it does not even know.

The difference between them

Here is the example that shows how these two evaluators differ.

With the dynamic evaluator:

? ((λ (f)
     ((λ (x)
        ;; bind x to 1 around the call to f
        (f))
      1))
   ((λ (x)
      ;; bind x to 2 when the function that will be f is created
      (λ () x))
    2))
1

The binding in effect is the dynamically current one, not the one that was in effect when the procedure was created.

With the lexical evaluator:

? ((λ (f)
     ((λ (x)
        ;; bind x to 1 around the call to f
        (f))
      1))
   ((λ (x)
      ;; bind x to 2 when the function that will be f is created
      (λ () x))
    2))
2

Now the binding in effect is the one that existed when the procedure was created.

Something more interesting is how you create recursive procedures in the lexical evaluator. With suitable bindings for primitives, it’s easy to see that this can’t work:

((λ (length)
   (length '(1 2 3)))
 (λ (l)
   (if (null? l)
       0
       (+ (length (cdr l)) 1))))

It can’t work because length is not in scope in the body of length. it will work in the dynamic evaluator.

The first fix, which is similar to what Scheme does with letrec, is to use assignment to mutate the binding so it is correct:

((λ (length)
   (set! length (λ (l)
                  (if (null? l)
                      0
                      (+ (length (cdr l)) 1))))
   (length '(1 2 3)))
 0)

Note the initial value of length is never used.

The second fix is to use something like the U combinator (you could use Y of course: I think U is simpler to understand):

((λ (length)
   (length '(1 2 3)))
 (λ (l)
   ((λ (c)
      (c c l 0))
    (λ (c t s)
      (if (null? t)
          s
          (c c (cdr t) (+ s 1)))))))

Source code

These two evaluators, together with a rudimentary REPL which can use either of them, can be found here.

Dynamic binding without special in Common Lisp

2023-02-27T09:53:27Z

In Common Lisp, dynamic bindings and lexical bindings live in the same namespace. They don’t have to.

Common Lisp has two sorts of bindings for variables: lexical binding and dynamic binding. Lexical binding has lexical scope — the binding is available where it is visible in source code — and indefinite extent — the binding is available as long as any code might reference it. Dynamic binding has indefinite scope — the binding is available to any code which runs between when the binding is established and when control leaves the form which established it — and dynamic extent — the binding ceases to exist when control leaves the binding form.

These are really two very different things. However CL places both of these kinds of bindings into the same namespace, relying on special declarations and proclamations to tell the system which sort of binding to create and reference for a given name.

That doesn’t have to be the case: it’s possible in CL to completely isolate these two namespaces from each other. This means you could write code where all variable references were to lexical bindings and where dynamic bindings were created and referenced by a completely different set of operators. Here is an example of that. Following practice in some old Lisps I will call this ‘fluid’ binding. I will also use / to delimit the names of fluid variables simply to distinguish them from normal variables.

(defun inner (varname value)
  (setf (fluid-value varname) value))

(defun outer (varname value)
  (call/fluid-bindings
   (lambda ()
     (values
      (fluid-value varname)
      (progn
        (inner varname (1+ value))
        (fluid-value varname))))
   (list varname)
   (list value)))

And now

> (outer '/v/ 1)
1
2

Here are a set of operators for dealing with these fluid variables:

fluid-value accesses the value of a fluid variable.

fluid-boundp tells you if a name is bound as a fluid variable.

call/fluid-bindings calls a function with one or more fluid variables bound.

define-fluid (not used above) defines a global value for a fluid variable.

Well, of course you can do something like this using an explicit binding stack and a single special variable to hang it from. But that’s not how this works: these ‘fluid variables’ are just CL’s dynamic variables:

(defun call/print-base (f base)
  (call/fluid-bindings  f '(*print-base*) (list base)))

> (call/print-base
   (lambda ()
     *print-base*)
   2)
2

So how does this work? Well fluid-value and fluid-boundp are obvious:

(defun fluid-value (s)
  (symbol-value s))

(defun (setf fluid-value) (n s)
  (setf (symbol-value s) n))

(defun fluid-boundp (s)
  (boundp s))

And the trick now is that CL gives you enough mechanism to bind named dynamic variables yourself, that mechanism being progv, which

[…] allows binding one or more dynamic variables whose names may be determined at run time […]

So now call/fluid-bindings just uses progv:

(defun call/fluid-bindings (f fluids values)
  (progv fluids values (funcall f)))

And finally define-fluid looks like this:

(defmacro define-fluid (var &optional (value nil)
                            (doc nil docp))
  `(progn
     (setf (fluid-value ',var) ,value)
     ,@(if docp
           `((setf (documentation ',var 'variable) ',doc))
         '())
     ',var))

The interesting thing here is that there are no special declarations or proclamations: you can create and bind new fluid variables without any recourse to special at all, in a way which is completely compatible with the existing dynamic variables, because fluid variables are dynamic variables.

So one way of thinking about special is that it is a declaration that says ‘for this variable name, access the namespace of dynamic bindings rather than lexical bindings’. This is not really what special was of course in Lisps before CL — it was historically closer to an instruction to use the interpreter’s variable binding mechanism in compiled code — but you can think of it this way in CL, where the interpreter and compiler do not have separate binding rules.

And, of course, using something like the above, you could write code in CL where all variable bindings were lexical and dynamic variables lived entirely in their own namespace. For instance this works fine:

(defun f ()
  (let ((x 2))
    (call/fluid-bindings
     (lambda ()
       (values x (fluid-value 'x)))
     '(x) '(3))))

> (f)
2
3

The reference to x as a variable refers to its lexical binding, while (fluid-value 'x) refers to its dynamic binding.

Whether writing code like that would be useful I am not sure: I think that the *-convention for dynamic variables is perfectly fine in fact. But it is perhaps interesting to see that you can think of dynamic bindings in CL this way.

How to understand closures in Common Lisp

2023-02-22T13:51:07Z

The first rule of understanding closures is that you do not talk about closures. The second rule of understanding closures in Common Lisp is that you do not talk about closures. These are all the rules.

There is a lot of elaborate bowing and scraping about closures in the Lisp community. But despite that a closure isn’t actually a thing: the thing people call a closure is just a function which obeys the language’s rules about the scope and extent of bindings. Implementors need to care about closures: users just need to understand the rules for bindings. So rather than obsessing about this magic invisible thing which doesn’t actually exist in the language, I suggest that it is far better simply to think about the rules which cover bindings.

Angels and pinheads

It’s easy to see why this has happened: the CL standard has a lot of discussion of lexical closures, lexical and dynamic environments and so on. So it’s tempting to think that this way of thinking about things is ‘the one true way’ because it has been blessed by those who went before us. And indeed CL does have objects representing part of the lexical environment which are given to macro functions. Occasionally these are even useful. But there are no objects which represent closures as distinct from functions, and no predicates which tell you if a function is a closure or not in the standard language: closures simply do not exist as objects distinct from functions at all. They were useful, perhaps, as part of the text which defined the language, but they are nowhere to be found in the language itself.

So, with the exception of the environment objects passed to macros, none of these objects exist in the language. They may exist in implementations, and might even be exposed by some implementations, but from the point of the view of the language they simply do not exist: if I give you a function object you cannot know if it is a closure or not.

So it is strange that people spend so much time worrying about these objects which, if they even exist in the implementation, can’t be detected by anyone using the standard language. This is worrying about angels and pinheads: wouldn’t it be simpler to simply understand what the rules of the language actually say should observably happen? I think it would.

I am not arguing that the terminology used by the standard is wrong! All I am arguing is that, if you think you want to understand closures, you might instead be better off understanding the rules that give rise to them. And when you have done that you may suddenly find that closures have simply vanished into the mist: all you need is the rules.

History

Common Lisp is steeped in history: it is full of traces of the Lisps which went before it. This is intentional: one goal of CL was to enable programs written in those earlier Lisps — which were all Lisps at that time of course — to run without extensive modification.

But one place where CL didn’t steep itself in history is in exactly the areas that you need to understand to understand closures. Before Common Lisp (really, before Scheme), people spent a lot of time writing papers about the funarg problem and describing and implementing more-or-less complicated ways of resolving it. Then Scheme came along and decided that this was all nonsense and that it could just be made to go away by implementing the language properly. And the Common Lisp designers, who knew about Scheme, said that, well, if Scheme can do this, then we can do this as well, and so they also made it the problem vanish, although not in quite such an extreme way as Scheme did.

And this is now ancient history: these predecessor Lisps to CL are all at least 40 years old now. I am, just, old enough to have used some of them when they were current, but for most CL programmers these questions were resolved before they were born. The history is very interesting, but you do not need to steep yourself in it to understand closures.

Bindings

So the notion of a closure is part of the history behind CL: a hangover from the time when people worried about the funarg problem; a time before they understood that the whole problem could simply be made to go away. So, again, if you think you want to understand closures, the best approach is to understand something else: to understand bindings. Just as with closures, bindings do not exist as objects in the language, although you can make some enquiries about some kinds of bindings in CL. They are also a concept which exists in many programming languages, not just CL.

A binding is an association between a name — a symbol — and something. The most common binding is a variable binding, which is an association between a name and a value. There are other kinds of bindings however: the most obvious kind in CL is a function binding: an association between a name and a function object. And for example within a (possibly implicit) block there is a binding between the name of the block and a point to which you can jump. And there are other kinds of bindings in CL as well, and the set is extensible. The CL standard only calls variable bindings ‘bindings’, but I am going to use the term more generally.

Bindings are established by some binding construct and are usually not first-class objects in CL: they are just as vaporous as closures and environments. Nevertheless they are a powerful and useful idea.

What can be bound?

By far the most common kind of binding is a variable binding: an association between a name and a value. However there are other kinds of bindings: associations between names and other things. I’ll mention those briefly at the end, but in everything else that follows it’s safe to assume that ‘binding’ means ‘variable binding’ unless I say otherwise.

Scope and extent

For both variable bindings and other kinds of bindings there are two interesting questions you can ask:

where is the binding available?
when is the binding visible?

The first question is about the scope of the binding. The second is about the extent of the binding.

Each of these questions has (at least) two possible answers giving (at least) four possibilities. CL has bindings which use three of these possibilities and the fourth in a restricted case: two and a restricted version of a third for variable bindings, the other one for some other kinds of bindings.

Scope. The two options are:

the binding may be available only in code where the binding construct is visible;
or the binding may be available during all code which runs between where the binding is established and where it ends, regardless of whether the binding construct is visible.

What does ‘visible’ mean? Well, given some binding form, it means that the bindings it establishes are visible to all the code that is inside that form in the source. So, in a form like (let ((x 1)) ...) the binding of x is visible to the code that replaces the ellipsis, including any code introduced by macroexpansion, and only to that code.

Extent. The two options are:

the binding may exist only during the time that the binding construct is active, and goes away when control leaves it;
or the binding may exist as long as there is any possibility of reference.

Unfortunately the CL standard is, I think, slightly inconsistent in its naming for these options. However I’m going to use the standard’s terms with one exception. Here they are.

Scope:

when a binding is available only when visible this called lexical scope;
when a binding available to all code within the binding construct this is called indefinite scope¹;

Extent:

when a binding ends at the end of the binding form this is called dynamic extent²;
when a binding available indefinitely this called indefinite extent.

The term from the standard I am not going to use is dynamic scope, which it defines to mean the combination of indefinite scope and dynamic extent. I am not going to use this term because I think it is confusing: although it has ‘scope’ in its name it concerns both scope and extent. Instead I will introduce better, commonly used, terms below for the interesting combinations of scope and extent.

The four possibilities for bindings are then:

lexical scope and dynamic extent;
lexical scope and indefinite extent;
indefinite scope and dynamic extent;
indefinite scope and indefinite extent.

The simplest kind of binding

So then let’s ask: what is the simplest kind of binding to understand? If you are reading some code and you see a reference to a binding then what choice from the above options will make it easiest for you to understand whether that reference is valid or not?

Well, the first thing is that you’d like to be able to know by looking at the code whether a reference is valid or not. That means that the binding construct should be visible to you, or that the binding should have lexical scope. Compare the following two fragments of code:

(defun simple (x)
  ...
  (+ x 1)
  ...)

and

(defun confusing ()
  ...
  (+ *x* 1)
  ...)

Well, in the first one you can tell, just by looking at the code, that the reference to x is valid: the function, when called, establishes a binding of x and you can see that when reading the code. In the second one you just have to assume that the reference to *x* is valid: you can’t tell by reading the code whether it is or not.

Lexical scope makes it easiest for people reading the code to understand it, and in particular it is easier to understand than indefinite scope. It is the simplest kind of scoping to understand for people reading the code.

So that leaves extent. Well, in the two examples above definite or indefinite extent makes no difference to how simple the code is to understand: once the functions return there’s no possibility of reference to the bindings anyway. To expose the difference we need somehow to construct some object which can refer to a binding after the function has returned. We need something like this:

(defun maker (x)
  ...
  <construct object which refers to binding of x>)

(let ((o (maker 1)))
  <use o somehow to cause it to reference the binding of x>)

Well, what it this object going to be? What sort of things reference bindings? Code references bindings, and the objects which contain code are functions³. What we need to do is construct and return a function:

(defun maker (x)
  (lambda (y)
    (+ x y)))

and then cause this function to reference the binding by calling it:

(let ((f (maker 1)))
  (funcall f 2))

So now we can, finally, ask: what is the choice for the extent of the binding of x which makes this code simplest to understand? Well, the answer is that unless the binding of x remains visible to the function that is created in maker, this code can’t work at all. It would have to be the case that it was simply not legal to return functions like this from other functions. Functions, in other words, would not be first-class objects.

Well, OK, that’s a possibility, and it makes the above code simple to understand: it’s not legal and it’s easy to see that it is not. Except consider this small variant on the above:

(defun maybe-maker (x return-identity-p)
  (if return-identity-p
      #'identity
    (lambda (y)
      (+ x y))))

There is no way to know from reading this code whether maybe-maker will return the nasty anonymous function or the innocuous identity function. If it is not allowed to return anonymous functions in this way then there is no way of knowing whether

(funcall (maybe-maker 1 (zerop (random 2)))
         2)

is correct or not. This is certainly not simple: in fact it is a horrible nightmare. Another way of saying this is that you’d be in a situation where

(let ((a 1))
  (funcall (lambda ()
             a)))

would work, but

(funcall (let ((a 1))
           (lambda ()
             a)))

would not. There are languages which work that way: those languages suck.

So what would be simple? What would be simple is to say that if a binding is visible, it is visible, and that’s the end of the story. In a function like maker above the binding of x established by maker is visible to the function that it returns. Therefore it’s visible to the function that maker returns: without any complicated rules or weird special cases. That means the binding must have indefinite extent.

Indefinite extent makes it easiest for people reading the code to understand it when that code may construct and return functions, and in particular it is easier to understand than dynamic extent, which makes it essentially impossible to tell in many cases whether such code is correct or not.

And that’s it: lexical scope and indefinite extent, which I will call lexical binding, is the simplest binding scheme to understand for a language which has first-class functions⁴.

And really that’s it: that’s all you need to understand. Lexical scope and indefinite extent make reading code simple, and entirely explain the things people call ‘closures’ which are, in fact, simply functions which obey these simple rules.

Examples of the simple binding rules

One thing I have not mentioned before is that, in CL, bindings are mutable, which is another way of saying that CL supports assignment: assignment to variables is mutation of variable bindings. So, as a trivial example:

(defun maximum (list)
  (let ((max (first list)))
    (dolist (e (rest list) max)
      (when (> e max)
        (setf max e)))))

This is very easy to understand and does not depend on the binding rules in detail.

But, well, bindings are mutable, so the rules which say they exist as long as they can be referred to also imply they can be mutated as long as they can be referred to: anything else would certainly not be simple. So here’s a classic example of this:

(defun make-incrementor (&optional (value 0))
  (lambda (&optional (increment 1))
    (prog1 value
      (incf value increment))))

And now:

> (let ((i (make-incrementor)))
    (print (funcall i))
    (print (funcall i))
    (print (funcall i -2))
    (print (funcall i))
    (print (funcall i))
    (values))

0
1
2
0
1

As you can see, the function returned by make-incrementor is mutating the binding that it can still see.

What happens when two functions can see the same binding?

(defun make-inc-dec (&optional (value 0))
  (values
   (lambda ()
     (prog1 value
       (incf value)))
   (lambda ()
     (prog1 value
       (decf value)))))

And now

> (multiple-value-bind (inc dec) (make-inc-dec)
    (print (funcall inc))
    (print (funcall inc))
    (print (funcall dec))
    (print (funcall dec))
    (print (funcall inc))
    (values))

0
1
2
1
0

Again, what happens is the simplest thing: you can see simply from reading the code that both functions can see the same binding of value and they are therefore both mutating this common binding.

Here is an example which demonstrates all these features: an implementation of a simple queue as a pair of functions which can see two shared bindings:

(defun make-queue ()
  (let ((head '())
        (tail nil))
    (values
     (lambda (thing)
       ;; Push thing onto the queue
       (if (null head)
           ;; It's empty currently so set it up
           (setf head (list thing)
                 tail head)
         ;; not empty: just adjust the tail
         (setf (cdr tail) (list thing)
               tail (cdr tail)))
       thing)
     (lambda ()
       (cond
        ((null head)
         ;; empty
         (values nil nil))
        ((null (cdr head))
         ;; will be empty: don't actually need this case but it is
         ;; cleaner
         (values (prog1 (car head)
                   (setf head '()
                         tail nil))
                 t))
        (t
         ;; will still have content
         (values (pop head) t)))))))

make-queue will return two functions:

the first takes one argument which it appends to the queue;
the second takes no argument and either the next element of the queue and t or nil and nil if the queue is empty.

So, with this little function to drain the queue

(defun drain-and-print (popper)
  (multiple-value-bind (value fullp) (funcall popper)
    (when fullp
      (print value)
      (drain-and-print popper))
    (values)))

we can see this in action

> (multiple-value-bind (pusher popper) (make-queue)
    (funcall pusher 1)
    (funcall pusher 2)
    (funcall pusher 3)
    (drain-and-print popper))

1
2
3

A less-simple kind of binding which is sometimes very useful

Requiring bindings to be simple usually makes programs easy to read and understand. But it also makes it hard to do some things. One of those things is to control the ‘ambient state’ of a program. A simple example would be the base for printing numbers. It’s quite natural to say that ‘in this region of the program I want numbers printed in hex’.

If all we had was lexical binding then this becomes a nightmare: every function you call in the region you want to cause printing to happen in hex needs to take some extra argument which says ‘print in hex’. And if you then decide that, well, you’d also like some other ambient parameter, you need to provide more arguments to every function⁵. This is just horrible.

You might think you can do this with global variables which you temporarily set: that is both fiddly (better make sure you set it back) and problematic in the presence of multiple threads⁶.

A better approach is to allow dynamic bindings: bindings with indefinite scope & dynamic extent. CL has these, and at this point history becomes unavoidable: rather than have some separate construct for dynamic bindings, CL simply says that some variable bindings, and some references to variable bindings, are to be treated as having indefinite scope and dynamic extent, and you tell the system which bindings this applies to withspecial declarations / proclamations. CL does this because that’s very close to how various predecessor Lisps worked, and so makes porting programs from them to CL much easier. To make this less painful there is a convention that dynamically-bound variable names have *stars* around them, of course.

Dynamic bindings are so useful that if you don’t have them you really need to invent them: I have on at least two occasions implemented a dynamic binding system in Python, for instance.

However this is not an article on dynamic bindings so I will not write more about them here: perhaps I will write another article later.

What else can be bound?

Variable bindings are by far the most common kind. But not the only kind. Other things can be bound. Here is a partial list⁷:

local functions have lexical scope and indefinite extent;
block names have lexical scope and definite extent (see below);
tag names have lexical scope and definite extent (see below);
catch tags have indefinite scope and definite extent;
condition handlers have indefinite scope and definite extent;
restarts have indefinite scope and definite extent.

The two interesting cases here are block names and tag names. Both of these have lexical scope but only definite extent. As I argued above this makes it hard to know whether references to them are valid or not. Look at this, for example:

(defun outer (x)
  (inner (lambda (r)
           (return-from outer r))
         x))

(defun inner (r rp)
  (if rp
      r
    (funcall r #'identity)))

So then (funcall (outer nil) 1) will: call inner with a function which wants to return from outer and nil, which will cause inner to call that function, returning the identity function, which is then called by funcall with argument 1: the result is 1.

But (funcall (outer t) 1) will instead return the function which wants to return from outer, which is then called by funcall which is an error since it is outside the dynamic extent of the call to outer.

And there is no way that either a human reading the code or the compiler can detect that this is going to happen: a very smart compiler might perhaps be able to deduce that the internal function might be returned from outer, but probably only because this is a rather simple case: for instance in

(defun nasty (f)
  (funcall f (lambda ()
               (return-from nasty t))))

the situation is just hopeless. So this is a case where the binding rules are not as simple as you might like.

What is simple?

For variable bindings I think it’s easy to see that the simplest rule for a person reading the code is lexical binding. The other question is whether that is simpler for the implementation. And the answer is that probably it is not: probably lexical scope and definite extent is the simplest implementationally. That certainly approximates what many old Lisps did⁸. It’s fairly easy to write a bad implementation of lexical binding, simply by having all functions retain all the bindings, regardless of whether they might refer to them. A good implementation requires more work. But CL’s approach here is that doing the right thing for people is more important than making the implementor’s job easier. And I think that approach has worked well.

On the other hand CL hasn’t done the right thing for blocks and tags: There are at least three reasons for this.

Implementational complexity. If the bindings had lexical scope and indefinite extent then you would need to be able to return from a block which had already been returned from, and go to a tag from outside the extent of the form that established it. That opens an enormous can of worms both in making such an implementation work at all but also handling things like dynamic bindings, open files and so on. That’s not something the CL designers were willing to impose on implementors.

Complexity in the specification. If CL had lexical bindings for blocks and tags then the specification of the language would need to describe what happens in all the many edge cases that arise, including cases where it is genuinely unclear what the correct thing to do is at all such as dealing with open files and so on. Nobody wanted to deal with that, I’m sure: the language specification was already seen as far too big and the effort involved would have made it bigger, later and more expensive.

Conceptual difficulty. It might seem that making block bindings work like lexical variable bindings would make things simpler to understand. Well, that’s exactly what Scheme did with call/cc and call/cc can give rise to some of the most opaque code I have ever seen. It is often very pretty code, but it’s not easy to understand.

I think the bargain that CL has struck here is at least reasonable: to make the common case of variable bindings simple for people, and to avoid the cases where doing the right thing results in a language which is harder to understand in many cases and far harder to implement and specify.

Finally, once again I think that the best way to understand how closures in CL is not to understand them: instead understand the binding rules for variables, why they are simple and what they imply.

indefinite scope is often called ‘dynamic scope’ although I will avoid this term as it is used by the standard to mean the combination of indefinite scope and dynamic extent. ↩
Dynamic extent could perhaps be called ‘definite extent’, but this is not the term that the standard uses so I will avoid it. ↩
Here and below I am using the term ‘function’ in the very loose sense that CL usually uses it: almost none of the ‘functions’ I will talk about are actually mathematical functions: they’re what Scheme would call ‘procedures’. ↩
For languages which don’t have first-class functions or equivalent constructs, lexical scope and definite extent is the same as lexical scope and indefinite extent, because it is not possible to return objects which can refer to bindings from the place those bindings were created. ↩
More likely, you would end up making every function have, for instance an ambient keyword argument whose value would be an alist or plist which mapped between properties of the ambient environment and values for them. All functions which might call other functions would need this extra argument, and would need to be sure to pass it down suitably. ↩
This can be worked around, but it’s not simple to do so. ↩
In other words ‘this is all I can think of right now, but there are probably others’. ↩
Very often old Lisps had indefinite scope and definite extent in interpreted code but lexical scope and definite extent in compiled code: yes, compiled code behaved differently to interpreted code, and yes, that sucked. ↩

A case-like macro for regular expressions

2023-01-11T18:17:29Z

I often find myself wanting a simple case-like macro where the keys are regular expressions. regex-case is an attempt at this.

I use CL-PPCRE for the usual things regular expressions are useful for, and probably for some of the things they should not really be used for as well. I often find myself wanting a case like macro, where the keys are regular expressions. There is a contributed package for Trivia which will do this, but Trivia is pretty overwhelming. So I gave in and wrote regex-case which does what I want.

regex-case is a case-like macro. It looks like

(regex-case <thing>
  (<pattern> (...)
   <form> ...)
  ...
  (otherwise ()
   <form> ...))

Here <pattern> is a literal regular expression, either a string or in CL-PPCRE’s s-expression parse-tree syntax for them. Unlike case there can only be a single pattern per clause: allowing the parse-tree syntax makes it hard to do anything else. otherwise (which can also be t) is optional but must be last.

The second form in a clause specifies what, if any, variables to bind on a match. As an example

(regex-case line
  ("fog\\s+(.*)\\s$" (:match m :registers (v))
    ...)
  ...)

will bind m to the whole match and v to the substring corresponding to the first register. You can also bind match and register positions. A nice (perhaps) thing is that you can not bind some register variables:

(regex-case line
  (... (:registers (_ _ v))
   ...)
  ...)

will bind v to the substring corresponding to the third register. You can use nil instead of _.

The current state of regex-case is a bit preliminary: in particular I don’t like the syntax for binding thngs very much, although I can’t think of a better one. Currently therefore it’s in my collection of toys: it will probably migrate from there at some point.

Currently documentation is here and source code is here.

The empty list

2022-12-16T17:14:32Z

My friend Zyni pointed out that someone has been getting really impressively confused and cross on reddit about empty lists, booleans and so on in Common Lisp, which led us to a discussion about what the differences between CL and Scheme really are here. Here’s a summary which we think is correct.

A peculiar object in Common Lisp¹

In Common Lisp there is a single special object, nil.

This represents both the empty list, and the special false value, all other objects being true.
This object is a list and is the only list object which is not a cons.
As such this object is an atom, and again it is the only list object which is an atom.
You can take the car and cdr of this object: both of these operations return the object itself.
This object is also a symbol, and it is the only object which is both a list and a symbol.
The empty list when written as an empty list, (), is self-evaluating.

Some comments.

It is necessary that there be a special empty-list object which is a list but not a cons: the things which are not necessary are that it be a symbol, and that it represent falsity.
Combining the empty list and the special false object can lead to particularly good implementations perhaps.
The implementation of this object is always going to be a bit weird.
It is clear that the empty list cannot be any kind of compound form so requiring it to be quoted — requiring you to write '() really — serves no useful purpose. Nevertheless I (Tim) would probably rather CL did that.
Not having to quote nil on the other hand is not at all strange: any symbol can be made self-evaluating simply by (defconstant s 's), for instance.
The graph of types in CL is a DAG, not a tree: it is not at all strange that there is an object whose type is both list and symbol.

Some entirely mundane things in Common Lisp

There is a symbol, t which represents the canonical true value. Nothing is magic about this symbol in any way: it could be defined by (defconstant t 't).
There is a type, boolean which could be defined by (deftype boolean () '(member nil t)), except that it is required that boolean be a recognisable subtype of symbol. All implementations we have tried recognise (member nil t) as a subtype of symbol, but the standard does not require them to do so. Additionally (type-of 't) must return boolean we think.
There is a type, null, which could be defined by (deftype null () '(member nil)) or (deftype null () '(eql nil)), with the same caveats as above, and (type-of nil) should return null.
There are types named t (top of the type graph) and nil (bottom of type graph).

These mundane things are just that: they don’t require implementational magic at all.

Three peculiar objects in Scheme

In Scheme there is an object, ().

() is the special object that represents the empty list.
It does not represent false.
It is not a symbol.
It is the only list object which is not a pair (cons): list? is true of it but pair? is false.
You can’t take the car or cdr of it.
It is not self-evaluatiing.

There is another object, #f.

#f is the distinguished false value and is the only false value in Scheme, all other objects being true.
It is not a symbol or a list but satisfies the boolean? predicate.
It is self-evaluating.

There is another object, #t.

#t represents the canonical true value, but all objects other than #f are true.
It is not a symbol or a list but satisfies the boolean? predicate.
It is self-evaluating.

Some comments. - Scheme does not have such an elaborate type system as CL and, apart from numbers, doesn’t really have subtype relations the way CL does.

A summary

CL’s treatment of nil clearly makes some people very unhappy indeed. In particular they seem to think CL is somehow inconsistent, which it clearly is not. Generally this is either because they don’t understand how it works, because it doesn’t work the way they want it to work, or (usually) both. Scheme’s treatment is often cited by these people as being better. But CL requires precisely one implementationally-weird object, while Scheme requires two, or three if you count #t which you probably should. Both languages have idiosyncratic evaluation rules around these objects. Additionally it’s worth understanding that things like CL’s boolean type mean essentially nothing implementationally: boolean is just a name for a set of symbols. The only thing preventing you from defining a type like this yourself is the requirement for type-of to return the type.

Is one better than the other? No: they’re just not the same. Certainly the CL approach carries more historical baggage. Equally certainly it is perfectly consistent, and changing it would break essentially all CL programs that exist.

Thanks to Zyni for most of this: I’m really writing it up just so we can remember it. We’re pretty confident about the CL part, less so about the Scheme bit.

peculiar, adjective: having eccentric or individual variations in relation to the general or predicted pattern, as in peculiar motion or velocity. noun: a parish or church exempt from the jurisdiction of the ordinary or bishop in whose diocese it is placed; anything exempt from ordinary jurisdiction. ↩

Closed as duplicate considered harmful

2022-12-05T16:10:07Z

The various Stack Exchange sites, and specifically Stack Overflow, seem to be some of the best places for getting reasonable answers to questions on a wide range of topics from competent people. They would be a lot better if they were not so obsessed about closing duplicates.

Closing duplicates seems like a good idea: having a single, canonical, question on a given topic with a single, canonical, answer seems like a good thing. It’s not.

The reason it’s not is that it makes two false assumptions:

that a given question has a single best answer;
that this answer does not change over time.

Neither of these assumptions is true for a large number of interesting questions.

Questions can have several good answers. I have at least three introductory books on analysis, and not because I didn’t find the good one on the first try: I have several because they give different perspectives — different answers, in the sense of Stack Exchange — to various aspects of the subject. I have several books on introductory quantum mechanics, several books on introductory general relativity, and so it goes on. It is, simply, a delusion that there exists a single most helpful answer to many questions: pretending that there is stupidly limiting.

And what constitutes a good answer can change over time. If you asked, for instance, what a macro was in Lisp and what macros are good for, you would have got very different answers in 1982 than in 2022¹. The same is true for many other subjects: human knowledge is not static.

All of this is made worse as only the person asking a question can accept an answer: they may not do so at all or, worse, they may be asking in bad faith and accept wrong or misleading answers (yes, this happens in various Stack Exchanges).

The true Stack Exchange believer will now explain in great detail² why none of this matters: people should just spend their time adding improved answers to questions which already have accepted answers rather than to new questions which will be closed as duplicates. Because, of course, the accepted answer will not be the one almost everyone looks at, and even if they don’t care about increasing their karma on Stack Exchange, they will be very happy to write answers that, in the real world, almost nobody will ever look at.

Yeah, right.

This is such a shame: Stack Exchange is a good thing, but it’s seriously damaged by this unnescessary problem. The answer is not simply to allow unrestricted duplicates, but to wait for a bit and see if a question which is, or is nearly, a duplicate has attracted new and interesting answers, and to not close it as a duplicate in that case. This would not be hard to do.

And even in 2022 you will get answers from people who seem not to have learned anything since 1982. ↩
Please, don’t: I don’t have a Stack Exchange account any more and, even if I did, I would not be interested. ↩

Package-local nicknames

2022-10-14T09:26:31Z

What follows is an opinion. Do not under any circumstances read it. Other opinions are available (but wrong).

Package-local nicknames are an abomination. They should be burned with nuclear fire, and their ashes launched into space on a trajectory which will leave the Solar System.

The only reason why package-local nicknames matter is if you are writing a lot of code with lots of package-qualified names in it. If you are doing that then you are writing code which is hard to read: the names in your code are longer than they need to be and the first several characters of them are package name noise (people read, broadly from left to right). Imagine me:a la:version ge:of oe:English oe:where la:people wrote like that: it’s just horrible. If you are writing code which is hard to read you are writing bad code.

Instead you should do the work to construct a namespace in which the words you intend to use are directly present. This means constructing suitable packages: the files containing the package definitions are then almost the only place where package names occur, and are a minute fraction of the total code. Doing this is a good practice in itself because the package definition file is then a place which describes just what names your code needs, from where, and what names it provides. Things like conduit packages (shameless self-promotion) can help with this, which is why I wrote them: being able to say ‘this package exports the combination of the exports of these packages, except …’ or ‘this package exports just the following symbols from these packages’ in an explicit way is very useful.

If you are now rehearsing a litany of things that can go wrong with this approach in rare cases¹, please don’t: this is not my first rodeo and, trust me, I know about these cases. Occasionally, the CL package system can make it hard or impossible to construct the namespace you need, with the key term here being being occasionally: people who give up because something is occasionally hard or impossible have what Erik Naggum famously called ‘one-bit brains’²: the answer is to get more bits for your brain.

Once you write code like this then the only place package-local nicknames can matter is, perhaps, the package definition file. And the only reason they can matter there is because people think that picking a name like ‘XML’ or ‘RPC’ or ‘SQL’ for their packages is a good idea. When people in the programming section of my hollowed-out-volcano lair do this they are … well, I will not say, but my sharks are well-fed and those things on spikes surrounding the crater are indeed their heads.

People should use long, unique names for packages. Java, astonishingly, got this right: use domains in big-endian order (org.tfeb.conduit-packages, org.tfeb.hax.metatronic). Do not use short nicknames. Never use names without at least one dot, which should be reserved for implementations and perhaps KMP-style substandards. Names will now not clash. Names will be longer and require more typing, but this will not matter because the only place package names are referred to are in package definition files and in in-package forms, which are a minute fraction of your code.

I have no idea where or when the awful plague of using package-qualified names in code arose: it’s not something people used to do, but it seems to happen really a lot now. I think it may be because people also tend to do this in Python and other dotty languages, although, significantly, in Python you never actually need to do this if you bother, once again, to actually go to the work of constructing the namespace you want: rather than the awful

import sys

... sys.argv ...

...

sys.exit(...)

you can simply say

from sys import argv, exit

... argv ...

exit(...)

and now the very top of your module lets anyone reading it know exactly what functionality you are importing and from where it comes.

It may also be because the whole constructing namespaces thing is a bit hard. Yes, it is indeed a bit hard, but designing programs, of which it is a small but critical part, is a bit hard.

OK, enough.

If, after reading the above, you think you should mail me about how wrong it all is and explain some detail of the CL package system to me: don’t, I do not want to hear from you. Really, I don’t.

in particular, if your argument is that someone has used, for instance, the name set in some package to mean, for instance, a set in the sense it is used in maths, and that this clashes with cl:set and perhaps some other packages, don’t. If you are writing a program and you think, ‘I know, I’ll use a symbol with the same name as a symbol exported from CL to mean something else’ in a context where users of your code also might want to use the symbol exported by CL (which in the case of cl:set is ‘almost never’, of course), then my shark pool is just over here: please throw yourself in. ↩
Curiously, I think that quote was about Scheme, which I am sure Erik hated. But, for instance, Racket’s module system lets you do just the things which are hard in the package system: renaming things on import, for instance. ↩

Bradshaw's laws

2022-10-03T19:50:51Z

There are two laws.

The laws

Bradshaw’s law. All sufficiently large software systems end up being programming languages.
Zyni’s corollary. Whenever you think the point is at which the first law will apply, it will apply before that.

Implications of the laws

When building software systems you should design them as programming langages. You should do this however small you think they will be. In order to make this practical for small systems you should therefore use a language which allows seamless extension into other languages with insignificant zero-point cost.

But because the laws are not widely known, most large software systems are built without understanding that what is being built is in fact a programming language. Because people don’t know they are building a programming language, don’t know how to build programming languages, and do not use languages which make the seamless construction of programming languages easy, the languages they build are usually terrible: they are hard to use, have opaque and inconsistent semantics and are almost always insecure.

Simple logging in Common Lisp

2022-09-26T11:26:32Z

slog is a simple logging framework for Common Lisp based on the observation that conditions can represent log events.

slog is based on an two observations about the Common Lisp condition system:

conditions do not have to represent errors, or warnings, but can just be a way of a program saying ‘look, something interesting happened’;
handlers can decline to handle a condition, and in particular handlers are invoked before the stack is unwound.

Well, saying ‘look, something interesting happened’ is really quite similar to what logging systems do, and slog is built on this idea.

slog is the simple logging system: it provides a framework on which logging can be built but does not itself provide a vast category of log severities &c. Such a thing could be built on top of slog, which aims to provide mechanism, not policy.

slog provides a couple of conditions representing log entries, which are designed to be subclassed in real life. Log entries are created using a slog function (this is why slog is called slog: log is already taken) which simply signals an appropriate condition. Handlers are set up by a logging form (this should really be called slogging but it is not), which associates conditions with handlers. There is fairly flexible file handling for logging to files, and in particular you can refer to file names which all get associated with the approprate stream, streams get closed automagically (and you can manually close them, when they will be reopened if need be), and the underlying mechanism for writing entries is exposed by a slog-to generic function which could be extended. Log entry formats can be controlled in various ways.

In addition slog tries to associate log entries with ‘precision time’, which is CL’s universal time expanded to the precision of a millisecond, or of internal time if it is less precise than a millisecond. Setting this up means that slog takes a second or so to load.

Once again: slog is a framework: it has no dealings with log severities, catagories, or anything like that. All that is meant to be provided on top of what slog provides.

Documentation is here, source code is here. It will be available from Quicklisp in due course.

Metatronic macros

2022-09-26T10:54:25Z

Metatronic macros are a simple hack which makes it a little easier to write less unhygienic macros in Common Lisp.

Common Lisp macros require you to avoid variable name capture yourself. So, for a macro which iterates over the lines in a file, this is wrong:

(defmacro with-file-lines ((line file) &body forms)
  ;; wrong
  `(with-open-file (in ,file)
     (do ((,line (read-line in nil in)
                 (read-line in nil in)))
         ((eq ,line in))
       ,@forms)))

It’s wrong because it binds in to the stream open to the file, and user code could perfectly legitimately refer to a variable of the same name.

The standard approach to dealing with this is to use gensyms:

(defmacro with-file-lines ((line file) &body forms)
  ;; righter
  (let ((inn (gensym)))
    `(with-open-file (,inn ,file)
     (do ((,line (read-line ,inn nil ,inn)
                 (read-line ,inn nil ,inn)))
         ((eq ,line inn))
       ,@forms))))

This creates a new symbol bound to inn (in’s name), and then uses it as the name of the variable bound to the stream. Code can’t then use any variable with this unique name.

This works, but it’s ugly. Metatronic macros let you write the above like this:

(defmacro/m with-file-lines ((line file) &body forms)
  ;; righter, easier
  `(with-open-file (<in> ,file)
     (do ((,line (read-line <in> nil <in>)
                 (read-line <in> nil <in>)))
         ((eq ,line <in>))
       ,@forms)))

In this macro all symbols which look like <…> (in any package) are rewritten to unique names, but all references to symbols with the same original name are to the same symbol¹. This makes this common case more pleasant to do: macros written using defmacro/m have less noise around their expansion.

Metatronic macros go to some lengths to avoid leaking the rewritten symbols. Given this silly macro

(defmacro/m silly ()
  ''<silly>)

then (eq (silly) (silly)) is false. Similarly given this:

(defmacro/m also-silly (f)
  `(eq ,f '<silly>))

Then (also-silly '<silly>) will be false of course.

There is defmacro/m, macrolet/m and define-compiler-macro/m, and the implementation of metatronization is exposed if you need it.

Documentation is here, source code is here. It will be available in Quicklisp in due course.

in fact, a symbol whose name is <> is rewritten as a unique gensym as a special case. I am not sure if this is a good thing but it’s what happens. ↩

Macros (from Zyni)

2022-08-27T10:12:33Z

It is the business of the future to be dangerous; and it is among the merits of science that it equips the future for its duties. — Alfred Whitehead

Once upon a time, long ago in a world far away, Lisp had many features which other languages did not have. Automatic storage management, dynamic typing, an interactive environment, lists, symbols … and macros, which allow you to seamlessly extend the language you have into the language you want and need.

But that was long long ago in a world far away where giants roamed the earth, trolls lurked under every bridge and, they say, gods yet lived on certain distant mountains.

Today, and in this world, many many languages have automatic storage management, are dynamically typed, have symbols, lists, interactive environments, and so and so and so. More of these languages arise from the thick, evil-smelling sludge that coats every surface each day: hundreds, if not thousands of them, like flies breeding on bad meat which must be swatted before they lay their eggs on your eyes.

Lisp, today and in this world not another, has exactly one feature which still distinguishes it from the endless buzz of these insect languages. That feature is seamless language extension by macros.

So yes, macros are dangerous, and they are hard and they are frightening. They are dangerous and hard and frightening because all powerful magic is dangerous and hard and frightening. They are dangerous because they are a thing which has escaped here from the future and it is the business of the future to be dangerous.

If macros are too dangerous, too hard and too frightening for you, do not use Lisp because macros are what Lisp is about.

This originated as a comment by my friend Zyni: it is used with her permission.

Two simple pattern matchers for Common Lisp

2022-07-21T09:17:45Z

I’ve written two pattern matchers for Common Lisp:

destructuring-match, or dsm, is a case-style construct which can match destructuring-bind-style lambda lists with a couple of extensions;
spam, the simple pattern matcher, does not bind variables but lets you match based on assertions about, for instance, the contents of lists.

Both dsm and spam strive to be simple and correct.

Simplicity

Both dsm and spam are simple: they do exactly one thing, and try to do that one thing well.

You could think of dsm as being to some other CL pattern matchers as Unix once was to Multics: dsm is the result of me looking at those other systems and thinking ‘please, not that’.

Those systems are vast, have several levels, and are extensible: some subset of them might do what I wanted to be able to do — make writing macros less unpleasant — but I’m not sure¹. They are obsessed with performance.

dsm does one thing, and exports a single macro. If you know how to use destructuring-bind and case you already know almost all there is to know about dsm: it’s a case construct whose cases are destructuring-bind lambda lists. dsm doesn’t care about performance at all, because macroexpansion performance never matters.

At least one of those matchers has almost as many commits in its repo as dsm has lines of code.

Like Multics was, those hairy pattern matchers are fine systems. But there was a good reason that Thompson and Ritchie wrote something very different².

`destructuring-match` / `dsm`

in CL destructuring-bind and, mostly equivalently, macro argument lists are both a blessing and a curse. They’re a blessing because they support destructuring, so you can write, for instance

(defmacro with-foo ((var &optional init) &body forms)
  ...)

They’re a curse because they are so fragile: with-foo can only support that syntax and will fail with an ugly error message from the implementation when it is fed anything else.

Writing robust macros in CL, especially macros which expect various different argument patterns, then turns into a great saga of manually checking argument patterns before using destructuring-bind to actually bind things. The result of that, of course, is that very many CL macros are not robust and have terrible error reporting.

destructuring-match does away with all this unpleasentness. It supports a slightly extended version of the lambda lists that destructuring-bind supports, has ‘guard’ clauses which allow additional checks, and will match a form against any number of lambda lists until one matches, with a fallback case.

As an example here is a version of with-foo which allows two patterns:

(defmacro with-foo (&body forms)
  (destructuring-match forms
    (((var &optional init) &body body)
     (:when (symbolp var))
     ...)
    ((((var &optional type) &optional init) &body body)
     (:when (symbolp var))
     ...)
    (otherwise
     (error ...))))

The guard clauses check that var is a symbol before the match succeeds, and will therefore ensure that the second match is the one chosen for (with-foo ((x y) 1) ...).

destructuring-match also supports ‘blank’ variables: any variable whose name is _ (in any package) is ignored, and all such variables are distinct. So for instance

(destructuring-match l
  ((_ _ _) ...))

will match if l is a proper list with exactly three elements.

Using destructuring-match it’s easy to write this macro³:

(defmacro define-matching-macro (name &body clauses)
  (let ((<whole> (make-symbol "WHOLE"))
        (<junk> (make-symbol "JUNK")))
    (destructuring-match clauses
      ((doc . the-clauses)
       (:when (stringp doc))
       `(defmacro ,name (&whole ,<whole> &rest ,<junk>)
          ,doc
          (destructuring-match ,<whole> ,@the-clauses)))
      (the-clauses
       `(defmacro ,name (&whole ,<whole> &rest ,<junk>)
          (destructuring-match ,<whole> ,@the-clauses))))))

And this then allows the above with-foo macro to be written like this:

(define-matching-macro with-foo
  ((_ (var &optional init) &body forms)
   (:when (symbolp var))
   ...)
  ((_ ((var &optional type) &optional init) &body forms)
   (:when (symbolp var))
   ...)
  (form
   (error "~S is bad syntax for with-foo" form)))

dsm was not written with performance in mind but it seems to be, typically, around a tenth to a half the speed of destructuring-bind while being far more powerful of course.

dsm can be found here. It will probably end up in Quicklisp in due course but currently it isn’t there, and some of its dependencies are also not up to date there.

`spam`, the simple pattern matcher

dsm has a lot of cases where it needs to check what the lambda list it is parsing and compiling looks like. To do this I wrote a bunch of predicate constructors and combinators, which return predicates which will check things. So for example:

(is 'foo) returns a function which checks its argument is eql to foo;
(some-of p1 ... pn) returns a function of one argument which will succeed if one of the predicates which are its arguments succeeds, so (some-of (is 'foo) (is 'bar));
(head-matches p1 ... pn) will succeed if the predicates which are its arguments succeed on the first elements of a list.

There are several other predicate constructrors and predicate combinators, but spam can use any predicate.

There is then a matches macro which uses these to match things, and a matchp function which simply invokes a predicate.

As an example, here’s part of a matcher for &rest specifications in lambda lists.

(matching ll
  ((head-matches (some-of (is '&rest) (is '&body))
                 (var)
                 (is '&key))
   ;; &rest x &key ...
   ...)
   ((head-matches (some-of (is '&rest) (is '&body))
                  (var)
                  (any))
    ;; &rest x with something else
    ...)
   ((list-matches (some-of (is '&rest) (is '&body))
                  (var))
    ;; &rest x and no more
    ...)
   (otherwise
    (error "oops")))

spam is pretty useful, and code written using it is much easier to read than doing the equivalent checks manually. It is used extensively in the implementation of dsm.

spam is now one of my CL hax.

At the time of writing Trivia supports lambda lists I think, but not destructuring-lambda lists: (match '(1 (1)) ((lambda-list a (b)) (values a b))) will fail, for instance. I don’t know whether is it meant to support destructuring lambda lists — comments in the sources imply it is, but it clearly does not in fact. ↩
I am aware of Gabriel’s ‘worse is better’ paper and its various afterthoughts. dsm is not like that: it is smaller and simpler, but is not intended to be worse. dsm is to these other systems perhaps as Scheme was to CL. Gabriel also talks about these two options, of course. ↩
Note this macro is 12 lines, half of which are handling the possible docstring. ↩

Macroexpansion in Common Lisp

2022-07-05T15:16:29Z

Yet another description of macroexpansion in Common Lisp. There is nothing particuarly new here and it partly duplicates some previous articles: I just wanted to rescue the text.

The following description is of how macroexpansion works in Common Lisp¹. It is slightly simplified and I have not always mentioned when it is². It is at least a partial duplicate of this previous article.

What macros are

Macros in CL are functions, written in ordinary CL, whose argument is source code, and whose value is other source code.

Source code is represented as s-expressions: symbols, conses, and so on. Macros don’t do string-rewriting.

The way to think slightly more abstractly about macros is that they are functions between languages: a macro is a function which takes as an argument fragments of a language which includes that macro, and returns as a value either a fragment of a language which doesn’t include the macro, or a fragment of a language which includes it in some weaker way.

The aim of macros is to build, on top of the language you are given, another language which is closer to the language in which you want to express your programs. CL itself is one such language, built-up using a number of standard macros on top of a substrate language.

People often think of macros as ‘functions which do not evaluate their arguments’: that’s really not right. They are functions — perfectly ordinary functions, written in CL — but their argument is source code, and their value is source code.

How macroexpansion happens

[This is simplified.]

Given some initial compound form (m ...), macroexpansion proceeds like this.

Start. Given a form, it should be one of

a compound form (m ...),
or a non-compound form.

Compound form. The form is (m ...)

Look at m: if it has an associated macro function (found using macro-function) then simply call that function on the whole form (m ...): its result is a new form³. Recurse on this form from Start.
If m is not a macro, then it may be a special operator, such as setq or if. Consider appropriate forms in the body of this form for expansion: which forms are known by the rules of the special operator. For instance all the forms in (if ...) are considered for expansion, while in (setq <x> <y>) only <y> is, and so on.
If it is not a macro and not a special form, then (m ...) is assumed to be a function call, with m denoting a function. All the forms in the body are now considered for macro expansion. Once that is done the expansion process is complete.
As a special case of the last case, m may be (lambda (...) ...), so the whole form will be ((lambda (...) ...) ...). In this case the forms in the body of the lambda are considered for macroexpansion; otherwise this is the same as the last case⁴.
There are no other cases.

Non-compound form. There is nothing to do here.

As I said, this is simplified: there are local macros for instance, and various other things. However one critical thing is that when expanding some macro form (m ...), the expansion carries on until it gets something which is not a macro form before looking at whatever is in the body of the form. That’s critical: although it’s tempting to think that expansion should happen inside-out, it can’t work that way, because until the outer macro has done its work you can’t know if the things in its body even should be candidates for macro expansion. There’s an example of this below.

Macros the hard way

OK, I said that macros were just functions, and I meant that. Let’s write a macro with-debugging which is like progn but it will perhaps print what it is doing.

So let’s write the macro function:

(defvar *debugging* t)

(defun expand-with-debugging (form environment)
  (declare (ignore environment))        ;I'm not mentioning environments
  `(progn
     ,@(loop for thing in (rest form)
             collect `(when *debugging*
                        (format *debug-io* "~&~S~%" ',thing))
             collect thing)))

And we can test it:

> (expand-with-debugging '(with-debugging (cons 1 2) 4) nil)
(progn
  (when *debugging* (format *debug-io* "~&~S~%" '(cons 1 2)))
  (cons 1 2)
  (when *debugging* (format *debug-io* "~&~S~%" '4))
  4)

And now we can install it as the macro function for with-debugging:

(setf (macro-function 'with-debugging) #'expand-with-debugging)

And now

> (with-debugging
   (cons 1 2)
   4)
(cons 1 2)
4
4

 (setf *debugging* nil)
nil

> (with-debugging
   (cons 1 2)
   4)
4

OK, here’s another macro done this way, and purpose of this one is to show you why macroexpansion has to happen outside in. Let’s say we want to be able to denote functions by (fun (arg ...) form ...), but we’d like to be able to debug the body with with-debugging. We can do that:

(defun expand-fun (form environment)
  (declare (ignore environment))        ;still not mentioning environments
  `(function (lambda ,(second form)
               ;; Not dealing with declarations
               (with-debugging ,@(cddr form)))))

(setf (macro-function 'fun) #'expand-fun)

And now

> (let ((*debugging* t))
    (funcall (fun (a) (+ a a)) 1))
(+ a a)
2

Now you can see why the macro expander has to work the way it does: the first form in the body of fun should not be macroexpanded at all, and the remaining forms are going to get wrapped in a macro which isn’t there in the source at all. So macroexpansion has to go outside in, as described above.

A better way

Well, you could write macros like that. Probably once they were written like that. But it’s a pain, because you almost never care about the first element of the form — the macros own name — and you have to manually take the rest of the form apart yourself. And also you need to deal with questions about making sure macros are defined at compile time and so on.

That’s what defmacro does. It is itself a macro, and its expansion will involve setting the macro-function of the macro to some appropriate thing. So using defmacro I can write the fun macro:

(defmacro fun ((&rest args) &body forms)
  ;; still not dealing with declarations
  `(function (lambda (,@args) (with-debugging ,@forms))))

This is easier to understand of course. But all it is is a (fairly elaborate!) wrapper around what I did above.

Watching the detectives

Using trace-macroexpand you can watch macroexpansion happen.

> (needs (:org.tfeb.hax.trace-macroexpand :compile t :use t))
; Loading [...]
((:org.tfeb.hax.trace-macroexpand t))

> (trace-macroexpand t)
nil

> (trace-macro fun with-debugging)
> (setf *trace-macroexpand-print-length* nil
        *trace-macroexpand-print-level* nil)
nil

> (trace-macro fun with-debugging)
(fun with-debugging)

> (setf *debugging* nil)                
nil

> (funcall (fun (a) a) 1)
(fun (a) a)
 -> #'(lambda (a) (with-debugging a))
(with-debugging a)
 -> (progn (when *debugging* (format *debug-io* "~&~S~%" 'a)) a)
(with-debugging a)
 -> (progn (when *debugging* (format *debug-io* "~&~S~%" 'a)) a)
1

Note that with-debugging is expanded twice: this is an artifact of the implementation: there’s no promise that macros only get expanded once in interpreted code.

This was once going to be a Stack Overflow answer, and I didn’t want to throw it away. ↩
And of course I might just be wrong about some details. ↩
I am not talking about the environment objects which get passed to macro functions. ↩
Another way of thinking about ((lambda (...) ...) ...) is that is is the same as (funcall (function (lambda (...) ...)) ...) and, since function is a special operator, its rules apply, and include expanding the forms in the body of the (lambda (...) ...) form (and of course lambda is itself a macro, so (lambda (...) ...) expands to (function (lambda (...) ...))) and then the rules for function apply again). I am old enough to remember adding the macro for lambda to various antique CLs. ↩

Avoiding circularity: a simple example

2022-03-23T17:54:40Z

Here’s a simple example of dealing with a naturally circular function definition.

Common Lisp has a predicate called some. Here is what looks like a natural definition of a slightly more limited version of this predicate, which only works on lists, in Racket:

(define (some? predicate . lists)
  ;; Just avoid the spread/nospread problem
  (some*? predicate lists))

(define (some*? predicate lists)
  (cond
    [(null? lists)
     ;; if there are no elements the predicate is not true
     #f]
    [(some? null? lists)
     ;; if any of the lists is empty we've failed
     #f]
    [(apply predicate (map first lists))
     ;; The predicate is true on the first elements
     #t]
    [else
     (some*? predicate (map rest lists))]))

Well, that looks neat, right? Except it is very obviously doomed because some*? falls immediately into an infinite recursion.

Well, the trick to avoid this is to check whether the predicate is null? and handle that case explicitly:

(define (some*? predicate lists)
  (cond
    [(null? lists)
     ;; 
     (error 'some? "need at least one list")]
    [(eq? predicate null?)
     ;; Catch the circularity and defang it
     (match lists
       [(list (? list? l))
        (cond
          [(null? l)
           #f]
          [(null? (first l))
           #t]
          [else
           (some? null? (rest l))])]
       [_ (error 'some? "~S bogus for null?" lists)])]
    [(some? null? lists)
     ;; if any of the lists is empty we've failed
     #f]
    [(apply predicate (map first lists))
     ;; The predicate is true on the first elements
     #t]
    [else
     (some*? predicate (map rest lists))]))

And this now works fine.

Of course this is a rather inefficient version of such a predicate, but it’s nice. Well, I think it is.

Note: a previous version of this had an extremely broken version of some*? which worked, by coincidence, sometimes.

Two understandable deficiencies in Common Lisp

2022-03-22T09:58:28Z

Common Lisp is, I think, a remarkably pleasant language, despite what some people like to say. Here are two small deficiencies, both of which are understandable in terms of the history of CL, and both of which ultimately hurt naïve programmers working in CL.

The default floating-point type is `single-float`

There are two things that make this true:

*read-default-float-format* is initially single-float, which means that, unless it is changed, 1.0 reads as 1.0f0, a single float¹;
The float function will convert to a single float unless it is given a prototype which is not a single float: (float 1) is 1.0f0, while to get a double float you would need (float 1 1.0d0).

In addition things like with-standard-io-syntax bind *read-default-float-format* to single-float, so you have to do a little more work to make doubles the default.

I think there are probably several historical reasons why this default was chosen:

a long time ago memory was very expensive and single floats take, usually, half the memory of double floats, thus pushing people towards single floats;
a long time ago, perhaps, on some machines, single float operations were significantly faster than double float operations even before possible float consing was taken into account;
Lisp hardware companies with significant influence on the standard, notably Symbolics, made hardware which allowed single (32 bit) floats to be immediate objects, while double floats were not, and had simple-minded compilers which were not capable of optimizing double float operations, thus making double float arithmetic extremely slow compared to single float arithmetic, and these companies wanted their machines to seem fast (they never, really, were) for naïve users;
it was not clear that implementations would choose single-float to mean ‘single precision IEEE 754 float’ and double-float to mean ‘double precision IEEE 754 float’, for instance it’s perfectly legal to have the short-float type mean single precision IEEE 754 and all of the single-float, double-float and long-float types mean double precision IEEE 754;
it wasn’t even even clear that IEEE 754 would come to dominate how machines implement floating-point: VAXes didn’t, and other machines of interest at the time also did not.

So there are good historical reasons for this. However all implementations I’m aware of now translate short-float to mean single-float, single-float to mean IEEE 754 single precision, double-float to mean IEEE 754 double precision and long-float to be the same as double-float.

So what is the problem with the default float type being single-float in the modern world? The answer is

> (log (/ 1 single-float-epsilon) 10)
7.22472

In other words, single precision IEEE 754 arithmetic has about 7 significant figures of precision. For many purposes, and especially for naïvely-written code that’s at best marginal and at worst less than that. On the other hand

> (log (/ 1 double-float-epsilon) 10)
15.954589770191001D0

which is almost 16 significant figures of precision, more than twice that of single precision.

That’s why the default should have been double precision: it makes naïve code more likely to work, and people who are writing non-naïve code can use single precision if they need it.

The `CL-USER` package is defined in an implementation-dependent way

From the spec:

The COMMON-LISP-USER package is the current package when a Common Lisp system starts up. This package uses the COMMON-LISP package. The COMMON-LISP-USER package has the nickname CL-USER. The COMMON-LISP-USER package can have additional symbols interned within it; it can use other implementation-defined packages.

(My emphasis.)

What this means is that when you start a CL environment, the current package may have all sorts of implementation-dependent symbols visible in it. You can see why this happened: if you’re implementing Super-Whizz-Bang CL which has all sorts of magic extra features, you want at least some of those features to be immediately available to users, rather than requiring them to pore over boring manuals to find them.

But for users, and especially for naïve users, it’s a terrible choice: naïve users don’t know about packages so they write their programs in CL-USER. And they also don’t really know which symbols available in CL-USER come from CL and are thus standard parts of the language, and which come from one of Super-Whizz-Bang CL’s implementation packages, and are not standard parts of the language. So their programs turn into a mess where the portable parts are not distinct from the non-portable parts. The way the CL-USER package is defined thus makes it harder for to write programs whose non-portable parts are well-isolated, and ultimately hurts the language.

This is a direct conflict between implementors and users: implementors both want their extra features immediately available so their implementation is shinier and want to encourage users to use these extra features in a way which makes it hard to move their programs to other implementations; users, when they think about it, generally don’t want this second thing, at least.

Instead, the language should have defined CL-USER as a package which only used CL, and perhaps have defined another standard package, perhaps IMPL-USER, which was defined the way CL-USER is today.

Can these be fixed?

While both of these problems could be fixed without changing the standard, I don’t think either can realistically be fixed.

For the single-float problem there is nothing to stop implementations simply defining short-float to mean IEEE 754 single precision and all the other types to mean IEEE 754 double precision. But all the existing code which assumes otherwise will then probably break in exciting ways. So this is unlikely to happen I expect.

The CL-USER problem could be fixed if implementations agree to define CL-USER to use only CL as it is allowed to do, and perhaps to define an IMPL-USER package as above. Of course that will make implementations slightly less convenient to use, so the chances of it happening would be small, even if implementors actually talked to each other in any useful way which I suspect they no longer do. Worse than that, this change will break many programs written by naïve users which live in CL-USER, and there are almost certainly lots of those.

A moment of convenience, a lifetime of regret, as the old saying goes.

An earlier version of this article had single floats written as, for instance 1.0s0: that’s wrong, those are short floats, single floats are 1.0f0 for instance. These are almost certainly the same type on any current implementation (and I think on any implementation I have ever used, hence the mistake) but they don’t have to be. Thanks to Prem Nirved for finding this stupidity. ↩

The endless droning: corrections and clarifications

2021-11-25T13:05:57Z

It seems that my article about the existence in the Lisp community of rather noisy people who seem to enjoy complaining rather than fixing things has atracted some interest. Some things in it were unclear, and some other things seem to have been misinterpreted: here are some corrections and clarifications.

First of all some people pointed out, correctly, that LispWorks is expensive if you live in a low-income country. That’s true: I should have been clearer that I believe the phenonenon I am describing is exclusively a rich-world one. I may be incorrect but I have never heard anyone from a non-rich-world country doing this kind of destructuve whining.

It may also have appeared that I am claiming that all Lisp people do this: I’m not. I think the number of people is very small, and that it has always been small. But they are very noisy and even a small number of noisy people can be very destructive.

Some people seem to have interpreted what I wrote as saying that the current situation was fine and that Emacs / SLIME / SLY was in fact the best possible answer. Given that my second sentence was

[Better IDEs] would obviously be desirable.

this is a curious misreading. Just in case I need to make the point any more strongly: I don’t think that Emacs is some kind of be-all and end-all: better IDEs would be very good. But I also don’t think Emacs is this insurmountable barrier that people pretend it is, and I also very definitely think that some small number of people are claiming it is because they want to lose.

I should point out that this claim that it is not an insurmountable barrier comes from some experience: I have taught people Common Lisp, for money, and I’ve done so based on at least three environments:

LispWorks;
Something based around Emacs and a CL running under it;
Genera.

None of those environments presented any significant barrier. I think that LW was probably the most liked but none of them got in the way or put people off.

In summary: I don’t think that the current situation is ideal, and if you read what I wrote as saying that you need to read more carefully. I do think that the current situation is not going to deter anyone seriously interested and is very far from the largest barrier to becoming good at Lisp. I do think that, if you want to do something to make the situation better then you should do it, not hang around on reddit complaining about how awful it is, but that there are a small number of noisy people who do exactly that because, for them, no situation would be ideal because what they want is to avoid being able to get useful work done. Those people, unsurprisingly, often become extremely upset when you confront them with this awkward truth about themselves. They are also extremely destructive influences on any discussion around Lisp. (Equivalents of these noisy people exist in other areas, of course.) That’s one of the reasons I no longer participate in the forums where these people tend to exist.

(Thanks to an ex-colleague for pointing out that I should perhaps post this.)

The endless droning

2021-11-22T12:36:25Z

Someone asked about better Lisp IDEs on reddit. Such things would obviously be desirable. But the comments are entirely full the usual sad endless droning from people who need there always to be something preventing them from doing what they pretend to want to do, and are happy to invent such barriers where none really exist. comp.lang.lisp lives on in spirit if not in fact.

[The rest of this article is a lot ruder than the above and I’ve intentionally censored it from the various feeds. See also corrections and clarifications.]

First of all it is nice to see people dismissing LispWorks because it’s ‘too expensive’. LW actually has an IDE and it actually does provide an editor which (while an Emacs inside) can pretend to be a native mac or windows editor. And it’s portable: you can develop on Windows and then build and deploy on Linux and that just works, and has done for at least two decades. But it’s ‘too expensive’: a new license for LW might cost the equivalent of a few days of employing a programmer, and the support on that license (which gets you upgrades for ever) might be a day or so. If that’s ‘too expensive’ then your costing is so fucked you might as well give up now and become a beggar. (The announcement of the Haskell IDE which triggered the post is for a commercial one, by the way, so let’s not have any ‘oh, but it’s not ideologically pure’ noise, thanks.)

And then we get the endless ‘things were better on ⟨ancient technology of your choice⟩’. Here’s the thing: I used both Symbolics and Interlisp-D based systems, extensively. They weren’t better than the LW IDE. They had one or two neat features that the LW IDE doesn’t because it’s hard to do on modern hardware, but they were not better. In the case of Interlisp-D systems it took a couple of weeks of practice before you could even use the thing for more than ten minutes without spending most of the time wondering what some front panel code meant (it always meant ‘I have crashed for reasons I cannot explain and you have lost your work and must now reload the sysout and that will take half an hour’) and how to restart it. That was … harder than learning Emacs. Those ancient systems might have been better than Emacs/SLIME … but they might not, I am not sure. But always, always there is the endless mindless droning from people mourning some distant lost golden age: well, I was there and that golden age never existed.

And then there’s the ‘but the new programmers find Emacs hard’. Seriously? Because people starting to learn Lisp are learning a language whose key idea is that it is a programming language in which you write programming languages. Lisp makes doing far more possible than other languages, but nothing is ever going to make it easy because designing programming languages turns out to be hard. Lisp is a language all of whose interesting features are intellectually difficult ideas. If you are put off Lisp by having to learn some different keys to press, give up now and learn Python or some other intellectually undemanding language instead, because Emacs is not remotely the hardest thing you are going to have do deal with. This is like people doing maths degrees complaining about the squiggly Greek characters: if that’s putting you off maths, don’t do maths. OK, ζ and ξ are kind of fiddly to write, but understanding what a Banach space is actually is hard. And, by the way, at some point you are going to have to learn LaTeX, and if you think Emacs is hard, you have a whole other think coming.

Oh, and by the way, I’ve worked somewhere where large numbers of people from non-programming backgrounds wrote vast masses of Python. How did they do it? They used Emacs: some of them probably used vi or vim. But they were actual scientists so they know what hard things are, and knew that learning Emacs was not one of those things.

And finally, there’s a long diatribe from someone listing all the steps they had to go through to get a CL IDE set up on a machine. This same person claims to have run teams of Lisp programmers. Well, there’s this idea called programming: if you have a long laborious set of tasks to do more than once you write a program to do that for you. And yes, I have done just that.

All of these people want to lose: they need there always to be something in the way that prevents them getting whatever it is they pretend to want to do done. If such a barrier is removed they will build a new one: I know this because I have done just that and watched them build their new barrier so they could avoid actually doing anything and keep complaining. These barriers do not exist: if you want a cross-platform IDE for Lisp that IDE exists. If you don’t want to use a commercial product, Emacs and SLIME/SLY are free, and fine. And yes there is a learning curve which is somewhat steep, but intellectually difficult things have steep learning curves: if you’re going to become a productive mathematician you are going to go through four years of very steep learning curve indeed, and if you’re going to become a productive Lisp programmer you’re going to go through a learning curve perhaps a tenth or less as hard as that, of which Emacs is one tiny part. If you’re not up to that, don’t write Lisp.

And if what you enjoy doing is whining in public about how things are always in your way then fuck off.

The proper use of macros in Lisp

2021-11-11T14:32:11Z

People learning Lisp often try to learn how to write macros by taking an existing function they have written and turning it into a macro. This is a mistake: macros and functions serve different purposes and it is almost never useful to turn functions into macros, or macros into functions.

Let’s say you are learning Common Lisp¹, and you have written a fairly obvious factorial function based on the natural mathematical definition: if $n \in \mathbb{N}$, then

\[ n! = \begin{cases} 1 &n \le 1\\ n \times (n - 1)! &n > 1 \end{cases} \]

So this gives you a fairly obvious recursive definition of factorial:

(defun factorial (n)
  (if (<= n 1)
      1
    (* n (factorial (1- n )))))

And so, you think you want to learn about macros so can you write factorial as a macro? And you might end up with something like this:

(defmacro factorial (n)
  `(if (<= ,n 1)
      1
    (* ,n (factorial ,(1- n )))))

And this superficially seems as if it works:

> (factorial 10)
3628800

But it doesn’t, in fact, work:

> (let ((x 3))
    (factorial x))

Error: In 1- of (x) arguments should be of type number.

Why doesn’t this work and can it be fixed so it does? If it can’t what has gone wrong and how are macros meant to work and what are they useful for?

It can’t be fixed so that it works. trying to rewrite functions as macros is a bad idea, and if you want to learn what is interesting about macros you should not start there.

To understand why this is true you need to understand what macros actually are in Lisp.

What macros are: a first look

A macro is a function whose domain and range is syntax.

Macros are functions (quite explicitly so in CL: you can get at the function of a macro with macro-function, and this is something you can happily call the way you would call any other function), but they are functions whose domain and range is syntax. A macro is a function whose argument is a language whose syntax includes the macro and whose value, when called on an instance of that language, is a language whose syntax doesn’t include the macro. It may work recursively: its value may be a language which includes the same macro but in some simpler way, such that the process will terminate at some point.

So the job of macros is to provide a family of extended languages built on some core Lisp which has no remaining macros, only functions and function application, special operators & special forms involving them and literals. One of those languages is the language we call Common Lisp, but the macros written by people serve to extend this language into a multitude of variants.

As an example of this I often write in a language which is like CL, but is extended by the presence of a number of extra constructs, one of which is called ITERATE (but it predates the well-known one and is not at all the same):

(iterate next ((x 1))
 (if (< x 10)
     (next (1+ x))
   x)

is equivalent to

(labels ((next (x)
          (if (< x 10)
              (next (1+ x))
            x)))
 (next 1))

Once upon a time when I first wrote iterate, it used to manually optimize the recursive calls to jumps in some cases, because the Symbolics I wrote it on didn’t have tail-call elimination. That’s a non-problem in LispWorks². Anyone familiar with Scheme will recognise iterate as named let, which is where it came from (once, I think, it was known as nlet).

iterate is implemented by a function which maps from the language which includes it to a language which doesn’t include it, by mapping the syntax as above.

So compare this with a factorial function: factorial is a function whose domain is natural numbers and whose range is also natural numbers, and it has an obvious recursive definition. Well, natural numbers are part of the syntax of Lisp, but they’re a tiny part of it. So implementing factorial as a macro is, really, a hopeless task. What should

(factorial (+ x y (f z)))

Actually do when considered as a mapping between languages? Assuming you are using the recursive definition of the factorial function then the answer is it can’t map to anything useful at all: a function which implements that recursive definition simply has to be called at run time. The very best you could do would seem to be this:

(defun fact (n)
 (if (< n 3)
     n
   (* n (fact (1- n)))))

(defmacro factorial (expression)
 `(fact ,expression))

And that’s not a useful macro (but see below).

So the answer is, again, that macros are functions which map between languages and they are useful where you want a new language: not just the same language with extra functions in it, but a language with new control constructs or something like that. If you are writing functions whose range is something which is not the syntax of a language built on Common Lisp, don’t write macros.

What macros are: a second look

Macroexpansion is compilation.

A function whose domain is one language and whose range is another is a compiler for the language of the domain, especially when that language is somehow richer than the language of the range, which is the case for macros.

But it’s a simplification to say that macros are this function: they’re not, they’re only part of it. The actual function which maps between the two languages is made up of macros and the macroexpander provided by CL itself. The macroexpander is what arranges for the functions defined by macros to be called in the right places, and also it is the thing which arranges for various recursive macros to actually make up a recurscive function. So it’s important to understand that the macroexpander is a critical part of the process: macros on their own only provide part of it.

An example: two versions of a recursive macro

People often say that you should not write recursive macros, but this prohibition on recursive macros is pretty specious: they’re just fine. Consider a language which only has lambda and doesn’t have let. Well, we can write a simple version of let, which I’ll call bind as a macro: a function which takes this new language and turns it into the more basic one. Here’s that macro:

(defmacro bind ((&rest bindings) &body forms)
 `((lambda ,(mapcar #'first bindings) ,@forms)
   ,@(mapcar #'second bindings)))

And now

> (bind ((x 1) (y 2))
    (+ x y))              
(bind ((x 1) (y 2)) (+ x y))
 -> ((lambda (x y) (+ x y)) 1 2)
3

(These example expansions come via use of my trace-macroexpand package, available in a good Lisp near you: see appendix for configuration).

So now we have a language with a binding form which is more convenient than lambda. But maybe we want to be able to bind sequentially? Well, we can write a let* version, called bind*, which looks like this

(defmacro bind* ((&rest bindings) &body forms)
 (if (null (rest bindings))
     `(bind ,bindings ,@forms)
   `(bind (,(first bindings))
      (bind* ,(rest bindings) ,@forms))))

And you can see how this works: it checks if there’s just one binding in which case it’s just bind, and if there’s more than one it peels off the first and then expands into a bind* form for the rest. And you can see this working (here both bind and bind* are being traced):

> (bind* ((x 1) (y (+ x 2)))
    (+ x y))
(bind* ((x 1) (y (+ x 2))) (+ x y))
 -> (bind ((x 1)) (bind* ((y (+ x 2))) (+ x y)))
(bind ((x 1)) (bind* ((y (+ x 2))) (+ x y)))
 -> ((lambda (x) (bind* ((y (+ x 2))) (+ x y))) 1)
(bind* ((y (+ x 2))) (+ x y))
 -> (bind ((y (+ x 2))) (+ x y))
(bind ((y (+ x 2))) (+ x y))
 -> ((lambda (y) (+ x y)) (+ x 2))
(bind* ((y (+ x 2))) (+ x y))
 -> (bind ((y (+ x 2))) (+ x y))
(bind ((y (+ x 2))) (+ x y))
 -> ((lambda (y) (+ x y)) (+ x 2))
4

You can see that, in this implementation, which is LW again, some of the forms are expanded more than once: that’s not uncommon in interpreted code: since macros should generally be functions (so, not have side-effects) it does not matter that they may be expanded multiple times. Compilation will expand macros and then compile the result, so all the overhead of macroexpansion happend ahead of run-time:

 (defun foo (x)
   (bind* ((y (1+ x)) (z (1+ y)))
     (+ y z)))
foo

> (compile *)
(bind* ((y (1+ x)) (z (1+ y))) (+ y z))
 -> (bind ((y (1+ x))) (bind* ((z (1+ y))) (+ y z)))
(bind ((y (1+ x))) (bind* ((z (1+ y))) (+ y z)))
 -> ((lambda (y) (bind* ((z (1+ y))) (+ y z))) (1+ x))
(bind* ((z (1+ y))) (+ y z))
 -> (bind ((z (1+ y))) (+ y z))
(bind ((z (1+ y))) (+ y z))
 -> ((lambda (z) (+ y z)) (1+ y))
foo
nil
nil

> (foo 3)
9

There’s nothing wrong with macros like this, which expand into simpler versions of themselves. You just have to make sure that the recursive expansion process is producing successively simpler bits of syntax and has a well-defined termination condition.

Macros like this are often called ‘recursive’ but they’re actually not: the function associated with bind* does not call itself. What is recursive is the function implicitly defined by the combination of the macro function and the macroexpander: the bind* function simply expands into a bit of syntax which it knows will cause the macroexpander to call it again.

It is possible to write bind* such that the macro function itself is recursive:

(defmacro bind* ((&rest bindings) &body forms)
  (labels ((expand-bind (btail)
             (if (null (rest btail))
                 `(bind ,btail
                    ,@forms)
               `(bind (,(first btail))
                  ,(expand-bind (rest btail))))))
    (expand-bind bindings)))

And now compiling foo again results in this output from tracing macroexpansion:

(bind* ((y (1+ x)) (z (1+ y))) (+ y z))
 -> (bind ((y (1+ x))) (bind ((z (1+ y))) (+ y z)))
(bind ((y (1+ x))) (bind ((z (1+ y))) (+ y z)))
 -> ((lambda (y) (bind ((z (1+ y))) (+ y z))) (1+ x))
(bind ((z (1+ y))) (+ y z))
 -> ((lambda (z) (+ y z)) (1+ y))

You can see that now all the recursion happens within the macro function for bind* itself: the macroexpander calls bind*’s macro function just once.

While it’s possible to write macros like this second version of bind*, it is normally easier to write the first version and to allow the combination of the macroexpander and the macro function to implement the recursive expansion.

Two historical uses for macros

There are two uses for macros — both now historical — where they were used where functions would be more natural.

The first of these is function inlining, where you want to avoid the overhead of calling a small function many times. This overhead was a lot on computers made of cardboard, as all computers were, and also if the stack got too deep the cardboard would tear and this was bad. It makes no real sense to inline a recursive function such as the above factorial: how would the inlining process terminate? But you could rewrite a factorial function to be explicitly iterative:

(defun factorial (n)
 (do* ((k 1 (1+ k))
       (f k (* f k)))
      ((>= k n) f)))

And now, if you have very many calls to factorial, you wanted to optimise the function call overhead away, and it was 1975, you might write this:

(defmacro factorial (n)
 `(let ((nv ,n))
    (do* ((k 1 (1+ k))
          (f k (* f k)))
         ((>= k nv) f))))

And this has the effect of replacing (factorial n) by an expression which will compute the factorial of n. The cost of that is that (funcall #'factorial n) is not going to work, and (funcall (macro-function 'factorial) ...) is never what you want.

Well, that’s what you did in 1975, because Lisp compilers were made out of the things people found down the sides of sofas. Now it’s no longer 1975 and you just tell the compiler that you want it to inline the function, please:

(declaim (inline factorial))
(defun factorial (n) ...)

and it will do that for you. So this use of macros is now purely historicl.

The second reason for macros where you really want functions is computing things at compile time. Let’s say you have lots of expressions like (factorial 32) in your code. Well, you could do this:

(defmacro factorial (expression)
 (typecase expression
   ((integer 0)
    (factorial/fn expression))
   (number
    (error "factorial of non-natural literal ~S" expression))
   (t
    `(factorial/fn ,expression))))

So the factorial macro checks to see if its argument is a literal natural number and will compute the factorial of it at macroexpansion time (so, at compile time or just before compile time). So a function like

(defun foo ()
 (factorial 32))

will now compile to simply return 263130836933693530167218012160000000. And, even better, there’s some compile-time error checking: code which is, say, (factorial 12.3) will cause a compile-time error.

Well, again, this is what you would do if it was 1975. It’s not 1975 any more, and CL has a special tool for dealing with just this problem: compiler macros.

(defun factorial (n)
 (do* ((k 1 (1+ k))
       (f k (* f k)))
      ((>= k n) f)))

(define-compiler-macro factorial (&whole form n)
 (typecase n
   ((integer 0)
    (factorial n))
   (number
    (error "literal number is not a natural: ~S" n))
   (t form)))

Now factorial is a function and works the way you expect — (funcall #'factoial ...) will work fine. But the compiler knows that if it comes across (factorial ...) then it should give the compiler macro for factorial a chance to say what this expression should actually be. And the compiler macro does an explicit check for the argument being a literal natural number, and if it is computes the factorial at compile time, and the same check for a literal number which is not a natural, and finally just says ’I don’t know, call the function’. Note that the compiler macro itself calls factorial, but since the argument isn’t a literal there’s no recursive doom.

So this takes care of the other antique use of macros where you would expect functions. And of course you can combine this with inlining and it will all work fine: you can write functions which will handle special cases via compiler macros and will otherwise be inlined.

That leaves macros serving the purpose they are actually useful for: building languages.

Appendix: setting up `trace-macroexpand`

(use-package :org.tfeb.hax.trace-macroexpand)

;;; Don't restrict print length or level when tracing
(setf *trace-macroexpand-print-level* nil
      *trace-macroexpand-print-length* nil)

;;; Enable tracing
(trace-macroexpand)

;;; Trace the macros you want to look at ...
(trace-macro ...)

;;; ... and ntrace them
(untrace-macro ...)

All the examples in this article are in Common Lisp except where otherwise specified. Other Lisps have similar considerations, although macros in Scheme are not explicitly functions in the way they are in CL. ↩
This article originated as a message on the lisp-hug mailing list for LispWorks users. References to ‘LW’ mean LispWorks, although everything here should apply to any modern CL. (In terms of tail call elimination I would define a CL which does not eliminate tail self-calls in almost all cases under reasonable optimization settings as pre-modern: I don’t use such implementations.) ↩

The best Lisp

2021-11-03T12:03:44Z

People sometimes ask which is the best Lisp dialect? That’s a category error, and here’s why.

Programming in Lisp — any Lisp — is about building languages: in Lisp the way you solve a problem is by building a language — a jargon, or a dialect if you like — to talk about the problem and then solving the problem in that language. Lisps are, quite explicitly, language-building languages.

This is, in fact, how people solve large problems in all programming languages: Greenspun’s tenth rule isn’t really a statement about Common Lisp, it’s a statement that all sufficiently large software systems end up having some hacked-together, informally-specified, half-working language in which the problem is actually solved. Often people won’t understand that the thing they’ve built is in fact a language, but that’s what it is. Everyone who has worked on large-scale software will have come across these things: often they are very horrible, and involve much use of language-in-a-string¹.

The Lisp difference is two things: when you start solving a problem in Lisp, you know, quite explicitly, that this is what you are going to do; and the language has wonderful tools which let you incrementally build a series of lightweight languages, ending up with one or more languages in which to solve the problem.

So, after that preface, why is this question the wrong one to ask? Well, if you are going to program in Lisp you are going to be building languages, and you want those languages not to be awful. Lisp makes it it far easier to build languages which are not awful, but it doesn’t prevent you doing so if you want to. And again, anyone who has dealt with enough languages built on Lisps will have come across some which are, in fact, awful.

If you are going to build languages then you need to understand how languages work — what makes a language habitable to its human users (the computer does not care with very few exceptions). That means you will need to be a linguist. So the question then is: how do you become a linguist? Well, we know the answer to that, because there are lots of linguists and lots of courses on linguistics. You might say that, well, those people study natural languages, but that’s irrelevant: natural languages have been under evolutionary pressure for a very long time and they’re really good for what they’re designed for (which is not the same as what programming languages are designed for, but the users — humans — are the same).

So, do you become a linguist by learning French? Or German? Or Latin? Or Cuzco Quechua? No, you don’t. You become a linguist by learning enough about enough languages that you can understand how languages work. A linguist isn’t someone who speaks French really well: they’re someone who understands that French is a Romance language, that German isn’t but has many Romance loan words, that English is closer to German than it is French but got a vast injection of Norman French, which in turn wasn’t that close to modern French, that Swiss German has cross-serial dependencies but Hochdeutsch does not and what that means, and so on. A linguist is someone who understands things about the structure of languages: what do you see, what do you never see, how do different languages do equivalent things? And so on.

The way you become a linguist is not by picking a language and learning it: it’s by looking at lots of languages enough to understand how they work.

If you want to learn to program in Lisp, you will need to become a linguist. The very best way to ensure you fail at that is to pick a ‘best’ Lisp and learn that. There is no best Lisp, and in order to program well in any Lisp you must be exposed to as many Lisps and as many other languages as possible.

If you think there’s a distinction between a ‘dialect’, a ‘jargon’ and a ‘language’ then I have news for you: there is. A language is a dialect with a standards committee. (This is stolen from a quote due to Max Weinrich that all linguists know:

אַ שפּראַך איז אַ דיאַלעקט מיט אַן אַרמיי און פֿלאָט

a shprakh iz a dyalekt mit an armey un flot.)

‘Language-in-a-string’ is where a programming language has another programming language embedded in strings in the outer language. Sometimes programs in that inner programming language will be made up by string concatenation in the outer language. Sometimes that inner language will, in turn, have languages embedded in its strings. It’s a terrible, terrible thing. ↩

Generic interfaces in Racket

2021-01-08T18:25:59Z

Or: things you do to distract yourself from watching an attempted fascist coup.

A thing that exists in many languages with a notion of a sequence of objects is a function variously known as fold or reduce: this takes another function of two arguments, some initial value, and walks along the sequence successively reducing it using the function. So, for instance:

(fold + 0 '(1 2 3)) turns into (fold + (+ 0 1) '(2 3)) which turns into …
(fold + 1 '(2 3)) turns into (fold + (+ 1 2) '(3)) which turns into …
(fold + 3 '(3)) turns into (fold + (+ 3 3) '()) which turns into …
6.

It’s pretty easy to write a version of fold for lists:

(define (fold op initial l)
  (if (null? l)
      initial
      (fold op (op initial (first l)) (rest l))))

Racket calls this (or a more careful version of this) foldl: there is also foldr which works from the other end of the list and is more expensive as a result.

Well, one thing you might want to do is have a version of fold which works on trees rather than just lists. One definition of a tree is:

it’s a collection of nodes;
nodes have values;
nodes have zero or more unique children, which are nodes.
no node is the descendant of more than one node;
there is exactly one root node which is the descendant of no other nodes.

A variant of this (which will matter below) is that the children of a node are either nodes or any other object, and there is some way of knowing if something is a node or not¹.

You can obviously represent trees as conses, with the value of a cons being its car, and the children being its cdr. Whatever builds the tree needs to make sure that (3), (4) and (5) are true, or you get a more general graph structure.

But you might want to have other sorts of trees, and you’d want the fold function not to care about what sort of tree it was processing: just that it was processing a tree. Indeed, it would be nice if it was possible to provide special implementations for, for instance, binary trees where rather than iterating over some sequence of children you’d know there were exactly two.

So, I wondered if there was a nice way of expressing this in Racket and it turns out there mostly is. Racket has a notion of generic interfaces which are really intended as a way for different structure types to provide common interfaces, I think. But it turns out they can be (ab?)used to do this, as well.

Generic interfaces are not provided by racket but by racket/generic: everything below assumed (require racket/generic).

A generic `treelike` interface

A treelike object supports two operations:

node-value returns the value of a node;
node-children returns a list of the node’s children.

The second of these is a bit nasty: it would be better perhaps to either provide an interface for mapping over a node’s children, or to return some general, possibly lazy, sequence of children. But this is just playing, so I don’t mind.

Here is a definition of a generic treelike interface, which includes default methods for lists:

(define-generics treelike
  ;; treelike objects have values and children
  (node-value treelike)
  (node-children treelike)
  #:fast-defaults
  (((λ (t)
      (and (cons? t) (list? t)))
    ;; non-null proper lists are trees: their value is their car;
    ;; their children are their cdr.
    (define node-value car)
    (define node-children cdr))))

Notes:

This uses #:fast-defaults instead of #:defaults, which means that the dispatch to objects which satisfy list? happens. This is fine in this case: lists are never going to be confused with any other tree type.
This relies on Racket’s (and Scheme’s?) list? predicate returning true only for proper lists rather than CL’s cheap listp which just returns true for anything which is either nil or a cons.
There are lots of other options to define-generics which I’m not using and many of which I don’t understand.

With this definition:

> (treelike? '())
#f
> (treelike? '(1 2 3))
#t
> (treelike? '(1 2 . 3))
#f
> (node-children '(1 2 3))
'(2 3)

So, OK.

A `treelike` binary tree

We could then define a binary-tree type which implements this generic interface:

(struct binary-tree (value left right)
  #:transparent
  #:methods gen:treelike
  ((define (node-value bt)
     (binary-tree-value bt))
   (define (node-children bt)
     (list (binary-tree-left bt)
           (binary-tree-right bt)))))

The #:methods gen:treelike tells the structure we’re defining the methods needed for this thing to be a treelike object.

And now we can check things:

> (treelike? (binary-tree 1 2 3))
#t
> (node-value (binary-tree 1 2 3))
1
> (node-children (binary-tree 1 2 3))
'(2 3)

OK.

Two attempts at a generic `foldable` interface

So now I want to define another interface for things which can be folded. And the first thing I tried is this:

(define-generics foldable
  ;; broken
  (fold operation initial foldable)
  #:defaults
  ((treelike?
    (define (fold op initial treelike)
      (let ([current (op initial (node-value treelike))]
            [children (node-children treelike)])
        (if (null? children)
            current
            (fold op (fold op current (first children))
                  (rest children))))))
   ((const true)
    (define (fold op initial any)
      (op initial any)))))

So this tries to define a fold generic function, which has two implementations: one for treelike objects and one for all other objects. So this means that all objects are foldable, and, for instance (fold + 0 1) simply turns into (+ 0 1). This is a bit odd but it simplifies the implementation of the interface for treelike objects on the assumption that the children of nodes may not themselves be nodes (see above).

There is another complexity: if the list of a treelike node’s children isn’t null, then it’s a treelike, so it can safely be recursed over rather than explicitly iterated over. This is a slightly questionable pun I think, but, well, I am a slightly questionable programmer.

And this … doesn’t work:

> (fold + 0 '(1 2 3))
; node-value: contract violation:
; expected: treelike?
; given: 2
; argument position: 1st

It took me a long time to understand this, and the answer is that the definitions of fold inside the define-generic form aren’t adding methods to a generic function: what they are doing is defining a little local function, fold which then gets glued into the generic function. So references to fold in the definition refer to the little local function. It is exactly as if you had done this, in fact:

(define-generics foldable
  ;; this is why it's broken
  (fold operation initial foldable)
  #:defaults
  ((treelike?
    (define fold
      (letrec ([fold (λ (op initial treelike)
                       (let ([current (op initial (node-value treelike))]
                             [children (node-children treelike)])
                         (if (null? children)
                             current
                             (fold op (fold op current (first children))
                                   (rest children)))))])
        fold)))
   ((const true)
    (define (fold op initial any)
      (op initial any)))))

And you can see why this can’t work: the fold bound by the letrec calls itself rather than going through the generic dispatch.

The way to fix this is to use the magic define/generic form to get a copy of the generic function, and then call that. This is syntactically horrid, but you can see why it is needed given the above. So a working version of this interface purports to be:

(define-generics foldable
  ;; not broken
  (fold operation initial foldable)
  #:defaults
  ((treelike?
    (define/generic fold/g fold)
    (define (fold op initial treelike)
      (let ([current (op initial (node-value treelike))]
            [children (node-children treelike)])
        (if (null? children)
            current
            (fold op (fold/g op current (first children))
                  (rest children))))))
   ((const true)
    (define (fold op initial any)
      (op initial any)))))

And indeed it is not broken:

> (fold + 0 '(1 2 3))
6

and with some tracing added:

> (fold + 0 '(1 2 3))
fold/treelike + 0 (1 2 3)
fold/any + 1 2
fold/treelike + 3 (3)
6

Adding a special case to `fold` for the binary tree

So now, finally, we can add a special case to fold to the binary tree defined above, rather than needlessly consing a list of children. We will need the same explicit-generic-function hack as before as the children of a binary tree may not be binary trees.

(struct binary-tree (value left right)
  #:transparent
  #:methods gen:treelike
  ((define (node-value bt)
     (binary-tree-value bt))
   (define (node-children bt)
     (list (binary-tree-left bt)
           (binary-tree-right bt))))
  #:methods gen:foldable
  ((define/generic fold/g fold)
   (define (fold op initial bt)
     (fold/g op
             (fold/g op (op initial (binary-tree-value bt))
                     (binary-tree-left bt))
             (binary-tree-right bt)))))

And now

> (fold + 0 (binary-tree 1
                         (binary-tree 2 3 4)
                         (binary-tree 5 6 7)))
28

and with some tracing

> (fold + 0 (binary-tree 1
                         (binary-tree 2 3 4)
                         (binary-tree 5 6 7)))
fold/bt + 0 #(struct:binary-tree 1 #(struct:binary-tree 2 3 4) #(struct:binary-tree 5 6 7))
fold/bt + 1 #(struct:binary-tree 2 3 4)
fold/any + 3 3
fold/any + 6 4
fold/bt + 10 #(struct:binary-tree 5 6 7)
fold/any + 15 6
fold/any + 21 7
28

Missing CLOS

In some ways this makes me miss CLOS: the explicit-generic-function hack is very annoying, single dispatch is annoying, not being able to define predicate-based methods separately from the define-generics form is annoying. But on the other hand predicate-based dispatch is pretty cool.

Perhaps these should be called ‘sloppy trees’ or something. ↩

The U combinator

2020-03-09T17:45:22Z

The U combinator allows you to define recursive functions and I think it is simpler to understand than the Y combinator.

It’s not obvious how things like letrec get defined in Scheme, without using secret assignment. In fact I think they are defined using secret assignment:

(letrec ([f (λ (...) ... (f ...) ...)])
  ...)

turns into

(let ([f ...])
  (set! f (λ (...) ... (f ...) ...))
  ...)

But it’s interesting to see how you can define recursive functions without relying on assignment, including mutually-recursive collections of functions. One way is using the U combinator.

I suspect that there is lots of information about this out there, but it’s seriously hard to search for anything which looks like ’*-combinator’ now (even now I am starting a set of companies called ‘integration by parts’, ‘the quotient rule’ &c).

You can famously do this with the Y combinator, but I didn’t want to do that because Y is something I find I can understand for a few hours at a time and then I have to work it all out again. But it turns out that you can use something much simpler: the U combinator. It seems to be even harder to search for this than Y, but here is a quote about it:

In the theory of programming languages, the U combinator, $U$, is the mathematical function that applies its argument to its argument; that is $U(f) = f(f)$, or equivalently, $U = \lambda f \cdot f(f)$.

Self-application permits the simulation of recursion in the λ-calculus, which means that the U combinator enables universal computation. (The U combinator is actually more primitive than the more well-known fixed-point Y combinator.)

The expression $U(U)$ is the smallest non-terminating program.

(Text mildly edited from here, which unfortunately is not a site all about the U combinator other than this quote.)

Prerequisites

All of the following code samples are in Racket. The macros are certainly Racket-specific and some of the other code probably is as well. To make the macros work you will need syntax-parse via:

(require (for-syntax syntax/parse))

However note that my use of syntax-parse is naïve in the extreme: I’m really just an unfrozen CL caveman pretending to understand Racket’s macro system.

Also note I have not ruthlessly turned everything into λ: Rather than ((λ (...) ...) ...) there is (let ([... ...] ...) ...) in this code; there is use of multiple values including let-values; there is (define (f ...) ...) rather than (define f (λ (...) ...)) and so on.

Two versions of U

The first version of U is the obvious one:

(define (U f)
  (f f))

But this will run into some problems with an applicative-order language, which Racket is by default. To avoid that we can make the assumption that (f f) is going to be a function, and wrap that form in another function to delay its evaluation until it’s needed: this is the standard trick that you have to do for Y in an applicative-order language as well. I’m only going to use the applicative-order U when I have to, so I’ll give it a different name:

(define (U/ao f)
  (λ args (apply (f f) args)))

Note also that I’m allowing more than one argument rather than doing the pure-λ-calculus thing.

Using U to construct a recursive functions

To do this we do a similar trick that you do with Y: write a function which, if given a function as argument which deals with the recursive cases, will return a recursive function. And obviously I’ll use the Fibonacci function as the canonical recursive function.

So, consider this thing:

(define fibber
  (λ (f)
    (λ (n)
      (if (<= n 2)
          1
          (+ ((U f) (- n 1))
             ((U f) (- n 2)))))))

This is a function which, given another function, U of which computes smaller Fibonacci numbers, will return a function which will compute the Fibonacci number for n.

In other words, U of this function is the Fibonacci function!

And we can test this:

> (define fibonacci (U fibber))
> (fibonacci 10)
55

So that’s very nice.

Wrapping U in a macro

So, to hide all this the first thing to do is to remove the explicit calls to U in the recursion. We can lift them out of the inner function completely:

(define fibber/broken
  (λ (f)
    (let ([fib (U f)])
      (λ (n)
        (if (<= n 2)
            1
            (+ (fib (- n 1))
               (fib (- n 2))))))))

Don’t try to compute U of this: it will recurse endlessly because (U fibber/broken) -> (fibber/broken fibber/broken) and this involves computing (U fibber/broken), and we’re doomed.

Instead we can use U/ao:

(define fibber
  (λ (f)
    (let ([fib (U/ao f)])
      (λ (n)
        (if (<= n 2)
            1
            (+ (fib (- n 1))
               (fib (- n 2))))))))

And this is all fine ((U fibber) 10) is 55 (and terminates!).

Purists can then turn let into λ in the usual way:

(define fibber
  (λ (f)
    ((λ (fib)
       (λ (n)
         (if (<= n 2)
             1
             (+ (fib (- n 1))
                (fib (- n 2))))))
     (U/ao f))))

And this is really all you need to be able to write the macro:

(define-syntax (with-recursive-binding stx)
  (syntax-parse stx
    [(_ (name:id value:expr) form ...+)
     #'(let ([name (U (λ (f)
                        (let ([name (U/ao f)])
                          value)))])
         form ...)]))

Or, for the pure of heart:

(define-syntax (with-recursive-binding stx)
  (syntax-parse stx
    [(_ (name:id value:expr) form ...+)
     #'((λ (name)
          form ...)
        (U (λ (f)
             ((λ (name)
                value)
              (U/ao f)))))]))

And this works fine:

(with-recursive-binding (fib (λ (n)
                               (if (<= n 2)
                                   1
                                   (+ (fib (- n 1))
                                      (fib (- n 2))))))
  (fib 10))

A caveat on bindings

One fairly obvious thing here is that there are two bindings constructed by this macro: the outer one, and an inner one of the same name. And these are not bound to the same function in the sense of eq?:

(with-recursive-binding (ts (λ (it)
                              (eq? ts it)))
  (ts ts))

is #f. This matters only in a language where bindings can be mutated: a language with assignment in other words. Both the outer and inner bindings, unless they have been mutated, are to functions which are identical as functions: they compute the same values for all values of their arguments. In fact, it’s hard to see what purpose eq? would serve in a language without assignment.

This caveat will apply below as well.

Two versions of U for many functions

The obvious generalization of U, U*, to many functions is that $U^*(f_1, \ldots, f_n)$ is the tuple $(f_1(f_1, \ldots, f_n), f_2(f_1, \ldots, f_n), \ldots)$. And a nice way of expressing that in Racket is to use multiple values:

(define (U* . fs)
  (apply values (map (λ (f)
                       (apply f fs))
                     fs)))

And we need the applicative-order one as well:

(define (U*/ao . fs)
  (apply values (map (λ (f)
                       (λ args (apply (apply f fs) args)))
                     fs)))

Note that U* is a true generalization of U: (U f) and (U* f) are the same.

Using U* to construct mutually-recursive functions

I’ll work with a trivial pair of functions:

an object is a numeric tree if it is a cons and its car and cdr are numeric objects;
an objct is a numeric object if it is a number, or if it is a numeric tree.

So we can define ‘maker’ functions (with an ’-er’ convention: a function which makes an x is an xer, or, if x has hyphens in it, an x-er) which will make suitable functions:

(define numeric-tree-er
  (λ (nter noer)
    (λ (o)
      (let-values ([(nt? no?) (U* nter noer)])
        (and (cons? o)
             (no? (car o))
             (no? (cdr o)))))))

(define numeric-object-er
  (λ (nter noer)
    (λ (o)
      (let-values ([(nt? no?) (U* nter noer)])
        (cond
          [(number? o) #t]
          [(cons? o) (nt? o)]
          [else #f])))))

Note that for both of these I’ve raised the call to U* a little, simply to make the call to the appropriate value of U* less opaque.

And this works:

(define-values (numeric-tree? numeric-object?)
  (U* numeric-tree-er numeric-object-er))

And now:

> (numeric-tree? 1)
#f
> (numeric-object? 1)
#t
> (numeric-tree? '(1 . 2))
#t
> (numeric-tree? '(1 2 . (3 4)))
#f

Wrapping U* in a macro

The same problem as previously happens when we raise the inner call to U* with the same result: we need to use U*/ao. In addition the macro becomes significantly more hairy and I’m moderately surprised that I got it right so easily. It’s not conceptually hard: it’s just not obvious to me that the pattern-matching works.

(define-syntax (with-recursive-bindings stx)
  (syntax-parse stx
    [(_ ((name:id value:expr) ...) form ...+)
     #:fail-when (check-duplicate-identifier (syntax->list #'(name ...)))
     "duplicate variable name"
     (with-syntax ([(argname ...) (generate-temporaries #'(name ...))])
       #'(let-values
             ([(name ...) (U* (λ (argname ...)
                                (let-values ([(name ...)
                                              (U*/ao argname ...)])
                                  value)) ...)])
           form ...))]))

And now, in a shower of sparks, we can write:

(with-recursive-bindings ((numeric-tree?
                           (λ (o)
                             (and (cons? o)
                                  (numeric-object? (car o))
                                  (numeric-object? (cdr o)))))
                          (numeric-object?
                           (λ (o)
                             (cond [(number? o) #t]
                                   [(cons? o) (numeric-tree? o)]
                                   [else #f]))))
  (numeric-tree? '(1 2 3 (4 (5 . 6) . 7) . 8)))

and get #t.

As I said, I am sure there are well-known better ways to do this, but I thought this was interesting enough not to lose. This originated as an answer to this Stack Overflow question.

Function calling conventions and bindings

2019-01-04T10:19:36Z

An attempt to describe three well-known function calling conventions in terms of bindings.

A little while ago I wrote an article on bindings which, in turn, was based on my answer to this Stack Overflow question. I have since written another answer to a more recent question and I thought it would be worth summarising part of that to describe how three famous function calling conventions can be described in terms of bindings¹.

Bindings in brief

A binding is an association between a name (a variable) and a value, where the value can be any object the language can talk about. In most Lisps (and other languages) bindings are not first-class: the language can not talk about bindings directly, and in particular bindings can not be values. Bindings are, or may be, mutable: their values (but not their names) can be changed by assignment. Many bindings can share the same value. Bindings have scope (where they are accessible) and extent (how long they are accessible for) and there are rules about that.

Call by value

In call by value the value of a binding is passed to a procedure. This means that the procedure can not mutate the binding itself. If the value is a mutable object it can be altered by the procedure, but the binding can not be.

Call by value is the convention used by all Lisps I know of. Here is a function which demonstrates that call by value can not mutate bindings:

(defun pbv (&optional (fn #'identity))
  ;; If FN returns then the first value of this function will be T
  (let ((c (cons 0 0)))                 ;first binding
    (let ((cc c))                       ;second binding, shares value with first
      (funcall fn c)                    ;FN gets the *value* of C
      (values (eq c cc) c))))           ;C and CC still refer to the same object

Call by reference

In call by reference, procedures get the bindings themselves as arguments. If a procedure modifies the binding by assignment, then it is modified in the calling procedure as well.

Lisp does not use call by reference: Fortran does, or can, use a calling mechanism which is equivalent to call by reference².

It is possible to implement what is essentially call by reference in Lisp (here Common Lisp, but any Lisp with lexical scope, indefinite extent & macros can do this) using some macrology:

(defmacro capture-binding (var)
  ;; Construct an object which captures a binding
  `(lambda (&optional (new-val nil new-val-p))
     (when new-val-p
       (setf ,var new-val))
     ,var))

(declaim (inline captured-binding-value
                 (setf captured-binding-value)))

(defun captured-binding-value (cb)
  ;; value of a captured binding
  (funcall cb))

(defun (setf captured-binding-value) (new cb)
  ;; change the value of a captured binding
  (funcall cb new))

And now, given

(defun mutate-binding (b v)
  (setf (captured-binding-value b) v))

(defun sort-of-call-by-reference ()
  (let ((c (cons 1 1)))
    (let ((cc c))
      (mutate-binding (capture-binding cc) 3)
      (values c cc))))

> (sort-of-call-by-reference)
(1 . 1)
3

The trick here is that the procedure created by the capture-binding macro has access to the binding being captured, and can mutate it.

Call by name

Call by name is the same as call by value, except the value of a binding is only computed at the point it is needed. Call by name is a form of delayed evaluation or normal-order evaluation strategy.

Lisp (at least Common Lisp: Lisps which have normal-order evaluation strategies exist) does not have call by name, but again it can be emulated with some macrology:

(defmacro delay (form)
  ;; simple-minded DELAY.  FORM is assumed to return a single value,
  ;; and will be evaluated no more than once.
  (let ((fpn (make-symbol "FORCEDP"))
        (vn (make-symbol "VALUE")))
    `(let ((,fpn nil) ,vn)
       (lambda ()
         (unless ,fpn
           (setf ,fpn t
                 ,vn ,form))
         ,vn))))

(declaim (inline force))

(defun force (thunk)
  ;; forcd a thunk
  (funcall thunk))

(defmacro funcall/delayed (fn &rest args)
  ;; call a function with a bunch of delayed arguments
  `(funcall ,fn ,@(mapcar (lambda (a)
                            `(delay ,a))
                          args)))

And now

(defun return-first-thunk-value (t1 t2)
  (declare (ignorable t2))
  (force t1))

(defun surprisingly-quick ()
  (funcall/delayed #'return-first-thunk-value
                   (cons 1 2)
                   (loop repeat 1000000
                         collect
                         (loop repeat 1000000
                               collect
                               (loop repeat 1000000
                                     collect 1)))))

> (time (surprisingly-quick))
Timing the evaluation of (surprisingly-quick)

User time    =        0.000
System time  =        0.000
Elapsed time =        0.001
Allocation   = 224 bytes
3 Page faults
(1 . 2)

The second argument to return-first-thunk-value was never forced, and so the function completes in reasonable time.

This, in turn, is distantly descended from a post on comp.lang.lisp by Erik Naggum. ↩
I think Fortran is allowed to implement its ‘by reference’ calls by copying any modified bindings back to the bindings in the parent procedure, and this is largely equivalent, at least for single-threaded code. ↩

Call by value in Scheme and Lisp

2018-12-11T10:50:28Z

I find the best way to think about this is to think in terms of bindings, rather than environments or frames, which are simply containers for bindings.

Bindings

A binding is an association between a name and a value. The name is often called a ‘variable’ and the value is, well, the value of the variable. The value of a binding can be any object that the language can talk about at all. Bindings, however, are behind-the-scenes things (sometimes this is called ‘not being first-class objects’): they’re not things that can be represented in the language but rather things that you can use as part of the model of how the language works. So the value of a binding can’t be a binding, because bindings are not first-class: the language can’t talk about bindings.

There are some rules about bindings:

there are forms which create them, of which the most important two are lambda and define;
bindings are not first-class — the language can not represent bindings as values;
bindings are, or may be, mutable — you can change the value of a binding once it exists — and the form that does this is set!;
there is no operator which destroys a binding;
bindings have lexical scope — the bindings available to a bit of code are the ones you can see by looking at it, not ones you have to guess by running the code and which may depend on the dynamic state of the system;
only one binding for a given name is ever accessible from a given bit of code — if more than one is lexically visible then the innermost one shadows any outer ones;
bindings have indefinite extent — if a binding is ever available to a bit of code, it is always available to it.

Obviously these rules need to be elaborated significantly (especially with regards to global bindings & forward-referenced bindings) and mare formal, but these are enough to understand what happens. In particular I don’t really think you need to spend a lot of time worrying about environments: the environment of a bit of code is just the set of bindings accessible to it, so rather than worry about the environment just worry about the bindings.

Call by value

So, what ‘call by value’ means is that when you call a procedure with an argument which is a variable (a binding) what is passed to it is the value of the variable binding, not the binding itself. The procedure then creates a new binding with the same value. Two things follow from that:

the original binding can not be altered by the procedure — this follows because the procedure only has the value of it, not the binding itself, and bindings are not first-class so you can’t cheat by passing the binding itself as the value;
if the value is itself a mutable object (arrays & conses are example of objects which usually are mutable, numbers are examples of objects which are not) then the procedure can mutate that object.

Examples of the rules about bindings

So, here are some examples of these rules.

(define (silly x)
  (set! x (+ x 1))
  x)

(define (call-something fn val)
  (fn val)
  val))

> (call-something silly 10)
10

So, here we are creating two top-level bindings, for silly and call-something, both of which have values which are procedures. The value of silly is a procedure which, when called:

creates a new binding whose name is x and whose value is the argument to silly;
mutates this binding so its value is incremented by one;
returns the value of this binding, which is one more than the value it was called with.

The value of call-something is a procedure which, when called:

creates two bindings, one named fn and one named val;
calls the value of the fn binding with the value of the val binding;
returns the value of the val binding.

Note that whatever the call to fn does, it can not mutate the binding of val, because it has no access to it. So what you can know, by looking at the definition of call-something is that, if it returns at all (it may not return if the call to fn does not return), it will return the value of its second argument. This guarantee is what ‘call by value’ means: a language (such as Fortran) which supports other call mechanisms can’t always promise this.

(define (outer x)
  (define (inner x)
    (+ x 1))
  (inner (+ x 1)))

Here there are four bindings: outer is a top-level binding whose value is a procedure which, when it is called, creates a binding for x whose value is its argument. It then creates another binding called inner whose value is another procedure, which, when it is called, creates a new binding for x to its argument, and then returns the value of that binding plus one. outer then calls this inner procedure with the value of its binding for x.

The important thing here is that, in inner, there are two bindings for x which are potentially lexically visible, but the closest one — the one established by inner — wins, because only one binding for a given name can ever be accessible at one time.

Here is the previous code (this would not be equivalent if inner was recursive) expressed with explicit lambdas:

(define outer
  (λ (x)
    ((λ (inner)
       (inner (+ x 1)))
     (λ (x)
       (+ x 1)))))

And finally an example of mutating bindings:

(define (make-counter val)
  (λ ()
    (let ((current val))
      (set! val (+ val 1))
      current)))

> (define counter (make-counter 0))
> (counter)
0
> (counter)
1
> (counter)
2

So, here, make-counter (is the name of a binding whose value is a procedure which, when called,) establishes a new binding for val and then returns a procedure it has created. This procedure makes a new binding called current which catches the current value of val, mutates the binding for val to add one to it, and returns the value of current. This code exercises the ‘if you can ever see a binding, you can always see it’ rule: the binding for val created by the call to make-counter is visible to the procedure it returns for as long as that procedure exists (and that procedure exists at least as long as there is a binding for it), and it also mutates a binding with set!.

Why not environments?

SICP, in chapter 3, introduces the ‘environment model’, where at any point there is an environment, consisting of a sequence of frames, each frame containing bindings. Obviously this is a fine model, but it introduces three kinds of thing — the enviromnent, the frames in the environment and the bindings in the frame — two of which are utterly intangible. At least for a binding you can get hold of it in some way: you can see it being created in the code and you can see references to it. So I prefer not to think in terms of these two extra sorts of thing which you can never get any kind of handle on.

However this is a choice which makes no difference in practice: thinking purely in terms of bindings helps me, thinking in terms of environments, frames & bindings may well help other people more.

Shorthands

In what follows I am going to use a shorthand for talking about bindings, especially top-level ones:

’x is a procedure which …’ means ’x is the name of a binding whose value is a procedure which, when called, …’;
’y is …’ means ’y is the name of a binding the value of which is …’;
’x is called with y’ means ‘the value of the binding named by x is called with the value of the binding named by y’;
’… binds x to …’ means ’… creates a binding whose name is x and whose value is …’;
’x’ means ‘the value of x’;
and so on.

Describing bindings like this is common, as the fully-explicit way is just painful: I’ve tried (but probably failed in places) to be fully explicit above.

The answer

And finally, after this long preamble, here’s the answer to the question you asked¹.

(define (make-withdraw balance)
  (λ (amount)
    (if (>= balance amount)
        (begin (set! balance (- balance amount))
               balance)
        "Insufficient funds")))

make-withdraw binds balance to its argument and returns a procedure it makes. This procedure, when called:

binds amount to its argument;
compares amount with balance (which it can still see because it could see it when it was created);
if there’s enough money then it mutates the balance binding, decrementing its value by the value of the amount binding, and returns the new value;
if there’s not enough money it returns "Insuficient funds" (but does not mutate the balance binding, so you can try again with a smaller amount: a real bank would probably suck some money out of the balance binding at this point as a fine).

Now

(define x (make-withdraw 100))

creates a binding for x whose value is one of the procedures described above: in that procedure balance is initially 100.

(define (f y) (y 25))

f is a procedure (is the name of a binding whose value is a procedure, which, when called) which binds y to its argument and then calls it with an argument of 25.

(f x)

So, f is called with x, x being (bound to) the procedure constructed above. In f, y is bound to this procedure (not to a copy of it, to it), and this procedure is then called with an argument of 25. This procedure then behaves as described above, and the results are as follows:

> (f x)
75
> (f x)
50
> (f x)
25
> (f x)
0
> (f x)
"Insufficient funds"

Note that:

no first-class objects are copied anywhere in this process: there is no ‘copy’ of a procedure created;
no first-class objects are mutated anywhere in this process;
bindings are created (and later become inacessible and so can be destroyed) in this process;
one binding is mutated repeatedly in this process (once for each call);
I have not anywhere needed to mention ‘environments’, which are just the set of bindings visible from a certain point in the code and I think not a very useful concept.

I hope this makes some kind of sense.

A more elaborate version of the above code

Something you might want to be able to do is to back out a transaction on your account. One way to do that is to return, as well as the new balance, a procedure which undoes the last transaction. Here is a procedure which does that (this code is in Racket):

(define (make-withdraw/backout
         balance
         (insufficient-funds "Insufficient funds"))
  (λ (amount)
    (if (>= balance amount)
        (let ((last-balance balance))
          (set! balance (- balance amount))
              (values balance
                      (λ ()
                       (set! balance last-balance)
                       balance)))
            (values
             insufficient-funds
             (λ () balance)))))

When you make an account with this procedure, then calling it returns two values: the first is the new balance, or the value of insufficient-funds (defaultly "Insufficient funds"), the second is a procedure which will undo the transaction you just did. Note that it undoes it by explicitly putting back the old balance, because you can’t necessarily rely on (= (- (+ x y) y) x) being true in the presence of floating-point arithmetic I think. If you understand how this works then you probably understand bindings.

This originated as an answer to this Stack Overflow question. ↩

Dynamic scope and macros

2017-01-26T13:56:36Z

I’ve recently been writing some Emacs Lisp code to do some massaging of files. Quite apart from having forgotten how primitive elisp is, I hadn’t realised before how hostile dynamic scope was for macros in particular.

A very common pattern for macros is call-with-* / with-*, in which there is a functional level which is wrapped by a more syntacticlly-friendly macro level. For instance, in Common Lisp you can map over lists with mapcar:

(mapcar
 (lambda (e)
   ...)
 ...)

but you might want to map over them with a syntax like

(mapping (e ...)
  ...)

Well, it’s easy to implement this:

(defmacro mapping ((e l) &body forms)
  `(mapcar (lambda (,e) ,@forms) ,l))

Even with CL’s unhygienic macro system & without a mass of gensymmery such a macro is safe.

A good example where CL exposes one side of a pattern like this is with-open-file: you can easily see how to implement this in terms of a function:

(defun call/open-file (fn filespec &rest keys
                          &key &allow-other-keys)
  (let ((s nil))
    (unwind-protect
        (progn
          (setf s (apply #'open filespec keys))
          (funcall fn s))
      (when s (close s)))))

(defmacro with-open-file* ((sn filespecn &rest keysn 
                               &key &allow-other-keys)
                           &body forms)
  `(call/open-file (lambda (,sn) ,@forms)
                   ,filespecn ,@keysn))

(This is probably not completely robust code: it’s just meant to get the idea across.)

Scheme exposes the other side of this pattern with call/cc:

(define-syntax-rule (with-cc (c) form ...)
  (call/cc (λ (c) form ...)))

(define-syntax-rule may be specific to Racket but, again, this is just meant to get the idea across.)

Well, now think about something like the above call/open-file / with-open-file* in a Lisp dialect with dynamic scope. In particular, what does this do:

(let ((s t))
  (with-open-file* (h ...)
    (when s ...)))

This expands to

(let ((s t))
  (call/open-file (lambda (h) (when s ...))))

But call/open-file binds s: so the binding of s in the called function is different than the outer binding, and nothing works.

Well, of course, this is something that happens pervasively with dynamically-scoped languages: every binding above you (or below you, depending on your viewpoint) matters, and can infect your namespace. But it’s particularly toxic for macros, because macros very often interpose bits of code into your code, and that code can include bindings which are dynamically, but not lexically, visible, even in the expansion of the macro. Dynamic scope enormously increases the hygiene problems of a macro system.

Dynamic scope is really useful as an option, and systems written in languages which don’t have it generally have to reinvent it, usually badly. But it’s just toxic and horrible as the only option. I can’t understand any more how I managed to use lisps with dynamic scope at all: perhaps I never wrote macros or just expected things to behave in a mysterious and strange way occasionally. Fortunately, even elisp now has the option of being lexically scoped.

Python instead of Lisp

2016-06-09T18:43:40Z

Lots of people, even famous Lisp hackers, like to claim that ‘Python can be seen as a dialect of Lisp with “traditional” syntax’.

Being famous does not make them right.

Python is nothing like Lisp

Expression language. Lisp is an expression language: everything in the language is an expression and has a value, and there is no distinction between expressions and statements, because there are no statements. Python is not: it has expressions, such as 2+3, lambda x: x*2 and statements such as x = 3. If expressions and statements are different things then writing macros and any kind of general-purpose lambda becomes very difficult.

Conses. Lisp has conses, Python does not. Conses are not everything¹, but unless you have them you can’t implement them reasonably, and they are extremely useful data structures for many purposes. In particular for conses to be useful you need two things:

a good syntax for them and for lists built from them;
good performance — conses should be extremely cheap, so you can’t implement them as a special case of some heavyweight data structure such as a Python list, because there is an enormous header.

This means that conses need to be wired into the language: you can’t take a language without conses and add them, because even if you can get the first (you can’t in Python) you can’t get the second.

Symbols. Lisp has symbols, Python does not. You can use strings, and this works sometimes.

Lambda. Lisp has lambda, Python has an extremely limited version. Not being an expression language (see above) and the lack of scoping and block constructs in Python cripples its lambda.

Source code available as a low-commitment data structure. Lisp has this, Python does not. ‘Low-commitment’ means that it is available before it has been decided what it means, but after it has been turned from a stream of characters into something more interesting. This matters because it makes macros possible: macros which work by transforming streams of characters are doomed to the sort of unspeakable horror of which Jinja2 is a good example, while macros which work after it has been decided what the code means then can’t make their own decision about what it means, which is half the point of macros.

Scoping. Lisp has a multiplicity of scoping constructs and all modern Lisps have lexical scope, with some (Scheme) extending this to control constructs. Binding and assignment are irreparably confused in Python: scope does not work properly and this can never be fixed. A language which requires a global declaration is not going to be fixed by adding nonlocal.

Macros. Lisp has them, Python doesn’t. Since macros are the point of Lisp, it is really hard to see how the above quote makes any kind of sense.

There is a terrible truth about the percieved arrogance of Lisp hackers that it has taken me a long time to understand. The arrogance is justified: Lisp is, in fact, a better programming language.

In particular conses are not a useful universal data structure in the way that, perhaps, early Lisp people thought they were. ↩

Macros in Racket, part three: checking boolean operators

2015-12-12T10:59:54Z

I wanted to see if I could write a mildly complicated macro in Racket without becoming too confused. I can, although I am not sure it is terribly idiomatic.

This is the third part of a series on writing macros in Racket for someone used to Common Lisp, although it is mostly independent of the previous parts. The previous parts are part one & part two.

One of the nice things about Lisp-family languages is that you can write your own control constructs, and it’s essentially easy to do so: if when did not exist then you could write it:

(define-syntax-rule (when test form ...)
  (and test
       (begin form ...)))

This kind of extensibility is one of the wonders of Lisp and Scheme: it’s tempting to say that it makes them better than programming languages which can’t do this but that’s not correct: it makes them incomparable to such languages: Lisp¹ programs can reason about themselves and often do². Everything about Lisp really leads to this ability.

When I taught (Common) Lisp to people one of the things I would try to get across was this ability of macros to extend the control constructs in the language: people often thought of macros as a way of essentially inlining code³, but that’s not what they’re actually good for. If you can add control constructs to your language, then you can make a new language, and that’s what Lisp macros are about, and therefore what Lisp is about.

A good way to get this across to people is to pretend that Lisp doesn’t have some control construct, and write it as a macro. This is easier than inventing new control constructs both because it doesn’t require thinking of a domain where they might be useful and because the existing control constructs have clear semantics. Reimplementing existing control constructs also demonstrates how the language is already built up from a more primitive language by macros and how the approach to solving problems in Lisp is to design and implement a language in which to talk about the problem, where that language is seamlessly built on the underlying Lisp, and can inherit all of its power and flexibiliy, including the ability to extend the language.

An advantage of reimplementing existing control constructs for teaching Lisp is that you can compare the new construct to the existing one, and with some small constraints you can do this exhaustively, so you can know whether you have actually implemented it right. This is, obviously, not possible in general, but if the operator has trivial syntax (so not cond) and if you limit the arguments of the operator to booleans then you can enumerate all the possible arguments in the obvious way, and so long as it returns a result for all combinations of arguments (does not fail to halt in other words) and is deterministic then there are only two things you need to check:

does the operator produce the same result for all combinations of arguments ($2^n$ possibilities for $n$ arguments) as the existing one?
does the operator evaluate its arguments the same number of times as the existing one for all these combinations?

So, for instance, if takes three arguments (in Racket) and should evaluate the first exactly once, and the others at most once, as well as returning the correct value.

Obviously such a check is not a full check of the operator — it does not tell you what it does with non-boolean arguments for instance. But I was interested in writing the check largely because it’s clearly a reasonably hairy macro which I know how to write in CL and wanted to see if I could write in Racket (I’m not very likely to teach people Lisp again).

What the macro needs to do

The idea is that to compare two boolean operators o1 and o2 which take n arguments you need to generate code which looks like this:

(for/and ([c (expt 2 n)])
  (let ([a1 (bitwise-bit-set? c 0)] ...)
    (let ([o1c1 0] ...)
      (let ([o2c1 0] ...)
        (and (eq? (o1 (begin (set! o1c1 (+ o1c1 1)) a1) ...)
                  (o2 (begin (set! o2c1 (+ o2c1 1)) a1) ...))
             (= o1c1 o2c1) ...)))))

So a1 is the first argument, o1c1 counts how many times o1 evaluates it, and o2c1 counts how many times o2 evaluates it, and so on. I decided to compare the operators with eq? rather than eqv? for no very good reason except that it works for operators whose results are booleans, which is what I was interested in. I should almost certainly use eqv? I think — certainly the -equivalent in the name would imply that — but I’m not.

It’s clear that a loop like that checks all of the $2^n$ possibilities for the arguments, where each argument can be either #f or #t only. So this does an exhaustive check of all the possibilities, and provided o1 and o2 are deterministic and halt on all their arguments it will tell you whether they are equivalent.

And finally, this must be written as a macro, because the operators it is testing are themselves not generally functions: in particular things like if and or are obviously themselves not functions.

Things I did not know how to do

The big thing I didn’t know how to do here was to make up new identifiers: all the counters need to be created, and possibly also the argument names. In CL you’d do this with make-symbol or gensym or something like that. Assuming I want to use syntax-case rather than writing a CL-style construct-the-form-with-backquote-and-use-datum->syntax macro (which I very much do want to do) then there are two problems:

constructing the names of the counters;
making them available as pattern variables.

Well, (2) is easy: you can use nested syntax-cases, or equivalently but much more prettily, with-syntax to bind the pattern variables. And it turns out that with-syntax is willing to do a lot of work on your behalf: if you give it something which is not a syntax object it will massage it into one for you. So, in particular, this works:

(with-syntax ([(o1c ...) (list ...)])
  ...)

It takes the list it is given, turns it into a syntax object (with datum->syntax I suppose) and then does the matching. So you can be really lazy here: all you need to invent is a list of identifier syntax objects, and with-syntax will do the rest, making the program a lot less noisy. This is a really neat feature, although it might lead you to get confused about what is, and what is not, a syntax object I suppose. Anyway, I used it ruthlessly.

So this leaves (1). You could obviously do this with something like (datum->syntax ctx (string->symbol (format ...))), but Racket provides a nice shorthand for that in the form of format-id: (format-id ctx "~a-count" v) will construct an identifier syntax object from v using ctx as lexical context. And it will do the appropriate magic if v is an identifier syntax object: extract the symbol from it and use it as the argument to format in the appropriate way.

So it looks pretty straightforward to construct lists of identifiers and bind them to pattern variables. The final thing that confuses me is what lexical context to use for the identifiers. The macro should be hygenic, which means they can’t have the context of the syntax object it is working on, but I think can have more-or-less any other context where they have no existing meaning: I just invented an object for them, which I think is safe, although I am a bit confused about this.

What users see

I spent a really long time stuck on what the syntax of the macro should be: this is entirely stupid because it just does not matter that much. The reason I got stuck is that it would matter if this was a real library and I am constitutionally incapable of writing things without worrying about that kind of thing. Eventually I decided that it would be best if the user provided the argument names as a list, because they generally make sense to users and because I didn’t want to get into something which looked as if you could pass it an integer when in fact what it needs is a literal integer. So I decided on a syntax like this:

(boolean-operators-equivalent? o1 o2 (a1 ...))

So, for instance:

(boolean-operators-equivalent? if my-if (test then else))

I still don’t really like this; but I’m just playing so, well, it will do.

Additional cleverness

I wanted to report syntax errors in a reasonable way: apparently the proper way to do this is using syntax-parse but I am not ready to understand that yet, so I used wrong-syntax and the current-syntax-context parameter to get reasonable-looking errors.

I thought it would be nice to be able to report failures of equivalence, so there is a parameter which controls that and the expansion of the macro includes a check for the parameter and prints the failed cases if it’s true. All this happens at run time (phase 0) of course.

The macro itself

So, finally, here it is.

(require (for-syntax (only-in racket/syntax format-id
                              current-syntax-context wrong-syntax)))

(define boe-report-failure? (make-parameter #f))

(define-syntax (boolean-operators-equivalent? stx)
  ;; Given the names of two boolean operators and a list of argument
  ;; names, expand to a form which tests that they are equivalent, by
  ;; evaluating the with arguments bound to all the combinations of #t
  ;; and #f, and also checking that they evaluate the same arguments
  ;; in each case.
  ;;
  (parameterize ([current-syntax-context stx])
    (syntax-case stx ()
      [(_ o1 o2 (v ...))
       (let* ([vars (syntax->list #'(v ...))]
              [nvars (length vars)])
         ;; This check could be a guard, but we need the bindings
         ;; anyway, so.
         (for ([var vars])
           (unless (identifier? var)
             (wrong-syntax var "not an identifier")))
         ;; vars is now a list of identifiers, and nvars is how many
         ;; there are.  We need to construct syntax for check
         ;; variables for each var and and operator, as well as
         ;; construct 2^n and a list of bit numbers.]  This is being
         ;; fairly fast and loose: it turns out that various things
         ;; get automagically converted into syntax objects, and I
         ;; have not cared about the context for numbers (what is
         ;; it?).  In general I am a bit confused about what the
         ;; context should be here, but it clearly should *not* be
         ;; stx.
         ;;
         (with-syntax ([(o1c ...) (for/list ([v vars])
                                    (format-id #'boe "~a-1-eval-count" v))]
                       [(o2c ...) (for/list ([v vars])
                                    (format-id #'boe "~a-2-eval-count" v))]
                       [2^n (expt 2 nvars)]
                       [(b ...) (for/list ([i nvars]) i)])
           ;; And now just write the pattern we want.  '...' is pretty
           ;; clever, it turns out
           #'(for/and ([c 2^n])
               (let ([v (bitwise-bit-set? c b)] ...)
                 (let ([o1c 0] ...)
                   (let ([o2c 0] ...)
                     (or (and (eq? (o1 (begin (set! o1c (+ o1c 1)) v) ...)
                                   (o2 (begin (set! o2c (+ o2c 1)) v) ...))
                              (= o1c o2c) ...)
                         (begin
                           (when (boe-report-failure?)
                             (eprintf "Not equivalent:~% ~a~% ~a~%"
                                      (list 'o1 `(,v ,o1c) ...)
                                      (list 'o2 `(,v ,o2c) ...)))
                           #f))))))))]
      [else
       (wrong-syntax #'else "expecting o1 o2 (a1 ...)")])))

To my astonishment, this worked pretty much first time (it did not initially have the wrong-syntax stuff, but this was easy compared to the rest of it):

> (define-syntax-rule (if/broken test then else)
    (or (and test then) else))
> (boe-report-failure? #t)
> (boolean-operators-equivalent? if if/broken (test then else))
Not equivalent:
 (if (#t 1) (#f 1) (#f 0))
 (if/broken (#t 1) (#f 1) (#f 1))
#f

The macro, complete with some tests and other infrastructure can be found here⁴.

Notes and queries

I still don’t know whether this is really idiomatic Racket, although I am reasonably happy that I understand what is going on. There are a couple of things I am not sure about:

is the context for the count variables right? I think it is, but I am not sure;
the macro relies heavily on Racket’s extremely smart behaviour with ... — I am still unclear just how smart this is and whether I am relying on things which are not actually specified to happen;
similarly it relies on with-syntax being willing to convert things to syntax objects for you, which I am not sure is safe.

However, even with these worries, I think it’s pretty clear that Racket macros are significantly nicer than CL macros, if also significantly more opaque.

I am going to use ‘Lisp’ to mean ‘Lisp-family’ from now on. This is not meant to denigrate Scheme — this post is about Racket, after all — I just need a term which is not too clumsy. ↩
Of course, programs in other languages often do end up reasoning about themselves: people end up writing little languages all the time. But you only have to look at most examples of this sort of thing to realise how far ahead Lisp is: I’m currently having to deal with a system whose configuration files are in a mutant version of Windows ini file syntax, with a preprocessor which is entirely unaware of that syntax, and an entire other language which lives in strings in the base language. The preprocessor does not know about the string syntax so it pokes down into this inner language as well. I’d like to say that Greenspun’s tenth law applies, but that would imply a level of sophistication entirely missing in this horrible thing: all I want to do is leave this job and never think about it again. ↩
Macros were often used to inline code in the days of primitive compilers of course, but that’s a long time ago now. ↩
I may move it somewhere more permanent in due course, so bookmark this at your peril. ↩

Greenspunning

2015-10-08T15:16:56Z

Three approaches to solving problems on computers.

When faced with a computational problem there are three common approaches:

write a program to solve the problem;
write a tool to solve the problem and other problems of the same kind;
write a programming language in which you can then write tools which solve problems of the same, and other, kinds.

Most people start by doing the first. Bradshaw’s corollory to Greenspun’s tenth law states:

for problems of size $s \ge s_1$, then, regardless of the initial approach, the final result is as if the third approach had been taken, even if this is not understood by the people solving the problem;
there is a problem size $s_0$ above which it is most efficient to take the third approach from the beginning;
$s_0 \lt s_1$.

What this means is that, if you have a sufficiently large problem ($s \ge s_1$) to solve then, whatever your intentions, you will inevitably end up creating a programming language as part of the solution. And there is a range of problems smaller than this ($s \in (s_0, s_1)$) for which the quickest way to solve the problem is to design and implement a programming language.

So, when approaching a problem, it is important to understand the values of $s_0$ & $s_1$ and how they compare to $s$. These values are hard to discover: a good trick is to start with a platform which makes $s_0$ very small and always take the third approach.

Rumours of my death

2015-02-01T20:54:34Z

When I first used Lisp, the common refrain was that Lisp was dead.

There was a single free implementation of CL (which required you to physically sign a license of some kind and return it, in exchange for a tape) which was deficient in many respects. The two or three commercial implementations cost about a year’s salary each. Enormous effort had been spent on implementations which ran on special hardware. One variant of these cost more than your house: the other rather less, but turned out to have been implemented by the fey — you seriously did not want to spend too much time with it if you did not want problems involving having your firstborn somehow changed into a strange and somehow absent creature.

(And there was a terrible, unspeakable truth about even the expensive hardware: the people who implemented it didn’t understand computer performance very well with the result you would expect. The systems were faster than a VAX, but everything was faster than a VAX, including some PDP–11s. A Sun 3/260 ate them alive, and you could buy several of those for the cost of a house, with bundled licenses.)

Performance was pretty grim: of course nothing was fast on machines that, on a good day, could execute a few million instructions a second, but Lisp implementations were problematic at best. You spent a lot of time turning recursive code into iterative code by hand and writing macros (no inlining) to get performance to be reasonable and worrying about the primitive garbage collectors.

There was no standard: existing implementations differed in basic details like error handling (not in the aluminium book) and a standard object system was a distant dream. The news from the standards committee was ominous: the special-hardware people were exerting pressure and there were serious worries that the object system would not be efficiently implementable on stock hardware. The language was going to be huge.

Standard or semi-standard libraries were not really thought of.

Everyone knew Lisp was dead: the coming thing was, perhaps, Scheme — tail-call elimination in the language, a small language (yet MIT Scheme somehow had a bigger footprint than the CLs we used) — or C++ or some functional language whose name no-one now remembers. But Lisp was dead: no question about it.

Fast forward.

I have two high-quality CL implementations on my machine and one Scheme-derived system, also of very high quality, which created this blog: I have long ago stopped counting the number of good-quality free implementations. One of the implementations I use is commercial: the annual support is about 10% of my monthly rent. I can run dozens of instances of each without the machine noticing, and I could happily run a full CL development system on a system less powerful and smaller than my phone. Performance is a solved problem: yes, highly-optimised code is, perhaps, slower than optimised C or Fortran but since almost all performance problems are design problems no-one older than about 19 cares any more. CL has an advanced, performant and standard object system and, in effect, a standard metaobject system as well. The library problem has been solved by Quicklisp and a large number of good-quality standard libraries. I am still using code I wrote over twenty-five years ago with essentially no modification: meanwhile the Python code I wrote ten years ago is long rendered obsolete by gratuitous changes in the language (the Perl code I wrote at the same time is doing fine, however).

And yet still the cry goes up: Lisp is dead; Lisp is dead.

Macros in Racket, part two

2015-01-28T19:31:18Z

The second part of my notes on writing macros in Racket.

This is the second part of at least three: the first part is here, and the third part is here. This won’t make much sense unless you’ve read that. As before I make no claims to be an expert in Racket’s macro system although I am familiar with Lisp macros in general: this is just some more notes I wrote while learning it.

The unwashed Lisp hacker’s version of `collecting`

So, we can write clet: can we write collecting? Yes, we can:

(require (for-syntax racket/list))

(define-syntax (collecting stx)
  (datum->syntax
   (quote-syntax collecting)
   `(let ([r '()])
      (define (,(datum->syntax stx 'collect) it)
        (set! r (cons it r)) it)
      ,@(rest (syntax->list stx))
      (reverse r))))

This works because, in the internal definition of collect, we’ve intentionally given it a name which uses the context of the syntax object we’re transforming, not the context of the macro. It’s easy to confirm that this works the way you would expect, and in particular that it’s safe in both directions: for instance

> (let ((reverse (λ (x) x)))
    (collecting (collect 1) (collect 2)))
'(1 2)

shows that the binding of reverse when the macro is called has not ‘infected’ the macro definition.

It seems as if that should be all you need: so long as you are careful about which context you choose, and you make sure that the ‘default’ context is the one from the macro not from where it is used. In fact it isn’t, quite: see below. However even if it were, it’s clearly a pain to write macros this way.

Pattern matching

Pretty much all macros do two things:

deconstruct their arguments in some more-or-less complicated way, but almost always in a way which is significantly more complicated than anything that needs to be done for the arguments of a function;
construct a form which is the result of the macro and which, again, may be complicated.

The beauty of traditional Lisp macros is that since the arguments and results of the macro were just what the reader spat out — lists and symbols and so on — and since Lisp was kind of good at doing things to these structures as it was designed for that, and finally since the whole power of the language was available in the macro, this was not horrible even without special tools, although it was not particularly pleasant for complicated macros.

Hygienic macros make this much less pleasant because the objects that need to be deconstructed and constructed are now opaque syntax objects, and there is additional worrying about context to do. The answer to this is to provide special tools which do the boring bits for you: this makes everything simpler, at the cost of making it still more opaque what is actually happening. In almost all cases that’s a tradeoff worth making. Pattern matching is also a fashionable thing amongst the young and hip, of course.

The way this is done in Racket is via syntax-case, its slightly simpler friend syntax-rules, and by syntax and variants on it.

syntax-case takes a bit of syntax and matches it against patterns, binding matches, which can then be used in syntax forms lexically within it to return syntax objects, whose context is that of the syntax-case form (so hygienic). There is syntactic sugar for syntax: (syntax ...) can be written #'... in the same way that (quote ...) can be written '.... There is also quasisyntax which works the same way as quasiquote, except that the various unquoting things are preceeded with #. quasisyntax, unsurprisingly also has syntactic sugar coating: (quasisyntax ...) can be written #`....

I’m not going to describe the patterns in any detail, largely because I only understand the simple cases. However the simple cases are relatively easy to understand and pleasant to use.

Once a case has matched in syntax-case the corresponding expression is evaluated, and its value is the value of the form. Generally that wants to be a bit of syntax.

The first important thing to understand is that syntax is not quote-for-syntax: it interpolates things which matched in a lexically surrounding syntax-case, if there is one (if there isn’t, then I think it is quote-for-syntax).

The second important thing to understand is that syntax-case and syntax turn Racket into a sort of bodged Lisp–2: the things matched by syntax-case can be used only in syntax forms. But it’s not actually a separate namespace, because if you refer to them outwith such a form you get a compile-time error. I don’t know why this is — perhaps to avoid accidentally naming matches outside a syntax form — but it is certainly annoying.

So, here are some examples.

A simple while form:

(define-syntax (while stx)
  (syntax-case stx ()
    [(_ test body ...)
     #'(let loop ()
         (when test
           body ...
           (loop)))]))

A simple implementation of let, leaving out the named-let case, which shows how good the pattern matching is:

(define-syntax (with stx)
  (syntax-case stx ()
    [(_ ([var val] ...) body ...)
     #'((λ (var ...) body ...) val ...)]))

A better implementation which deals with the empty body case ((λ (...)) is illegal in Racket) and also optimises a simple case:

(define-syntax (with stx)
  (syntax-case stx ()
    [(_ () body ...)
     ;; no vars: trivial case
     #'(begin body ...)]
    [(_ ([var val] ...))
     ;; null body: make sure vars are evaluated
     #'(begin val ... (void))]
    [(_ ([var val] ...) body ...)
     #'((λ (var ...) body ...) val ...)]))

One thing which syntax-case allows is the notion of literal names which must occur in the source. So for instance let’s say I wanted to write some mutant loop macro whose syntax was (loop for x in y do ...): where for, in, do are literals. Well, I can write something to match this:

> (define-syntax (loop stx)
    (syntax-case stx (for in do)
    [(_ for v in l do body ...)
     #'(for ([v (in-list l)]) body ...)]))
> (loop for x in '(1 2 3) do (print x))
123
> (loop with x in '(1 2 3) do (print x))
loop: bad syntax in: (loop with x in (quote (1 2 3)) do (print x))

The syntax object that corresponds to stx here is the whole form: the equivalent to CL’s &WHOLE. It’s almost never necessary to worry about the car of this since it will obviously be loop. However I’m always tempted to provide it as a literal.

syntax-rules is (almost: there is some complexity I think) a wrapper around syntax-case which provides the function wrapper for it and which implicitly wraps the right hand side of the cases, which must be just one form, in a syntax form. So the above definition of with could be written:

(define-syntax with
  (syntax-rules ()
    [(_ () body ...)
     ;; no vars: trivial case
     (begin body ...)]
    [(_ ([var val] ...))
     ;; null body: make sure vars are evaluated
     (begin val ... (void))]
    [(_ ([var val] ...) body ...)
     ((λ (var ...) body ...) val ...)]))

syntax-rules can be defined something like this (this is due to bmastenbrook):

(require (for-syntax 
          (rename-in racket 
                     [syntax-rules racket:syntax-rules])))

(begin-for-syntax
  (define-syntax syntax-rules
    (racket:syntax-rules ()
      [(_ literals (pattern expansion) ...)
       (lambda (s)
         (syntax-case s literals
           (pattern #'expansion) ...))])))

define-syntax-rule combines define-syntax and a single rule for syntax-rules. I think it might be equivalent to this:

(define-syntax define-syntax-rule
  (syntax-rules ()
    [(_ (name pat ...) expansion)
     (define-syntax name
       (syntax-rules ()
         [(name pat ...) expansion]))]))

although I am probably missing some complexity here.

There is a useful variant on syntax-case called with-syntax: it looks more like let-style thing, and all the patterns in the clauses must match, when all the pattern variables will be bound.

So, what about our desirable macros?

collect is pretty easy. Here are two different versions. The first uses quasisyntax:

(define-syntax (collecting stx)
  (syntax-case stx ()
    [(_) #'(void)]
    [(_ body ...)
     #`(let ([r '()])
         (define (#,(datum->syntax stx 'collect) it)
           (set! r (cons it r)) it)
         body ...
         (reverse r))]))

The second uses with-syntax:

(define-syntax (collecting stx)
  (syntax-case stx ()
    [(_) #'(void)]
    [(_ body ...)
     (with-syntax ([collect (datum->syntax stx 'collect)])
       #'(let ([r '()])
         (define (collect it)
           (set! r (cons it r)) it)
           body ...
           (reverse r)))]))

This is pretty nice, I think. Note that you could not do this with syntax-rules, or at least I can’t see how to do it: syntax-rules is quite a lot less general than syntax-case.

clet is harder, because each element of the binding list may be either an identifier or a two-element list. If we insisted on a two-element list it would be easy (see above). Here is the best I can do:

(require racket/undefined)        

(define-syntax (clet stx)
  (syntax-case stx ()
    [(_ ()) #'(void)]
    [(_ () body ...) #'(begin body ...)]
    [(_ (b ...) body ...)
     (let-values ([(vars vals)
                   (for/lists (as vs) ([binding (syntax->list #'(b ...))])
                     (syntax-case binding ()
                       [(var val) 
                        (identifier? #'var)
                        (values #'var #'val)]
                       [var
                        (identifier? #'var)
                        (values #'var #'undefined)]
                       [_ (raise-syntax-error #f "bad binding" stx)]))])
       #`((λ #,vars body ...) #,@vals))]))

Well, this is still quite hairy, but almost all of the hair involves processing the binding list, which is done using syntax-case again, using an additional feature of it whereby it can use a ‘guard’ expression to decide whether a clause matches: identifer? returnt true if a syntax object refers to an identifier. I think there must be a way of using with-syntax to avoid the quasisyntax form.

Even with all this hair, this version of clet is far easier to read than the previous one, and not harder to read than the CL equivalent.

A better version of clet would, I think, need a proper parser for syntax. I think that is what syntax-parse is, although I have not investigated that.

Macro composition

As mentioned above, we don’t yet have quite all the tools we need to write some kinds of macros: specifically macros which are intentionally slightly unygienic, such as collecting. As an example, let’s suppose we wanted a general purpose, intentionally-unhygenic, with-abort macro which provided an abort function which would, well, abort. Without thinking too hard about the implications of call/cc we could write this as:

(define-syntax (with-abort stx)
  (syntax-case stx ()
    [(_ body ...)
     #`(call/cc (λ (#,(datum->syntax stx 'abort))
                  body ...))]))

So now (with-abort (abort 2) (end-the-world)) returns 2 and does not end the world.

Well, we might want to use this macro in another macro:

(define-syntax-rule (while/abort test body ...)
  (with-abort
    (let loop ([r test])
      (when r
        body ...
        (loop test)))))

Now something like the following will work:

> (let ([x 0])
    (while/abort (< x 10) (set! x (+ x 1)) (print x)))
12345678910

But the whole point was to be able to use abort in the body, and that doesn’t work:

> (let ([x 0])
    (while/abort (< x 10) (set! x (+ x 1)) (when (> x 1) (abort 'done))))
abort: undefined;
 cannot reference an identifier before its definition

Oh, dear. The problem here is that while/abort is hygenic, so the abort binding that is introduced by with-abort is not visible in the body.

We could fix this by better design:

(define-syntax-rule (with-named-abort (abort) body ...)
  ;; a better macro
  (call/cc (λ (abort) body ...)))

(define-syntax (with-abort stx)
  ;; backwards compatible
  (syntax-case stx ()
    [(_ body ...)
     #`(with-abort (#,(datum->syntax stx 'abort)) body ...)]))

(define-syntax (while/abort stx)
  ;; the end result
  (syntax-case stx ()
    [(_ test body ...)
     #`(with-named-abort (#,(datum->syntax stx 'abort))
         (let loop ([r test])
           (when r
             body ...
             (loop test))))]))

But that’s not the solution we’re after.

Racket’s answer to this is syntax parameters. I don’t completely understand these, but they are at least close to dynamic variables, except at macro-expansion time. What you do is to define a syntax parameter, and then rebind it during the expansion: the rebound value is visible to macros which are expanded dynamically within the rebinding form. As with Racket’s ordinary special variables these look like functions (yet another namespace in disguise).

So we can define a syntax parameter called abort using define-syntax-parameter:

(require racket/stxparam)

(define-syntax-parameter abort
  (λ (stx)
    (raise-syntax-error #f "not available" stx)))

So now any reference to abort will result in a syntax error:

> (abort)
abort: not available in: (abort)
> abort
abort: not available in: abort

And we can now try to use syntax-parameterize, to rebind abort as a macro:

(define-syntax with-abort
  (syntax-rules (with-abort)
    [(with-abort) (void)]
    [(with-abort body ...)
     (call/cc
      (λ (a)
        (syntax-parameterize ([abort
                               (syntax-rules ()
                                 [(_ ...) (a ...)])])
          body ...)))]))

And this fails horribly, because the outer syntax-rules thinks it owns the patterns and sees ...s that it does not expect. So much for that.

Well, we could at least check this works with a specific number of arguments:

(define-syntax with-abort
  (syntax-rules (with-abort)
    [(with-abort) (void)]
    [(with-abort body ...)
     (call/cc
      (λ (a)
        (syntax-parameterize ([abort
                               (λ (stx)
                                 (syntax-case stx (abort)
                                   [(abort) #'(a)]
                                   [(abort x) #'(a x)]
                                   [_ (raise-syntax-error #f "I give up" stx)]))])
          body ...)))]))

But this is obviously just a rubbish answer.

Well, there is an answer to this: all we really need to do is to make the abort macro attach itself to a, and there is a special hack, make-rename-transformer, to do this:

(define-syntax with-abort
  (syntax-rules (with-abort)
    [(with-abort) (begin)]
    [(with-abort body ...)
     (call/cc
      (λ (a)
        (syntax-parameterize ([abort (make-rename-transformer #'a)])
          body ...)))]))

And this now works:

> (with-abort (abort 1 2 3))
     
1
2
3

And we can use this to write a really robust version of collecting

(require racket/stxparam)

(define-syntax-parameter collect
  (λ (stx)
    (raise-syntax-error #f "not collecting" stx)))

(define-syntax collecting
  (syntax-rules ()
    [(_) (void)]
    [(_ body ...)
     (let ([r '()])
       (define (clct it)
         (set! r (cons it r)) it)
       (syntax-parameterize ([collect (make-rename-transformer #'clct)])
         body ...
         (reverse r)))]))

As far as I can see there is still a problem, however: it is very hard to write macros which expand to other macros which themselves do pattern-matching, since the patterns get acquired by the outer macros. There must be some answer to this, but I can’t see what it is.

On the other hand, this is also extremely painful in CL: here is a version of collecting where collect is a local macro:

(defmacro collecting (&body forms)
  ;; collect lists forwards using a tail pointer
  ;; local macro version
  (let ((rn (make-symbol "R"))
        (rtn (make-symbol "RT"))
        (itn (make-symbol "IT")))
    `(let ((,rn '())
           (,rtn nil))
       (macrolet ((collect (form)
                    `(let ((,',itn ,form))
                       (if (not (null ,',rn))
                           (setf (cdr ,',rtn) (cons ,',itn nil)
                                 ,',rtn (cdr ,',rtn))
                         (setf ,',rn (cons ,',itn nil)
                               ,',rtn ,',rn))
                       ,',itn)))
         ,@forms)
       ,rn)))

This is not easy to understand.

Additionally, the problem almost always comes from ellipses, and in many interesting cases they can be avoided by using dotted pairs as patterns — here is yet another version of with-abort that does this:

(require racket/stxparam)

(define-syntax-parameter abort
  (λ (stx)
    (raise-syntax-error #f "not available" stx)))

(define-syntax with-abort
  (syntax-rules (with-abort)
    [(with-abort) (void)]
    [(with-abort body ...)
     (call/ec
      (λ (a)
        (syntax-parameterize ([abort
                               (syntax-rules (abort)
                                 [(abort . args) (a . args)])])
                             

          body ...)))]))

This is clearly better than the CL version.

Summary

Well, I think I now know enough about Racket’s macros to be going on with: I can certainly write the macros I need to be able to write now without it just being cargo-cult programming. There are still things I don’t understand, and the whole system smells to me as if, by trying remain ideologically pure, it has become vast and essentially incomprehensible. This seems to be a common problem with Scheme, unfortunately.

Small notes

Macro definitions scope properly, so you can define a local macro the same way you can define a local function, so this works:

(define (foo ...)
  (define-syntax-rule (while test body ...)
    (let loop ()
      (when test
        body ...
        (loop))))
  ... (while ... ...) ...)

This makes the equivalent of CL’s MACROLET easy to do.

For fun, here is a version of with which can deal with named-let: There must be a way of implementing this without assignment, but I can never work out what it is.

(define-syntax (with stx)
  (syntax-case stx ()
    [(_ ())
     ;; all null
     #'(void)]
    [(_ () body ...)
     ;; no vars: trivial case
     #'(begin body ...)]
    [(_ ([var val] ...))
     ;; null body: make sure vars are evaluated
     #'(begin val ... (void))]
    [(_ ([var val] ...) body ...)
     ;; normal let
     #'((λ (var ...) body ...) val ...)]
    [(_ n ())
     (identifier? #'n)
     ;; named null
     #'(void)]
    [(_ n ([var val] ...))
     (identifier? #'n)
     ;; named null body
     #'(begin val ... (void))]
    [(_ n ([var val] ...) body ...)
     ;; named let with arguments
     ;; (is there an implementation without assignment?
     (identifier? #'n)
     #'((λ (n)
          ((λ (l)
             (set! n l)
             (l val ...))
           (λ (var ...) body ...)))
        #f)]
    [_ (raise-syntax-error #f "bad syntax" stx)]))

Things I still do not know or understand

At this point I’m mostly comfortable writing macros in Racket, but there are things I still do not understand:

protecting and arming syntax objects — I just don’t understand what this is about at all;
syntax-parse is, I think, not difficult but I have not bothered to learn about it as it seems to add yet another layer.
there are probably other things that I don’t even know I don’t know.

At some point I might write a further part of this series on some of that.

Pointers

Eli Barilay’s paper on syntax-parameterize.

Fear of Macros, again.

Macros in Racket, part one

2015-01-13T14:45:48Z

I’ve written in Lisp for a long time, but I’ve never used a hygienic macro system in any way other than the most simple. Here are some initial notes on my experiences learning Racket’s macro system.

This is the first part of several: see part two and part three. I’m not completely fluent with Racket macros yet: there are almost certainly mistakes and confusions here. Despite appearances, I also have no axe to grind: I’m learning Racket because I want to and I have time. Finally this is not a tutorial: look at Greg Hendershott’s Fear of Macros for something closer to that. This is just some notes which were useful to me, and might be useful to other CL people.

Macros in Common Lisp

Common Lisp’s macro system is, in essence, simple: it’s what you’d end up writing if you had to write a macro system for a Lisp. That’s not surprising because it is the descendent of the first macro systems people wrote for Lisp. In CL what happens is this:

the reader ingests the source text and produces data structures which represent the source of the program;
these structures are possibly transformed by macros, which are simply Lisp functions which are given the Lisp representation of the source and return some other representation;
once all macros are expanded, then the code is compiled, evaluated or both.

(I have missed out some subtleties here, but they don’t matter for my purposes.)

In CL, what the reader produces is exactly what you would expect. If it reads "(defun foo (a) a)" then, with standard settings, it returns a list whose car is the symbol DEFUN (in the CL package) and so on. It is this structure that macros transform.

CL provides relatively limited support for writing macros: there is backquote, which is critical to being able to write macros which are even slightly readable, limited pattern matching in the form of destructuring, and there are mechanisms to generate unique names as well a few other things. There is a semi-standard way of enquiring about bindings in the environment at macro expansion time, although this is not in the standard.

In practice, CL’s macro system has turned out to work very well; in theory it has all sorts of problems, the most important being that the programmer is entirely responsible for making sure that macros don’t introduce or accidentally use names they should not. Consider this:

(defmacro collecting (&body forms)
  ;; collect lists forwards using a tail pointer
  ;; polluting version
  `(let ((r '())
         (rt nil))
     (flet ((collect (form)
              (if (not (null r))
                  (setf (cdr rt) (cons form nil)
                        rt (cdr rt))
                (setf r (cons form nil)
                      rt r))
              form))
       ,@forms)
     r))

This intentionally introduces a function binding, collect, but also accidentally introduces bindings for r and rt.

(let ((r 2))
  (collecting
    (+ r r)))

Does not do what it should. One right way to write the collecting macro is like this:

(defmacro collecting (&body forms)
  ;; collect lists forwards using a tail pointer
  ;; non-polluting version
  (let ((rn (make-symbol "R"))
        (rtn (make-symbol "RT")))
    `(let ((,rn '())
           (,rtn nil))
       (flet ((collect (form)
                (if (not (null ,rn))
                    (setf (cdr ,rtn) (cons form nil)
                          ,rtn (cdr ,rtn))
                  (setf ,rn (cons form nil)
                        ,rtn ,rn))
                form))
         ,@forms)
       ,rn)))

And now the above form does not signal an error and correctly returns ().

Note that the problem is with names and not just bindings. Consider this CL code:

(defvar *stashes* '())
(defvar *mark* nil)
  
(defun stash (name thing)
  ;; Stash something under a name
  (setf *stashes* (acons name thing *stashes*))
  (values name thing))

(defun retrieve (name)
  ;; Retrieve the value of a name, dropping everything stashed more
  ;; recently, and stopping at the mark, if any.
  (let ((mark *mark*))
    (labels ((rl (tail)
               (if (or (null tail)
                       (eq (first tail) mark))
                   (values nil nil)
                 (destructuring-bind ((n . v) . r) tail
                   (if (eql n name)
                       (progn
                         (setf *stashes* r)
                         (values v t))
                     (rl r))))))
      (rl *stashes*))))

(defmacro with-marked-stash (&body forms)
  ;; mark the stack of stashes for the dynamic extent of FORMS
  (let ((mn (make-symbol "MARK")))
    `(let ((*stashes* (cons ',mn *stashes*))
           (*mark* ',mn))
       ,@forms)))

In this code the marks on the stack of stashes established by with-marked-stash are not bound anywhere: they are just names. But it’s important to the correct functioning of the code that they are unique names. (There are better ways of doing this such as using a fresh cons for the mark: I just wanted an example where a name mattered other than as the name of a variable.)

The politically correct way of saying that we’re talking about names is to talk about ‘lexical context’ or ‘lexical information’: it’s the same thing but more confusing to those not initiated into the cult, which is always good.

The disadvantages of the CL macro system are this problem with hygiene and the lack of any clever tools to do pattern matching on macro forms. The second of these is easily overcome by using any of a number of tools, while the first is generally not a problem in practice: CL being a Lisp–2 (separate namespaces for functions and variables) helps here.

The advantage of the CL macro system is that there is no magic: macros get passed the things that the source code looks like — generally a structure whose interesting parts are lists and symbols — which you process using the normal list-processing tools to produce some other structure which is the expansion of the macro. It’s easy enough that you could write it yourself: there are no special opaque objects being handed around.

That being said, having a standard set of tools for pattern matching in macros and a way of dealing with the hygiene problems which is less ugly than in CL might well be worth the cost in transparency.

Macros in Scheme

I am not a native Scheme person, but it has clearly taken the whole hygiene thing very seriously: Scheme, as a set of languages, treats purity as much more than CL, which revels in being a fairly grungy language, does. However these posts are not about Scheme: the only reason I am mentioning it is to say that I have not cared at all whether anything here applies generally to Scheme or is specific to Racket.

Macros in Racket: baby steps

For a long time the only kind of macros that I’ve really been able to define in Racket are annoyingly trivial ones using define-syntax-rule, things like:

(define-syntax-rule (while test body ...)
  (let loop ()
    (when test
      body ...
      (loop))))

That’s all very well, but the ‘obvious’ (and obviously wrong) definition of collect then looks like this:

(define-syntax-rule (collecting body ...)
  ;; horribly wrong	
  (let ([s '()])
    (define (collect it)
      (set! s (cons it s))
      it)
    body ...
    (reverse s)))

(There’s no obvious way to build lists backwards in Racket: reversing the list is probably as cheap as anything). This is either introducing a spurious binding for s or not introducing a deliberate one for collect, and in fact, of course, it’s the latter.

Quite apart from this, define-syntax-rule gives the strong impression that it lets you write only the sort of macros that would give people who write C++ great pride: simple ones. (Actually you can do reasonably hairy things even with this because the pattern matching is very competent:

(define-syntax-rule (mlet ([var val] ...) body ...)
  ((λ (var ...) body ...) val ...))

is an implementation of simple let, for instance. Indeed we can defined named let as well:

(define-syntax-rule (nlet label ([var val] ...) body ...)
  (mlet ()
    (define (label var ...) body ...)
    (label val ...)))

What I can’t work out how to do is to make mlet do both things: I think this is too hard for define-syntax-rule although I might be wrong.)

But for a long time I was stuck with that: whenever I looked at Racket macros in more detail I walked into a wall of opaque terminology and just decided that I had better things to do that year. This year, I don’t.

Two desirable macros

There are many ways people use macros in Lisp: some of them are good. I decided that if I could write two macros and understand them then I would be well on my way.

collecting / collect. This is the macro given above in CL. It’s interesting not for what it does — the tail-pointer stuff is less interesting now than it once was and is hard to implement in Racket anyway — but because it introduces a binding: it is intentionally not completely hygienic, while having an essentially trivial expansion: no complicated destructuring is needed.
CL’s let, which I’ll call clet. This is interesting because it requires destructuring of arguments which is not completely simple, but it does not present problems of hygiene. The reason it’s not just a subset of Racket’s let is that CL allows variables with no initial value, which get bound to nil and should, I think, become undefined in Racket. So (clet ((x 1) y) body ...) should expand to (let ([x 1] [y undefined]) body ...) or something equivalent to that.

Here is a simple implementation of clet in CL, missing any error checking:

(defmacro clet (bindings &body forms)
  (multiple-value-bind (args vals)
      (loop for binding in bindings
            for consp = (consp binding)
            collect (if consp (first binding) binding) into as
            collect (if consp (second binding) nil) into vs
            finally (return (values as vs)))
    `((lambda (,@args) ,@forms) ,@vals)))

Like most macros in CL it’s not particularly pretty but it is reasonably clear what it does.

I will use these two macros as examples below.

Phases

To understand macros in any Lisp you need to develop a strong idea of the various ‘times’ that things happen and the relationships between them: for CL these are things like read time, macro expansion time, compilation time (compiler-macro expansion time), load time, run time and so on. Racket has formalised the parts of this after read time into a notion of ‘phase’:

phase 0 is run-time;
phase 1 is macro expansion time;
phase 2 would, I think, be macros used in macro expansion;
and so on.

However I am not sure how this ties in to read time: is that phase 1? For CL read time is before macro expansion time although the two are, or may be, interleaved at the granularity of forms (rather than a per-file or per-compilation-unit). Also there are negative phases which I don’t understand, although I think they must be to do with code which exists at macro expansion time (phase 1) wanting to make things available at run time (phase 0). All of this is integrated into the module system (and CL gets away without it mostly because it does not have a formalised module system).

Bindings exist at a phase, and the same name can have different bindings at different phases.

Modules can say what they provide at which phase, and, importantly, the racket module does indeed provide different things at different phases: if you look at it you’ll find:

(provide ...
         (for-syntax (all-from-out racket/base)))

Which means that, at phase 1, what is available is racket/base: a significantly smaller language than racket itself. If you need things in macros which are in racket but not racket/base you need to require them:

(require (for-syntax ...))

An example of this is first & rest, both of which are provided at phase 0 by racket but not at phase one: if you want them you need to say (require (for-syntax racket/list)).

Syntax objects

As in CL, Racket macros are source-to-source functions. The difference is that in Racket the source is represented by a syntax object and a macro needs to produce another syntax object, while in CL source is represented as it looks: usually as nested lists.

So then a Racket macro is simply a function which maps from syntax objects to other syntax objects. The reason for having an opaque syntax object is that it can carry around all sorts of information around with it, and in particular it can carry information about names, which help the system maintain hygiene. (There is also information about source location and so on, but this isn’t so important.)

So the Racket macro system needs tools to transform syntax objects into other syntax objects, ultimately by digging around inside them to find out what the source code actually was. This is necessarily more complicated than it is in CL both because the objects are opaque and because they contain information which is not present at all in the objects CL macros get.

Additionally, and mostly independently, there is a layer on top of this which does not exist in CL (without libraries) at all: pattern matching and template filling. This means that for many purposes you can write macros in Racket simply by specifying patterns that the source must match and filling templates with the results of those matches. This is a very nice way of writing macros, although it renders what is actually going on even more opaque. For a CL person, used to feeling the bits between their toes, this can be quite disconcerting at first since what is actually happening can become entirely obscure.

Syntax objects for the unwashed Lisp hacker

Well, of course it is possible to ignore all this terrifyingly modern pattern matching stuff and write macros almost the way you do in CL, and it’s worth doing that at least once, perhaps. So here is clet:

(require (for-syntax racket/list)
         racket/undefined)

(define-syntax clet
  (λ (stx)
    (define ctx (quote-syntax clet))
    (define top-level (syntax->list stx))
    (define bindings (second top-level))
    (define body (rest (rest top-level)))
    (define-values (args vals)
      (for/lists (as vs) ([binding (syntax->list bindings)])
        (define it (syntax->list binding))
        (if it
            (values (first it) (second it))
            (values binding (datum->syntax ctx 'undefined)))))
    (datum->syntax 
     ctx
     `((λ (,@args) ,@body) ,@vals))))

So how does this work? Well, it uses some functions provided by Racket to look inside the syntax object (getting the ‘datum’ in the syntax object) and in turn to construct a new one:

syntax->list takes a syntax object which wraps a proper list and unpacks one level of it, returning a list of syntax objects, or #f if it does not wrap a proper list;
datum->syntax takes a context object and a datum and wraps it into a syntax object, leaving any syntax objects in the datum as they are;
quote-syntax is like quote but it creates a syntax object, and this object contains the lexical information present in the source.

So the macro pulls apart the syntax object in a fairly straightforward way: making it into a list, extracting the second element and all the remaining elements, which will be the binding specifications, and then grinding over the binding specifications, using syntax->list both to work out if the bindings are a list or not and to extract the variable and value if it is, and then reassembles everything as a call to an anonymous function.

The critical trick is that the context that datum->syntax needs is a syntax object and you need to pick the right one: you can use the syntax object you got given, which provides the context of the place where the macro was expanded, or you can use a syntax object of your own devising which provides that object’s context. And in this case we want our own context, not the context of place where the macro was expanded. This is what ctx is for: providing a suitable context.

Notice the require:

we need racket/list at phase 1 (macro expansion time) because the macro uses first and so on;
we need racket/undefined at phase 0 (run time) as the expansion of the macro uses undefined.

So we can try this:

(clet ((x 12) y) (values x y))
12
#<undefined>
> (let ((undefined 'hello)) (clet (x) x))
#<undefined>
> (clet ((undefined 'hello)) (clet (x) x))
#<undefined>
> (clet ((x 1)))
λ: bad syntax in: (λ (x))
> (clet (1) 1)
λ: not an identifier, identifier with default, or keyword in: 1

The second and third examples show why we need the macro context: we don’t want a binding of undefined to alter what the clet picks as the undefined value. The fourth and fifth examples show that the macro isn’t very robust, and has terrible error reporting.

Some notes:

I’ve deliberately written (define-syntax clet (λ (stx) ...) rather than the more pleasant (define-syntax (clet stx) ...) to make it clear that clet is a function which transforms a syntax object;
but I’ve used internal define where in CL there would be let* or nested lets — I’m not sure why other than reducing indentation;
the destructuring of the syntax object is done in a way which is primitive even by the standards of CL;
it should be evident that the macro is not very robust — something like (clet ((x 1) 2) ...) will fail horribly;
it’s not much less clear than the CL version, although I think it is a bit less clear.

I am fairly but not completely sure that this macro is right: I am slightly confused by the handling of undefined: although it is easy to check, by wrapping clet into a module, that clients of that module don’t themselves need to import racket/undefined and do get the right initial values in forms like (clet (x) ...) I am still a bit queasy about what it’s doing.

What is very clear is that this macro is just horrible: even by the standards of CL macros it’s horrible, because there is so much explcit unpacking and repacking going on. Things would be even worse if there was any significant error checking. Something better than this is needed to deal with syntax objects, in a way that it isn’t needed for CL macros. In next week’s exciting episode I’ll look at ways of making this better.

Pointers

Writing ‘syntax-case’ Macros by Eli Barzilay. This was the article that first helped me understand what was going on.

Fear of Macros by Greg Greg Hendershott. This is an introduction to macros, and macros in Racket in particular, by the author of Frog.

The cult of programming

2015-01-05T19:24:26Z

Programming is not meant to be easy and it’s important to make sure that it is as cryptic as possible otherwise people other than cult members might be able to understand it. Of course, you also need to make sure it’s pure, because otherwise cult members will laughingly throw you into a pit full of spikes and the rotting remains of other heretics.

For instance, you can’t be writing this sort of thing:

(defun ss (n)
  (let ((s 0) (i 0))
    (tagbody
     loop
     (when (> i n) (go done))
     (setf s (+ s (* i i))
           i (+ i 1))
     (go loop)
     done
     (return-from ss s))))

This is just terrible code. Non cult members may well be able to understand it, and the cultists will have you in the pit before you know it.

You might think this was better

(defun ss (n)
  (loop for i from 0 to n
    summing (* i i)))

But in fact it’s far worse. Fellow cultists will definitely still be at the laughing and pit-throwing, and the others will certainly understand it and laugh at you because you don’t know the closed form.

Instead, you must write this:

(define (ss n)
  (let-values ([(a i l) (call/cc (λ (c) (values 0 0 c)))])
    (l (+ a (* i i))
       (+ i 1)
       (if (< i (- n 1))
           l
           (λ (a i l) a)))))

This is almost a perfect solution. It’s so achingly pure and cryptic that you will be immediately appointed king of the cult and be able to do your own laughing, and throw other members into pits you have first made them dig, for which they will thank you as they slide down the spikes. Non cult members stand essentially no chance of understanding what it does and sniping about the whole silly closed-form thing: certainly the only way they will be able to learn what it does is by first joining the cult, at which point, as king, you can just throw them straight into the pit.

It’s important you understand this.