Lisp hacks

Some small programs in Lisp

• Conduit packages
• Collecting lists forwards
• Memoizing functions
• Applicative iteration
• HTML generation
• Abstract classes
• Singleton classes
• Maintaining dynamic state
• Read-time packages
• Hierachical packages
• Defaulting optional arguments for generic functions

Conduits

The `conduits' system tries to generalise the CL package system to allow for more flexible construction of packages. In particular it lets you define packages which are `like' other packages in the sense that they share some or many symbols with them, but they extend them in some way, perhaps by having some symbols from another package. These packages are `conduits' for the packages they extend.

Example of conduits 1: variant Lisps

As an example, consider someone who wanted to have the basic arithmetic operators be generic functions so they could do the usual stupid thing of defining + on strings or something. They could start by defining a package which had its own versions of the operators:

(defpackage :generic-maths-implementation
  (:use :cl)
  (:shadow #:+ #:- #:* #:/)
  (:export #:+ #:- #:* #:/))

This is quite easy. But to use the generic-maths-implementation package, you have to do something like this:

(defpackage :gm-user
  (:use :cl)
  (:shadowing-import-from :generic-maths-implementation
                          #:+ #:- #:* #:/))

You can't do the apparently obvious thing:

(defpackage :gm-user
  (:use :cl)
  (:use :generic-maths-implementation))

because you get symbol clashes.

What you want to be able to construct is a package which is `just like' the cl package, but which has its own versions of the symbols of interest. It's possible to do this by understanding the distinction between using a package - which simply says to look for its external symbols when looking up a symbol -- and importing symbols from it - which makes them directly present in the current package, from which they can then be exported in turn.

So you can construct a package which imports all the symbols except the interesting ones from cl, and the interesting ones from generic-maths-implemementation, and then rexports all these symbols. This package then looks just like the cl package, except it has different versions of the interesting symbols.

This could be done `by hand' -- by looping over the exported symbols of the packages you are extending -- but the conduits system provides a version of defpackage which supports this directly:

(defpackage :common-lisp/generic-maths
  (:nicknames :cl/gm)
  (:use)
  (:extends/excluding :cl #:+ #:- #:* #:/)
  (:extends :generic-maths-implementation))

(defpackage :gm-user
  (:use :cl/gm))

The cl/gm package extends cl, except for a certain set of symbols, and extends generic-maths-implementation.

Example of conduits 2: sub-common-lisps

Another thing that the conduit system can do is to allow you extend a package including only some symbols. This allows you to define subset dialects of the language in a fairly flexible way:

(defpackage :tiny-lisp
  (:use)
  (:extends/including :cl #:lambda #:t #:nil #:if #:eq))

Other things the package can do

Various features have accreted to conduits since the original implementation. These are much less well tested than the main functionality, which has now been used for a fairly large system. Most of the reason for these features being here is that I want to have just one defpackage macro. The documentation for these things is also even more sketchy than the conduit documentation proper.

Package cloning

You can `clone' a package by creating a new package which has the same symbols, package use lists, imports, exports and so on. This is useful for taking a `snapshot' of a package - things newly interned in the cloned package will not perturb the original package. Cloning is not compatible with extending.

(defpackage :foo
  (:use :cl)
  (:export #:grun #:grob))

(defpackage :foo-clone
  (:use)
  (:clones :foo))

;;; Now we could do things in FOO-CLONE such as intern new symbols &c
;;; which would not change FOO.  But FOO-CLONE:GRUN is EQ to FOO:GRUN

Per-package aliases

It is (arguably) nice to be able to have `shorthands' for package names which are defined per-package: `if I'm in the ORG.TFEB.CLC-USER package, I want CL to refer to ORG.TFEB.CLC, so CL:DEFPACKAGE will mean ORG.TFEB.CLC:DEFPACKAGE'. The conduit system can now do this, in conjunction with the hierarchical packages hack.

(defpackage :com.cley.cl-user
  (:nicknames :com.cley.user)
  (:use :org.tfeb.clc)
  (:aliases 
   ;; This means: if *PACKAGE* is COM.CLEY.CL-USER then CL:x means
   ;; ORG.TFEB.CLC:x.  Note the order is (alias real-name).
   (:cl :org.tfeb.clc)))

Because this trick depends on the hierarchical packages it will not work in all Lisps. Further, you need to have compiled and loaded the hierarchical packages code before compiling conduits to get this support, and you need to have the hierarchical packages fasl file in the same directory as the conduits fasl file at load time, if it is not already loaded, so it can be automatically dragged in.

Notes on conduits

it's quite easy to define a `static' conduit system, which allows you to define conduit packages as above. This is adequate for many purposes. However since Lisp is a dynamic language, it is desirable that the conduit packages have dynamic behavior: if a package which they extend changes the symbols it exports, they too should change the symbols they export. The current implementation tries to do this by defining its own versions of many of the package-manipulating functions. In particular the cl/conduits package -- nicknamed clc -- as well as defining a version of defpackage, defines versions of export, unexport, delete-package and rename-package which are `conduits-aware'. The clc package is itself a conduit, of course.

Because of obscure implementation bugs in the handling of nested eval-when forms, the conduits package can not be compiled as one file in (at least) CLISP and old versions of CMUCL. You need to put the definition of the clc package in a separate file and compile it only after the main drag is loaded. Allegro CL, Genera, and recent versions of CMUCL can compile it successfully. I'd appreciate feedback on any other implementations.

There is no documented function-level interface to any of the conduits functiionality. There should be.

The conduits system has been used in a fairly large application, and many smaller ones, and I'm fairly sure that the basic functionality works pretty well. The dynamic behaviour is less well-tested, and the per package alias code has had minimal testing. Please report any bugs!

Downloading the conduits package, contact information

Collecting lists

It's often the case that you are performing some search and accumulating results into a list. If this doesn't fit into one of the conventional mapping paradigms, you end up doing something like this:

(defun find-gribbles (in)
  (let ((results '()))
    (labels ((findit (thing)
	       (typecase thing
		 (gribble
		  (push thing results))
		 (gribble-bag
		  (mapc #'findit (gribbles-of thing)))
		 (t nil)))))
    (findit in)
    ;; we care about the ordering.
    (nreverse gribbles)))

The natural way of accumulating objects into a list in Lisp is backwards, since access to the head of the list is fast, but access to the end takes time linear in the length of the list. If the ordering of the results matters, you then need to reverse the list of results.

It's quite possible to accumulate results into a list forwards, by keeping a `tail pointer' -- a pointer to the last cons of the list, which you then destructively modify to add elements to the end of the list. This can be done `by hand' but it can also be wrapped up into a macro, which makes life much easier.

The code here defines two macros:

collecting

Allows collection of a single list, which is returned from the form. Within the body of a collecting form, the collect macro collects an element into the list

with-collectors

Allows collection into multiple lists. The first argument of with-collectors is a list of symbols, which are defined as macros within the body to collect into as many lists as there are symbols. The collected lists are returned as multiple values from the form.

Examples:

(defun find-gribbles/collecting (in)
  (collecting
   (labels ((findit (thing)
	      (typecase thing
		 (gribble
		  (collect thing))
		 (gribble-bag
		  (mapc #'findit (gribbles-of thing)))
		 (t nil)))))
   (findit in)))

(defun find-gribbles-and-wibbles/collecting (in)
  ;; 2 vals
  (with-collectors (cw cg)
    (labels ((findit (thing)
	       (typecase thing
		 (gribble
		  (cg thing))
		 (wibble
		  (cw thing))
		 (bag
		  (mapc #'findit (gribbles-of thing)))
		 (t nil)))))
    (findit in)))

Although the collecting operators are locally-defined macros, they can be `wrapped' in functions and then passed into non-lexically-visible contexts, as in this slightly silly example:

(defun collect-gribbles (thing collector)
  (typecase thing
    (gribble
     (funcall collector thing)
     (gribble-bag
      (mapc #'findit (gribbles-of thing)))
     (t nil)))))

(defun find-gribbles/silly-collecting (in)
  (collecting
   (collect-gribbles in #'(lambda (e)
			    (collect e)))))

Finally, with slight constraints you can actually return a function which will collect items into the end of a list. The constraint is that you have to have collected at least one thing into the list already -- otherwise you get NIL as the list, and the function collects into somethign to which you don't have access!

(defun make-collector ()
  ;; returns a cons whose car is the collector function, and whose cdr
  ;; is the list it collects into.
  (collecting
   (collect #'(lambda (e)
		(collect e)))))

Downloading the file, contact information

Memoizing functions

The technique of `memoizing' a function is a well-known way of enormously improving the performance of certain kinds of recursive (pure) functions without going to the effort of rewriting them in a more efficient way. For example, consider this (very naive) version of a function to compute the nth Fibonacci number:

(defun fib (n)
  (if (<= n 1)
      1
      (+ (fib (- n 1))
	 (fib (- n 2)))))

This takes time exponential in n. But it only does this because it repeatedly computes the same result in recursive calls -- the tree of recursion is actually a DAG with a great deal of sharing, but the function as written doesn't know this.

Memoizing the function causes it to remember the value it computes for a given argument, and, before computing a new answer, to look in this table and return the result stored there if it finds one. A memoized version of fib is close-to linear in n, even though the source looks almost identical

Memoization is well-described in Paradigms of Artificial Indelligence Programming, by Peter Norvig, pp269-275. This code simply generalises that given there. The interesting thing that it does is provide a macro called memoized-labels which allows local functions to be memoized.

The relative ease with which memoization can be integrated into the language is a testement to Lisp's syntactic and semantic flexibility.

Functions and macros

All of these are exported from the memoize package.

memoize-function (fn-name &key (key #'first) (test #'eql))

Memoize fn-name, a symbol, causing its results to be stashed. key is a function which is given the arglist of fn-name, and should return a key to hash on for memoizing. test is a function which the test for the hashtable. See Norvig P269-275.
Note this function may not work on self-recursive functions because the compiler can optimize away self-calls in various ways. def-memoized-function should work for those cases as it is careful to ensure the function can not be inlined like this.

unmemoize-function (fn-name)

Remove memoization for fn-name.

unmemoize-functions ()

Unmemoize all functions

clear-memoized-function (fn-name)

Clear any memoized results for fn-name (empty the hash table).

clear-memoized-functions ()

Clear results for all memoized functions.

function-memoized-p (fn-name)

Return true if fn-name is memoized.

def-memoized-function (fnspec args &body bod)

Define a memoized function. fnspec is either the name of the function, or a list suitable as an arglist for memoize-function. args and bod are passed off to defun.
This will declare the function notinline, which may be necessary to prevent good compilers optimizing away self calls and stuff like that.

memoized-labels ((&rest labdefs) &body bod)

A version of labels that memoizes the local functions. See memoize-function and def-memoized-function. If code that uses this is compiled (either by compile or compile-file, then the table of memoized results will be unique, if interpreted then a new table may be generated for each use. The function `names' are generalised in the same way as for def-memoized-function.

examples

A memoized version of fib:

(def-memoized-function fib (n)
  (if (<= n 1)
      1
      (+ (fib (- n 1))
	 (fib (- n 2)))))

A rather silly version of the function which uses memoized-labels:

(defun fib (n)
  (memoized-labels ((fi (m)
		      (if (<= m 1)
			  1
			  (+ (fi (- m 1))
			     (fi (- m 2))))))
    (fi n)))

Notes

Memoized functions should be `pure' -- they should depend only on their arguments. If they depend on other information, then the stash of results can be simply wrong. This is particularly insidious for memoized-labels because the local function can't rely on lexically visible bindings defined outside itself. (Actually there are some devious ways around this in some cases using the key function to generate a unique hash key using information other than the argument values.)

The implementation of memoization here isn't thread-safe yet, since calling a memoized function can modify the table of memoized results, which needs to be synchronized.

This code hasn't been that well tested, and it does some slightly devious things with the compiler, so it may be buggy. Please let me know if so!

Downloading the file, contact information

Applicative iteration

In a lisp which does tail-call elimination, the following code is essentially a loop:

(defun repeat (n)
  (labels ((loupe (i)
	     (if (= i n)
		 i
		 (loupe (1+ i)))))
    (loupe 1)))

Scheme, which requires tail-call elimination, provides an optional special syntax for this, `named let'. In Scheme you can say:

(define (repeat n)
  (let loupe ((i 1))
    (if (= i n)
	i
	(loupe (1+ i)))))

This syntax is clearly only trivially different than that using labels in CL, but it is easier to use. The iterate macro provides exactly the syntax that named let privides for CL:

(defun repeat (n)
  (iterate loupe ((i 1))
    (if (= i n)
        i
        (loupe (1+ i))))))

Of course, CL does not require tail-call elimination, so the iterate macro may not work. I used to really like it however, so I went as far as writing a special, hacky, version of it which will compile a loop even in an implementation without tail-call elimination. However, it will only compile a loop -- non-tail `calls' compile to jumps just as tail `calls' do, and so will do something cvompletely unexpected. To reduce this problem, the special loop-only code is only generated if the name has `loop' in it (in either case).

In an implementation that does tail-call elimination iterate is quite powerful. It is possible to combine iteration and recursion in a fairly flexible way. In fact, it's too flexible -- it's basically a `goto with arguments', and it's possible to write very opaque code using it. For this reason and others, I don't use it any more, but it is interesting as an example of writing a new sort of control structure in Lisp.

Downloading the file, contact information

HTML generation

This documentation is incomplete

Lisp's syntax can be used to express the structure of SGML/XML-based documents very concisely:

<doc>
  <heading>Foo</heading>
  <para>this is a paragraph</para>
</doc>

Becomes

(:doc
 (:heading "Foo")
 (:para "This is a paragraph"))
  

Because Lisp is also a programming language it is possible to mingle Lisp code and SGML/XML markup in Lisp syntax to programmatically generate documents.

In order to do this `right' for XML you need to deal with issues of Unicode, case, namespaces and so on. But almost all documents don't need to worry about this complexity. This code provides a simple way to mix HTML markup with Lisp code. It does not enforce anything except the matching of tags, so it knows nothing about document types or anything like that. In fact the only thing you need to tell it is which elements have empty content models, so it generates <br> rather than <br></br>.

Here is an example (included with the code) of Lisp source, and following it the generated HTML (edited slightly so it can be spliced into this document).

(defun count-numbers (n w &optional (s *standard-output*))
  (with-html-output (s)
    (:html
     (:head (:title 
	     (fmt "Numbers from zero below ~R" n)))
     (:body
      (:h1  (fmt "Numbers from zero below ~R" n))
      ;; Forms beginning with non-keyword symbols are code to be evaluated.
      (lfd)
      (:p "Table border width "
	  (princ w s))
      ;; isolated keywords are empty tags.
      :br
      (lfd)
      ;; empty tags with attributes need this slightly crufty syntax, 
      ;; and also need to be defined as empty. 
      ((:hr :noshade))
      (:center
       ;; the values of atttributes are evaluated (in fact the whole 
       ;; attribute list is, but attribute names asre keywords).
       ((:table :border w
		:width "90%")
	(:tbody				;html 4, bah.
	 (:tr
	  ((:th :align :left) "English")
	  ((:th :align :right) "Arabic")
	  ((:th :align :right) "Roman"))
	 ;; you can leap into Lisp...
	 (dotimes (i n)
	   (let ((c (if (evenp i) "blue" "white")))
	     ;; ... and then back into HTML: the local HTML macro is shorthand
	     ;; for WITH-HTML-OUTPUT to the same stream.
	     (htm
	      ((:tr :bgcolor c)
	       ((:td :align :left)
		(fmt "~R" i))
	       ((:td :align :right)
		(fmt "~D" i))
	       ((:td :align :right)
		(if (zerop i)
		    (fmt "")
		    (fmt "~:@R" i))))
	      (lfd)))))))
       ((:hr :noshade))))))

Calling (count-numbers 10 0) yields HTML which is rendered as follows:

Numbers from zero below ten

Table border width 0

---

English	Arabic	Roman
zero	0
one	1	I
two	2	II
three	3	III
four	4	IIII
five	5	V
six	6	VI
seven	7	VII
eight	8	VIII
nine	9	VIIII

---

Downloading the file, contact information

Abstract classes

A common complaint about CLOS is that it is somehow `not properly object-oriented'. As usual with such complaints it's often hard to find out what is really meant. However it is clear that one of the things people don't like is that CLOS is too liberal. Politically-correct object-oriented languages differ in a number of ways from CLOS, among them:

• They have single-dispatch;
• They either have single inheritance or make multiple inheritance very hard to use;
• They specify all the methods of a class within the class definition;
• They mingle in various kinds of mandatory data hiding with the object system;
• They allow classes to be defined with various characteristics, such as being `abstract' -- instances of the class may not be made, though instances of subclasses may be, or `final' -- the class may not be subclassed.

Most of these aspects of `modern' object-oriented languages are simply beneath contempt. They seem to be restrictions placed on the language out of some misplaced desire for an efficient implementation without doing very much work, restrictions because it's impossible to trust programmers, especially with classes they didn't write, restrictions because it's just nice to be restricted and not to have to think too hard, or a gratuitous overloading of several ideas into one construct.

CLOS is quite different. CLOS assumes some intelligence and maturity in its users, and provides something much more like a toolkit for constructing object-oriented programs that actually fit the domain rather than having to deform the domain into some ludicrously restrictive framework.

Of course, this doesn't help, because everyone now `knows' that an object system has to have all these properties, so CLOS is just inherently no good. Well, all is not lost: CLOS is a toolkit for constructing object-oriented programs that fit the domain, and one of the domains it can produce programs to fit is the domain of `modern' object-oriented languages. It's fairly obvious for instance that by a little judicous use of the package system and macros you could easily define a single-dispatch, all-methods-are-in-the-class language based on CLOS. Data hiding can also be done by devious use of gensymed slot names, and so on. Almost everything is possible

Finally, you can use the MOP to change CLOS in all-sorts of interesting ways, most of which are simply beyond what `modern' systems can imagine. Here is a small example of this. By defining suitable metaclasses, it is almost trivial to add `abstract' and `final' classes to CLOS.

Abstract class

A class which can be subclassed, but not directly implemented. Defined using either the metaclass abstract-class or the macro define-abstract-class.

(define-abstract-class container ()
  ;; May not be instantiated
  ((children :initarg :children
	     :accessor children
	     :initform '())))

(define-abstract-class contained ()
  ;; May not be instantiated
  ((parent :initarg :parent
	     :accessor parent
	     :initform nil)))

(defclass exchange-object (container contained)
  ;; may be instantiated
  ())

Final class

A class which may not be further subclassed. Defined either using the metaclass final-class or the macro define-final-class.

(defclass exchange-object (container contained)
  ;; may be instantiated and subclassed
  ())

(define-final-class mux (exchange-object)
  ;; may be instantiated but not subclassed
  ((id :initarg :id
       :reader id)))

(defclass shelf (exchange-object)
  ;; may be instantiated but not subclassed
  ()
  (:metaclass final-class))

Both abstract and final classes are implemented very simply by subclassing standard-class and writing suitable methods on make-instance and validate-superclass. Although the MOP is not completely standardised, the implementation here works on three commercial, and one non-commercial, CLs with only minor conditionalisation.

Downloading the file, contact information

Singleton classes

Along the lines of abstract classes above, it's quite easy using the MOP to create classes which only have a single instance. For these classes, make-instance always returns the same object. It's somewhat questionable whether this is a good thing to do, since it violates a lot of expectations - expecting (eq (make-instance 'foo) (make-instance 'foo)) to be true is almost as strange as expecting (eq (cons 1 1) (cons 1 1)) to be true. However, it's easy enough to do, and it gives another example of using the MOP to alter CLOS.

Example:

(in-package :cl-user)

(use-package :org.tfeb.hax.singleton-classes)

(defclass foo ()
  ((x :initform (progn
		  (format *debug-io* "~&Initialising x~%")
		  1)))
  (:metaclass singleton-class))

Downloading the file, contact information

Maintaining dynamic state

Special variables are very useful in CL for establishing dynamic state which needs to be accessible throughout some process. They let you do things which are quite hard to do in many other languages. However, if they're used in an uncontrolled way they lead to an awful proliferation of global variables and obscure assignments to free variables which can lead to wonderfully bug-prone code.

One way to get around this is to not declare things globally special but rather to declare specific bindings special. Since there is no global special declaration, the bindings will only be visible if you explicitly ask to see them. An example is something like this:

(defun foo (x)
  (let ((%state% '()))
    (declare (special %state%))
    (bar x)
    (dolist (s %state&)
      (deal-with s))))

(defun bar (x)
  ;; %STATE% is not visible here
  ...
  (crun y)
  ...)

(defun crun (y)
  (declare (special %state%))
  (push y %state%))

In this code, %state% is the state variable, which is bound in some outer function, and then can be accessed by callees who declare it special, without cluttering up the global namespace or allowing uncontrolled access. A good use of this technique is in a web server which is computing a web page. If an error happens during generation then the error handler needs to know if the HTTP header has already been generated. If it has not, then it can generate a 5xx response to the client indicating that something bad happened, if it has, then it can't do that, but needs to either say noting or try an spit out some indication that something went wrong during the generation. A sketch of this follows.

(defun generate-page (stream function)
  ;; STREAM is the stream to generate on, FUNCTION does the work,
  ;; including generating the header.
  (let ((%sent-header% nil))
    (declare (special %sent-header%))
    (handler-case (funcall function stream)
      (error (e)
	(handler-case			;because something can die here, too
	    (if %sent-header%
		(error-page-body stream e)
		(error-page-header-and-body stream e))
	  (error ()
	    nil))))))

(defun generate-http-header (stream ...)
  ...
  (locally
      (declare (special %sent-header%))
    (setf %sent-header% t)))

This kind of thing is fairly hard in, say C, because of the requirement that the binding of %sent-header% be per-thread. (See here for some notes on why special variables are hard to implement if you don't have them already.)

However, this still isn't very pretty code. To make it easier to follow what is going on, I have a special macro which defines a `dynamic state' - a set of variables - and macros for establishing and accessing this state. The previous example rewritten to use this macro looks like this:

(define-dynamic-state (with-http-state with-http-state-access)
    %sent-header%)

(defun generate-page (stream function)
  ;; STREAM is the stream to generate on, FUNCTION does the work,
  ;; including generating the header.
  (with-http-state ((%sent-header% nil))
    (handler-case 
	(funcall function stream)
      (error (e)
	(handler-case			;because something can die here, too
	    (if %sent-header%
		(error-page-body stream e)
		(error-page-header-and-body stream e))
	  (error ()
	    nil))))))

(defun generate-http-header (stream ...)
  ...
  (with-http-state-access (%sent-header%)
    (setf %sent-header% t)))

The idea is that you can define a set of variables which make up the dynamic state of part of a program, and then define a form for establishing this state and one for accessing it. Both of these forms check that the variables they are given belong to the set they know about, thus making things a little easier to debug. Mostly however the benefit is in the readability of the code - with these named dynamic-state forms it's much easier to tell just what it is is being done, instead of having to work it out from some obscure declarations.

The convention of %variable% for these non-global specials is just something I made up.

The code defines a single macro:

define-dynamic-state ((binder accessor) &rest varnames)

Defines two macros:

binder ((&rest bindings) &body forms)

A let-like macro which binds some or all of the variables from varnames and declares those bindings special. It checks that only variables from varnames are bound.

accessor (&rest vars) &body forms)

A form for accessing the bindings established by binder. It has syntax like let with no initialization allowed, and essentially expands into (locally (declare (special ...)) ...) to allow access to the special bindings. It checks that vars come from varnames.

An example of the use of this macro is above.

Downloading the file, contact information

Read-time packages

It's possible to specify in Lisp code that a symbol comes from a specific package using the standard package-prefix notation: foo:bar or foo::bar. Sometimes it's useful to specify that an entire form be read in the context of a given package. It would be nice to be able to say something like foo:(x y z) but this isn't possible in CL (although something like it was possible on the LispMs).

In the special case of top-level forms this can be achieved by using in-package and relying on the compile-time behaviour of this:

(in-package :foo)

(this-form-is-read-in-package-foo)

(in-package :cl-user)

However this isn't very nice. Firstly you have to remember the old package - which could be fixed by defining push-package and pop-package macros in a fairly obvious way. Secondly and more importantly this completely fails to work other than at top-level:

(in-package :cl-user)

(defun wont-work (x)
  (in-package :foo)
  (this-form-is-not-read-in-package-foo *crun*)
  (in-package :cl-user)
  x)

In order to get this to work you have to be able to switch packages at read time. This code lets you do that, using a special bit of reader syntax:

(in-package :cl-user)

(defun will-work (x)
  #@foo (this-form-is-read-in-package-foo *crun*)
  ;; but this is read in CL-USER.
  x)

does what the previous example probably wanted to do.

#@package-name form

Reads form in the package named by package-name. package-name is read by read and is expected to be read as an unqualified symbol (with no package prefix). Symbols with package prefixes or strings are accepted with a warning. An error is signalled at read-time for any other type, or if no package named package-name exists.

The terminology used above is bad: symbols don't have package prefixes. What I mean is that #@cl-user::foo (car x) will cause a warning, whereas #@foo (car x) is OK. The whole mechanism is pretty hacky as it uses a secret package to read the package name, so in fact #@read-package-package::foo (car x) will work.

Further, this isn't really like what package prefixes do on symbols at all. What it does is, at read time, make the required package be current, then read the form in that context. So there is no distinction between internal and external symbols. Really, the whole thing is a hack to allow various ugly-but-necessary hacks to be slightly less ugly.

Downloading the file, contact information

Hierachical packages

Languages like Java have a hierarchically structured namespace: you can say com.cley.grovelling.explode(x);, which means something like `call the explode method of the class grovelling in the namespace com.cley'. If you ignore the standard message-passing orthodoxy there is still something useful here. Not only can the namespace be defined hierarchically in Java, but depending on `where you are' in the namespace you can refer to other bits of the namespace with short names. In CL, it's easy enough to define a hierarchically structured namespace - just pick a convention for package names - but without the ability to refer to packages by short names, this isn't terribly useful. For instance if there were two packages which implemented a given chunk of functionality, then you need to specify the `full pathname' of each one, rather than being able to specify a relative name, which makes code somewhat more brittle.

Franz have implemented a hierarchical package naming system in Allegro CL 6.0. This uses `.' characters to separate name components, and supports relative package names. Documentation for this is available here (this is for 6.1, which I think has not changed anything from 6.0). Sample code to implement their system is provided on that page.

I have taken this sample code and slightly cleaned it up, and made it work in a couple of other CL systems. Note that this code is inherently not completely portable since it needs to intervene in fairly low-level parts of the package system. The code as given here will work in those systems which both allow redefinition of cl:find-package, and actually go through this function in the reader. Not all implementations do so, and it's not a bug that they don't.

For the (intended) behaviour of this code, please see the Franz documentation.

This code works in CMUCL and LispWorks. However I believe that CMUCL 18d (not released at the time I'm writing this) will have its own hierarchical packages support, compatible, I hope, with this code.

Even without this code, I think that hierarchically structuring package names in a way like Java does is a sensible thing to do.

The hierarchical package system now exports a hashtable which maps from packages to an alist of (alias . real-name): if the current package (the value of *package* is found in the table, then find-package will interpret alias as if it meant real-name. This substitution happens once, before any other lookups are done. This table is used by the conduits system to support per-package aliases.

Downloading the file, contact information

Defaulting optional arguments for generic functions

I've always found it a pain to provide good defaults for optional and keyword arguments to generic functions. It's horrible for each method to have to do this, but around methods don't really do what you want. The problem is that the most specific around method is outermost, so if you write a method to default arguments you have to cope with another method wrapping itself around you.

The solution to this is `wrapping' methods. Wrapping methods are like around methods but happen outside them (so: before and after), and the least specific wrapping method is outermost. Thus, if you write a completely unspecific wrapping method, it will always wrap around any other methods. This makes it a very suitable place to do optional argument defaulting.

Wrapping methods are also useful for running things like `hooks' - things that should run before or after a GF is called. This is what they were originally called in fact.

CLOS is powerful enough to define wrapping methods completely portably (no use of the MOP, in other words) and in a way which is completely compatible with standard method comnbination. Tests on a couple of implementations show no performance difference between this and standard method combination.

Downloading the file, contact information