ciel/docs/language-extensions.md

Data structures

Generic and nested access to datastructures (access)

From Access, we import access and accesses (plural).

It's always

(access my-structure :elt)

for an alist, a hash-table, a struct, an object… Use accesses for nested access (specially useful with JSON). See also json-pointer.

Hash-table utilities (Alexandria and Serapeum)

We import functions from Alexandria and Serapeum.

To see their full list with their documentation, see alexandria serapeum.

;; alexandria
hash-table-keys
hash-table-values
ensure-gethash

;; serapeum
:dict
:do-hash-table ;; see also trivial-do
:dict*
:dictq  ;; quoted
:href  ;; for nested lookup.
:href-default
:pophash
:swaphash
:hash-fold
:maphash-return
:merge-tables
:flip-hash-table
:set-hash-table
:hash-table-set
:hash-table-predicate
:hash-table-function
:make-hash-table-function
:delete-from-hash-table
:pairhash

Here's how we can create a hash-table with keys and values:

;; create a hash-table:
(dict :a 1 :b 2 :c 3)
;; =>
(dict
 :A 1
 :B 2
 :C 3
)

In default Common Lisp, you would do:

  (let ((ht (make-hash-table :test 'equal)))
    (setf (gethash :a ht) 1)
    (setf (gethash :b ht) 2)
    (setf (gethash :c ht) 3)
    ht)
;; #<HASH-TABLE :TEST EQUAL :COUNT 3 {1006CE5613}>

As seen above, hash-tables are pretty-printed by default.

You can toggle the representation with toggle-pretty-print-hash-table, or by setting

(setf *pretty-print-hash-tables* nil)

in your configuration file.

Sequences utilities (Alexandria, Serapeum)

From Serapeum we import:

:assort
:batches
:filter
:runs
:partition
:partitions
:split-sequence

And from Alexandria:

:iota
:flatten
:shuffle
:random-elt
:length=
:last-elt
:emptyp

From alexandria-2 we import:

:subseq*  (the end argument can be larger than the sequence's length)

and some more.

String manipulation (str)

Available with the str prefix.

It provides functions such as: trim, join, concat, split, repeat, pad, substring, replace-all, emptyp, blankp, alphanump, upcase, upcasep, remove-punctuation, from-file, to-file…

See https://github.com/vindarel/cl-str/ and https://lispcookbook.github.io/cl-cookbook/strings.html

Arrow macros

We provide the Clojure-like arrow macros and "diamond wands" from the arrow-macros library.

CIEL

;; -> inserts the previous value as its first argument:
(-> "  hello macros   "
  str:upcase
  str:words) ; => ("HELLO" "MACROS")

;; ->> inserts it as its second argument:
(->> "  hello macros   "
  str:upcase
  str:words
  (mapcar #'length)) ; => (5 6)


;; use as-> to be flexible on the position of the argument:
(as-> 4 x
  (1+ x)
  (+ x x)) ; => 10

CL

;; In pure CL, just wrap function calls…
(mapcar #'length (str:words (str:upcase "  hello world ")))

;; … or use let* and intermediate variables:
(let* ((var "hello macros")
       (upcased (str:upcase var))
       (words (str:words upcased)))
  words)

And there is more. All the available macros are:

:->
:->>
:some->
:some->>
:as->
:cond->
:cond->>
:-<>
:-<>>
:some-<>
:some-<>>

Functions

Partial application

We import Serapeum's partial and Alexandria's rcurry. They allow partial application of functions.

partial is similar to alexandria:curry but is always inlined.

We import Serapeum's juxt.

You can check Serapeum's function helpers, the non-imported ones are only a serapeum package prefix away in CIEL if you need them. Other noticeable functions: distinct, once, throttle, trampoline, do-nothing…

Memoization

We import defcached from function-cache, a library that provides extensible caching/memoization.

For example:

(defcached (foo :timeout 10) (arg)
  (sleep 3)
  arg)

Run (foo 1), it sleeps 3 seconds and returns 1. Run (foo 1) again and it returns 1 immediately. Its result (for this argument) is cached for 10 seconds.

Bind, more destructuring in let (metabang-bind)

We import the bind macro from metabang-bind (GitHub).

The idiomatic way to declare local variables is let (and let*), the way to declare local functions is flet (and labels). Use them if it is fine for you.

However, if you ever noticed you write convoluted let forms, adding list destructuring, multiple values or slot access into the mix, and if you use a flet and then a let, then read on.

bind integrates more variable binding and list destructuring idioms. It has two goals. Quoting:

1. reduce the number of nesting levels

2. make it easier to understand all of the different forms of destructuring and variable binding by unifying the multiple forms of syntax and reducing special cases.

Bind in CIEL

We import the bind macro. However, the package has more external symbols that we don't import, such as its error type (bind-error) and its extension mechanism.

Note: if you like object destructuring in general, you'll like pattern matching.

Bind is a replacement for let and let*.

You can use bind in lieue of let*.

So, its simpler form is:

(bind (a)
  (do-something a))

a is initialized to nil.

To give it a default value, use a pair, as in (a 1) below:

(bind ((a 1)
       (b 2))
  ...)

Below, we'll use indicators for a, the binding on the left-hand side, so we can have a bind form that starts with three parenthesis. But it's OK, you know how to read them.

(bind (((:values a b c) (a-function)))
  ...)

Bind with multiple-values: (:values ...)

Use (:values ...) in the left-hand side of the binding:

(bind (((:values a b) (truncate 4.5))
       ...

In pure CL, you'd use multiple-value-bind (aka mvb for completion).

Ignore values: the _ placeholder

As in:

(bind (((_ value-1 value-2) (some-function-returning-3-values)))
       ...)

Destructuring lists

Use a list in the left-hand side:

(defun return-list (a b) (list a b))

(bind (((a b) (return-list 3 4)))
  (list a b))
;; => (3 4)

You can use usual lambda parameters for more destructuring:

(bind ((a 2)
       ((b &rest args &key (c 2) &allow-other-keys) '(:a :c 5 :d 10 :e 54))
       ((:values d e) (truncate 4.5)))
  (list a b c d e args))

Bind with plists, arrays, classes, structures, regular expressions, flet and labels

It's all well explained in the documentation!

Conditions

See https://lispcookbook.github.io/cl-cookbook/error_handling.html

From Serapeum, we import ignoring.

An improved version of ignore-errors. The behavior is the same: if an error occurs in the body, the form returns two values, nil and the condition itself.

ignoring forces you to specify the kind of error you want to ignore:

(ignoring parse-error
          ...)

Iteration

See https://lispcookbook.github.io/cl-cookbook/iteration.html for examples, including about the good old loop.

We import macros from trivial-do, that provides dolist-like macro to iterate over more structures:

• dohash: dohash iterates over the elements of an hash table and binds key-var to the key, value-var to the associated value and then evaluates body as a tagbody that can include declarations. Finally the result-form is returned after the iteration completes.

• doplist: doplist iterates over the elements of an plist and binds key-var to the key, value-var to the associated value and then evaluates body as a tagbody that can include declarations. Finally the result-form is returned after the iteration completes.

• doalist: doalist iterates over the elements of an alist and binds key-var to the car of each element, value-var to the cdr of each element and then evaluates body as a tagbody that can include declarations. Finally the result-form is returned after the iteration completes.

• doseq*: doseq* iterates over the elements of an sequence and binds position-var to the index of each element, value-var to each element and then evaluates body as a tagbody that can include declarations. Finally the result-form is returned after the iteration completes.

• doseq: doseq iterates over the elements of an sequence and binds value-var to successive values and then evaluates body as a tagbody that can include declarations. Finally the result-form is returned after the iteration completes.

• dolist*: dolist* iterates over the elements of an list and binds position-var to the index of each element, value-var to each element and then evaluates body as a tagbody that can include declarations. Finally the result-form is returned after the iteration completes.

We ship for so you can try it, but we don't import its symbols. for's over keyword allows to loop across all data structures (lists, hash-tables…).

Lambda shortcut

^ is a synonym macro for lambda.

(^ (x) (+ x 10))
=>
(lambda (x) (+ x 10))

Pythonic triple quotes docstring

We can enable the syntax to use triple quotes for docstrings, and double quotes within them:

CIEL

(ciel:enable-pythonic-string-syntax)

(defun foo ()
  """foo "bar"."""
  t)

CL

;; Normally single quotes must be escaped.
(defun foo ()
   "foo \"bar\"."
   t)

;; use:
(pythonic-string-reader:enable-pythonic-string-syntax)

To disable this syntax, do:

(ciel:disable-pythonic-string-syntax)

We use pythonic-string-reader.

!> This syntax conflicts with libraries that use cl-syntax to use triple quotes too, even only internally. It happens with the Jonathan library.

Packages

defpackage is nice and well, until you notice some shortcomings. That's why we import UIOP's define-package. You'll get:

• less warnings when you remove an exported symbol
• a :reexport option (as well as :use-reexport and :mix-reeport)
• :recycle and :mix options.

It is a drop-in replacement.

Here's uiop:define-package documentation.

Pattern matching

We use Trivia (see its wiki), from which we import match.

You can start typing "match", followed by the object to match against, and the clauses, which are similar to a cond. Here's an example to match a list:

(match '(1 2)
  ((list x y)  ;; <-- pattern
   (print x)
   (print y))
  (_           ;; <-- failover clause
    :else))
;; 1
;; 2

On the above example, (list x y) is the pattern. It binds x to 1 and y to 2. Pay attention that the list pattern is "strict": it has two subpatterns (x and y) and it will thus match against an object of length 2.

If you wanted y to match the rest of the list, use list*:

(match '(1 2 3)
  ((list* x y)
   (print x)
   (print y))
  (_ :else))
;; 1
;; (2 3)

This could also be achieved with the cons pattern:

(match '(1 2 3)
   ((cons x y)
    (print x)
    (print y))
   (_ :else))
;; 1
;; (2 3)

You can of course use _ placeholders:

(match '(1 2 3)
  ((list* x _)
   (print x))
  (_ :else))
;; 1

As we saw with list and cons, Trivia has patterns to match against types (vectors, alists, plists, arrays), including classes and structures.

You can use numeric patterns (=, < and friends, that behave as you expect):

(let ((x 5))
   (match x
     ((< 10)
      :lower)))
;; :LOWER

Then, you can combine them with logic based patterns and guards. For example:

(match x
  ((or (list 1 a)
       (cons a 3))
   a))

guards allow to check the matches against a predicate. For example:

(match (list 2 5)
  ((guard (list x y)     ; subpattern1
          (= 10 (* x y))) ; test-form
   t))

Above we use the list pattern, and we verify a predicate.

Trivia has more tricks in its sleeve. See the special patterns (access and change objects), the ppcre contrib, etc.

You migth also be interested in exhaustiveness type checking explained just below.

Regular expressions

Use ppcre.

See https://common-lisp-libraries.readthedocs.io/cl-ppcre and https://lispcookbook.github.io/cl-cookbook/regexp.html

Type declarations

Use the --> macro to gradually add type declarations.

Alternatively, use defun*, defgeneric*, defmethod*, defparameter* and defvar* to add type declarations directly in the lambda list.

These notations are not strictly equivalent though.

--> comes from Serapeum. It is a shortcut for (declaim (ftype my-function (… input types …) … return type …))

CIEL

(--> mod-fixnum+ (fixnum fixnum) fixnum)
(defun mod-fixnum+ (x y) ...)

;; --> comes straight from serapeum:->

CL

(declaim (ftype (function (fixnum fixnum) fixnum) mod-fixnum+))
(defun mod-fixnum+ (x y) ...)

Now defun* and friends allow to add type declarations directly in the lambda list. They add the (declaim (ftype as above, and additionnaly declare types inside the function body:

CIEL

(defun* foo ((a integer))
  (:returns integer)
  (* 10 a))

CL

;; In pure CL, type the functions at its boundaries with ftype.
;; It is a bit verbose, but it has the advantage, being not tied to defun,
;; that we can easily refine types during development.
(declaim (ftype (function (integer) integer)             foo))
;;                        ^^ inputs ^^ output [optional] ^^ function name

;; defstar adds the internal "declare" and "the…".
;; "the" is a promise made to the compiler, that will optimize things out.
(defun foo (a)
  (declare (type integer a))
  (the integer (* 10 a)))

A type declaration for a parameter:

CIEL

(defparameter* (*file-position* (integer 0)) 0)

CL

;; Normal defparameter:
(defparameter *file-position* 0)

;; Assigning a bad value works:
(setf *file-position* "8")
;; "8"

;; We add a type declaration:
(declaim (type (integer 0) *file-position*))

;; and now:
(setf *file-position* "8")
;;
;; Value of #1="8" in (THE INTEGER "8") is #1#, not a INTEGER.
;;   [Condition of type SIMPLE-TYPE-ERROR]
;;
;; we get a type error.

We can use any type specifier:

(deftype natural () '(real 0))
(defun* sum  ((a natural) (b natural))
  (:returns natural)
  (+ a b))

Now, we get type errors at compile time:

(foo "3")
;; =>
The value
  "3"
is not of type
  INTEGER
when binding A
   [Condition of type TYPE-ERROR]

Restarts: […]

Backtrace:
  0: (FOO "3") [external]
  1: (SB-INT:SIMPLE-EVAL-IN-LEXENV (FOO "a") #<NULL-LEXENV>)
  2: (EVAL (FOO "3"))

and we get compile-time warnings on type mismatches (but to be honest on simple cases like this SBCL is already quite good):

(defun* bad-foo ((a integer))
  (:returns integer)
  (format t "~a" (* 10 a)))
;
; in: DEFUN* BAD-FOO
;     (THE INTEGER (FORMAT T "~a" (* 10 CIEL::A)))
;
; caught WARNING:
;   Constant NIL conflicts with its asserted type INTEGER.
;   See also:
;     The SBCL Manual, Node "Handling of Types"
;
; compilation unit finished
;   caught 1 WARNING condition
BAD-FOO

We could add extra protection and a check-type, evaluated at runtime. Defstar can add them automatically if defstar:*check-argument-types-explicitly?* is non-nil.

In theory, such declarations don't guarantee that Lisp will do type checking but in practice the implementations, and in particular SBCL, perform type checking.

We use the defstar library. Its README has many more examples and even more features (adding assertions, :pre and :post clauses).

Note: we are not talking thorough ML-like type checking here (maybe the Coalton library will bring it to Common Lisp). But in practice, the compiler warnings and errors are helpful during development, "good enough", and SBCL keeps improving in that regard.

Note: there is no "undeclaim" form :] You can unintern a symbol and re-define it.

See also:

• declarations in the Common Lisp Hyper Spec.
• https://lispcookbook.github.io/cl-cookbook/type.html
• the article Static type checking in SBCL, by Martin Cracauer
• the article Typed List, a Primer - let's explore Lisp's fine-grained type hierarchy! with a shallow comparison to Haskell.

Type checking: exhaustiveness type checking

Write a "case" and get a compile-time warning if you don't cover all cases.

From Serapeum, we import:

:etypecase-of
:ctypecase-of
:typecase-of
:case-of
:ccase-of

etypecase-of allows to do compile-time exhaustiveness type checking.

Example with enums

We may call a type defined using member an enumeration. Take an enumeration like this:

(deftype switch-state ()
  '(member :on :off :stuck :broken))

Now we can use ecase-of to take all the states of the switch into account.

(defun flick (switch)
  (ecase-of switch-state (state switch)
    (:on (switch-off switch))
    (:off (switch-on switch))))
=> Warning

(defun flick (switch)
  (ecase-of switch-state (state switch)
    (:on (switch-off switch))
    (:off (switch-on switch))
    ((:stuck :broken) (error "Sorry, can't flick ~a" switch))))
=> No warning

Example with union types

(defun negative-integer? (n)
  (etypecase-of t n
    ((not integer) nil)
    ((integer * -1) t)
    ((integer 1 *) nil)))
=> Warning

(defun negative-integer? (n)
  (etypecase-of t n
    ((not integer) nil)
    ((integer * -1) t)
    ((integer 1 *) nil)
    ((integer 0) nil)))
=> No warning

See Serapeum's reference.

trivial-types: more type definitions

From trivial-types, we import

• association-list-p
• type-expand
• string-designator
• property-list
• tuple
• association-list
• character-designator
• property-list-p
• file-associated-stream-p
• type-specifier-p
• list-designator
• package-designator
• tuplep
• non-nil
• file-associated-stream
• stream-designator
• function-designator
• file-position-designator
• pathname-designator