ciel/README.org

* CIEL Is an Extended Lisp                                         :noexport:

STATUS: the API WILL change, but it is usable.


* What is this ?                                                   :noexport:

  CIEL is a collection of useful libraries.

  It's Common Lisp, batteries included.

  Questions, doubts? See the [[file:FAQ.org][FAQ]].


* TODOs                                                            :noexport:

- settle on libraries that help newcomers
- automate the documentation
- distribute (Quicklisp, Qlot, Quicklisp distribution, Ultralisp,
  Ultralisp distribution (upcoming)…)
- ship a core image and a binary
- optionnal: create a tool that, given a CIEL code base, explains what
  packages to import in order to switch to "plain CL".

How to procede ?

This is an experiment. I'd be happy to give push rights to more
maintainers. We will send pull requests, discuss, and in case we don't
find a consensus for what should be on by default, we can create other
packages.

Rules

- don't install libraries that need a Slime helper to work in the REPL (cl-annot).
- reader syntax changes may not be enabled by default.

* Table of contents :TOC:
- [[#install][Install]]
  - [[#with-quicklisp][With Quicklisp]]
  - [[#with-a-core-image][With a core image]]
  - [[#with-a-binary-use-ciels-custom-repl][With a binary. Use CIEL's custom REPL.]]
- [[#libraries][Libraries]]
  - [[#data-structures][Data structures]]
  - [[#data-formats][Data formats]]
  - [[#date-and-time][Date and time]]
  - [[#databases][Databases]]
  - [[#iteration][Iteration]]
  - [[#pattern-matching][Pattern matching]]
  - [[#numbers][Numbers]]
  - [[#regular-expressions][Regular expressions]]
  - [[#threads-monitoring-scheduling][Threads, monitoring, scheduling]]
  - [[#http-and-uri-handling][HTTP and URI handling]]
  - [[#web][Web]]
  - [[#conditions][Conditions]]
  - [[#types-type-checking-exhaustiveness-type-checking][Types, type checking, exhaustiveness type checking]]
  - [[#syntax-extensions][Syntax extensions]]
  - [[#development][Development]]
  - [[#generic-cl][generic-cl]]
- [[#final-words][Final words]]

* Install

** With Quicklisp

You need a Lisp implementation and Quicklisp installed.

CIEL is not yet on Quicklisp (but it is on [[https://ultralisp.org][Ultralisp]]), so clone this
repository and load the .asd (with =load= or =C-c C-k= in
Slime).

: git clone https://github.com/ciel-lang/CIEL ~/quicklisp/local-projects/CIEL

Then, quickload it:

#+BEGIN_SRC lisp
(ql:quickload "ciel")
#+end_src

and enter the =ciel-user= package, instead of the default
=common-lisp-user= (or =cl-user=):

#+BEGIN_SRC lisp
(in-package :ciel-user)
#+end_src

** With a core image

You need a Lisp implementation, but you don't need Quicklisp.

Build a /core image/ for your lisp with all CIEL's dependencies:

: sbcl --load build-image.lisp

and use it:

: sbcl --core ciel --eval '(in-package :ciel-user)'

TODO: we will distribute ready-to-use core images.

** With a binary. Use CIEL's custom REPL.

   You don't need anything, just download the CIEL executable and run
   its REPL.

   TODO: build it on CI for different platforms.

   To build it, clone this repository and run =make build=.

   Start it with =./ciel-repl=.

   You are dropped into a custom Lisp REPL, freely based on [[https://github.com/hellerve/sbcli][sbcli]].

   This REPL is more user friendly than the default SBCL one:

- it has readline capabilities, meaning that the arrow keys work (!)
  and there is a persistent history, like in any shell.
- it has *multiline input* and reset.
- it has *TAB completion*.
- it handles errors gracefully: you are not dropped into the debugger
  and its sub-REPL, you simply see the error message.

  It also defines short helper commands:

#+begin_src txt
  :help => Prints this general help message
  :doc => Prints the available documentation for this symbol
  :? => Gets help on a symbol <sym>: :? str
  :w => Writes the current session to a file <filename>
  :d => Dumps the disassembly of a symbol <sym>
  :t => Prints the type of a expression <expr>
  :q => Ends the session.
#+end_src

Note: the documentation is also available by appending a "?" after a
function name:

#+begin_src txt
ciel-user> (dict ?
#+end_src


# update the TOC with toc-org

* Libraries

  To see the full list of dependencies, see the =ciel.asd= project
  definition or this [[file:doc/dependencies.md][dependencies list]].

** Data structures
*** Generic and nested access to datastructures (access)

From [[https://github.com/AccelerationNet/access/%0A][Access]], we import =access= and =accesses= (plural).

It's always

#+BEGIN_SRC lisp
(access my-structure :elt)
#+end_src

for an alist, a hash-table, a struct, an object… Use =accesses= for
nested access (specially useful with JSON).

*** Hash-table utilities (Serapeum)

We import functions from Serapeum.
https://github.com/ruricolist/serapeum/blob/master/REFERENCE.md#hash-tables

#+begin_src txt
:dict
:do-hash-table ;; see also trivial-do
:dict*
:dictq  ;; quoted
:href  ;; for nested lookup.
:href-default
:pophash
:swaphash
:hash-fold
:maphash-return
:merge-tables
:flip-hash-table
:set-hash-table
:hash-table-set
:hash-table-predicate
:hash-table-function
:make-hash-table-function
:delete-from-hash-table
:pairhash
#+end_src

#+BEGIN_SRC lisp
;; create a hash-table:
(dict :a 1 :b 2 :c 3)
#+end_src

*** Sequences utilities (Alexandria, Serapeum)

From [[ https://github.com/ruricolist/serapeum/blob/master/REFERENCE.md#sequences][Serapeum]] we import:

#+begin_src txt
:assort
:batches
:runs
:partition
:partitions
:split-sequence
#+end_src

And from [[https://common-lisp.net/project/alexandria/draft/alexandria.html][Alexandria]]:

#+begin_src text
:iota
:flatten
:shuffle
:random-elt
:length=
:last-elt
:emptyp
#+end_src

and some more.

*** String manipulation (str)

    Available with the =str= prefix.

    https://github.com/vindarel/cl-str/

** Data formats
*** CSV

    You have [[https://github.com/AccelerationNet/cl-csv][cl-csv]], under its =cl-csv= package name and the =csv=
    local nickname.

    #+BEGIN_SRC lisp
      ;; read a file into a list of lists
      (cl-csv:read-csv #P"file.csv")
      => (("1" "2" "3") ("4" "5" "6"))

      ;; read csv from a string (streams also supported)
      (cl-csv:read-csv "1,2,3
      4,5,6")
      => (("1" "2" "3") ("4" "5" "6"))

      ;; read a file that's tab delimited
      (cl-csv:read-csv #P"file.tab" :separator #\Tab)

      ;; loop over a CSV for effect
      (let ((sum 0))
        (cl-csv:do-csv (row #P"file.csv")
          (incf sum (parse-integer (nth 0 row))))
        sum)
    #+end_src

See also:

- [[https://github.com/defunkydrummer/auto-text][auto-text]], automatic detection for text files (encoding, end of
  line, column width, csv delimiter etc). [[https://github.com/t-sin/inquisitor][inquisitor]] for detection of
  asian and far eastern languages.
- [[https://github.com/sharplispers/clawk][CLAWK]], an AWK implementation embedded into Common Lisp, to parse
  files line-by-line.

*** JSON

    We use [[https://common-lisp.net/project/cl-json/cl-json.html][cl-json]] ([[https://github.com/hankhero/cl-json][GitHub]]). It has a =json= nickname.

To encode an object to a string, use =encode-json-to-string=:

#+BEGIN_SRC lisp
(json:encode-json-to-string (list (dict :a 1)))
;; "[{\"A\":1}]"
#+end_src

To decode from a string: =decode-json-from-string=.

To encode or decode objects from a /stream/, use:

- =encode-json object &optional stream=
- =decode-json &optional stream=

as in:

#+BEGIN_SRC lisp
(with-output-to-string (s)
   (json:encode-json (dict :foo (list 1 2 3)) s))
;; "{\"FOO\":[1,2,3]}"

(with-input-from-string (s "{\"foo\": [1, 2, 3], \"bar\": true, \"baz\": \"!\"}")
  (json:decode-json s))
;; ((:|foo| 1 2 3) (:|bar| . T) (:|baz| . "!"))
#+end_src

cl-json can encode and decode from objects. Given a simple class:

#+BEGIN_SRC lisp
(defclass person ()
  ((name :initarg :name)
   (lisper :initform t)))
#+end_src

We can encode an instance of it:

#+BEGIN_SRC lisp
(json:encode-json-to-string (make-instance 'person :name "you"))
;; "{\"NAME\":\"you\",\"LISPER\":true}"
#+end_src

By default, cl-json wants to convert our lisp symbols to camelCase,
and the JSON ones to lisp-case. We disable that in the =ciel-user= package.

You can set this behaviour back with:

#+BEGIN_SRC lisp
(setf json:*json-identifier-name-to-lisp* #'json:camel-case-to-lisp)
(setf json:*lisp-identifier-name-to-json* #'json:lisp-to-camel-case)
#+end_src

** Date and time

   The [[https://common-lisp.net/project/local-time/][local-time]] package is available.

   See also [[https://github.com/CodyReichert/awesome-cl#date-and-time][awesome-cl#date-and-time]] and the [[https://lispcookbook.github.io/cl-cookbook/dates_and_times.html][Cookbook]].

** Databases

Mito and SxQL are available.

https://lispcookbook.github.io/cl-cookbook/databases.html

** Iteration

   We ship =iterate= and =for= so you can try them, but we don't import
   their symbols.

   See https://lispcookbook.github.io/cl-cookbook/iteration.html for
   examples, including about the good old =loop=.

   We import macros from [[https://github.com/yitzchak/trivial-do/][trivial-do]], that provides =dolist=-like macro
   to iterate over more structures:

- =dohash=: dohash iterates over the elements of an hash table and binds key-var to the key,
value-var to the associated value and then evaluates body as a tagbody that can include
declarations. Finally the result-form is returned after the iteration completes.
- =doplist=: doplist iterates over the elements of an plist and binds key-var to the key, value-var to
the associated value and then evaluates body as a tagbody that can include declarations.
Finally the result-form is returned after the iteration completes.
- =doalist=: doalist iterates over the elements of an alist and binds key-var to the car of each element,
value-var to the cdr of each element and then evaluates body as a tagbody that can include
declarations. Finally the result-form is returned after the iteration completes.
- =doseq*=: doseq* iterates over the elements of an sequence and binds position-var to the index of each
element, value-var to each element and then evaluates body as a tagbody that can include
declarations. Finally the result-form is returned after the iteration completes.
- =doseq=: doseq iterates over the elements of an sequence and binds value-var to successive values
and then evaluates body as a tagbody that can include declarations. Finally the result-form
is returned after the iteration completes.
- =dolist*=: dolist* iterates over the elements of an list and binds position-var to the index of each
element, value-var to each element and then evaluates body as a tagbody that can include
declarations. Finally the result-form is returned after the iteration completes.

** Pattern matching

Use Trivia, also available with the =match= local nickname.

** Numbers

   We import =mean=, =variance=, =median= and =clamp= from Alexandria.

   We import functions to parse numbers (Common Lisp only has
   =parse-integer= by default).

[[https://github.com/soemraws/parse-float][parse-float]]

Similar to PARSE-INTEGER, but parses a floating point value and
returns the value as the specified TYPE (by default
=*READ-DEFAULT-FLOAT-FORMAT*=). The DECIMAL-CHARACTER (by default #.)
specifies the separator between the integer and decimal parts, and the
EXPONENT-CHARACTER (by default #e, case insensitive) specifies the
character before the exponent. Note that the exponent is only parsed
if RADIX is 10.

  #+begin_src text
ARGLIST: (string &key (start 0) (end (length string)) (radix 10) (junk-allowed nil)
        (decimal-character .) (exponent-character e)
        (type *read-default-float-format*))
  #+end_src

From [[https://github.com/sharplispers/parse-number][parse-number]], we import:

#+begin_src text
  :parse-number
  :parse-positive-real-number
  :parse-real-number
#+end_src

#+begin_src text
PARSE-NUMBER
  FUNCTION: Given a string, and start, end, and radix parameters,
  produce a number according to the syntax definitions in the Common
  Lisp Hyperspec.
  ARGLIST: (string &key (start 0) (end nil) (radix 10)
          ((float-format *read-default-float-format*)
           ,*read-default-float-format*))
#+end_src

See also [[https://github.com/tlikonen/cl-decimals][cl-decimals]] to parse and format decimal numbers.

** Regular expressions

Use =ppcre=.

** Threads, monitoring, scheduling

[[https://common-lisp.net/project/bordeaux-threads/][Bordeaux-Threads]] (=bt= prefix)

[[https://lparallel.org/][Lparallel]]

[[https://github.com/ruricolist/moira][Moira]]  (monitor and restart background threads)

[[http://quickdocs.org/trivial-monitored-thread/][trivial-monitored-thread]]

#+begin_quote
Trivial Monitored Thread offers a very simple (aka trivial) way of
spawning threads and being informed when one any of them crash and
die.
#+end_quote

[[http://quickdocs.org/cl-cron/api][cl-cron]] (with [[https://github.com/ciel-lang/cl-cron][our fork here]])

For example, run a function every minute:

#+BEGIN_SRC lisp
  (defun say-hi () (print "Hi!"))
  (cl-cron:make-cron-job #'say-hi)
  (cl-cron:start-cron)
#+end_src

Wait a minute to see some output.

Stop all jobs with =stop-cron=.

=make-cron='s keyword arguments are:

#+BEGIN_SRC lisp
(minute :every) (step-min 1) (hour :every) (step-hour 1) (day-of-month :every)
(step-dom 1) (month :every) (step-month 1) (day-of-week :every)
(step-dow 1)
(boot-only nil) (hash-key nil))
#+end_src

** HTTP and URI handling

   See:

- Dexador (=dex= nickname)
- Quri
- Lquery

** Web

Imported:

- Hunchentoot
- Easy-routes

https://lispcookbook.github.io/cl-cookbook/web.html


** Conditions

   From Serapeum, we import [[https://github.com/ruricolist/serapeum/blob/master/REFERENCE.md#ignoring-type-body-body][=ignoring=]].

   An improved version of =ignore-errors=. The behavior is the same:
   if an error occurs in the body, the form returns two values, nil
   and the condition itself.

   =ignoring= forces you to specify the kind of error you want to ignore:

   #+BEGIN_SRC lisp
     (ignoring parse-error
               ...)
   #+end_src

** Types, type checking, exhaustiveness type checking

   From Serapeum, we import:

   #+begin_src text
     :etypecase-of
     :ctypecase-of
     :typecase-of
     :case-of
     :ccase-of
   #+end_src

   =etypecase-of= allows to do [[https://github.com/ruricolist/serapeum#compile-time-exhaustiveness-checking%0A][compile-time exhaustiveness type checking]].


*** Example with enums

We may call a type defined using member an enumeration. Take an enumeration like this:

#+BEGIN_SRC lisp
(deftype switch-state ()
  '(member :on :off :stuck :broken))
#+end_src

Now we can use =ecase-of= to take all the states of the switch into account.

#+BEGIN_SRC lisp
(defun flick (switch)
  (ecase-of switch-state (state switch)
    (:on (switch-off switch))
    (:off (switch-on switch))))
=> Warning
#+end_src

#+BEGIN_SRC lisp
(defun flick (switch)
  (ecase-of switch-state (state switch)
    (:on (switch-off switch))
    (:off (switch-on switch))
    ((:stuck :broken) (error "Sorry, can't flick ~a" switch))))
=> No warning
#+end_src

*** Example with union types

    #+BEGIN_SRC lisp
(defun negative-integer? (n)
  (etypecase-of t n
    ((not integer) nil)
    ((integer * -1) t)
    ((integer 1 *) nil)))
=> Warning

(defun negative-integer? (n)
  (etypecase-of t n
    ((not integer) nil)
    ((integer * -1) t)
    ((integer 1 *) nil)
    ((integer 0) nil)))
=> No warning
    #+end_src

   See [[https://github.com/ruricolist/serapeum/blob/master/REFERENCE.md#control-flow][Serapeum's reference]].

** Syntax extensions
*** Arrow macros

    We provide the Clojure-like arrow macros and "diamond wands" from
    the [[https://github.com/hipeta/arrow-macros][arrow-macros]] library.

    #+BEGIN_SRC lisp
      ;; -> inserts the previous value as its first argument:
      (-> "  hello macros   "
        str:upcase
        str:words) ; => ("HELLO" "MACROS")

      ;; ->> inserts it as its second argument:
      (->> "  hello macros   "
        str:upcase
        str:words
        (mapcar #'length)) ; => (5 6)


      ;; use as-> to be flexible on the position of the argument:
      (as-> 4 x
        (1+ x)
        (+ x x)) ; => 10
    #+end_src

    And there is more. All the available macros are:

    #+begin_src txt
:->
:->>
:some->
:some->>
:as->
:cond->
:cond->>
:-<>
:-<>>
:some-<>
:some-<>>
    #+end_src

*** Pythonic triple quotes docstring

https://github.com/smithzvk/pythonic-string-reader

We can use triple quotes for docstrings, and double quotes within them.

#+BEGIN_SRC lisp
(defun foo ()
  """foo "bar"."""
  t)
#+end_src
*** Lambda shortcuts

You have to enable cl-punch's syntax yourself.

 https://github.com/windymelt/cl-punch/ - Scala-like anonymous lambda literal.

: (cl-punch:enable-punch-syntax)

#+BEGIN_SRC lisp
;; ^() is converted into (lambda ...) .
;; Each underscore is converted into a lambda argument.

(mapcar ^(* 2 _) '(1 2 3 4 5))
;; => '(2 4 6 8 10)

;; One underscore corresponds one argument.

(^(* _ _) 2 3)
;; => 6

;; <_ reuses last argument.

(mapcar ^(if (oddp _) (* 2 <_) <_) '(1 2 3 4 5))
;; => '(2 2 6 4 10)

;; _! corresponds one argument but it is brought to top of the argument list.
;; It can be useful when you want to change argument order.

(^(cons _ _!) :a :b)
;; => (:b . :a)

(^(list _! _! _!) 1 2 3)
;; => '(3 2 1)
#+end_src

** Development
*** Testing (Fiveam)

    The [[https://common-lisp.net/project/fiveam/docs/][FiveAM]] test framework is available for use.

    Below we create a package to contain our tests and we define the
    most simple one:

    #+BEGIN_SRC lisp
(defpackage ciel-5am
  (:use :cl :5am))

(in-package :ciel-5am)

(test test-one
  (is (= 1 1)))
    #+end_src

    Run the test with:

#+begin_src txt
(run! 'test-one)

Running test TEST-ONE .
 Did 1 check.
    Pass: 1 (100%)
    Skip: 0 ( 0%)
    Fail: 0 ( 0%)

T
NIL
NIL
#+end_src

If the test fails you will see explanations:

#+begin_src txt
> (run! 'test-one)

Running test TEST-ONE .f
 Did 2 checks.
    Pass: 1 (50%)
    Skip: 0 ( 0%)
    Fail: 1 (50%)

 Failure Details:
 --------------------------------
 TEST-ONE []:

1

 evaluated to

1

 which is not

=

 to

2


 --------------------------------

NIL
(#<IT.BESE.FIVEAM::TEST-FAILURE {1007307ED3}>)
NIL
#+end_src

Use =run= to not print explanations.

You can use =(!)= to re-run the last run test.

You can ask 5am to open the interactive debugger on an error:

: (setf *debug-on-error* t)

*** Logging (log4cl)

https://github.com/sharplispers/log4cl/

: (log:info …)

*** Discoverability of documentation (repl-utilities' readme, summary,…)

We use =readme= and =summary= from [[http://quickdocs.org/repl-utilities/][repl-utilities]].

Learn more with:

: (readme repl-utilities)

*** printv

   [[https://github.com/danlentz/printv][printv]]

    #+BEGIN_SRC lisp
 (:printv
  (defvar *y*)
  (defparameter *x* 2)
  (setf *y* (sqrt *x*))
  (setf *y* (/ 1 *y*)))

;; This produces the following text to PRINTV's output stream, and still results in the same returned value: 0.70710677.

;;;   (DEFVAR *Y*) => *Y*
;;;   (DEFPARAMETER *X* 2) => *X*
;;;   (SETF *Y* (SQRT *X*)) => 1.4142135
;;;   (SETF *Y* (/ 1 *Y*)) => 0.70710677

    #+end_src

*** Getting a function's arguments list (trivial-arguments)

 https://github.com/Shinmera/trivial-arguments

 #+BEGIN_SRC emacs-lisp
 (defun foo (a b c &optional d) nil)
 (arglist #'foo)
 ;; (a b c &optional d)
 #+END_SRC

** generic-cl

 https://github.com/alex-gutev/generic-cl/

 todo:

 : generic-ciel

 Example:

 #+BEGIN_SRC emacs-lisp
 ;; with a struct or class "point":
 (defmethod equalp ((p1 point) (p2 point))
    (…))
 #+END_SRC


* Final words

That was your life in CL:

#+html: <p align="center"><img src="before.jpeg" /></p>

and now:

#+html: <p align="center"><img src="after-plus.jpeg" /></p>