ciel/README.org

917 lines
23 KiB
Org Mode

* CIEL Is an Extended Lisp :noexport:
STATUS: the API WILL change, but it is usable.
* What is this ? :noexport:
CIEL is a collection of useful libraries.
It's Common Lisp, batteries included.
Questions, doubts? See the [[file:FAQ.org][FAQ]].
* TODOs :noexport:
- settle on libraries that help newcomers
- automate the documentation
- distribute (Quicklisp, Qlot, Quicklisp distribution, Ultralisp,
Ultralisp distribution (upcoming)…)
- ship a core image and a binary
- optionnal: create a tool that, given a CIEL code base, explains what
packages to import in order to switch to "plain CL".
How to procede ?
This is an experiment. I'd be happy to give push rights to more
maintainers. We will send pull requests, discuss, and in case we don't
find a consensus for what should be on by default, we can create other
packages.
Rules
- don't install libraries that need a Slime helper to work in the REPL (cl-annot).
- reader syntax changes may not be enabled by default.
* Table of contents :TOC:
- [[#install][Install]]
- [[#with-quicklisp][With Quicklisp]]
- [[#with-a-core-image][With a core image]]
- [[#with-a-binary-use-ciels-custom-repl][With a binary. Use CIEL's custom REPL.]]
- [[#libraries][Libraries]]
- [[#data-structures][Data structures]]
- [[#data-formats][Data formats]]
- [[#date-and-time][Date and time]]
- [[#databases][Databases]]
- [[#gui-ltk][GUI (ltk)]]
- [[#iteration][Iteration]]
- [[#pattern-matching][Pattern matching]]
- [[#numerical-and-scientific][Numerical and scientific]]
- [[#regular-expressions][Regular expressions]]
- [[#threads-monitoring-scheduling][Threads, monitoring, scheduling]]
- [[#http-and-uri-handling][HTTP and URI handling]]
- [[#web][Web]]
- [[#conditions][Conditions]]
- [[#types-type-checking-exhaustiveness-type-checking][Types, type checking, exhaustiveness type checking]]
- [[#syntax-extensions][Syntax extensions]]
- [[#development][Development]]
- [[#generic-cl][generic-cl]]
- [[#final-words][Final words]]
- [[#how-to-generate-the-documentation][How to generate the documentation]]
* Install
** With Quicklisp
You need a Lisp implementation and Quicklisp installed.
CIEL is not yet on Quicklisp (but it is on [[https://ultralisp.org][Ultralisp]]), so clone this
repository and load the .asd (with =load= or =C-c C-k= in
Slime).
: git clone https://github.com/ciel-lang/CIEL ~/quicklisp/local-projects/CIEL
Then, quickload it:
#+BEGIN_SRC lisp
(ql:quickload "ciel")
#+end_src
and enter the =ciel-user= package, instead of the default
=common-lisp-user= (or =cl-user=):
#+BEGIN_SRC lisp
(in-package :ciel-user)
#+end_src
** With a core image
You need a Lisp implementation, but you don't need Quicklisp.
Build a /core image/ for your lisp with all CIEL's dependencies:
: sbcl --load build-image.lisp
and use it:
: sbcl --core ciel --eval '(in-package :ciel-user)'
TODO: we will distribute ready-to-use core images.
** With a binary. Use CIEL's custom REPL.
You don't need anything, just download the CIEL executable and run
its REPL.
TODO: build it on CI for different platforms.
To build it, clone this repository and run =make build=.
Start it with =./ciel-repl=.
You are dropped into a custom Lisp REPL, freely based on [[https://github.com/hellerve/sbcli][sbcli]].
This REPL is more user friendly than the default SBCL one:
- it has readline capabilities, meaning that the arrow keys work by
default (wouhou!) and there is a persistent history, like in any shell.
- it has *multiline input*.
- it has *TAB completion*.
- it handles errors gracefully: you are not dropped into the debugger
and its sub-REPL, you simply see the error message.
- it has optional *syntax highlighting*.
It also defines short helper commands:
#+begin_src txt
:help => Prints this general help message
:doc => Prints the available documentation for this symbol
:? => Gets help on a symbol <sym>: :? str
:w => Writes the current session to a file <filename>
:d => Dumps the disassembly of a symbol <sym>
:t => Prints the type of a expression <expr>
:q => Ends the session.
#+end_src
Note: the documentation is also available by appending a "?" after a
function name:
#+begin_src txt
ciel-user> (dict ?
#+end_src
Syntax highlighting is currently off by default. To enable it, install
[[https://pygments.org/][pygments]] and add this in your =~/.cielrc=:
#+BEGIN_SRC lisp
(in-package :sbcli)
(setf *syntax-highlighting* t)
;; and, optionally:
;; (setf *pygmentize* "/path/to/pygmentize")
;; (setf *pygmentize-options* (list "-s" "-l" "lisp"))
#+end_src
You can also switch it on and off from the REPL:
#+BEGIN_SRC lisp
(setf sbcli:*syntax-highlighting* t)
#+end_src
# update the TOC with toc-org
* Libraries
To see the full list of dependencies, see the =ciel.asd= project
definition or this [[file:doc/dependencies.md][dependencies list]].
** Data structures
*** Generic and nested access to datastructures (access)
From [[https://github.com/AccelerationNet/access/%0A][Access]], we import =access= and =accesses= (plural).
It's always
#+BEGIN_SRC lisp
(access my-structure :elt)
#+end_src
for an alist, a hash-table, a struct, an object… Use =accesses= for
nested access (specially useful with JSON).
*** Hash-table utilities (Serapeum)
We import functions from Serapeum.
https://github.com/ruricolist/serapeum/blob/master/REFERENCE.md#hash-tables
#+begin_src txt
:dict
:do-hash-table ;; see also trivial-do
:dict*
:dictq ;; quoted
:href ;; for nested lookup.
:href-default
:pophash
:swaphash
:hash-fold
:maphash-return
:merge-tables
:flip-hash-table
:set-hash-table
:hash-table-set
:hash-table-predicate
:hash-table-function
:make-hash-table-function
:delete-from-hash-table
:pairhash
#+end_src
Here's how we can create a hash-table with keys and values:
#+BEGIN_SRC lisp
;; create a hash-table:
(dict :a 1 :b 2 :c 3)
;; =>
(dict
:A 1
:B 2
:C 3
)
#+end_src
In default Common Lisp, you would do:
#+BEGIN_SRC lisp
(let ((ht (make-hash-table :test 'equal)))
(setf (gethash :a ht) 1)
(setf (gethash :b ht) 2)
(setf (gethash :c ht) 3)
ht)
;; #<HASH-TABLE :TEST EQUAL :COUNT 3 {1006CE5613}>
#+end_src
As seen above, hash-tables are pretty-printed by default.
You can toggle the representation with =toggle-print-hash-table=, or
by setting
#+BEGIN_SRC lisp
(setf *pretty-print-hash-tables* nil)
#+end_src
in your configuration file.
*** Sequences utilities (Alexandria, Serapeum)
From [[ https://github.com/ruricolist/serapeum/blob/master/REFERENCE.md#sequences][Serapeum]] we import:
#+begin_src txt
:assort
:batches
:runs
:partition
:partitions
:split-sequence
#+end_src
And from [[https://common-lisp.net/project/alexandria/draft/alexandria.html][Alexandria]]:
#+begin_src text
:iota
:flatten
:shuffle
:random-elt
:length=
:last-elt
:emptyp
#+end_src
and some more.
*** String manipulation (str)
Available with the =str= prefix.
https://github.com/vindarel/cl-str/
** Data formats
*** CSV
You have [[https://github.com/AccelerationNet/cl-csv][cl-csv]], under its =cl-csv= package name and the =csv=
local nickname.
#+BEGIN_SRC lisp
;; read a file into a list of lists
(cl-csv:read-csv #P"file.csv")
=> (("1" "2" "3") ("4" "5" "6"))
;; read csv from a string (streams also supported)
(cl-csv:read-csv "1,2,3
4,5,6")
=> (("1" "2" "3") ("4" "5" "6"))
;; read a file that's tab delimited
(cl-csv:read-csv #P"file.tab" :separator #\Tab)
;; loop over a CSV for effect
(let ((sum 0))
(cl-csv:do-csv (row #P"file.csv")
(incf sum (parse-integer (nth 0 row))))
sum)
#+end_src
See also:
- [[https://github.com/defunkydrummer/auto-text][auto-text]], automatic detection for text files (encoding, end of
line, column width, csv delimiter etc). [[https://github.com/t-sin/inquisitor][inquisitor]] for detection of
asian and far eastern languages.
- [[https://github.com/sharplispers/clawk][CLAWK]], an AWK implementation embedded into Common Lisp, to parse
files line-by-line.
*** JSON
We use [[https://common-lisp.net/project/cl-json/cl-json.html][cl-json]] ([[https://github.com/hankhero/cl-json][GitHub]]). It has a =json= nickname.
To encode an object to a string, use =encode-json-to-string=:
#+BEGIN_SRC lisp
(json:encode-json-to-string (list (dict :a 1)))
;; "[{\"A\":1}]"
#+end_src
To decode from a string: =decode-json-from-string=.
To encode or decode objects from a /stream/, use:
- =encode-json object &optional stream=
- =decode-json &optional stream=
as in:
#+BEGIN_SRC lisp
(with-output-to-string (s)
(json:encode-json (dict :foo (list 1 2 3)) s))
;; "{\"FOO\":[1,2,3]}"
(with-input-from-string (s "{\"foo\": [1, 2, 3], \"bar\": true, \"baz\": \"!\"}")
(json:decode-json s))
;; ((:|foo| 1 2 3) (:|bar| . T) (:|baz| . "!"))
#+end_src
cl-json can encode and decode from objects. Given a simple class:
#+BEGIN_SRC lisp
(defclass person ()
((name :initarg :name)
(lisper :initform t)))
#+end_src
We can encode an instance of it:
#+BEGIN_SRC lisp
(json:encode-json-to-string (make-instance 'person :name "you"))
;; "{\"NAME\":\"you\",\"LISPER\":true}"
#+end_src
By default, cl-json wants to convert our lisp symbols to camelCase,
and the JSON ones to lisp-case. We disable that in the =ciel-user= package.
You can set this behaviour back with:
#+BEGIN_SRC lisp
(setf json:*json-identifier-name-to-lisp* #'json:camel-case-to-lisp)
(setf json:*lisp-identifier-name-to-json* #'json:lisp-to-camel-case)
#+end_src
** Date and time
The [[https://common-lisp.net/project/local-time/][local-time]] package is available.
See also [[https://github.com/CodyReichert/awesome-cl#date-and-time][awesome-cl#date-and-time]] and the [[https://lispcookbook.github.io/cl-cookbook/dates_and_times.html][Cookbook]].
** Databases
Mito and SxQL are available.
https://lispcookbook.github.io/cl-cookbook/databases.html
** GUI (ltk)
We ship [[http://www.peter-herth.de/ltk/ltkdoc/][ltk]].
The Tk toolkit is nearly ubiquitous and simple to use. It doesn't
have a great deal of widgets, but it helps anyways for utility
GUIs. Moreover, it doesn't look aweful (as it did back), it has
themes to look nearly native on the different platforms.
You have other GUI options a quickload away (Qt4, Gtk, IUP, Nuklear, not
mentioning LispWorks CAPI…):
https://lispcookbook.github.io/cl-cookbook/gui.html
Here's how to start with Ltk:
- either put yourself in the =ltk-user= package:
#+BEGIN_SRC lisp
(in-package :ltk-user)
#+end_src
- either =use= ltk:
#+BEGIN_SRC lisp
(use-package :ltk)
#+end_src
Use the =with-ltk= macro to define your GUI, use =make-instance= +
a widget name to create it, and use the =grid= to position widgets.
#+BEGIN_SRC lisp
(with-ltk ()
(let ((button (make-instance 'button :text "hello")))
(grid button 0 0)))
#+end_src
Read more: https://lispcookbook.github.io/cl-cookbook/gui.html#tk
** Iteration
We ship =iterate= and =for= so you can try them, but we don't import
their symbols.
See https://lispcookbook.github.io/cl-cookbook/iteration.html for
examples, including about the good old =loop=.
We import macros from [[https://github.com/yitzchak/trivial-do/][trivial-do]], that provides =dolist=-like macro
to iterate over more structures:
- =dohash=: dohash iterates over the elements of an hash table and binds key-var to the key,
value-var to the associated value and then evaluates body as a tagbody that can include
declarations. Finally the result-form is returned after the iteration completes.
- =doplist=: doplist iterates over the elements of an plist and binds key-var to the key, value-var to
the associated value and then evaluates body as a tagbody that can include declarations.
Finally the result-form is returned after the iteration completes.
- =doalist=: doalist iterates over the elements of an alist and binds key-var to the car of each element,
value-var to the cdr of each element and then evaluates body as a tagbody that can include
declarations. Finally the result-form is returned after the iteration completes.
- =doseq*=: doseq* iterates over the elements of an sequence and binds position-var to the index of each
element, value-var to each element and then evaluates body as a tagbody that can include
declarations. Finally the result-form is returned after the iteration completes.
- =doseq=: doseq iterates over the elements of an sequence and binds value-var to successive values
and then evaluates body as a tagbody that can include declarations. Finally the result-form
is returned after the iteration completes.
- =dolist*=: dolist* iterates over the elements of an list and binds position-var to the index of each
element, value-var to each element and then evaluates body as a tagbody that can include
declarations. Finally the result-form is returned after the iteration completes.
** Pattern matching
Use Trivia, also available with the =match= local nickname.
** Numerical and scientific
We import =mean=, =variance=, =median= and =clamp= from Alexandria.
We import functions to parse numbers (Common Lisp only has
=parse-integer= by default).
[[https://github.com/soemraws/parse-float][parse-float]]
Similar to PARSE-INTEGER, but parses a floating point value and
returns the value as the specified TYPE (by default
=*READ-DEFAULT-FLOAT-FORMAT*=). The DECIMAL-CHARACTER (by default #.)
specifies the separator between the integer and decimal parts, and the
EXPONENT-CHARACTER (by default #e, case insensitive) specifies the
character before the exponent. Note that the exponent is only parsed
if RADIX is 10.
#+begin_src text
ARGLIST: (string &key (start 0) (end (length string)) (radix 10) (junk-allowed nil)
(decimal-character .) (exponent-character e)
(type *read-default-float-format*))
#+end_src
From [[https://github.com/sharplispers/parse-number][parse-number]], we import:
#+begin_src text
:parse-number
:parse-positive-real-number
:parse-real-number
#+end_src
#+begin_src text
PARSE-NUMBER
FUNCTION: Given a string, and start, end, and radix parameters,
produce a number according to the syntax definitions in the Common
Lisp Hyperspec.
ARGLIST: (string &key (start 0) (end nil) (radix 10)
((float-format *read-default-float-format*)
,*read-default-float-format*))
#+end_src
See also [[https://github.com/tlikonen/cl-decimals][cl-decimals]] to parse and format decimal numbers.
We don't ship [[Numbers][Numcl]], a Numpy clone in Common Lisp, but we invite you
to install it right now with Quicklisp:
#+BEGIN_SRC lisp
(ql:quickload "numcl")
#+end_src
** Regular expressions
Use =ppcre=.
See https://common-lisp-libraries.readthedocs.io/cl-ppcre and https://lispcookbook.github.io/cl-cookbook/regexp.html
** Threads, monitoring, scheduling
We ship:
[[https://common-lisp.net/project/bordeaux-threads/][Bordeaux-Threads]] (=bt= prefix)
[[https://lparallel.org/][Lparallel]]
[[https://github.com/ruricolist/moira][Moira]] (monitor and restart background threads)
[[http://quickdocs.org/trivial-monitored-thread/][trivial-monitored-thread]]
#+begin_quote
Trivial Monitored Thread offers a very simple (aka trivial) way of
spawning threads and being informed when one any of them crash and
die.
#+end_quote
[[http://quickdocs.org/cl-cron/api][cl-cron]] (see the sources on [[https://github.com/ciel-lang/cl-cron][our fork here]])
For example, run a function every minute:
#+BEGIN_SRC lisp
(defun say-hi ()
(print "Hi!"))
(cl-cron:make-cron-job #'say-hi)
(cl-cron:start-cron)
#+end_src
Wait a minute to see some output.
Stop all jobs with =stop-cron=.
=make-cron='s keyword arguments are:
#+BEGIN_SRC lisp
(minute :every) (step-min 1) (hour :every) (step-hour 1) (day-of-month :every)
(step-dom 1) (month :every) (step-month 1) (day-of-week :every)
(step-dow 1)
(boot-only nil) (hash-key nil))
#+end_src
** HTTP and URI handling
See:
- Dexador. Use the =dex= nickname or the =http= local nickname.
- Quri
- Lquery
#+BEGIN_SRC lisp
(dex:get "http://my.url")
#+end_src
** Web
We ship:
- Hunchentoot
- Easy-routes
https://lispcookbook.github.io/cl-cookbook/web.html
** Conditions
See https://lispcookbook.github.io/cl-cookbook/error_handling.html
From Serapeum, we import [[https://github.com/ruricolist/serapeum/blob/master/REFERENCE.md#ignoring-type-body-body][=ignoring=]].
An improved version of =ignore-errors=. The behavior is the same:
if an error occurs in the body, the form returns two values, nil
and the condition itself.
=ignoring= forces you to specify the kind of error you want to ignore:
#+BEGIN_SRC lisp
(ignoring parse-error
...)
#+end_src
** Types, type checking, exhaustiveness type checking
From Serapeum, we import:
#+begin_src text
:etypecase-of
:ctypecase-of
:typecase-of
:case-of
:ccase-of
#+end_src
=etypecase-of= allows to do [[https://github.com/ruricolist/serapeum#compile-time-exhaustiveness-checking%0A][compile-time exhaustiveness type checking]].
*** Example with enums
We may call a type defined using member an enumeration. Take an enumeration like this:
#+BEGIN_SRC lisp
(deftype switch-state ()
'(member :on :off :stuck :broken))
#+end_src
Now we can use =ecase-of= to take all the states of the switch into account.
#+BEGIN_SRC lisp
(defun flick (switch)
(ecase-of switch-state (state switch)
(:on (switch-off switch))
(:off (switch-on switch))))
=> Warning
#+end_src
#+BEGIN_SRC lisp
(defun flick (switch)
(ecase-of switch-state (state switch)
(:on (switch-off switch))
(:off (switch-on switch))
((:stuck :broken) (error "Sorry, can't flick ~a" switch))))
=> No warning
#+end_src
*** Example with union types
#+BEGIN_SRC lisp
(defun negative-integer? (n)
(etypecase-of t n
((not integer) nil)
((integer * -1) t)
((integer 1 *) nil)))
=> Warning
(defun negative-integer? (n)
(etypecase-of t n
((not integer) nil)
((integer * -1) t)
((integer 1 *) nil)
((integer 0) nil)))
=> No warning
#+end_src
See [[https://github.com/ruricolist/serapeum/blob/master/REFERENCE.md#control-flow][Serapeum's reference]].
*** More type definitions (trivial-types)
From [[https://github.com/m2ym/trivial-types][trivial-types]], we import
- =association-list-p=
- =type-expand=
- =string-designator=
- =property-list=
- =tuple=
- =association-list=
- =character-designator=
- =property-list-p=
- =file-associated-stream-p=
- =type-specifier-p=
- =list-designator=
- =package-designator=
- =tuplep=
- =non-nil=
- =file-associated-stream=
- =stream-designator=
- =function-designator=
- =file-position-designator=
- =pathname-designator=
** Syntax extensions
*** Arrow macros
We provide the Clojure-like arrow macros and "diamond wands" from
the [[https://github.com/hipeta/arrow-macros][arrow-macros]] library.
#+BEGIN_SRC lisp
;; -> inserts the previous value as its first argument:
(-> " hello macros "
str:upcase
str:words) ; => ("HELLO" "MACROS")
;; ->> inserts it as its second argument:
(->> " hello macros "
str:upcase
str:words
(mapcar #'length)) ; => (5 6)
;; use as-> to be flexible on the position of the argument:
(as-> 4 x
(1+ x)
(+ x x)) ; => 10
#+end_src
And there is more. All the available macros are:
#+begin_src txt
:->
:->>
:some->
:some->>
:as->
:cond->
:cond->>
:-<>
:-<>>
:some-<>
:some-<>>
#+end_src
*** Pythonic triple quotes docstring
https://github.com/smithzvk/pythonic-string-reader
We can use triple quotes for docstrings, and double quotes within them.
#+BEGIN_SRC lisp
(defun foo ()
"""foo "bar"."""
t)
#+end_src
*** Lambda shortcuts
You have to enable cl-punch's syntax yourself.
https://github.com/windymelt/cl-punch/ - Scala-like anonymous lambda literal.
: (cl-punch:enable-punch-syntax)
#+BEGIN_SRC lisp
;; ^() is converted into (lambda ...) .
;; Each underscore is converted into a lambda argument.
(mapcar ^(* 2 _) '(1 2 3 4 5))
;; => '(2 4 6 8 10)
;; One underscore corresponds one argument.
(^(* _ _) 2 3)
;; => 6
;; <_ reuses last argument.
(mapcar ^(if (oddp _) (* 2 <_) <_) '(1 2 3 4 5))
;; => '(2 2 6 4 10)
;; _! corresponds one argument but it is brought to top of the argument list.
;; It can be useful when you want to change argument order.
(^(cons _ _!) :a :b)
;; => (:b . :a)
(^(list _! _! _!) 1 2 3)
;; => '(3 2 1)
#+end_src
** Development
*** Testing (Fiveam)
The [[https://common-lisp.net/project/fiveam/docs/][FiveAM]] test framework is available for use.
Below we create a package to contain our tests and we define the
most simple one:
#+BEGIN_SRC lisp
(defpackage ciel-5am
(:use :cl :5am))
(in-package :ciel-5am)
(test test-one
(is (= 1 1)))
#+end_src
Run the test with:
#+begin_src txt
(run! 'test-one)
Running test TEST-ONE .
Did 1 check.
Pass: 1 (100%)
Skip: 0 ( 0%)
Fail: 0 ( 0%)
T
NIL
NIL
#+end_src
If the test fails you will see explanations:
#+begin_src txt
> (run! 'test-one)
Running test TEST-ONE .f
Did 2 checks.
Pass: 1 (50%)
Skip: 0 ( 0%)
Fail: 1 (50%)
Failure Details:
--------------------------------
TEST-ONE []:
1
evaluated to
1
which is not
=
to
2
--------------------------------
NIL
(#<IT.BESE.FIVEAM::TEST-FAILURE {1007307ED3}>)
NIL
#+end_src
Use =run= to not print explanations.
You can use =(!)= to re-run the last run test.
You can ask 5am to open the interactive debugger on an error:
: (setf *debug-on-error* t)
*** Logging (log4cl)
https://github.com/sharplispers/log4cl/
: (log:info …)
*** Discoverability of documentation (repl-utilities' readme, summary,…)
We use =readme= and =summary= from [[http://quickdocs.org/repl-utilities/][repl-utilities]].
Learn more with:
: (readme repl-utilities)
*** printv
[[https://github.com/danlentz/printv][printv]]
#+BEGIN_SRC lisp
(:printv
(defvar *y*)
(defparameter *x* 2)
(setf *y* (sqrt *x*))
(setf *y* (/ 1 *y*)))
;; This produces the following text to PRINTV's output stream, and still results in the same returned value: 0.70710677.
;;; (DEFVAR *Y*) => *Y*
;;; (DEFPARAMETER *X* 2) => *X*
;;; (SETF *Y* (SQRT *X*)) => 1.4142135
;;; (SETF *Y* (/ 1 *Y*)) => 0.70710677
#+end_src
*** Getting a function's arguments list (trivial-arguments)
https://github.com/Shinmera/trivial-arguments
#+BEGIN_SRC emacs-lisp
(defun foo (a b c &optional d) nil)
(arglist #'foo)
;; (a b c &optional d)
#+END_SRC
** generic-cl
https://github.com/alex-gutev/generic-cl/
todo:
: generic-ciel
Example:
#+BEGIN_SRC emacs-lisp
;; with a struct or class "point":
(defmethod equalp ((p1 point) (p2 point))
())
#+END_SRC
* Final words
That was your life in CL:
#+html: <p align="center"><img src="docs/before.jpeg" /></p>
and now:
#+html: <p align="center"><img src="docs/after-plus.jpeg" /></p>
* How to generate the documentation
See =src/ciel.lisp= and run =(generate-dependencies-page-reference)=.