
@node System building
@section System building

@menu
* Compiling with ECL::
* Compiling with ASDF::

@c * Compiling with Matroska::

@end menu

@cindex System building
@cindex Creating executables and libraries

@node Compiling with ECL
@subsection Compiling with ECL

In this section we will introduce topics on compiling Lisp programs. ECL
is especially powerful on combining lisp programs with C programs. You
can embed ECL as a lisp engine in C programs, or call C functions via
@ref{Foreign Function Interface}. We explain file types generated by
some compilation approaches. GNU/Linux system and gcc as a development
environment are assumed.

You can generate following files with ECL.

@enumerate
@item Portable FASL file (.fasc)
@item Native FASL file (.fas, .fasb)
@item Object file (.o)
@item Static library
@item Shared library
@item Executable file
@end enumerate

Relations among them are depicted below:

@float Figure,fig:file_types
@caption{Build file types}
@image{figures/file-types}
@end float

@node Portable FASL (fasc)
@subsubsection Portable FASL
@cindex Portable FASL

ECL provides two compilers (bytecodes compiler, and C/C++
compieler). Portable FASL files are build from source lisp files by the
bytecodes compiler. Generally FASC files are portable across
architectures and operating systems providing convenient way of shipping
portable modules. Portable FASL files may be concatenated, what leads to
bundles. FASC files are faster to compile, but generally slower to run.

@exindex Building Portable FASL file
@lisp
;; install bytecodes compiler
(ext:install-bytecodes-compiler)

;; compile hello.lisp file to hello.fasc
(compile-file "hello1.lisp")
(compile-file "hello2.lisp")

;; reinitialize C/C++ compiler back
(ext:install-c-compiler)

;; FASC file may be loaded dynamically from lisp program
(load "hello1.fasc")

;; ... concatenated into a bundle with other FASC
(with-open-file (output "hello.fasc"
                        :direction :output
                        :if-exists :supersede)
  (ext:run-program
   "cat" '("hello1.fasc" "hello2.fasc")  :output output))

;; ... and loaded dynamically from lisp program
(load "hello.fasc")
@end lisp

@node Native FASL
@subsubsection Native FASL

@cindex Native FASL
@ftindex DLOPEN
@cfindex --enable-shared [YES|no]]


If you want to make a library which is loaded dynamically from lisp
program, you should choose fasl file format. Under the hood native fasls
are just a shared library files.

This means you can load fasl files with @code{dlopen} and initialize it
by calling a init function from C programs, but this is not an intended
usage. Recommended usage is loading fasl files by calling load lisp
function. To work with @emph{Native FASL files} ECL has to be compiled
with @code{--enable-shared} configure option (enabled by default).

Creating a fasl file from one lisp file is very easy.

@lisp
(compile-file "hello.lisp")
@end lisp

To create a fasl file from more lisp files, firstly you have to compile
each lisp file into an object file, and then combine them with
c:build-fasl.

@exindex Building native FASL
@lisp
;; generates hello.o
(compile-file "hello.lisp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lisp" :system-p t)

;; generates hello-goodbye.fas
(c:build-fasl "hello-goodbye"
              :lisp-files '("hello.o" "goodbye.o"))

;; fasls may be built from mix of objects and libraries (both shared and
;; static)
(c:build-fasl "mixed-bundle"
              :lisp-files '("hello1.o" "hello2.a" "hello3.so"))
@end lisp

@node Object file
@subsubsection Object file

Object file works as an intermediate file format. If you want to compile
more than two lisp files, you might better to compile with a :system-p t
option, which generates object files (instead of a fasl).

On linux systems, ECL invokes gcc -c for generating object files.

An object file consists of some functions in C:

@itemize
@item Functions corresponding to Lisp functions
@item The initialization function which registers defined functions on the lisp environment
@end itemize

Consider the example below.

@lisp
(defun say-hello ()
  (print "Hello, world"))
@end lisp

@cindex Object file internal layout
During compilation, this simple lisp program is translated into the C
program, and then compiled into the object file. The C program contains
two functions:

@itemize

@item @code{static cl_object L1say_hello}:
'say-hello' function

@item @code{ECL_DLLEXPORT void _eclwm2nNauJEfEnD_CLSxi0z(cl_object flag)}:
initialization function

@end itemize

In order to use these object files from your C program, you have to call
initialization functions before using lisp functions (such as
@code{say-hello}). However the name of an init function is seemed to be
randomized and not user-friendly. This is because object files are not
intended to be used directly.

ECL provides other user-friendly ways to generate compiled lisp programs
(as static/shared libraries or executable), and in each approach, object
files act as intermediate files.

@node Static library
@subsubsection Static library

ECL can compile lisp programs to static libraries, which can be linked
with C programs. A static library is created by c:build-static-library
with some compiled object files.

@exindex Building static library
@lisp
;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates libhello-goodbye.a
(c:build-static-library "hello-goodbye"
                        :lisp-files '("hello.o" "goodbye.o")
                        :init-name "init_hello_goodbye")
@end lisp

When you use static/shared library, you have to call init functions. The
name of the function is specified by @code{:init-name} option. In this
example, @code{init_hello_goodbye} is it. The usage of this function is
shown below:

@exindex Initializing static/shared library in C/C++
@example
@verbatim
#include <ecl/ecl.h>
extern void init_hello_goodbye(cl_object cblock);

int
main(int argc, char **argv)
{
    /* setup the lisp runtime */
    cl_boot(argc, argv);

    /* call the init function via read_VV */
    read_VV(OBJNULL, init_hello_goodbye);

    /* ... */

    /* shutdown the lisp runtime */
    cl_shutdown();

    return 0;
}
@end verbatim
@end example

Because the program itself does not know the type of the init function,
a prototype declaration is inserted. After booting up the lisp
environment, invoke @code{init_hello_goodbye} via
@code{read_VV}. @code{init_hello_goodbye} takes a argument, and read_VV
supplies an appropriate one. Now that the initialization is finished, we
can use functions and other stuffs defined in the library.

@node Shared library
@subsubsection Shared library

Almost the same as the case of static library. User has to use
@code{c:build-shared-library}:

@exindex Building shared library
@lisp
;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates libhello-goodbye.so
(c:build-shared-library "hello-goodbye"
                        :lisp-files '("hello.o" "goodbye.o")
                        :init-name "init_hello_goodbye")
@end lisp

@node Executable
@subsubsection Executable

ECL supports executable file generation. To create a standalone
executable from lisp programs, compile all lisp files to object
files. After that, calling @code{c:build-program} creates the
executable.

@exindex Building executable
@lisp
;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates hello-goodbye
(c:build-program "hello-goodbye"
                 :lisp-files '("hello.o" "goodbye.o"))
@end lisp

Like native FASL, program may be built also from libraries.

@node Summary
@subsubsection Summary

In this post, some file types that can be compiled to with ECL were introduced. Each file type has adequate purpose:

@itemize
@item Object file: intermediate file format for others
@item Fasl files: loaded dynamically via load lisp function
@item Static library: linked with and used from C programs
@item Shared library: loaded dynamically and used from C programs
@item Executable: standalone executable
@end itemize

ECL provides a high-level interface @code{c:build-*} for each native
format. In case of @emph{Portable FASL} bytecodes compiler is needed.

@node Compiling with ASDF
@subsection Compiling with ASDF

Besides the simple situation that we write Lisp without depending on any other Lisp libraries, a more practical example is build a library depends on other asdf systems or Quicklisp projects. ECL provides a useful extension for asdf called @code{asdf:make-build}, it's almost as easy as build a library without dependencies. Because Quicklisp also uses asdf to load systems with dependencies, just make sure you have successfully load and run your library in ECL REPL (or @code{*slime-repl*}). Don't worry Quicklisp, asdf, swank and other unused libraries are packed into the executable or library, ECL will only build and pack libraries your project depends on (that is, all dependence you put in your @code{.asd} file, and their dependencies, nothing more even you are build in a image already load with lots of other libraries).

@node Example code to build
@subsubsection Example code to build

We use a simple project depends on alexandria to demonstrate the steps. Consists of @code{example-with-dep.asd}, @code{package.lisp} and @code{example.lisp}. For convenience, we list these files here:

@lisp
;;;; example-with-dep.asd
(defsystem :example-with-dep
  :serial t
  :depends-on (:alexandria)
  :components ((:file "package")
	       (:file "example")))
@end lisp

@lisp
;;;; package.lisp
(in-package :cl-user)

(defpackage :example
  (:use :cl)
  (:export :test-function))
@end lisp

@lisp
;;;; example.lisp
(in-package :example)

(defun test-function (n)
  (format t "Factorial of ~a is: ~a~%" n (alexandria:factorial n)))
@end lisp

Before any kind you build, you need to push full path of this directory (@code{asdf_with_dependence/}) into @code{asdf:*central-registry*}.

@node Build it as an single executable
@subsubsection Build it as an single executable

Use this in REPL to make a executable:

@lisp
(asdf:make-build :example-with-dep
		 :type :program
		 :move-here #P"./"
		 :epilogue-code '(progn (example:test-function 5)
					(si:exit)))
@end lisp
Here the @code{:epilogue-code} is what to do after loading our library, we can use arbitrary Lisp forms here. You can also write this code in your Lisp files and directly build them without this @code{:epilogue-code} option to have the same effect.
Run the program in console will display the following and exit:

@example
Factorial of 5 is: 120
@end example

@node Build it as shared library and use in C
@subsubsection Build it as shared library and use in C

Use this in REPL to make a shared library:
@lisp
(asdf:make-build :example-with-dep
		 :type :shared-library
		 :move-here #P"./"
		 :monolithic t)
@end lisp

Here @code{:monolithic t} means to let ECL solve dependence and build all dependence into one library named @code{example-with-dep--all-systems.so} in this directory.

To use it, we use a simple C program:

@example
/* test.c */
#include <ecl/ecl.h>

int main (int argc, char **argv) @{
  extern void init_dll_EXAMPLE_WITH_DEP__ALL_SYSTEMS(cl_object);

  cl_boot(argc, argv);
  ecl_init_module(NULL, init_dll_EXAMPLE_WITH_DEP__ALL_SYSTEMS);

  /* do things with the Lisp library */
  cl_eval(c_string_to_object("(example:test-function 5)"));

  cl_shutdown();
  return 0;
@}
@end example

Note the name convention here: an asdf system named @code{example-with-dep} will compiled to @code{example-with-dep--all-systems.so} and in the C code should be init with @code{init_dll_EXAMPLE_WITH_DEP__ALL_SYSTEMS}. Compile it using:

@example
gcc test.c example-with-dep--all-systems.so -o test -lecl
@end example

ECL's library path and current directory may not be in your @code{LD_LIBRARY_PATH}, so call @code{./test} using:

@example
LD_LIBRARY_PATH=/usr/local/lib/:. ./test
@end example

This will show:

@example
Factorial of 5 is: 120
@end example

You can also build all dependent libraries separately as several @code{.so} files and link them together. For example, if you are building a library called @code{complex-example}, that depends on @code{alexandria} and @code{cl-fad}, you can also do these in ECL REPL:

@lisp
(asdf:make-build :complex-example
		 :type :shared-library
		 :move-here #P"./")
(asdf:make-build :alexandria
		 :type :shared-library
		 :move-here #P"./")
(asdf:make-build :cl-fad
		 :type :shared-library
		 :move-here #P"./")
(asdf:make-build :bordeaux-threads
		 :type :shared-library
		 :move-here #P"./")
@end lisp

Note here is no @code{:monolithic t} and we also need to build @code{bordeaux-threads} because @code{cl-fad} depends on it. The building sequence doesn't matter and the result @code{.so} files can also be used in your future program if these libraries are not modified.
And We need to initialize all these modules using @code{ecl_init_module}, the name convention is to initialize @code{cl-fad} you need:

@example
extern void init_dll_CL_FAD(cl_object);

/* after cl_boot(argc, argv); 
   and if B depends on A, you should first init A then B. */
ecl_init_module(NULL, init_dll_CL_FAD);
@end example

You can easily figure out name conventions with other libraries.

@node Build it as static library and use in C
@subsubsection Build it as static library and use in C

To build a static library, use:

@lisp
(asdf:make-build :example-with-dep
		 :type :static-library
		 :move-here #P"./"
		 :monolithic t)
@end lisp

That will generate a @code{example-with-dep--all-systems.a} in current directory and we need to replace @code{init_dll_EXAMPLE_WITH_DEP__ALL_SYSTEMS} with @code{init_lib_EXAMPLE_WITH_DEP__ALL_SYSTEMS}. (The code is given in test-static.c) And compile it using:

@example
gcc test-static.c example-with-dep--all-systems.a -o test-static -lecl
@end example

Then run it:

@example
LD_LIBRARY_PATH=/usr/local/lib/ ./test-static
@end example

Note we don't need to give current path in @code{LD_LIBRARY_PATH} here, since our Lisp library is statically bundled to the executable. 
The result is same as the shared library example above. You can also build all dependent libraries separately to static libraries. To use them you also need replace names like @code{init_dll_CL_FAD} to @code{init_lib_CL_FAD}.
