@node System building
@section System building

@menu
* Compiling with ECL::
* Compiling with ASDF::
* C compiler configuration::
@end menu
@c * Compiling with Matroska::

@cindex System building
@cindex Creating executables and libraries

@node Compiling with ECL
@subsection Compiling with ECL

In this section we will introduce topics on compiling Lisp
programs. ECL is especially powerful on combining lisp programs with C
programs. You can embed ECL as a lisp engine in C programs, or call C
functions via @ref{Foreign Function Interface}. We explain file types
generated by some compilation approaches. For the examples,
a GNU/Linux system and gcc as a development environment are assumed.

You can generate the following files with ECL:

@enumerate
@item Portable FASL file (.fasc)
@item Native FASL file (.fas, .fasb)
@item Object file (.o)
@item Static library
@item Shared library
@item Executable file
@end enumerate

Relations among them are depicted below:

@float Figure,fig:file_types
@caption{Build file types}
@image{figures/file-types}
@end float

@menu
* Portable FASL (fasc)::
* Native FASL::
* Object file::
* Static library::
* Shared library::
* Executable::
* Summary::
@end menu

@node Portable FASL (fasc)
@subsubsection Portable FASL
@cindex Portable FASL

ECL provides two compilers (bytecodes compiler, and C/C++
compiler). Portable FASL files are built from source lisp files by the
bytecodes compiler. Generally FASC files are portable across
architectures and operating systems providing a convenient way of shipping
portable modules. Portable FASL files may be concatenated, what leads to
bundles. FASC files are faster to compile, but generally slower to run.

@exindex Building Portable FASL file
@lisp
;; install bytecodes compiler
(ext:install-bytecodes-compiler)

;; compile hello.lisp file to hello.fasc
(compile-file "hello1.lisp")
(compile-file "hello2.lisp")

;; reinitialize C/C++ compiler back
(ext:install-c-compiler)

;; FASC file may be loaded dynamically from lisp program
(load "hello1.fasc")

;; ... concatenated into a bundle with other FASC
(with-open-file (output "hello.fasc"
                        :direction :output
                        :if-exists :supersede)
  (ext:run-program
   "cat" '("hello1.fasc" "hello2.fasc")  :output output))

;; ... and loaded dynamically from lisp program
(load "hello.fasc")
@end lisp

@node Native FASL
@subsubsection Native FASL

@cindex Native FASL
@ftindex DLOPEN
@cfindex --enable-shared [YES|no]


If you want to make a library which is loaded dynamically from a lisp
program, you should choose the fasl file format. Under the hood native fasls
are just shared library files.

This means you can load fasl files with @code{dlopen} and initialize it
by calling a init function from C programs, but this is not an intended
usage. The recommended usage is to load fasl files by calling the load lisp
function. To work with @emph{Native FASL files} ECL has to be compiled
with @code{--enable-shared} configure option (enabled by default).

Creating a fasl file from one lisp file is very easy.

@lisp
(compile-file "hello.lisp")
@end lisp

To create a fasl file from more lisp files, firstly you have to compile
each lisp file into an object file, and then combine them with
@code{c:build-fasl}.

@exindex Building native FASL
@lisp
;; generates hello.o
(compile-file "hello.lisp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lisp" :system-p t)

;; generates hello-goodbye.fas
(c:build-fasl "hello-goodbye"
              :lisp-files '("hello.o" "goodbye.o"))

;; fasls may be built from mix of objects and libraries (both shared and
;; static)
(c:build-fasl "mixed-bundle"
              :lisp-files '("hello1.o" "hello2.a" "hello3.so"))
@end lisp

@node Object file
@subsubsection Object file

Object files work as an intermediate file format. If you want to compile
more than two lisp files, you might better to compile with a
@code{:system-p t} option, which generates object files (instead of a
fasl).

On linux systems, ECL invokes @code{gcc -c} to generate object files.

An object file consists of some functions in C:

@itemize
@item Functions corresponding to Lisp functions
@item The initialization function which registers defined functions on the lisp environment
@end itemize

Consider the example below.

@lisp
(defun say-hello ()
  (print "Hello, world"))
@end lisp

@cindex Object file internal layout
During compilation, this simple lisp program is translated into the C
program, and then compiled into the object file. The C program contains
two functions:

@itemize

@item @code{static cl_object L1say_hello}:
'say-hello' function

@item @code{ECL_DLLEXPORT void _eclwm2nNauJEfEnD_CLSxi0z(cl_object flag)}:
initialization function

@end itemize

In order to use these object files from your C program, you have to call
initialization functions before using lisp functions (such as
@code{say-hello}). However the name of an init function is seemed to be
randomized and not user-friendly. This is because object files are not
intended to be used directly.

ECL provides other user-friendly ways to generate compiled lisp programs
(as static/shared libraries or executables), and in each approach, object
files act as intermediate files.

@node Static library
@subsubsection Static library

ECL can compile lisp programs to static libraries, which can be linked
with C programs. A static library is created by
@code{c:build-static-library} with some compiled object files.

@exindex Building static library
@lisp
;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates libhello-goodbye.a
(c:build-static-library "hello-goodbye"
                        :lisp-files '("hello.o" "goodbye.o")
                        :init-name "init_hello_goodbye")
@end lisp

When you use a static/shared library, you have to call its init function. The
name of this function is specified by the @code{:init-name} option. In this
example, it is then @code{init_hello_goodbye}. The usage of this function is
shown below:

@cppdef ecl_init_module
@exindex Initializing static/shared library in C/C++
@example
@verbatim
#include <ecl/ecl.h>
extern void init_hello_goodbye(cl_object cblock);

int
main(int argc, char **argv)
{
    /* setup the lisp runtime */
    cl_boot(argc, argv);

    /* call the init function via ecl_init_module */
    ecl_init_module(NULL, init_hello_goodbye);

    /* ... */

    /* shutdown the lisp runtime */
    cl_shutdown();

    return 0;
}
@end verbatim
@end example

Because the program itself does not know the type of the init function,
a prototype declaration is inserted. After booting up the lisp
environment, it invokes @code{init_hello_goodbye} via
@coderef{ecl_init_module}. @code{init_hello_goodbye} takes an argument,
and @coderef{ecl_init_module} supplies an appropriate one. Now that the
initialization is finished, we can use functions and other stuff defined
in the library.

@strong{DEPRECATED} @code{read_VV} - equivalent to @coderef{ecl_init_module}

@node Shared library
@subsubsection Shared library

Almost the same as with a static library. The user has to use
@code{c:build-shared-library}:

@exindex Building shared library
@lisp
;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates libhello-goodbye.so
(c:build-shared-library "hello-goodbye"
                        :lisp-files '("hello.o" "goodbye.o")
                        :init-name "init_hello_goodbye")
@end lisp

@node Executable
@subsubsection Executable

ECL supports the generation of executable files. To create a standalone
executable from a lisp program, compile all lisp files to object
files. After that, calling @code{c:build-program} creates the
executable:

@exindex Building executable
@lisp
;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates hello-goodbye
(c:build-program "hello-goodbye"
                 :lisp-files '("hello.o" "goodbye.o"))
@end lisp

Like with native FASL, the program may be built also from libraries.

@node Summary
@subsubsection Summary

In this section, some file types that can be compiled with ECL were
introduced. Each file type has an adequate purpose:

@itemize
@item Object file: intermediate file format for others
@item Fasl files: loaded dynamically via the @code{load} lisp function
@item Static library: linked with and used from C programs
@item Shared library: loaded dynamically and used from C programs
@item Executable: standalone executable
@end itemize

ECL provides a high-level interface @code{c:build-*} for each native
format. In case of @emph{Portable FASL} the bytecodes compiler is needed.

@node Compiling with ASDF
@subsection Compiling with ASDF

For larger systems involving more complex file dependencies, or for systems
that are portable across different Common Lisp implementations, it may be
better to define systems using @uref{https://common-lisp.net/project/asdf/,asdf}.

ECL provides a useful extension for asdf called @code{asdf:make-build}, which
offers an abstraction for building libraries directly from system definitions.
Note that this extension is only available in the ASDF that is shipped with
ECL; it may not be available from an ASDF installed from the system or from
Quicklisp.

To download dependencies you may use
@uref{https://www.quicklisp.org,Quicklisp} to load your system (with
dependencies defined). Make sure you can successfully load and run your
library in the ECL REPL (or @code{*slime-repl*}). Don't worry about
other libraries loaded in your image – ECL will only build and pack
libraries your project depends on (that is, all dependencies you put in
your @code{.asd} file, and their dependencies - nothing more, despite
the fact that other libraries may be loaded).

@menu
* Example code to build::
* Build it as an single executable::
* Build it as shared library and use in C::
* Build it as static library and use in C::
@end menu

@node Example code to build
@subsubsection Example code to build

An example project is included in the ECL source distribution in the
@code{examples/asdf_with_dependence/} directory.

This project depends on the @code{alexandria} library and consists of a system
definition (@code{example-with-dep.asd}), package definition
(@code{package.lisp}), and the actual library code (@code{example.lisp}).

Before following the steps below, you must
@uref{https://asdf.common-lisp.dev/asdf/Configuring-ASDF-to-find-your-systems.html,configure ASDF to find your systems}.
You can either copy or symlink the example directory in one of the standard
ASDF locations, or push the path of the example directory to your
@code{asdf:*central-registry*}, for example:

@lisp
(push "./" asdf:*central-registry*)
@end lisp

@node Build it as an single executable
@subsubsection Build it as an single executable

Use this in the REPL to make an executable:

@lisp
(asdf:make-build :example-with-dep
                 :type :program
                 :move-here #P"./"
                 :epilogue-code '(progn (example:test-function 5)
                                        (si:exit)))
@end lisp

Here the @code{:epilogue-code} is executed after loading our library;
we can use arbitrary Lisp forms here. You can also put this code in
your Lisp files and directly build them without this
@code{:epilogue-code} option to achieve the same result. Running the
program in a console will display the following and exit:

@example
Factorial of 5 is: 120
@end example

@node Build it as shared library and use in C
@subsubsection Build it as shared library and use in C

Use this in the REPL to make a shared library:

@lisp
(asdf:make-build :example-with-dep
                 :type :shared-library
                 :move-here #P"./"
                 :monolithic t
                 :init-name "init_dll_example")
@end lisp

Here @code{:monolithic t} means that ECL will compile the library and
all its dependencies into a single library named
@code{example-with-dep--all-systems.so}. The @code{:move-here}
parameter is self-explanatory. @code{:init-name} sets the name of the
initialization function. Each library linked from C/C++ code must be
initialized, and this is a mechanism to specify the initialization
function's name.

To use it, we write a simple C program:

@example
@verbatim
/* test.c */
#include <ecl/ecl.h>
extern void init_dll_example(cl_object);

int main (int argc, char **argv) {
  
  cl_boot(argc, argv);
  ecl_init_module(NULL, init_dll_example);

  /* do things with the Lisp library */
  cl_eval(c_string_to_object("(example:test-function 5)"));

  cl_shutdown();
  return 0;
}
@end verbatim
@end example

Compile the file using a standard C compiler (note we're linking to
@code{libecl.so} with @code{-lecl}, which provides the lisp
runtime@footnote{You may also link ECL runtime statically. That is not
covered in this walkthrough.}):

@example
gcc test.c example-with-dep--all-systems.so -o test -lecl
@end example

If ECL is installed in a non-standard location you may need to provide
flags for the compiler and the linker. You may read them with:

@example
ecl-config --cflags
ecl-config --libs
@end example

Since our shared object is not in the standard location, you need to
provide @code{LD_LIBRARY_PATH} pointing to the current directory to
run the application:

@example
LD_LIBRARY_PATH=`pwd` ./test
@end example

This will show:

@example
Factorial of 5 is: 120
@end example

You can also build all dependent libraries separately as a few
@code{.so} files and link them together. For example, if you are
building a library called @code{complex-example}, that depends on
@code{alexandria} and @code{cl-fad}, you can do the following (in the
REPL):

@lisp
(asdf:make-build :complex-example
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_example")

(asdf:make-build :alexandria
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_alexandria")

(asdf:make-build :cl-fad
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_fad")

(asdf:make-build :bordeaux-threads
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_bt")
@end lisp

Note that we haven't specified @code{:monolithic t}, so we need to build
@code{bordeaux-threads} as well because @code{cl-fad} depends on it. The
building sequence doesn't matter and the resultant @code{.so} files can
also be used in your future programs if these libraries are not
modified.

We need to initialize all these modules using @coderef{ecl_init_module}
in the correct order. (@code{bordeaux-threads} must be initialized
before @code{cl-fad}; @code{cl-fad} and @code{alexandria} must be
initialized before @code{complex-ecample}.)

Here is a code snippet (not a full program):

@example
extern void init_fad(cl_object);
extern void init_alexandria(cl_object);
extern void init_bt(cl_object);
extern void init_example(cl_object);

/* call these *after* cl_boot(argc, argv); 
   if B depends on A, you should first init A then B. */
ecl_init_module(NULL, init_bt);
ecl_init_module(NULL, init_fad);
ecl_init_module(NULL, init_alexandria);
ecl_init_module(NULL, init_example);
@end example

@node Build it as static library and use in C
@subsubsection Build it as static library and use in C

To build a static library, use:

@lisp
(asdf:make-build :example-with-dep
                 :type :static-library
                 :move-here #P"./"
                 :monolithic t
                 :init-name "init_example")
@end lisp

This will generate @code{example-with-dep--all-systems.a} in the
current directory which we need to initialize with the
@code{init_example} function. Compile it using:

@example
gcc test.c example-with-dep--all-systems.a -o test-static -lecl
@end example

Then run it:

@example
./test-static
@end example

This will show:

@example
Factorial of 5 is: 120
@end example

Note we don't need to pass the current path in @code{LD_LIBRARY_PATH}
here, since our Lisp library is statically bundled with the executable.
The result is the same as the shared library example above. You can also
build all dependent libraries separately as static libraries.

@node C compiler configuration
@subsection C compiler configuration

ECL provides some global variables to customize which C compiler and
compiler options to use:

@subsubsection Compiler flags

It is not required to surround the compiler flags with quotes or use
slashes before special characters.

@deftypevar string {c:*user-cc-flags*}
Flags and options to be passed to the C compiler when building FASL,
shared libraries and standalone programs.
@end deftypevar

@deftypevar string {c:*user-linker-flags*}
Flags for options (e.g. @code{-Wl,foo} flags, usually in the
@code{$LDFLAGS} variable in autoconf) to be passed to the linker when
building FASL, shared libraries and standalone programs.
@end deftypevar

@deftypevar string {c:*user-linker-libs*}
Flags for libraries (e.g. @code{-lfoo} flags, usually in the
@code{$LIBS} variable in autoconf) to be passed to the linker when
building FASL, shared libraries and standalone programs.
@end deftypevar

@deftypevar string {c:*cc-optimize*}
Optimize options to be passed to the C compiler.
@end deftypevar

@deftypevar string {c:*user-ld-flags*}
@strong{DEPRECATED} Flags and options to be passed to the linker when
building FASL, shared libraries and standalone programs.
@end deftypevar

@subsubsection Compiler & Linker programs

@deftypevar string {c::*cc*}
This variable controls how the C compiler is invoked by ECL. One can
set the variable appropriately adding for instance flags which the C
compiler may need to exploit special hardware features (e.g. a
floating point coprocessor).
@end deftypevar

@deftypevar string {c::*ld*}
This variable controls the linker which is used by ECL.
@end deftypevar

@deftypevar string {c::*ranlib*}
Name of the `ranlib' program on the hosting platform.
@end deftypevar

@deftypevar string {c::*ar*}
Name of the `ar' program on the hosting platform.
@end deftypevar

@deftypevar string {c::*ecl-include-directory*}
Directory where the ECL header files for the target platform are located.
@end deftypevar

@deftypevar string {c::*ecl-library-directory*}
Directory where the ECL library files for the target platform are located.
@end deftypevar
