@node System building
@section System building

@menu
* Compiling with ECL::
* Compiling with ASDF::
* Cross compilation::
* C compiler configuration::
@end menu
@c * Compiling with Matroska::

@cindex System building
@cindex Creating executables and libraries

@node Compiling with ECL
@subsection Compiling with ECL

In this section we will introduce topics on compiling Lisp
programs. ECL is especially powerful on combining lisp programs with C
programs. You can embed ECL as a lisp engine in C programs, or call C
functions via @ref{Foreign Function Interface}. We explain file types
generated by some compilation approaches. For the examples,
a GNU/Linux system and gcc as a development environment are assumed.

You can generate the following files with ECL:

@enumerate
@item Portable FASL file (.fasc)
@item Native FASL file (.fas, .fasb)
@item Object file (.o)
@item Static library
@item Shared library
@item Executable file
@end enumerate

Relations among them are depicted below:

@float Figure,fig:file_types
@caption{Build file types}
@image{figures/file-types}
@end float

@menu
* Portable FASL (fasc)::
* Native FASL::
* Object file::
* Static library::
* Shared library::
* Executable::
* Summary::
@end menu

@node Portable FASL (fasc)
@subsubsection Portable FASL
@cindex Portable FASL

ECL provides two compilers (bytecodes compiler, and C/C++
compiler). Portable FASL files are built from source lisp files by the
bytecodes compiler. Generally FASC files are portable across
architectures and operating systems providing a convenient way of shipping
portable modules. Portable FASL files may be concatenated, what leads to
bundles. FASC files are faster to compile, but generally slower to run.

@exindex Building Portable FASL file
@lisp
;; install bytecodes compiler
(ext:install-bytecodes-compiler)

;; compile hello.lisp file to hello.fasc
(compile-file "hello1.lisp")
(compile-file "hello2.lisp")

;; reinitialize C/C++ compiler back
(ext:install-c-compiler)

;; FASC file may be loaded dynamically from lisp program
(load "hello1.fasc")

;; ... concatenated into a bundle with other FASC
(with-open-file (output "hello.fasc"
                        :direction :output
                        :if-exists :supersede)
  (ext:run-program
   "cat" '("hello1.fasc" "hello2.fasc")  :output output))

;; ... and loaded dynamically from lisp program
(load "hello.fasc")
@end lisp

@node Native FASL
@subsubsection Native FASL

@cindex Native FASL
@ftindex DLOPEN
@cfindex --enable-shared [YES|no]


If you want to make a library which is loaded dynamically from a lisp
program, you should choose the fasl file format. Under the hood native fasls
are just shared library files.

This means you can load fasl files with @code{dlopen} and initialize it
by calling a init function from C programs, but this is not an intended
usage. The recommended usage is to load fasl files by calling the load lisp
function. To work with @emph{Native FASL files} ECL has to be compiled
with @code{--enable-shared} configure option (enabled by default).

Creating a fasl file from one lisp file is very easy.

@lisp
(compile-file "hello.lisp")
@end lisp

To create a fasl file from more lisp files, firstly you have to compile
each lisp file into an object file, and then combine them with
@code{c:build-fasl}.

@exindex Building native FASL
@lisp
;; generates hello.o
(compile-file "hello.lisp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lisp" :system-p t)

;; generates hello-goodbye.fas
(c:build-fasl "hello-goodbye"
              :lisp-files '("hello.o" "goodbye.o"))

;; fasls may be built from mix of objects and libraries (both shared and
;; static)
(c:build-fasl "mixed-bundle"
              :lisp-files '("hello1.o" "hello2.a" "hello3.so"))
@end lisp

@node Object file
@subsubsection Object file

Object files work as an intermediate file format. If you want to compile
more than two lisp files, you might better to compile with a
@code{:system-p t} option, which generates object files (instead of a
fasl).

On linux systems, ECL invokes @code{gcc -c} to generate object files.

An object file consists of some functions in C:

@itemize
@item Functions corresponding to Lisp functions
@item The initialization function which registers defined functions on the lisp environment
@end itemize

Consider the example below.

@lisp
(defun say-hello ()
  (print "Hello, world"))
@end lisp

@cindex Object file internal layout
During compilation, this simple lisp program is translated into the C
program, and then compiled into the object file. The C program contains
two functions:

@itemize

@item @code{static cl_object L1say_hello}:
'say-hello' function

@item @code{ECL_DLLEXPORT void _eclwm2nNauJEfEnD_CLSxi0z(cl_object flag)}:
initialization function

@end itemize

In order to use these object files from your C program, you have to call
initialization functions before using lisp functions (such as
@code{say-hello}). However the name of an init function is seemed to be
randomized and not user-friendly. This is because object files are not
intended to be used directly.

ECL provides other user-friendly ways to generate compiled lisp programs
(as static/shared libraries or executables), and in each approach, object
files act as intermediate files.

@node Static library
@subsubsection Static library

ECL can compile lisp programs to static libraries, which can be linked
with C programs. A static library is created by
@code{c:build-static-library} with some compiled object files.

@exindex Building static library
@lisp
;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates libhello-goodbye.a
(c:build-static-library "hello-goodbye"
                        :lisp-files '("hello.o" "goodbye.o")
                        :init-name "init_hello_goodbye")
@end lisp

When you use a static/shared library, you have to call its init function. The
name of this function is specified by the @code{:init-name} option. In this
example, it is then @code{init_hello_goodbye}. The usage of this function is
shown below:

@cppdef ecl_init_module
@exindex Initializing static/shared library in C/C++
@example
@verbatim
#include <ecl/ecl.h>
extern void init_hello_goodbye(cl_object cblock);

int
main(int argc, char **argv)
{
    /* setup the lisp runtime */
    cl_boot(argc, argv);

    /* call the init function via ecl_init_module */
    ecl_init_module(NULL, init_hello_goodbye);

    /* ... */

    /* shutdown the lisp runtime */
    cl_shutdown();

    return 0;
}
@end verbatim
@end example

Because the program itself does not know the type of the init function,
a prototype declaration is inserted. After booting up the lisp
environment, it invokes @code{init_hello_goodbye} via
@coderef{ecl_init_module}. @code{init_hello_goodbye} takes an argument,
and @coderef{ecl_init_module} supplies an appropriate one. Now that the
initialization is finished, we can use functions and other stuff defined
in the library.

@strong{DEPRECATED} @code{read_VV} - equivalent to @coderef{ecl_init_module}

@node Shared library
@subsubsection Shared library

Almost the same as with a static library. The user has to use
@code{c:build-shared-library}:

@exindex Building shared library
@lisp
;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates libhello-goodbye.so
(c:build-shared-library "hello-goodbye"
                        :lisp-files '("hello.o" "goodbye.o")
                        :init-name "init_hello_goodbye")
@end lisp

@node Executable
@subsubsection Executable

ECL supports the generation of executable files. To create a standalone
executable from a lisp program, compile all lisp files to object
files. After that, calling @code{c:build-program} creates the
executable:

@exindex Building executable
@lisp
;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates hello-goodbye
(c:build-program "hello-goodbye"
                 :lisp-files '("hello.o" "goodbye.o"))
@end lisp

Like with native FASL, the program may be built also from libraries.

@node Summary
@subsubsection Summary

In this section, some file types that can be compiled with ECL were
introduced. Each file type has an adequate purpose:

@itemize
@item Object file: intermediate file format for others
@item Fasl files: loaded dynamically via the @code{load} lisp function
@item Static library: linked with and used from C programs
@item Shared library: loaded dynamically and used from C programs
@item Executable: standalone executable
@end itemize

ECL provides a high-level interface @code{c:build-*} for each native
format. In case of @emph{Portable FASL} the bytecodes compiler is needed.

@node Compiling with ASDF
@subsection Compiling with ASDF

For larger systems involving more complex file dependencies, or for systems
that are portable across different Common Lisp implementations, it may be
better to define systems using @uref{https://common-lisp.net/project/asdf/,asdf}.

ECL provides a useful extension for asdf called @code{asdf:make-build}, which
offers an abstraction for building libraries directly from system definitions.
Note that this extension is only available in the ASDF that is shipped with
ECL; it may not be available from an ASDF installed from the system or from
Quicklisp.

To download dependencies you may use
@uref{https://www.quicklisp.org,Quicklisp} to load your system (with
dependencies defined). Make sure you can successfully load and run your
library in the ECL REPL (or @code{*slime-repl*}). Don't worry about
other libraries loaded in your image – ECL will only build and pack
libraries your project depends on (that is, all dependencies you put in
your @code{.asd} file, and their dependencies - nothing more, despite
the fact that other libraries may be loaded).

@menu
* Example code to build::
* Build it as an single executable::
* Build it as shared library and use in C::
* Build it as static library and use in C::
@end menu

@node Example code to build
@subsubsection Example code to build

An example project is included in the ECL source distribution in the
@code{examples/asdf_with_dependence/} directory.

This project depends on the @code{alexandria} library and consists of a system
definition (@code{example-with-dep.asd}), package definition
(@code{package.lisp}), and the actual library code (@code{example.lisp}).

Before following the steps below, you must
@uref{https://asdf.common-lisp.dev/asdf/Configuring-ASDF-to-find-your-systems.html,configure ASDF to find your systems}.
You can either copy or symlink the example directory in one of the standard
ASDF locations, or push the path of the example directory to your
@code{asdf:*central-registry*}, for example:

@lisp
(push "./" asdf:*central-registry*)
@end lisp

@node Build it as an single executable
@subsubsection Build it as an single executable

Use this in the REPL to make an executable:

@lisp
(asdf:make-build :example-with-dep
                 :type :program
                 :move-here #P"./"
                 :epilogue-code '(progn (example:test-function 5)
                                        (si:exit)))
@end lisp

Here the @code{:epilogue-code} is executed after loading our library;
we can use arbitrary Lisp forms here. You can also put this code in
your Lisp files and directly build them without this
@code{:epilogue-code} option to achieve the same result. Running the
program in a console will display the following and exit:

@example
Factorial of 5 is: 120
@end example

@node Build it as shared library and use in C
@subsubsection Build it as shared library and use in C

Use this in the REPL to make a shared library:

@lisp
(asdf:make-build :example-with-dep
                 :type :shared-library
                 :move-here #P"./"
                 :monolithic t
                 :init-name "init_dll_example")
@end lisp

Here @code{:monolithic t} means that ECL will compile the library and
all its dependencies into a single library named
@code{example-with-dep--all-systems.so}. The @code{:move-here}
parameter is self-explanatory. @code{:init-name} sets the name of the
initialization function. Each library linked from C/C++ code must be
initialized, and this is a mechanism to specify the initialization
function's name.

To use it, we write a simple C program:

@example
@verbatim
/* test.c */
#include <ecl/ecl.h>
extern void init_dll_example(cl_object);

int main (int argc, char **argv) {
  
  cl_boot(argc, argv);
  ecl_init_module(NULL, init_dll_example);

  /* do things with the Lisp library */
  cl_eval(c_string_to_object("(example:test-function 5)"));

  cl_shutdown();
  return 0;
}
@end verbatim
@end example

Compile the file using a standard C compiler (note we're linking to
@code{libecl.so} with @code{-lecl}, which provides the lisp
runtime@footnote{You may also link ECL runtime statically. That is not
covered in this walkthrough.}):

@example
gcc test.c example-with-dep--all-systems.so -o test -lecl
@end example

If ECL is installed in a non-standard location you may need to provide
flags for the compiler and the linker. You may read them with:

@example
ecl-config --cflags
ecl-config --libs
@end example

Since our shared object is not in the standard location, you need to
provide @code{LD_LIBRARY_PATH} pointing to the current directory to
run the application:

@example
LD_LIBRARY_PATH=`pwd` ./test
@end example

This will show:

@example
Factorial of 5 is: 120
@end example

You can also build all dependent libraries separately as a few
@code{.so} files and link them together. For example, if you are
building a library called @code{complex-example}, that depends on
@code{alexandria} and @code{cl-fad}, you can do the following (in the
REPL):

@lisp
(asdf:make-build :complex-example
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_example")

(asdf:make-build :alexandria
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_alexandria")

(asdf:make-build :cl-fad
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_fad")

(asdf:make-build :bordeaux-threads
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_bt")
@end lisp

Note that we haven't specified @code{:monolithic t}, so we need to build
@code{bordeaux-threads} as well because @code{cl-fad} depends on it. The
building sequence doesn't matter and the resultant @code{.so} files can
also be used in your future programs if these libraries are not
modified.

We need to initialize all these modules using @coderef{ecl_init_module}
in the correct order. (@code{bordeaux-threads} must be initialized
before @code{cl-fad}; @code{cl-fad} and @code{alexandria} must be
initialized before @code{complex-ecample}.)

Here is a code snippet (not a full program):

@example
extern void init_fad(cl_object);
extern void init_alexandria(cl_object);
extern void init_bt(cl_object);
extern void init_example(cl_object);

/* call these *after* cl_boot(argc, argv); 
   if B depends on A, you should first init A then B. */
ecl_init_module(NULL, init_bt);
ecl_init_module(NULL, init_fad);
ecl_init_module(NULL, init_alexandria);
ecl_init_module(NULL, init_example);
@end example

@node Build it as static library and use in C
@subsubsection Build it as static library and use in C

To build a static library, use:

@lisp
(asdf:make-build :example-with-dep
                 :type :static-library
                 :move-here #P"./"
                 :monolithic t
                 :init-name "init_example")
@end lisp

This will generate @code{example-with-dep--all-systems.a} in the
current directory which we need to initialize with the
@code{init_example} function. Compile it using:

@example
gcc test.c example-with-dep--all-systems.a -o test-static -lecl
@end example

Then run it:

@example
./test-static
@end example

This will show:

@example
Factorial of 5 is: 120
@end example

Note we don't need to pass the current path in @code{LD_LIBRARY_PATH}
here, since our Lisp library is statically bundled with the executable.
The result is the same as the shared library example above. You can also
build all dependent libraries separately as static libraries.

@node Cross compilation
@subsection Cross compilation

ECL supports cross compiling Lisp files for a target system that differs
from the host system on which the compilation takes place. This section
of the manual describes how to use this feature and explains important
things to keep in mind when cross compiling Common Lisp.

@node Getting started with cross compilation
@subsubsection Getting started with cross compilation

To get started, follow the steps described below:

@enumerate
@item
Cross compile ECL itself following the steps described in @ref{Building
ECL}.
@item
Step one will create a Lisp file called @file{target-info.lsp}
containing all the information necessary for the ECL compiler to create
compiled files for the chosen target system. Load this information into
a running ECL process on the host system by calling
@code{c:read-target-info} on this file.
@item
Supply the resulting target information to the @code{:target} option of
@coderef{compile-file} or @coderef{with-compilation-unit}.
@end enumerate

For an example, consider two files @file{a.lisp} and @file{b.lisp} which
are supposed to be linked into a shared library. This can be
accomplished with the following steps:

@example
(defvar *target* (c:read-target-info "/path/to/ecl/installation/lib/ecl-xx.x.x/target-info.lsp"))

(compile-file "a.lisp" :target *target* :system-p t)
(load "a.lisp") ; make macro definitions in "a.lisp" accessible to "b.lisp"
(compile-file "b.lisp" :target *target* :system-p t)

(with-compilation-unit (:target *target*)
  (c:build-shared-library "example"
                          :lisp-files '("a.o" "b.o")
                          :init-name "init_example"))
@end example

Cross compilation using ASDF is not supported yet but is planned for the
future.

@node Limitations and pitfalls for cross compiling Common Lisp
@subsubsection Limitations and pitfalls for cross compiling Common Lisp

The ubiquity with which typical Common Lisp programs run code at compile
time in macros or other similar constructs makes cross compilation
somewhat more challenging than in other programming languages. Since
compilation happens on a different system than the one in which the code
generated by a macro is run, differences between the host and target
environment can lead to bugs if the macro hasn't been written with cross
compilation in mind. Consider for instance the following example macro
iterating over prime number in the range from @code{a} to @code{b}:

@example
(defmacro do-prime-numbers ((a b) p &body body)
  `(loop with ,p = (next-prime ,a)
         until (> ,p ,b)
         do ,(if (typep b 'fixnum)
                 `(locally (declare (fixnum ,p))
                    ,@@body)
                 `(progn ,@@body))
            (setf ,p (next-prime (1+ ,p)))))
@end example

While this macro works fine in a standard setting, it will produce
an incorrect declaration when cross compiling if the size of a fixnum
is larger in the host than in the target and the upper bound @code{b} is
larger than @code{most-positive-fixnum} in the target but smaller than
@code{most-positive-fixnum} in the host. In general, any observable
difference between target and host system may contribute to such issues.
In the list below, we collect a number of such potential issues and
explain strategies to solve them.

@itemize
@item
Different type or subtype definitions. This may arise from differing
word sizes, as in the fixnum example above, or different feature support
such as complex floating point number support in the C compiler leading
to differing subtype relationships for complex numbers in Lisp. There
are two main strategies for dealing with this problem.
@enumerate
@item
Avoid the problem by rewriting the code not to rely on target specific
information. For instance, in the above example we can change the
declaration as follows to avoid relying on fixnums:
@example
(defmacro do-prime-numbers ((a b) p &body body)
  `(loop with ,p = (next-prime ,a)
         until (> ,p ,b)
         do (locally (declare (type (integer ,a ,b) ,p))
              ,@@body)
            (setf ,p (next-prime (1+ ,p)))))
@end example
If the declared integer type is a subtype of fixnum, the ECL compiler
will automatically take this into account and optimize the same as if a
fixnum declaration had been made.
@item
Use the lexical environment parameter to @clhs{f_typep.htm,typep} and
@clhs{f_subtpp.htm,subtypep}. Both functions take an optional parameter
which can be used to supply information about type relationships in the
target system. In the above example, this works as follows:
@example
(defmacro do-prime-numbers ((a b) p &body body &environment env)
  `(loop with ,p = (next-prime ,a)
         until (> ,p ,b)
         do ,(if (typep b 'fixnum env)
                 `(locally (declare (fixnum ,p))
                    ,@@body)
                 `(progn ,@@body))
            (setf ,p (next-prime (1+ ,p)))))
@end example
It is important, however, not to blindly modify every occurrence of
@code{typep} and @code{subtypep} in this way. Some calls to these
functions may check for types in the host environment, in which case
they should not receive an environment parameter. It is necessary to
know for which environment the call is meant in order to decide what to
do.
@end enumerate
@item
Different value of @code{*features*}, leading to mismatched @code{#+}
and @code{#-} read time conditionals. During cross compilation,
@code{*features*} will be rebound to the value in the target
system.@footnote{Changes to @code{*features*} during compilation are
carried over in the target information structure. This means that in a
scenario where one cross compiles a file @file{a.lisp} containing
@code{(eval-when (:compile-toplevel) (push :my-feature *features*))}
before cross compiling another file @file{b.lisp} containing
@code{#+my-feature}, the read time conditional in @file{b.lisp} will
evaluate to true.} However, macros that are defined in the host system
will see the host @code{*features*} during read time. Consider again the
example given above where @file{a.lisp} was cross compiled, then loaded
before @file{b.lisp} was cross compiled. If @file{a.lisp} contains a
macro definition
@example
(defmacro my-macro (...)
  #+android `(do-something ...)
  #-android `(do-something-else ...))
@end example
then @code{my-macro} will expand to @code{do-something} in @file{a.lisp}
but @code{do-something-else} in @file{b.lisp} if cross compiling for the
android target. There are several ways to deal with this issue:
@enumerate
@item
Load source files containing macro definitions in an environment where
@code{*features*} has the same value as in the target system. This can
be accomplished either by wrapping @code{load} in a
@coderef{with-compilation-unit} call with a @code{:target} option given
or by using the @code{:load} keyword option of @coderef{compile-file} in
conjunction with @code{:target}. In the above example, we would compile
as follows:
@example
(compile-file "a.lisp" :target *target* :system-p t :load t)
(compile-file "b.lisp" :target *target* :system-p t)
@end example
Note that this will not work if the file contains read time conditionals
selecting between different compile time code paths due to differences
in the host system (let's say for example some code generation from a
file that is located in a different location for different compilation
hosts).
@item
Replace read time conditionals by lookups that run during macro
expansion, e.g.
@example
(defmacro my-macro (...)
  (if (member :android *features*)
      `(do-something ...)
      `(do-something-else ...)))
@end example
@item
Add feature keywords for the target system to @code{*features*} in the
host before compilation. This is best suited for conditionals which only
add but not subtract code (keywords which only appear in @code{#+}
conditionals). In our example, we would call
@example
(push :android *features*)
@end example
before starting cross compilation.
@end enumerate
@item
Floating point accuracy issues. ECL relies on the C standard library for
numerical functions which may return slightly different values for
different systems. If your code depends sensitively on numerical
functions returning bitwise identical results, it is best not to run any
numerical code at compile time and restrict yourself to only running run
time code.
@end itemize

Another option to avoid the aformentioned issues entirely is to use
emulation instead of cross compilation. Even if full-blown emulation is
too complicated, simply cross compiling from a version of ECL compiled
for the same word size as the target system would for instance avoid the
fixnum issues in our example macro alltogether.

Whether the compiler is switched to cross compilation mode or not can be
diagnosed from the presence of a @code{:cross} keyword in
@code{*features*}. If necessary, this can be used to select different
code paths as well.


@node C compiler configuration
@subsection C compiler configuration

ECL provides some global variables to customize which C compiler and
compiler options to use:

@subsubsection Compiler flags

It is not required to surround the compiler flags with quotes or use
slashes before special characters.

@deftypevar string {c:*user-cc-flags*}
Flags and options to be passed to the C compiler when building FASL,
shared libraries and standalone programs.
@end deftypevar

@deftypevar string {c:*user-linker-flags*}
Flags for options (e.g. @code{-Wl,foo} flags, usually in the
@code{$LDFLAGS} variable in autoconf) to be passed to the linker when
building FASL, shared libraries and standalone programs.
@end deftypevar

@deftypevar string {c:*user-linker-libs*}
Flags for libraries (e.g. @code{-lfoo} flags, usually in the
@code{$LIBS} variable in autoconf) to be passed to the linker when
building FASL, shared libraries and standalone programs.
@end deftypevar

@deftypevar string {c:*cc-optimize*}
Optimize options to be passed to the C compiler.
@end deftypevar

@deftypevar string {c:*user-ld-flags*}
@strong{DEPRECATED} Flags and options to be passed to the linker when
building FASL, shared libraries and standalone programs.
@end deftypevar

@subsubsection Compiler & Linker programs

@deftypevar string {c::*cc*}
This variable controls how the C compiler is invoked by ECL. One can
set the variable appropriately adding for instance flags which the C
compiler may need to exploit special hardware features (e.g. a
floating point coprocessor).
@end deftypevar

@deftypevar string {c::*ld*}
This variable controls the linker which is used by ECL.
@end deftypevar

@deftypevar string {c::*ranlib*}
Name of the `ranlib' program on the hosting platform.
@end deftypevar

@deftypevar string {c::*ar*}
Name of the `ar' program on the hosting platform.
@end deftypevar

@deftypevar string {c::*ecl-include-directory*}
Directory where the ECL header files for the target platform are located.
@end deftypevar

@deftypevar string {c::*ecl-library-directory*}
Directory where the ECL library files for the target platform are located.
@end deftypevar
