@node Foreign Function Interface
@section Foreign Function Interface
@cindex Foreign function interface
@ftindex FFI

@menu
* What is a FFI?:: FFI introduction
* Two kinds of FFI:: ECL's FFI brief description
* Foreign objects:: Handling the foreign objects
* Higher level interfaces:: Usage examples
* SFFI Reference:: Static FFI reference
@c * LFFI Reference:: Library FFI reference /dlopen/
* DFFI Reference:: Dynamic FFI reference
* UFFI Reference:: UFFI reference manual
@end menu

@node What is a FFI?
@subsection What is a FFI?
A Foreign Function Interface, or FFI for short, is a means for a programming language to
interface with libraries written in other programming languages, the foreign code. You will
see this concept most often being used in interpreted environments, such as Python, Ruby or
Lisp, where one wants to reuse the big number of libraries written in C and C++ for dealing
with graphical interfaces, networking, filesystems, etc.

A FFI is made of at least three components:

@table @dfn
@item Foreign objects management
This is the data that the foreign code will use. A FFI needs to provide
means to build and manipulate foreign data, with automatic conversions
to and from lisp data types whenever possible, and it also has to deal
with issues like garbage collection and finalization.
@item Foreign code loader
To actually use a foreign routine, the code must reside in memory. The
process of loading this code and finding out the addresses of the
routines we want to use is normally done by an independent component.
@item Foreign function invocation
This is the part of the FFI that deals with actually calling the foreign
routines we want to use. For that one typically has to tell the FFI what
are the arguments that these routines expect, what are the calling
conventions and where are these routines to be found.
@end table

On top of these components sits a higher level interface written
entirely in lisp, with which you will actually declare and use foreign
variables, functions and libraries. In the following sections we
describe both the details of the low-level components (@xref{Two kinds
of FFI} and @ref{Foreign objects}), and of the higher level interface
(@xref{Higher level interfaces}). It is highly recommended that you read
all sections.

@node Two kinds of FFI
@subsection Two kinds of FFI
@cindex Two kinds of FFI
ECL allows for two different approaches when building a FFI. Both
approaches have a different implementation philosophy and affect the
places where you can use the FFI and how.

@table @dfn
@item Static FFI (SFFI)
For every foreign function and variable you might need to use, a wrapper
is automatically written in C with the help of ffi:c-inline. These
wrappers are compiled using an ordinary C compiler and linked against
both the foreign libraries you want to use and against the ECL
library. The result is a FASL file that can be loaded from ECL and where
the wrappers appear as ordinary lisp functions and variables that the
user may directly invoked.

@c XXX: should we document DLOPEN from the FFI perspective here?
@c @item Library FFI (LFFI)

@item Dynamic FFI (DFFI)
@cfindex --with-libffi-prefix=path
@cfindex --with-dffi [system|included|AUTO|no]

First of all, the foreign libraries are loaded in memory using the
facilities of the operating system. Similar routines are used to find
out and register the memory location of all the functions and variables
we want to use. Finally, when actually accessing these functions, a
little piece of assembly code does the job of translating the lisp data
into foreign objects, storing the arguments in the stack and in CPU
registers, calling the function and converting back the output of the
function to lisp.

ECL for this purpose utilizes
@emph{@uref{https://sourceware.org/libffi/, libffi}}, a portable
foreign-function interface library.
@end table

@float Figure,fig:ffi_components
@caption{FFI components}
@image{figures/ffi,,1.5in}
@end float

As you see, the first approach uses rather portable techniques based on a
programming language (C, C++) which is strongly supported by the
operating system. The conversion of data is performed by a calling
routines in the ECL library and we need not care about the precise
details (organizing the stack, CPU registers, etc) when calling a
function: the compiler does this for us.

On the other hand, the dynamic approach allows us to choose the
libraries we load at any time, look for the functions and invoke them
even from the toplevel, but it relies on unportable techniques and
requires the developers to know very well both the assembly code of the
machine the code runs on and the calling conventions of that particular
operating system. For these reasons ECL doesn't maintain it's own
implementation of the DFFI but rather relies on the third party library.

ECL currently supports the static method on all platforms, and the
dynamical one a wide range of the most popular ones, shown in the
section @emph{Supported Platforms} at
@url{https://sourceware.org/libffi/}.

@ftindex DFFI
You can test if your copy of ECL was built with DFFI by inspecting
whether the symbol @code{:dffi} is present in the list from variable
@code{*features*}.

@node Foreign objects
@subsection Foreign objects

While the foreign function invocation protocols differ strongly between
platforms and implementations, foreign objects are pretty easy to handle
portably. For ECL, a foreign object is just a bunch of bytes stored in
memory. The lisp object for a foreign object encapsulates several bits
of information:

@itemize @bullet
@item
A list or a symbol specifying the C type of the object.
@item
The pointer to the region of memory where data is stored.
@item
A flag determining whether ECL can automatically manage that piece of
memory and deallocated when no longer in use.
@end itemize

A foreign object may contain many different kinds of data: integers,
floating point numbers, C structures, unions, etc. The actual type of
the object is stored in a list or a symbol which is understood by the
higher level interface (@xref{Higher level interfaces}).

The most important component of the object is the memory region where
data is stored. By default ECL assumes that the user will perform
manual management of this memory, deleting the object when it is no
longer needed. The first reason is that this block may have been
allocated by a foreign routine using @code{malloc()}, or
@code{mmap()}, or statically, by referring to a C constant. The second
reason is that foreign functions may store references to this memory
which ECL is not aware of and, in order to keep these references
valid, ECL should not attempt to automatically destroy the object.

In many cases, however, it is desirable to automatically destroy foreign
objects once they have been used. The higher level interfaces UFFI and
@url{https://common-lisp.net/project/cffi/,CFFI} provide tools for doing
this. For instance, in the following example adapted from the UFFI
documentation, the string @var{name} is automatically deallocated

@lisp
(ffi:def-function ("gethostname" c-gethostname)
  ((name (* :unsigned-char))
   (len :int))
  :returning :int)

(ffi:with-foreign-object (name '(:array :unsigned-char 256))
    (if (zerop (c-gethostname (ffi:char-array-to-pointer name) 256))
        (format t "Hostname: ~S" (ffi:convert-from-foreign-string name))
        (error "gethostname() failed.")))
@end lisp

@subsubsection C Reference

@cppindex ecl_make_pointer
@cppdef ecl_make_foreign_data
@deftypefun cl_object ecl_make_foreign_data (cl_object tag, cl_index size, void *data)

@paragraph Description
This function creates a Lisp ``foreign object'' that points to a C data. Use
this function to pass a data from C to Lisp.

@var{tag} denotes the data type (@xref{Primitive Types})
@var{size} is a number of elements in @var{data} (or 0 for a raw pointer)
@var{data} is a pointer to C data (either an object or an array)

The C macro @code{ecl_make_pointer(pointer)} expands to
@code{ecl_make_foreign_data(ECL_NIL, 0, (pointer))}.
@end deftypefun

@cppdef ecl_to_pointer
@cppdef ecl_foreign_data_pointer_safe
@deftypefun void *ecl_foreign_data_pointer_safe(cl_object f)

@paragraph Description
This function returns a C pointer for the given Lisp foreign object. Lisp
foreign objects are constructed with functions @code{ecl_make_foreign_data}
and @code{ecl_allocate_foreign_data}.

The wrapper created with @code{ecl_make_foreign_data} is a subject of
garbage collection, but the pointer itself is not ``freed'', because
the data producer is ``C world'' and the consumer is ``Lisp world''.

This is different from @code{si:allocate-foreign-object} where the
data producer is ``Lisp world'' and the consumer is ``C world''. In
that case the wrapper is not collected unless explicitly released with
@code{si:free-foreign-object} in which case the allocated foreign data
is also released.
@end deftypefun

@cppdef ecl_base_string_pointer_safe

@deftypefun char *ecl_base_string_pointer_safe(cl_object f)

@paragraph Description
This function returns a pointer to a simple base string @var{f}. If @var{f} is
not a simple base string this function signals an error.

@end deftypefun

@cppdef ecl_null_terminated_base_string
@deftypefun cl_object ecl_null_terminated_base_string(cl_object s)

@paragraph Description
Tries to coerce a string to a simple base string suitable for
@code{ecl_base_string_pointer_safe}. This function may cons data.

@end deftypefun

@c FIXME these functions operate on the enum ecl_ffi_tag but other functions
@c operate on keyword tags (see ecl_make_foreign_data). Also both functions
@c are part of the exported interface, but conversion keyword->ecl_ffi_tag is
@c done by the function ecl_foreign_type_code that is declared in internal.h.
@c
@c Do one of either:
@c 1. move ecl_foreign_data_*_elt to internal.h
@c 2. move ecl_foreign_type_code from internal.h to external.h
@c 3. document the ecl_ffi_tag enum and change the ecl_make_foreign_data
@c 4. change functions ecl_foreign_data_*_elt to work on tags
@c
@c The cleanest solution will be 4. Perhaps to remain backward compatible with
@c people who used ecl_foreign_data_*_elt we should make these two C macros
@c and expand to a new function interface using tags. Until this is fixed
@c let's not advertise these functions in the documentation. -- jd 2021-11-22

@c @cppdef ecl_foreign_data_ref_elt
@c @cppdef ecl_foreign_data_set_elt
@c @deftypefun cl_object ecl_foreign_data_ref_elt(void *p, enum ecl_ffi_tag tag)
@c @deftypefunx cl_object ecl_foreign_data_set_elt(void *p, enum ecl_ffi_tag tag, cl_object value)

@c @paragraph Description
@c These functions are used to operate directly on C data in @var{p}.
@c The referenced C object is coerced to a Lisp type and returned.
@c The Lisp value that is set to C object is coerced to a C type.

@c @end deftypefun

@node Higher level interfaces
@subsection Higher level interfaces
Up to now we have only discussed vague ideas about how a FFI works, but you are probably more
interested on how to actually code all these things in lisp. You have here three
possibilities:

@itemize @bullet
@item
ECL supplies a high level interface which is compatible with UFFI up to
version 1.8 (api for >=v2.0 is provided by cffi-uffi-compat system
shipped with CFFI). Code designed for UFFI library should run mostly
unchanged with ECL. Note, that api resides in ffi package, not uffi, to
prevent conflicts with cffi-uffi-compat. New code shouldn't use this
interface preferring @url{https://common-lisp.net/project/cffi/,CFFI}.
@item
The @url{https://common-lisp.net/project/cffi/,CFFI} library features a
complete backend for ECL. This method of interfacing with the foreign
libraries is preferred over using UFFI.
@item
ECL's own low level interface. Only to be used if ECL is your deployment
platform. It features some powerful constructs that allow you to mix
arbitrary C and lisp code.
@end itemize

In the following two subsections we will discuss two practical examples of using the native
UFFI and the CFFI library.

@c XXX: we should describe here, how to use SFFI for interactive C/C++ development
@c @node Lower level interfaces

@subsubheading UFFI example
@exindex UFFI usage

The example below shows how to use UFFI in an application. There are several important
ingredients: 

@itemize @bullet
@item
You need to specify the libraries you use and do it at the toplevel, so
that the compiler may include them at link time.
@item
Every function you will use has to be declared using
@coderef{ffi:def-function}.
@item
In the cases of headers not used by ECL, a header to include might need
to be specified using @coderef{ffi:clines}.
@end itemize

@lisp
#|
Build and load this module with (compile-file "uffi.lsp" :load t)
|#
;;
;; This toplevel statement notifies the compiler that we will
;; need this shared library at runtime. We do not need this
;; statement in windows or modern macOS.
;; The actually needed path to libm might be different on different systems.
;;
#-(or ming32 windows darwin)
(ffi:load-foreign-library "/usr/lib/libm.so")
;;
;; With this other statement, we import the C function sin(),
;; which operates on IEEE doubles.
;;
(ffi:def-function ("sin" c-sin) ((arg :double))
                  :returning :double)
;;
;; We now use this function and compare with the lisp version.
;;
(format t "~%Lisp sin:~t~d~%C sin:~t~d~%Difference:~t~d"
	(sin 1.0d0) (c-sin 1.0d0) (- (sin 1.0d0) (c-sin 1.0d0)))
@end lisp

@subsubheading CFFI example
@exindex CFFI usage

The @url{https://common-lisp.net/project/cffi/,CFFI} library is an
independent project and it is not shipped with ECL. If you wish to use
it you can go to their homepage, download the code and build it using
ASDF.

CFFI differs slightly from UFFI in that functions may be used even
without being declared beforehand.

@lisp
#|
Build and load this module with (compile-file "cffi.lsp" :load t)
|#
;;
;; This toplevel statement notifies the compiler that we will
;; need this shared library at runtime. We do not need this
;; statement in windows or macOS.
;;
#-(or ming32 windows darwin)
(cffi:load-foreign-library "/usr/lib/libm.so")
;;
;; With this other statement, we import the C function sin(),
;; which operates on IEEE doubles.
;;
(cffi:defcfun ("sin" c-sin) :double '(:double))
;;
;; We now use this function and compare with the lisp version.
;;
(format t "~%Lisp sin:~t~d~%C sin:~t~d~%Difference:~t~d"
	(sin 1.0d0) (c-sin 1.0d0) (- (sin 1.0d0) (c-sin 1.0d0)))
;;
;; The following also works: no declaration!
;;
(let ((c-cos (cffi:foreign-funcall "cos" :double 1.0d0 :double)))
   (format t "~%Lisp cos:~t~d~%C cos:~t~d~%Difference:~t~d"
	(cos 1.0d0) c-cos (- (cos 1.0d0) c-cos)))
@end lisp

@subsubheading SFFI example (low level inlining)
@exindex SFFI usage

To compare with the previous pieces of code, we show how the previous programs would be
written using @coderef{ffi:clines} and @coderef{ffi:c-inline}.

@lisp
#|
Build and load this module with (compile-file "ecl.lsp" :load t)
|#
;;
;; With this other statement, we import the C function sin(), which
;; operates on IEEE doubles. Notice that we include the C header to
;; get the full declaration.
;;
(defun c-sin (x)
  (ffi:clines "#include <math.h>")
  (ffi:c-inline (x) (:double) :double "sin(#0)" :one-liner t))
;;
;; We now use this function and compare with the lisp version.
;;
(format t "~%Lisp sin:~t~d~%C sin:~t~d~%Difference:~t~d"
	(sin 1.0d0) (c-sin 1.0d0) (- (sin 1.0d0) (c-sin 1.0d0)))
@end lisp

@include extensions/ffi_sffi.txi
@include extensions/ffi_dffi.txi
@include extensions/ffi_uffi.txi
