Interpreted closures currently just grab a reference to the complete
lexical environment, so (lambda (x) (+ x y)) can end up looking like
(closure ((foo ...) (y 7) (bar ...) ...)
(x) (+ x y))
where the foo/bar/... bindings are not only useless but can prevent
the GC from collecting that memory (i.e. it's a representation that is
not "safe for space") and it can also make that closure "unwritable"
(or more specifically, it can cause the closure's print
representation to be u`read`able).
Compiled closures don't suffer from this problem because `cconv.el`
actually looks at the code and only stores in the compiled closure
those variables which are actually used.
So, we fix this discrepancy by letting the existing code in `cconv.el` tell
`Ffunction` which variables are actually used by the body of the
function such that it can filter out the irrelevant elements and
return a closure of the form:
(closure ((y 7)) (x) (+ x y))
* lisp/loadup.el: Preload `cconv` and set
`internal-filter-closure-env-function` once we have a usable `cconv-fv`.
* lisp/emacs-lisp/bytecomp.el (byte-compile-preprocess): Adjust to new
calling convention of `cconv-closure-convert`.
(byte-compile-not-lexical-var-p): Delete function, moved to `cconv.el`.
(byte-compile-bind): Use `cconv--not-lexical-var-p`.
* lisp/emacs-lisp/cconv.el (cconv--dynbound-variables): New var.
(cconv-closure-convert): New arg `dynbound-vars`
(cconv--warn-unused-msg): Remove special case for `ignored`,
so we don't get confused when a function uses an argument called
`ignored`, e.g. holding a list of things that it should ignore.
(cconv--not-lexical-var-p): New function, moved from `bytecomp.el`.
Don't special case keywords and `nil` and `t` since they are already
`special-variable-p`.
(cconv--analyze-function): Use `cconv--not-lexical-var-p`.
(cconv--dynbindings): New dynbound var.
(cconv-analyze-form): Use `cconv--not-lexical-var-p`.
Remember in `cconv--dynbindings` the vars for which we used
dynamic scoping.
(cconv-analyze-form): Use `cconv--dynbound-variables` rather than
`byte-compile-bound-variables`.
(cconv-fv): New function.
* src/eval.c (Fsetq, eval_sub): Remove optimization designed when
`lexical-binding == nil` was the common case.
(Ffunction): Use `internal-filter-closure-env-function` when available.
(eval_sub, Ffuncall): Improve error info for `excessive_lisp_nesting`.
(internal-filter-closure-env-function): New defvar.
Recent changes have caused bootstrapping to fail for certain
configurations, and it was likely getting close to the limits
for others. This change raises the limits to those previously
used when configured for nativecomp:
max-specpdl-size raised from 1800 to 2500
max-lisp-eval-depth raised from 800 to 1600
* src/eval.c (init_eval_once): Raise limits.
* doc/lispref/eval.texi (Eval):
* doc/lispref/variables.texi (Local Variables): Document new values.
Setting backtrace-on-redisplay-error to non-nil enables the generation of a
Lisp backtrace in buffer *Redisplay-trace* following an error in Lisp called
from redisplay.
* doc/lispref/debugging.texi (Debugging Redisplay): New subsection.
(Error Debugging): Reference to the new subsection.
* etc/NEWS: New entry for the new facility.
* src/eval.c (redisplay_deep_handler): New variable.
(init_eval): Initialize redisplay_deep_handler.
(call_debugger): Don't throw to top-level after calling debug-early
(internal_condition_case_n): "Bind" redisplay_deep_handler to the current
handler.
(backtrace_yet): New boolean variable.
(signal_or_quit): New code section to handle Lisp errors occurring in
redisplay.
(syms_of_eval): New DEFVAR_BOOL backtrace-on-redisplay-error.
* src/keyboard.c (command_loop_1): Set backtrace_yet to false each time around
the loop.
(safe_run_hooks_error): Allow args to be up to four Lisp_Objects long.
(safe_run_hooks_2): New function.
* src/lisp.h (top level): declare as externs backtrace_yet and
safe_run_hooks_2.
* src/xdisp.c (run_window_scroll_functions): Replace a call to
run_hook_with_args_2 with one to safe_run_hooks_2.
* src/fns.c (Frequire):
* src/eval.c (Fautoload_do_load): Avoid further errors while
outputting the error about not being able to autoload/require
while bootstrapping.
This fixes errors caused by invalid error traps being left on
the error handler stack if an IO error causes a non-local exit
out of the protected code, and another crash caused by
delete_frame trying to read async input.
* src/eval.c (unwind_to_catch, push_handler_nosignal): Save and
restore the X error handler stack.
* src/lisp.h (struct handler): [HAVE_X_WINDOWS]: New field
`x_error_handler_depth'.
* src/xterm.c (struct x_error_message_stack): Make string a
regular string.
(x_unwind_errors_to): New function.
(x_error_catcher, x_catch_errors_with_handler)
(x_uncatch_errors_after_check, x_uncatch_errors): Update the
stack depth.
(x_check_errors): Stop manually unwinding since unwind_to_catch
now does that for us.
(x_had_errors_p, x_clear_errors): Update for new type of
`string'.
(x_connection_closed): Block input between just before
delete_frame to when the terminal is unlinked.
* src/xterm.h: Update prototypes.
The bytecode interpreter can't directly call special forms, so
the byte-compiler usually converts special forms into some sequence of
byte codes (basically, providing a duplicate definition of the special
form). There are still two exceptions to this: `defconst` and `defvar`,
where the compiler instead generates a convoluted chunk of code like:
(funcall '(lambda (x) (defvar <sym> x <doc>)) <value>)
where the quote makes sure we keep the function non-compiled, so as
to end up running the special form at run time.
Get rid of this workaround by introducing `defvar-1` and `defconst-1`
which provide a *functional* interface to the functionality of the
corresponding special form.
* src/eval.c (defvar, Fdefvar_1, Fdefconst_1): New functions, extracted from
`Fdefvar` and `Fdefconst`.
(Fdefvar, Fdefconst): Use them.
(syms_of_eval): `defsubr` the new functions.
* lisp/emacs-lisp/bytecomp.el (byte-compile-tmp-var): Delete const.
(byte-compile-defvar): Simplify using the new functions.
* doc/lispref/variables.texi (Defining Variables): Adjust the doc of
`defvar` to reflect the actual semantics implemented.
It's used by `interactive-form` when it encounters an OClosure.
This lets one compute the `interactive-form` of OClosures
dynamically by adding appropriate methods.
This does not include support for `command-modes` for Oclosures.
* lisp/simple.el (oclosure-interactive-form): New generic function.
* src/data.c (Finteractive_form): Delegate to
`oclosure-interactive-form` if the arg is an OClosure.
(syms_of_data): New symbol `Qoclosure_interactive_form`.
* src/eval.c (Fcommandp): Delegate to `interactive-form` if the arg is
an OClosure.
* src/lisp.h (VALID_DOCSTRING_P): New function, extracted from
`store_function_docstring`.
* src/doc.c (store_function_docstring): Use it.
* lisp/kmacro.el (kmacro): Don't carry any interactive form.
(oclosure-interactive-form) <kmacro>: New method, instead.
* test/lisp/emacs-lisp/oclosure-tests.el (oclosure-interactive-form)
<oclosure-test>: New method.
(oclosure-test-interactive-form): New test.
* doc/lispref/commands.texi (Using Interactive):
Document `oclosure-interactive-form`.
Inlining these trivial functions gives a healthy speed boost to many
common functions such as `sort`, `mapcar` etc.
* src/eval.c (call0, ..., call8): Move functions...
* src/lisp.h (call0, ..., call8): ...here and declare them inline.
Add a new `record_unwind_protect_ptr_mark` function for use with C data
structures that use the specpdl for clean-up but also contain possibly
unique references to Lisp objects.
* src/eval.c (record_unwind_protect_ptr_mark): New.
(record_unwind_protect_module, set_unwind_protect_ptr):
Set the mark function to NULL.
(mark_specpdl): Call the mark function if present.
* src/lisp.h (unwind_ptr): Add a mark function pointer to the
SPECPDL_UNWIND_PTR case.
We here just add the new type. It is not fully self-contained.
It requires cooperation from `cconv.el` on the one hand, and it
hijacks the docstring info to hold the type of OClosure objects.
This does imply that OClosures can't have docstrings, tho this
limitation will be lifted in subsequent patches.
* lisp/emacs-lisp/oclosure.el: New file.
* test/lisp/emacs-lisp/oclosure-tests.el: New file.
* doc/lispref/functions.texi (OClosures): New section.
* src/eval.c (Ffunction): Accept symbols instead of strings for docstrings.
* src/doc.c (store_function_docstring): Avoid overwriting an OClosure type.
* lisp/emacs-lisp/cconv.el (cconv--convert-function): Tweak ordering of
captured variables.
(cconv-convert): Add case for `oclosure--fix-type`.
* src/comp.c (directory_files_matching) [WINDOWSNT]: New function.
(eln_load_path_final_clean_up) [WINDOWSNT]: Use it.
This removes the need for internal_condition_case_5.
* src/eval.c (internal_condition_case_3)
(internal_condition_case_4, internal_condition_case_5): Remove.
The first two were never used; the last only in now-removed code.
Use a dedicated stack for bytecode, instead of using the C stack.
Stack frames are managed explicitly and we stay in the same
exec_byte_code activation throughout bytecode function calls and
returns. In other words, exec_byte_code no longer uses recursion
for calling bytecode functions.
This results in better performance, and bytecode recursion is no
longer limited by the size of the C stack. The bytecode stack is
currently of fixed size but overflow is handled gracefully by
signalling a Lisp error instead of the hard crash that we get now.
In addition, GC marking of the stack is now faster and more precise.
Full precision could be attained if desired.
* src/alloc.c (ATTRIBUTE_NO_SANITIZE_ADDRESS): Make non-static.
* src/bytecode.c (enum stack_frame_index, BC_STACK_SIZE)
(sf_get_ptr, sf_set_ptr, sf_get_lisp_ptr, sf_set_lisp_ptr)
(sf_get_saved_pc, sf_set_saved_pc, init_bc_thread, free_bc_thread)
(mark_bytecode, Finternal_stack_stats, valid_sp): New.
(exec_byte_code): Adapt to use the new bytecode stack.
(syms_of_bytecode): Add defsubr.
* src/eval.c (unwind_to_catch): Restore saved stack frame.
(push_handler_nosignal): Save stack frame.
* src/lisp.h (struct handler): Add act_rec member.
(get_act_rec, set_act_rec): New.
* src/thread.c (mark_one_thread): Call mark_bytecode.
(finalize_one_thread): Free bytecode thread state.
(Fmake_thread, init_threads): Set up bytecode thread state.
* src/thread.h (struct bc_thread_state): New.
(struct thread_state): Add bytecode thread state.
Pass the function object and encoded arity, not the other components.
This speeds up several call paths and is necessary for improvements to
come.
* src/bytecode.c (Fbyte_code): Make a new byte code object for
execution. This is slower but performance isn't critical here.
(exec_byte_code): Retrieve components from the passed function.
* src/eval.c (fetch_and_exec_byte_code):
* src/lisp.h (exec_byte_code): Update signature.
Keep track of the end of specpdl explicitly since that is what we are
comparing against on critical code paths.
* src/eval.c (init_eval_once_for_pdumper, signal_or_quit)
(grow_specpdl_allocation):
* src/fileio.c (Fdo_auto_save):
* src/lisp.h (grow_specpdl):
* src/thread.c (run_thread, Fmake_thread):
* src/thread.h (struct thread_state):
Replace specpdl_size with specpdl_end, according to the equation
specpdl_end = specpdl + specpdl_size.
It's critical in several function call paths.
* src/eval.c (grow_specpdl_allocation): Make non-static.
(grow_specpdl, record_in_backtrace): Move from here...
* src/lisp.h (grow_specpdl, record_in_backtrace): ... to here,
and declare inline.
Make the context switch code handle buffer-local variables more
correctly by reusing the code originally written for `backtrace-eval`.
This has the side benefit of making the `saved_value` field unused.
* src/lisp.h (enum specbind_tag): Remove `saved_value` field.
(rebind_for_thread_switch, unbind_for_thread_switch): Delete decls.
(specpdl_unrewind): Declare function.
* src/eval.c (specpdl_saved_value): Delete function.
(specbind): Delete the code related to `saved_value`, and consolidate
common code between the different branches.
(rebind_for_thread_switch, -unbind_for_thread_switch): Move to `thread.c`.
(specpdl_unrewind): New function, extracted from `backtrace_eval_unrewind`.
Use `SET_INTERNAL_THREAD_SWITCH`. Skip the buffer & excursion unwinds
depending on new arg `vars_only`.
(backtrace_eval_unrewind): Use it.
(mark_specpdl): Don't mark `saved_value`.
* src/thread.c (rebind_for_thread_switch, unbind_for_thread_switch):
Move from `eval.c` and rewrite using `specpdl_unrewind`.
* test/src/thread-tests.el (threads-test-bug48990): New test.
* test/Makefile.in (test_template): Add a + as suggested by make:
"warning: jobserver unavailable: using -j1. Add '+' to parent make rule".
This is also used in batch mode in general.
* lisp/debug-early.el (debug-early-backtrace, debug-early): New functions.
* lisp/loadup.el (top level): Load debug-early.el as first file.
* src/eval.c (signal_or_quit): Remove the condition in the batch mode section
of not being in dumping or bootstrap, since it is no longer needed. Test that
'debug-early's symbol-function is bound. Ensure there is enough working space
in specpdl and eval_depth.
(syms_of_eval): New DEFSYM for Qdebug_early. Initialise Vdebugger to
Qdebug_early rather than Qnil.
* src/eval.c (funcall_subr): Test most likely cases first (conforming
calls to finite-arity or n-adic SUBRs), and the error cases last,
instead of doing it the other way around. Simplify.
* src/eval.c (signal_or_quit): Bind the correct variable, Qdebugger (not
Vdebugger) to Qdebug in the section for errors in batch jobs.
(syms_of_eval): New DEFSYM for Qdebugger.
Rework the code we have in Fdefalias that tries to keep track
of definitions so as to be able to undo them later.
We used to store in `load-history` when an autoload is redefined as
a non-autoload and in the `autoload` symbol property we used to store
the autoload data that used to be used before it got overriden.
Instead, store the history of the function definition of
a symbol in its `function-history` symbol property.
To make this list cheap in the default case, the latest value is not stored
in the list (since it's in the `symbol-function`) and neither is the first
file. So if there's only been a single definition (the most common case),
the list is empty and the property is just not present at all.
The patch also gets rid of the `autoload` vs `defun` distinction in
`load-history` which seems unnecessary (a significant part of the
motivation for this patch was to get rid of the special handling of
autoloads in this part of the code).
* src/data.c (add_to_function_history): New function.
(defalias): Use it. Don't add the `t` entries for autoloads and always
use `defun` regardless of the kind of definition.
Change `Vautoload_queue` to only hold the function
symbols since the rest is now available from `function-history`.
* src/eval.c (un_autoload): Adjust accordingly.
* src/lread.c (load-history): Udate docstring.
* lisp/loadhist.el (loadhist-unload-filename): New var.
(unload-feature): Bind it.
(loadhist-unload-element): Document its availability.
(loadhist--restore-autoload): Delete var.
(loadhist--unload-function): Delete function.
(loadhist-unload-element): Delete the `t` and `autoload` methods.
Rewrite the `defun` method using `function-history`.
* lisp/help-fns.el: Require `seq`.
(help-fns--autoloaded-p): Rewrite.
(help-fns-function-description-header): Adjust call accordingly.
* doc/lispref/loading.texi (Where Defined): Remove `autoload` and `t`
entries from `load-history` since we don't generate them any more.
Document the `function-history` which replaces the `autoload` property.
(Unloading): Adjust symbol property name accordingly.
* test/lisp/loadhist-resources/loadhist--bar.el:
* test/lisp/loadhist-resources/loadhist--foo.el: New files.
* test/lisp/loadhist-tests.el (loadhist-tests-unload-feature-nested)
(loadhist-tests-unload-feature-notnested): New tests.
* src/data.c (defalias): New function, extracted from `Fdefalias`.
(Fdefalias): Use it.
(Ffset): Don't handle `Vautoload_queue` here, handle it in
`defalias` instead.
* src/comp.c (comp--register-subr): Use `defalias` instead of
duplicating its code.
* src/eval.c (load_with_autoload_queue): New function, extracted from
`Fautoload_do_load`.
(Fautoload_do_load): Use it.
(un_autoload): Mark it as static.
* src/fns.c (Frequire): Use it as well.
* src/lisp.h (defalias, load_with_autoload_queue): New declarations.
(un_autoload): Remove declaration.
Since `maybe_quit` is called on many critical paths, inline a slightly
cheaper condition (that ignores Vinhibit_quit).
* src/eval.c (maybe_quit): Rename to `probably_quit`.
* src/lisp.h (maybe_quit): New simplified inline function.
The common case is just to increment `specpdl_ptr`; do that in-line,
but move the uncommon reallocation to a separate subroutine.
* src/eval.c (grow_specpdl): Now inline, most code moved...
(grow_specpdl_allocation): ...here.
* src/eval.c (funcall_general, Ffuncall): Delegate the actual work in
Ffuncall to funcall_general which does exactly this.
This slows down some less used function call paths by a small amount
but the code duplication was just silly.
Since we pass no arguments to a non-lexbind bytecode function, we can
specify its arity as 0 instead of nil and save a test and branch.
* src/bytecode.c (Fbyte_code, exec_byte_code):
* src/eval.c (fetch_and_exec_byte_code, funcall_lambda):
* src/lisp.h:
Change the args_template parameter type to ptrdiff_t, since it is now
always a small integer, in exec_byte_code and
fetch_and_exec_byte_code, all callers adjusted.
Inline parts of the code for function calls to speed up the common
case of calling lexbound byte-code. By eliminating intermediate
functions, this also reduces C stack usage a little.
* src/bytecode.c (exec_byte_code): Inline parts of Ffuncall,
funcall_lambda and fetch_and_exec_byte_code in the Bcall opcode
handler.
* src/eval.c (backtrace_debug_on_exit): Inline and move to lisp.h.
(do_debug_on_call): Make global so that it can be called from
bytecode.c.
(funcall_general): New function, essentially the meat of Ffuncall.
* src/lisp.h (backtrace_debug_on_exit): Moved here from eval.c.
1. Check the type (symbol with position) of the argument given to the native
compiled version of SYMBOL_WITH_POS_SYM.
2. Handle infinite recursion caused by circular lists, etc., in
macroexp-strip-symbol-positions by using hash tables.
3. Read byte compiled functions without giving symbols positions.
* lisp/emacs-lisp/comp.el (comp-finalize-relocs): Add symbol-with-pos-p into
the list of relocated symbols.
* lisp/emacs-lisp/macroexp.el (macroexp--ssp-conses-seen)
(macroexp--ssp-vectors-seen, macroexp--ssp-records-seen): Renamed, and
animated as hash tables.
(macroexp--strip-s-p-2): Optionally tests for the presence of an argument in
one of the above hash tables, so as to handle otherwise infinite recursion.
(byte-compile-strip-s-p-1): Add a condition-case to handle infinite recursion
caused by circular lists etc., using the above hash tables as required.
* src/comp.c (comp_t): New element symbol_with_pos_sym.
(emit_SYMBOL_WITH_POS_SYM): Amend just to call the new SYMBOL_WITH_POS_SYM.
(emit_CHECK_SYMBOL_WITH_POS, define_SYMBOL_WITH_POS_SYM): New functions.
(Fcomp__init_ctxt): Register an emitter for Qsymbol_with_pos_p.
(Fcomp__compile_ctxt_to_file): Call define_SYMBOL_WITH_POS_SYM.
(syms_of_comp): Define Qsymbol_with_pos_p.
* src/data.c (syms_of_data): Define a new error symbol Qrecursion_error, an
error category for the new error symbols Qexcessive_variable_binding and
Qexcessive_lisp_nesting.
* src/eval.c (grow_specpdl): Change the signal_error call to an xsignal0 call
using the new error symbol Qexcessive_variable_binding.
(eval_sub, Ffuncall): Change the `error' calls to xsignal using the new error
symbol Qexcessive_lisp_nesting.
* src/lread.c (read1): When reading a compiled function, read the components
of the vector without giving its symbols a position.
