Add line-column tracking for tree-sitter parsers. Copied from
comments in treesit.c:
Technically we had to send tree-sitter the line and column
position of each edit. But in practice we just send it dummy
values, because tree-sitter doesn't use it for parsing and
mostly just carries the line and column positions around and
return it when e.g. reporting node positions[1]. This has
been working fine until we encountered grammars that actually
utilizes the line and column information for
parsing (Haskell)[2].
[1] https://github.com/tree-sitter/tree-sitter/issues/445
[2] https://github.com/tree-sitter/tree-sitter/issues/4001
So now we have to keep track of line and column positions and
pass valid values to tree-sitter. (It adds quite some
complexity, but only linearly; one can ignore all the linecol
stuff when trying to understand treesit code and then come
back to it later.) Eli convinced me to disable tracking by
default, and only enable it for languages that needs it. So
the buffer starts out not tracking linecol. And when a
parser is created, if the language is in
treesit-languages-require-line-column-tracking, we enable
tracking in the buffer, and enable tracking for the parser.
To simplify things, once a buffer starts tracking linecol, it
never disables tracking, even if parsers that need tracking
are all deleted; and for parsers, tracking is determined at
creation time, if it starts out tracking/non-tracking, it
stays that way, regardless of later changes to
treesit-languages-require-line-column-tracking.
To make calculating line/column positons fast, we store
linecol caches for begv, point, and zv in the
buffer (buf->ts_linecol_cache_xxx); and in the parser object,
we store linecol cache for visible beg/end of that parser.
In buffer editing functions, we need the linecol for
start/old_end/new_end, those can be calculated by scanning
newlines (treesit_linecol_of_pos) from the buffer point
cache, which should be always near the point. And we usually
set the calculated linecol of new_end back to the buffer
point cache.
We also need to calculate linecol for the visible_beg/end for
each parser, and linecol for the buffer's begv/zv, these
positions are usually far from point, so we have caches for
all of them (in either the parser object or the buffer).
These positions are far from point, so it's inefficient to
scan newlines from point to there to get up-to-date linecol
for them; but in the same time, because they're far and
outside the changed region, we can calculate their change in
line and column number by simply counting how much newlines
are added/removed in the changed
region (compute_new_linecol_by_change).
* doc/lispref/parsing.texi (Using Parser): Mention line-column
tracking in manual.
* etc/NEWS: Add news.
* lisp/treesit.el:
(treesit-languages-need-line-column-tracking): New variable.
* src/buffer.c: Include treesit.h (for TREESIT_EMPTY_LINECOL).
(Fget_buffer_create):
(Fmake_indirect_buffer): Initialize new buffer fields.
(Fbuffer_swap_text): Add new buffer fields.
* src/buffer.h (ts_linecol): New struct.
(buffer): New buffer fields.
(BUF_TS_LINECOL_BEGV):
(BUF_TS_LINECOL_POINT):
(BUF_TS_LINECOL_ZV):
(SET_BUF_TS_LINECOL_BEGV):
(SET_BUF_TS_LINECOL_POINT):
(SET_BUF_TS_LINECOL_ZV): New inline functions.
* src/casefiddle.c (casify_region): Record linecol info.
* src/editfns.c (Fsubst_char_in_region):
(Ftranslate_region_internal):
(Ftranspose_regions): Record linecol info.
* src/insdel.c (insert_1_both):
(insert_from_string_1):
(insert_from_gap_1):
(insert_from_buffer):
(replace_range):
(del_range_2): Record linecol info.
* src/treesit.c (TREESIT_BOB_LINECOL):
(TREESIT_EMPTY_LINECOL):
(TREESIT_TS_POINT_1_0): New constants.
(treesit_debug_print_linecol):
(treesit_buf_tracks_linecol_p):
(restore_restriction_and_selective_display):
(treesit_count_lines):
(treesit_debug_validate_linecol):
(treesit_linecol_of_pos):
(treesit_make_ts_point):
(Ftreesit_tracking_line_column_p):
(Ftreesit_parser_tracking_line_column_p): New functions.
(treesit_tree_edit_1): Accept real TSPoint and pass to
tree-sitter.
(compute_new_linecol_by_change): New function.
(treesit_record_change_1): Rename from treesit_record_change,
handle linecol if tracking is enabled.
(treesit_linecol_maybe): New function.
(treesit_record_change): New wrapper around
treesit_record_change_1 that handles some boilerplate and sets
buffer state.
(treesit_sync_visible_region): Handle linecol if tracking is
enabled.
(make_treesit_parser): Setup parser's linecol cache if tracking
is enabled.
(Ftreesit_parser_create): Enable tracking if the parser's
language requires it.
(Ftreesit__linecol_at):
(Ftreesit__linecol_cache_set):
(Ftreesit__linecol_cache): New functions for debugging and
testing.
(syms_of_treesit): New variable
Vtreesit_languages_require_line_column_tracking.
* src/treesit.h (Lisp_TS_Parser): New fields.
(TREESIT_BOB_LINECOL):
(TREESIT_EMPTY_LINECOL): New constants.
* test/src/treesit-tests.el (treesit-linecol-basic):
(treesit-linecol-search-back-across-newline):
(treesit-linecol-col-same-line):
(treesit-linecol-enable-disable): New tests.
* src/lisp.h: Declare display_count_lines.
* src/xdisp.c (display_count_lines): Remove static keyword.
* etc/NEWS: Announce 'window-restore-killed-buffer-windows'.
* src/buffer.h (struct buffer) : New field last_name_.
* src/buffer.c (Fbuffer_last_name): New function to return last
name of buffer before it was killed or renamed.
(bset_last_name, Fget_buffer_create, Fmake_indirect_buffer)
(Frename_buffer, Fkill_buffer, init_buffer_once): Set buffer's
last_name_ field accordingly.
* src/window.c (window_restore_killed_buffer_windows): New
variable replacing Vwindow_kept_windows_functions.
(Fset_window_configuration): Use
window_restore_killed_buffer_windows instead of
Vwindow_kept_windows_functions.
* lisp/window.el (window--state-put-2, window-state-put): Use
'window-restore-killed-buffer-windows' instead of
'window-kept-windows-functions'.
* doc/lispref/windows.texi (Window Configurations): Describe
'window-restore-killed-buffer-windows' which replaces
'window-kept-windows-functions'.
Use `ITREE_FOREACH` instead of `overlays_in` if that can save us from
allocating an array.
* src/buffer.c (overlays_in): Mark as static.
(mouse_face_overlay_overlaps): Use `ITREE_FOREACH` instead of `overlays_in`.
(disable_line_numbers_overlay_at_eob): Same, and also change return
value to a boolean.
* src/buffer.h (overlays_in): Don't declare.
* src/editfns.c (overlays_around): Delete function.
(Fget_pos_property): Use `ITREE_FOREACH` and keep the "best so far"
instead of using `overlays_in` and sorting the elements.
* src/lisp.h (disable_line_numbers_overlay_at_eob): Change return
type to a boolean.
* src/xdisp.c (should_produce_line_number): Adjust accordingly.
Be more systematic about putting space before paren in calls,
and in avoiding unnecessary parentheses in macros.
This was partly inspired by my wading through gcc -E output
while debugging something else, and seeing too many parens.
This patch does not change the generated .o files on my platform.
* src/buffer.h: Remove case_fold_search_ buffer object slot.
* src/buffer.c (bset_case_fold_search): Remove - no longer needed.
(init_buffer_once): Remove removed buffer slot init.
(syms_of_buffer): Use DEFVAR_LISP to define `case-fold-search' and
declare it buffer-local.
* src/minibuf.c (syms_of_minibuf): Remove DEFSYM call for
`case-fold-search' symbol. It now lives in `syms_of_buffer'.
* src/editfns.c (Fcompare_buffer_substrings):
(Fchar_equal):
* src/search.c (looking_at_1):
(string_match_1):
(search_command):
(Fre__describe_compiled): Adjust C queries to `case-fold-search' value
to use C globals instead of BVAR macro.
* doc/lispref/internals.texi (Buffer Internals): Do not list
`case_fold_search' slot.
See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=66117#259
When used as buffer slot, let-binding `case-fold-search' would scale
with the number of live buffers and can be slow. This change makes
let-binding much faster at the cost of slightly slower `set-buffer'.
cae528457c ; Add 2023 to copyright years.
b394359261 Improve documentation of 'isearch-open-overlay-temporary'
ab3210e709 Document 'use-package' in the 2 main manuals
# Conflicts:
# etc/refcards/ru-refcard.tex
# lib/explicit_bzero.c
# m4/explicit_bzero.m4
This fixes bug#58928. The patch works by moving the
`evaporate_overlays` check closer to `itree_delete_gap`.
* src/buffer.c (adjust_overlays_for_delete_in_buffer): New function,
integrating the functionality of `evaporate_overlays`.
(adjust_overlays_for_delete): Use it.
(evaporate_overlays): Delete function.
* src/buffer.h (evaporate_overlays): Delete declaration.
* src/insdel.c (adjust_markers_for_replace): Minor optimization.
(adjust_after_replace, replace_range, replace_range_2, del_range_2):
Don't call `evaporate_overlays`.
* test/src/buffer-tests.el (buffer-tests--overlays-indirect-evaporate):
Remove the `:expected-result :failed` attribute.
See bug#58703.
* src/buffer.c (overlays_in): Add a new TRAILING arg expressing the
behavior wanted by `overlays-at', namely to include all overlays
beginning at the POS passed to `overlays-at', even if POS is the end
of the narrowed region. Pass true and the search range is extended to
ZV+1 if END is greater than ZV, just as is done for EMPTY.
(overlays_at): Pass 'true' for the new trailing arg. At present this
is the only caller passing 'true'.
(mouse_face_overlay_overlaps): Pass 'false' for the new trailing arg.
(disable_line_numbers_overlay_at_eob): ditto.
(Foverlays_in): ditto.
* src/editfns.c (overlays_around): ditto.
* test/src/buffer-tests.el (sorted-overlays): Add a spot test for
this.
For the most part, I replaced the interval_tree_ prefix with itree_,
interval_node_ with itree_node_, etc.
* src/alloc.c: Rename everywhere as appropriate.
* src/alloc.c: ditto.
* src/buffer.c: ditto.
* src/buffer.h: ditto.
* src/itree.c: ditto.
* src/itree.h: ditto.
Correct `get-char-property` which failed to ignore empty overlays.
* src/textprop.c (get_char_property_and_overlay): Make sure the
overlay covers the character.
* test/src/buffer-tests.el (buffer-tests--overlay-bug58479): New test.
* src/buffer.h (buffer_has_overlays): Avoid `interval_tree_size`.
Move args between `build_overlay` and `add_buffer_overlay`,
to try and keep buffer positions together with their buffer.
Be more strict in the `otick` values passed to `interval_tree_insert`.
Move a few things around to try and reduce dependencies through `.h` files.
Fix a thinko bug in `check_tree`.
* src/alloc.c (build_overlay): Remove `begin` and `end` args.
* src/buffer.c (add_buffer_overlay): Move from `buffer.h`.
Add `begin` and `end` args.
(copy_overlays): Adjust accordingly.
(Fmake_overlay): Use BUF_BEG and BUF_Z; adjust call to `build_overlay`
and `add_buffer_overlay`.
(Fmove_overlay): Use BUF_BEG and BUF_Z; Use the new `begin` and `end`
args of `add_buffer_overlay` so we don't need to use
`interval_node_set_region` when moving to a new buffer.
(remove_buffer_overlay, set_overlay_region): Move from `buffer.h`.
* src/buffer.h (set_overlay_region, add_buffer_overlay)
(remove_buffer_overlay): Move to `buffer.c`.
(build_overlay): Move from `lisp.h`.
(maybe_alloc_buffer_overlays): Delete function (inline into its only
caller).
* src/itree.c (interval_tree_insert): Move declaration `from buffer.h`.
(check_tree): Fix initial offset in call to `recurse_check_tree`.
Remove redundant check of the `limit` value.
(interval_node_init): Remove `begin` and `end` args.
(interval_tree_insert): Mark it as static.
Assert that the new node's `otick` should already be uptodate and its
new parent as well.
(itree_insert_node): New function.
(interval_tree_insert_gap): Assert the otick of the removed+added nodes
were uptodate and mark them as uptodate again after adjusting
their positions.
(interval_tree_inherit_offset): Check that the parent is at least as
uptodate as the child.
* src/lisp.h (build_overlay): Move to `buffer.h`.
* src/itree.h (interval_node_init): Adjust accordingly.
(interval_tree_insert): Remove declaration.
(itree_insert_node): New declaration.
* src/buffer.c (delete_all_overlays): Add comment.
* src/itree.c (struct interval_generator): New fields `running`,
`file`, and `line` moved from `interval_tree`.
(interval_stack_push_flagged): Adjust comment to resolve a FIXME.
(interval_tree_clear): Replace assignment with an a
(interval_tree_iter_next): Delete function.
(interval_tree_clear): Don't set `iter_running` here any more.
(interval_generator_create): Set it here instead.
(interval_tree_iter_start): Fetch `iter` once and for all.
(interval_generator_narrow): Mark it as non-static.
(interval_tree_iter_next, interval_tree_iter_narrow):
Delete functions. Inline their old bodies in the callers.
(interval_tree_iter_finish): Take the iter rather than
the whole tree. Adjust all callers.
(interval_generator_next): Move `running `assertion here from
`interval_tree_iter_next`.
* src/buffer.h: Adjust accordingly.
* src/itree.h (struct interval_tree): Remove fields `iter_running`,
`file`, and `line`, moved to `interval_generator`.
(interval_generator_narrow): Replace `interval_tree_iter_narrow`.
When starting an iteration, store __FILE__ and __LINE__ where this
happens in the interval_tree structure.
* src/buffer.h (buffer_overlay_iter_start): New macro adding __FILE
and __LINE__.
(buffer_overlay_iter_start1): Renamed from ..._start.
* src/itree.h (struct interval_tree): Add file and line info.
* src/itree.c: (interval_tree_contains, interval_tree_nodes,
interval_tree_insert_gap): Pass __FILE__ and __LINE__ to iter_start.
(interval_tree_iter_start): Record file and line info in tree.
* src/buffer.h (struct buffer_text): New field.
(BUF_CHARS_UNCHANGED_MODIFIED, CHARS_UNCHANGED_MODIFIED): New macros.
* src/buffer.c (Fget_buffer_create): Initialize the new field.
* src/xdisp.c (mark_window_display_accurate_1): Set the new field.
(redisplay_window): Use it, together with CHARS_MODIFF, instead of
MODIFF and UNCHANGED_MODIFIED to decide whether to check for long
lines.
* src/pdumper.c (dump_buffer): Dump the new field.
* src/xdisp.c (redisplay_window): Increase the threshold above which
long lines detection is performed in the buffer. This should avoid
triggering that detection for most simple editing operations.
* src/lisp.h (modiff_incr): Explain why the counter is incremented
logarithmically.
* src/buffer.h (struct buffer_text): Adapt the comment about the
'modiff' field accordingly.
* src/buffer.c (modify_overlay): Increase the counter by 1 instead of
the size of the buffer section on which the overlay is placed.
* etc/NEWS: Small improvement.
* src/buffer.h (struct buffer_text): New 'unchanged_size' field.
(BUF_UNCHANGED_SIZE): New macro to access the field.
* src/buffer.c (Fget_buffer_create): Initialize the field.
(Fbuffer_swap_text): Handle it.
* src/xdisp.c (mark_window_display_accurate_1): Set the field.
(redisplay_window): Use the field for long lines detection.
* src/buffer.h (struct buffer): New field 'long_line_optimizations_p'.
* src/buffer.c (syms_of_buffer): New variable 'long-line-threshold'.
(reset_buffer): Initialize the 'long_line_optimizations_p' field.
(Fbuffer_swap_text): Handle it.
* src/xdisp.c (redisplay_window): Set 'long_line_optimizations_p' when
a buffer contains long lines.
(init_iterator): Use 'long_line_optimizations_p'.
(get_narrowed_begv): Update.
(SET_WITH_NARROWED_BEGV): New macro.
(unwind_narrowed_begv): New internal function used by the new macro.
(back_to_previous_line_start, get_visually_first_element,
move_it_vertically_backward): Use the new macro.
* src/search.c (find_newline1): Make it externally visible.
* src/lisp.h: Make 'find_newline1' externally visible.
* src/dispextern.h (struct it): Update comment. Remove the
'WITH_NARROWED_BEGV' macro.
* etc/NEWS: Mention the 'long-line-threshold' variable.
