For my test app with 100K tiddlers and 20K tags, the tag indexer improvement gives a significant improvement:
| Metric | Before | After Tag Indexer | Improvement |
|--------|--------|-------------------|-------------|
| **Initial render** | 261.95ms | 83.51ms | **3.1x faster** |
| **Total refresh** | 2,496.25ms | 1,338.84ms | **1.9x faster (46%)** |
| **Mean refresh** | 416.04ms | 223.14ms | **1.9x faster** |
| **Max refresh** | 480.76ms | 287.26ms | **1.7x faster** |
The specific filter impact: `[subfilter{$:/core/config/GlobalImportFilter}]` went from **1,172ms** (the #1 bottleneck) to effectively **0ms** — it dropped out of the top 20 entirely. That single filter was consuming 47% of total refresh time.
The root cause was that `TagSubIndexer.prototype.update()` was setting `this.index = null` on every tiddler change, forcing a complete rebuild (iterating all 95k tiddlers) on the next tag lookup. The fix makes it incremental — just removing/adding the changed tiddler's tags — which is O(number of tags on the changed tiddler) instead of O(all tiddlers in the wiki).
* fix: apply automatic eslint fixes
* lint: allow hashbang comment for tiddlywiki.js
* lint: first back of manual lint fixes for unused vars
* lint: added more fixes for unused vars
* lint: missed files
* lint: updated eslint config with selected rules from #9669
* fix: ignore self-referential transclusion
* feat: support old <$transclude tiddler param
* fix: restore old behavior: include itself like backlinks[]
* refactor: use LinkedList in transcludes[] and backtranscludes[]
* fix: only fallback to title when {{!!xxx}}, not when input is empty
* refactor: move transcludes ast extractor to a file
* refactor: move links ast extractor to a file
* Revert "refactor: move links ast extractor to a file"
This reverts commit 5600a00cd8.
* Revert "refactor: move transcludes ast extractor to a file"
This reverts commit 61d5484f09.
* lint: use pushTop and remove space
* feat: add transcludes and backtranscludes filter and its relying indexer
* feat: add test about backtranscludes
* docs: add doc about transcludes and backtranscludes Operator
* refactor: merge backlinks and backtranscludes indexer
* fix: test not executed
* fix: latest transclude use $tiddler instead of tiddler
* feat: A tiddler transclude with template will still use the tiddler as result.
* docs: wrong comment
* Add tests for backlinks
* Add backlinks indexer
* Use backlinks indexer in getTiddlerBacklinks if available
* Extract link extraction into its own method
This way we can provide an arbitrary parse tree, rather than just a
title, which will allow us to compare lists of outgoing links between
versions of a single tiddler
* Use new extractLinks method in backlinks indexer
...rather than copy-pasting the implementation
* Remove ES6-isms
TiddlyWiki needs to work with browsers that only support ES5
Fixes#4082
This version removes selective updating of the tag index, instead completely clearing the index on each update. I'm investigating restoring that optimisation.
The test rig previously used a simplified implementation of shadow tiddlers which broke with the new indexing engine. There was also a problem that made that even if indexers were disabled they were still initialised.
This PR fixes both problems, in preparation for fixing #4082
* First pass at modular wiki indexes
An exploratory experiment
* Fix tests
* Faster checking for existence of index methods
We don't really need to check the type
* Use the index for the has operator
* Fix typo
* Move iterator index methods into indexer modules
Now boot.js doesn't know the core indexers
* Fix up the other iterator index functions
* Fix crash with missing index branch
* Limit the field indexer to values less than 128 characters
* Fallback to the old manual scan if the index method returns null
* Sadly, we can no longe re-use the field indexer to accelerate the `has` operator, because the index now omits tiddlers that have field values longer than the limit
Still need to make the index configuration exposed somehow
* Rearrange tests so that we can test with and without indexers
We also need to expose the list of enabled indexers as a config option
* Test the field indexer with different length fields
So that we test the indexed and non-indexed codepaths
