mirrors/TiddlyWiki5
1
Fork 0
mirror of https://github.com/Jermolene/TiddlyWiki5.git synced 2025-12-15 15:10:30 -08:00
Code Issues Projects Releases Packages Wiki Activity

Docs about the history of the filter syntax

Browse source
This commit is contained in:
Jeremy Ruston 2025-07-30 13:04:33 +01:00
parent 482af2c5d9
commit 29a567f0a8
5 changed files with 130 additions and 81 deletions
Download patch file Download diff file Expand all files Collapse all files

28
editions/tw5.com/tiddlers/about/Filter Syntax History.tid Normal file
View file

@ -0,0 +1,28 @@
title: Filter Syntax History
tags: [[History of TiddlyWiki]]
modifier: Jeremy Ruston
In response to [[a discussion|https://talk.tiddlywiki.org/t/filter-syntax-history/13058]] about the filter syntax in TiddlyWiki5 I posted this brief personal history.
For context, before TiddlyWikIi, I only had practical experience of a very small number of languages: machine code, assembly language, BASIC, FORTH, C, C++ and of course JavaScript. I had a smattering of Java, Pascal and one or two other ancient languages but no experience of actor based languages like Erlang, or modern functional languages.
The story starts with the double square bracket syntax used in wikitext for links. In 2004 this was already an established usage in wikis. However, I switched the ordering of pretty links because I thought Wikipedia's `[[link address|link text]]` was the wrong way around. It broke up sentences: `The file is [[https://site.com/thing|here]]` seems less readable than `The file is [[here|https://site.com/thing]]`. For a long time I regretted this decision, and wished that I had just gone with Wikipedia's established usage. Others have since pointed out that TiddlyWiki's ordering is consistent with [[Markdown]].
One way to look at the double square bracket link syntax is that it establishes a way to quote page/tiddler titles so that they may contain spaces, and don't have to use CamelCase. Thus, in TiddlyWiki Classic, when I was implementing the DefaultTiddlers feature it was natural to use double square brackets to quote titles containing spaces.
Then, when I was starting to think about TiddlyWiki 5 I wanted to extend the implementation of DefaultTiddlers so that more complex logic could be expressed while retaining backwards compatibility. A trick that I am apt to use in such situations is to try to engineer things so that the current behaviour is re-interpreted as a shortcut syntax for a new, richer syntax that provides more flexibility. In this case, the idea was that in filters we would interpret `[[mytiddler]]` as a shortcut for `[title[mytiddler]]`. Then we could put any keywords we like in place of "title", giving us an infinitely extensible syntax. A similar example is the way that we implemented filter run prefixes by retrospectively defining the absence of a prefix as implying a default prefix.
We implemented a very, very simple first version of the filter syntax in TiddlyWiki Classic that supported things like `[tag[mytag]]`, but it wasn't until TiddlyWiki 5 that it evolved into something approaching a programming language. As others have probably expressed much more eloquently, a characteristic of the programming languages that I love is that they start with a small number of principles that are consistently applied and combined. In the case of TiddlyWiki, the list would be very roughly:
* Double square brackets for linking and quoting
* Curly braces for transclusion
* Angle brackets for macros (which evolved into variables)
* Double exclamation marks to indicate fields `{{!!myfield}}`
* Double hashes to indicate indexes `{{##myindex}}`
* Smashing together adjacent filter operations by removing the combining `][`
* The dollar sign as a rough signifier of data owned by the system rather than the user
As I have written about elsewhere I was privileged to know Joe Armstrong, the co-inventor of Erlang, in the last few years of his life – we were working together on a book about TiddlyWiki when he passed away in 2019. Joe had contacted me out of the blue ten years before to express his admiration for TiddlyWiki, and we had developed a friendship. He was actually a big fan of TW5's filter syntax, and used to make me feel better about it by joking that I had (re-)invented the monad, which sounded impressive to me. That doesn't make the filter language any easier to learn, but it does mean that it is *worth* learning: it's a real language, based on the same principles as other languages.
I find it pleasing that the TW5 filter language has its roots in decisions that were taken in the TWC days. It's still hard to learn, but that's an ongoing paradox of programming: people want to do complicated things, and complicated things are complicated. It's hard to see how we could have made filters any simpler without depriving users of the possibility of doing complicated things.

83
editions/tw5.com/tiddlers/about/History of TiddlyWiki.tid
View file

@ -4,83 +4,8 @@ tags: About
title: History of TiddlyWiki
type: text/vnd.tiddlywiki
! Twenty Years of ~TiddlyWiki
We've held a number of livestreams to celebrate twenty years of ~TiddlyWiki. You can watch the recordings here:
* 19th September 2024 - https://youtube.com/live/z9slx92TyrU
* 20th September 2024 - https://youtube.com/live/puFdN-FgOjg
* 21st September 2024 - https://youtube.com/live/0SjsHvwjHGE
* 22nd September 2024 - https://youtube.com/live/oD7Jtq2D4lg
Over on GitHub, we celebrated the contributors to TiddlyWiki by [[asking them|https://github.com/TiddlyWiki/TiddlyWiki5/discussions/7983]] for their reflections on the anniversary. We received some interesting and thoughtful responses. For example, this from [[@FND|https://github.com/FND]]:
> TiddlyWiki had an immeasurable, enduring influence not just on my career, such as it is, but also on my values: To this day, I regularly find myself referring back to [[fundamental concepts|https://prepitaph.org/articles/creative-privacy/]] TiddlyWiki instilled in me - many of which are often forgotten or ignored elsewhere. Having this background thus helps me keep my bearings working in this industry, whether it's worshipping at the altar of technical complexity or even just remembering humans exist in the world of technology.
> By TiddlyWiki, I mean people. It was an immense privilege interacting with and learning from this community and the group Jeremy built around it. It also helps reminding myself that this privilege was afforded to me by sheer happenstance; I hope to be paying it forward.
Some recent podcasts featuring ~TiddlyWiki:
* The changelog podcast from 2016 - https://changelog.com/podcast/196 discussing ~TiddlyWiki's backstory
* Floss Weekly recording from 2021 - https://twit.tv/shows/floss-weekly/episodes/620
! Origins of TiddlyWiki
Back in 1997 a colleague introduced me to [[Ward Cunningham's original wiki|http://c2.com/cgi/wiki]]. I was impressed that something so powerful could fit into just 700 lines of Perl, and fascinated by the radical reimagining of security and permissions. Like many other developers, I took every opportunity I could to try out various wikis, and to explore their use at work.
The allure of the wiki for me was the feeling that it could eventually disrupt the prevailing paradigm of print-oriented documents and emails.
After watching people use wikis for a few years, I noticed that power users made extensive use of the ability to open multiple wiki pages at once in several browser tabs, making it easier for them to compare and review pages, to copy text between them and to act as a sort of queue of pages yet to be read.
I felt that this ability to manipulate multiple pages at once was central to the ability to refactor a wiki, and it is generally accepted that a wiki that is lovingly refactored tends to be more useful. And yet, standard wiki user interfaces have always been designed exclusively for the presentation and manipulation of single pages at once.
All of these thoughts came together when I saw GMail in April 2004, which used Ajax cleverly to blend individual emails into threaded conversations.
I started experimenting with HTML and JavaScript to explore the idea further. I'd had virtually no experience of either, just having put together some static pages and simple ASP sites in previous lives. Getting my head around these client-side technologies was painful; like everyone else, I was horrified to discover how appalling were the incompatibilities and inconsistencies of web programming.
! Launch of TiddlyWiki
So, in September 2004 I released a primitive [[first version of TiddlyWiki|https://classic.tiddlywiki.com/firstversion.html]]. It was the smallest possible thing that demonstrated the idea: it was a simple, self-contained static 48KB HTML file.
The downside of writing the first version of TiddlyWiki in this way was that it made it completely impractical to use for editing - when you click 'save changes' it just pops up a window showing the data that would be saved if it were possible for an HTML page to write to the file system.
Much of the early feedback was that TiddlyWiki was neat, but that it would be more useful when it was possible to properly save changes. I was a little frustrated, as I thought I knew that it was impossible for an HTML file running in the browser to save changes to the local file system.
Within a few months I saw an experimental Firefox extension that enabled TiddlyWiki to save changes in the browser. Examining the code, I realised that the APIs that it used to write to the file system were actually available in ordinary HTML files - as long as they were loaded via a `file://` URI.
I adapted the Firefox code into the core of TiddlyWiki, and soon added a similar ability for Internet Explorer (making use of an old [[ActiveX|https://en.wikipedia.org/wiki/ActiveX]] control that Microsoft distributed with Internet Explorer).
! Growth of TiddlyWiki
A major milestone in the growth of TiddlyWiki was the creation of "GTDTiddlyWiki" by Nathan Bowers. He took the vanilla TiddlyWiki product and adapted it for the specific application of keeping track of tasks using the popular Getting Things Done methodology. GTDTiddlyWiki was an immediate hit, being enthusiastically greeted on websites like [[LifeHacker|https://lifehacker.com/]].
Over the next couple of years TiddlyWiki continued to grow in popularity, and gained new features and capabilities. Within a year I was able to support myself by performing bespoke development work on TiddlyWiki, notably working with wiki pioneer [[SocialText|https://en.wikipedia.org/wiki/Socialtext]] on the ability to synchronise changes with an online server
! BT Acquisition
In May 2007, [[BT]] acquired [[Osmosoft]], my consultancy company. It was an unusual decision to acquire a company with a single employee and a tiny trickle of revenue - [[Osmosoft]] didn't even own the intellectual property in TiddlyWiki since I had handed it over to [[UnaMesa]] to assure its future for the community.
[[BT]]'s motivation was to help them understand community-based ecosystems. I joined the organisation as "Head of Open Source Innovation", taking responsibility for open source governance, and providing advice and expertise on how to participate in open soure communities.
! [[Osmosoft]] and TiddlySpace
I built a team in BT under the name [[Osmosoft]]. Our purpose was to evangelise the benefits of open source, and to help other teams realise those benefits in practice. We also found that it was necessary to evangelise the use of the web in general, and web standards in particular.
Our approach was to focus on showing rather than telling. We worked with the TiddlyWiki community to extend the ecosystem and we built numerous internal systems for BT (some based on TiddlyWiki and some not).
[[Osmosoft]]'s chief contribution to the TiddlyWiki community was the creation of TiddlyWeb and TiddlySpace. TiddlyWeb was a robust, internet scale server for tiddlers that could also compose TiddlyWiki views of those tiddlers. TiddlySpace was an attempt to package TiddlyWeb into a more directly usable form.
! Leaving BT
By the end of 2011 I was increasingly feeling that I would be better placed to realise the potential of TiddlyWiki outside of the corporate confines BT. Accordingly, I left and started work as an independent developer, primarily working on a brand new reboot of TiddlyWiki in the shape of TiddlyWiki5.
! Development of TiddlyWiki5
I worked on new release of TiddlyWiki from November 2011. As a programmer, working on "version 2.0" of something that I had already written is a very attractive proposition. It means that the requirements were fully understood, allowing me to focus on evolving the architecture needed to support the desired functionality.
! The Future
Now that TiddlyWiki5 has finally left "beta" status behind, my hope is that it will have a long life. Because it only uses standard features of HTML5 and Node.js, there is no reason why it cannot be fully operational for many years to come. My goal is for it to last for at least 25 years.
//Jeremy Ruston, 20th September 2014//
Here is a brief history of TiddlyWiki, its origins and its evolution since it was first released on 20th September 2004. Contributions and reminiscences are welcome.
* [[The Story of TiddlyWiki]] – a personal account of the story of TiddlyWiki, its origins and evolution
* [[TiddlyWiki Anniversaries]] – relive the celebrations of TiddlyWiki's major anniversaries
* [[Filter Syntax History]] – gives a brief history of the evolution of the filter syntax in TiddlyWiki5

67
editions/tw5.com/tiddlers/about/The Story of TiddlyWiki.tid Normal file
View file

@ -0,0 +1,67 @@
title: The Story of TiddlyWiki
tags: [[History of TiddlyWiki]]
modifier: Jeremy Ruston
This is a personal account of the story of TiddlyWiki, its origins and its evolution since it was first released on 20th September 2004.
! Origins of TiddlyWiki
Back in 1997 a colleague introduced me to [[Ward Cunningham's original wiki|http://c2.com/cgi/wiki]]. I was impressed that something so powerful could fit into just 700 lines of Perl, and fascinated by the radical reimagining of security and permissions. Like many other developers, I took every opportunity I could to try out various wikis, and to explore their use at work.
The allure of the wiki for me was the feeling that it could eventually disrupt the prevailing paradigm of print-oriented documents and emails.
After watching people use wikis for a few years, I noticed that power users made extensive use of the ability to open multiple wiki pages at once in several browser tabs, making it easier for them to compare and review pages, to copy text between them and to act as a sort of queue of pages yet to be read.
I felt that this ability to manipulate multiple pages at once was central to the ability to refactor a wiki, and it is generally accepted that a wiki that is lovingly refactored tends to be more useful. And yet, standard wiki user interfaces have always been designed exclusively for the presentation and manipulation of single pages at once.
All of these thoughts came together when I saw GMail in April 2004, which used Ajax cleverly to blend individual emails into threaded conversations.
I started experimenting with HTML and JavaScript to explore the idea further. I'd had virtually no experience of either, just having put together some static pages and simple ASP sites in previous lives. Getting my head around these client-side technologies was painful; like everyone else, I was horrified to discover how appalling were the incompatibilities and inconsistencies of web programming.
! Launch of TiddlyWiki
So, in September 2004 I released a primitive [[first version of TiddlyWiki|https://classic.tiddlywiki.com/firstversion.html]]. It was the smallest possible thing that demonstrated the idea: it was a simple, self-contained static 48KB HTML file.
The downside of writing the first version of TiddlyWiki in this way was that it made it completely impractical to use for editing - when you click 'save changes' it just pops up a window showing the data that would be saved if it were possible for an HTML page to write to the file system.
Much of the early feedback was that TiddlyWiki was neat, but that it would be more useful when it was possible to properly save changes. I was a little frustrated, as I thought I knew that it was impossible for an HTML file running in the browser to save changes to the local file system.
Within a few months I saw an experimental Firefox extension that enabled TiddlyWiki to save changes in the browser. Examining the code, I realised that the APIs that it used to write to the file system were actually available in ordinary HTML files - as long as they were loaded via a `file://` URI.
I adapted the Firefox code into the core of TiddlyWiki, and soon added a similar ability for Internet Explorer (making use of an old [[ActiveX|https://en.wikipedia.org/wiki/ActiveX]] control that Microsoft distributed with Internet Explorer).
! Growth of TiddlyWiki
A major milestone in the growth of TiddlyWiki was the creation of "GTDTiddlyWiki" by Nathan Bowers. He took the vanilla TiddlyWiki product and adapted it for the specific application of keeping track of tasks using the popular Getting Things Done methodology. GTDTiddlyWiki was an immediate hit, being enthusiastically greeted on websites like [[LifeHacker|https://lifehacker.com/]].
Over the next couple of years TiddlyWiki continued to grow in popularity, and gained new features and capabilities. Within a year I was able to support myself by performing bespoke development work on TiddlyWiki, notably working with wiki pioneer [[SocialText|https://en.wikipedia.org/wiki/Socialtext]] on the ability to synchronise changes with an online server
! BT Acquisition
In May 2007, [[BT]] acquired [[Osmosoft]], my consultancy company. It was an unusual decision to acquire a company with a single employee and a tiny trickle of revenue - [[Osmosoft]] didn't even own the intellectual property in TiddlyWiki since I had handed it over to [[UnaMesa]] to assure its future for the community.
[[BT]]'s motivation was to help them understand community-based ecosystems. I joined the organisation as "Head of Open Source Innovation", taking responsibility for open source governance, and providing advice and expertise on how to participate in open soure communities.
! [[Osmosoft]] and TiddlySpace
I built a team in BT under the name [[Osmosoft]]. Our purpose was to evangelise the benefits of open source, and to help other teams realise those benefits in practice. We also found that it was necessary to evangelise the use of the web in general, and web standards in particular.
Our approach was to focus on showing rather than telling. We worked with the TiddlyWiki community to extend the ecosystem and we built numerous internal systems for BT (some based on TiddlyWiki and some not).
[[Osmosoft]]'s chief contribution to the TiddlyWiki community was the creation of TiddlyWeb and TiddlySpace. TiddlyWeb was a robust, internet scale server for tiddlers that could also compose TiddlyWiki views of those tiddlers. TiddlySpace was an attempt to package TiddlyWeb into a more directly usable form.
! Leaving BT
By the end of 2011 I was increasingly feeling that I would be better placed to realise the potential of TiddlyWiki outside of the corporate confines BT. Accordingly, I left and started work as an independent developer, primarily working on a brand new reboot of TiddlyWiki in the shape of TiddlyWiki5.
! Development of TiddlyWiki5
I worked on new release of TiddlyWiki from November 2011. As a programmer, working on "version 2.0" of something that I had already written is a very attractive proposition. It means that the requirements were fully understood, allowing me to focus on evolving the architecture needed to support the desired functionality.
! The Future
Back in 2014, shortly after TiddlyWiki5 was first released, I wrote:
> Now that TiddlyWiki5 has finally left "beta" status behind, my hope is that it will have a long life. Because it only uses standard features of HTML5 and Node.js, there is no reason why it cannot be fully operational for many years to come. My goal is for it to last for at least 25 years.
As I write this, TiddlyWiki5 is 44% of the way to that goal. With the support and enthusiasm of the community I am confident the project will continue to thrive and evolve.

29
editions/tw5.com/tiddlers/about/TiddlyWiki Anniversaries.tid Normal file
View file

@ -0,0 +1,29 @@
title: TiddlyWiki Anniversaries
tags: [[History of TiddlyWiki]]
! Twentieth Anniversary of TiddlyWiki
We've held a number of livestreams to celebrate twenty years of ~TiddlyWiki. You can watch the recordings here:
* 19th September 2024 - https://youtube.com/live/z9slx92TyrU
* 20th September 2024 - https://youtube.com/live/puFdN-FgOjg
* 21st September 2024 - https://youtube.com/live/0SjsHvwjHGE
* 22nd September 2024 - https://youtube.com/live/oD7Jtq2D4lg
Over on GitHub, we celebrated the contributors to TiddlyWiki by [[asking them|https://github.com/TiddlyWiki/TiddlyWiki5/discussions/7983]] for their reflections on the anniversary. We received some interesting and thoughtful responses. For example, this from [[@FND|https://github.com/FND]]:
> TiddlyWiki had an immeasurable, enduring influence not just on my career, such as it is, but also on my values: To this day, I regularly find myself referring back to [[fundamental concepts|https://prepitaph.org/articles/creative-privacy/]] TiddlyWiki instilled in me - many of which are often forgotten or ignored elsewhere. Having this background thus helps me keep my bearings working in this industry, whether it's worshipping at the altar of technical complexity or even just remembering humans exist in the world of technology.
> By TiddlyWiki, I mean people. It was an immense privilege interacting with and learning from this community and the group Jeremy built around it. It also helps reminding myself that this privilege was afforded to me by sheer happenstance; I hope to be paying it forward.
Some recent podcasts featuring ~TiddlyWiki:
* The changelog podcast from 2016 - https://changelog.com/podcast/196 discussing ~TiddlyWiki's backstory
* Floss Weekly recording from 2021 - https://twit.tv/shows/floss-weekly/episodes/620
! Tenth Anniversary of TiddlyWiki
You can watch the livestream from 20th September 2014 celebrating the tenth anniversary of TiddlyWiki here:
https://www.youtube.com/watch?v=f_02ZV0J9NY

4
editions/tw5.com/tiddlers/concepts/Filters.tid
View file

@ -24,5 +24,5 @@ A filter output can change as tiddlers are added and deleted in the wiki. ~Tiddl
''Find out more:''
* <$linkcatcher message="tm-navigate" actions=<<openAdvancedSearch>> >[[Advanced Search|$:/AdvancedSearch]]</$linkcatcher> -- has a <<.advancedsearch-tab Filter>> tab that makes it easy to experiment with filters.
* [[Filtered Transclusions|Transclusion in WikiText]] -- If you want to use filter results in your text
* [[Filtered Transclusions|Transclusion in WikiText]] -- if you want to use filter results in your text
* [[TiddlyWiki Syntax History]] -- if you are curious why the filter syntax is the way it is