TidGi-Desktop/docs/internal/IPCSyncAdaptorAndFSAdaptor.md

Sync Architecture: IPC and Watch-FS Plugins

This document describes how the tidgi-ipc-syncadaptor (frontend) and watch-filesystem-adaptor (backend) plugins work together to provide real-time bidirectional synchronization between the TiddlyWiki in-memory store and the file system.

Architecture Overview

Frontend (Browser)          Backend (Node.js Worker)
┌─────────────────┐        ┌──────────────────────┐
│ TiddlyWiki      │        │ TiddlyWiki (Server)  │
│ In-Memory Store │◄──────►│ File System Adaptor  │
└────────┬────────┘        └──────────┬───────────┘
         │                            │
         │ IPC Sync                   │ Watch-FS
         │ Adaptor                    │ Adaptor
         │                            │
         ├─ Save to FS ──────────────►│
         │                            │
         │◄───── SSE Events ──────────┤
         │     (File Changes)         │
         │                            │
         │                            ├─ nsfw Watcher
         │                            │  (File System)
         └────────────────────────────┘

Key Design Principles

1. Single Source of Truth: File System

• Backend watch-fs monitors the file system using nsfw library
• All file changes (external edits, saves from frontend) flow through file system
• Backend wiki state reflects file system state

2. Two-Layer Echo Prevention

IPC Layer (First Defense):

• ipcServerRoutes.ts tracks recently saved tiddlers in recentlySavedTiddlers Set
• When wiki.addTiddler() triggers change event, filter out tiddlers in the Set
• Prevents frontend from receiving its own save operations as change notifications

Watch-FS Layer (Second Defense):

• When saving, watch-fs temporarily excludes file from monitoring for 200ms
• Prevents watcher from detecting the file write operation
• Combined with IPC filtering, ensures no echo even for rapid successive saves

3. SSE-like Change Notification (IPC Observable)

• Backend sends change events to frontend via IPC (not real SSE, but Observable pattern via ipc-cat)
• Frontend subscribes to getWikiChangeObserver$ observable from window.observables.wiki
• Change events trigger tw's syncFromServer() to pull updates from backend, it will in return call our getUpdatedTiddlers

Plugin Responsibilities

Frontend: tidgi-ipc-syncadaptor

Purpose: Bridge between frontend TiddlyWiki and backend file system

IPC Communication Flow:

• Frontend plugin calls window.service.wiki methods (provided by preload script)
• Preload script uses setupIpcServerRoutesHandlers.ts (runs in view context) to set up IPC protocol handlers
• These handlers forward requests to ipcServerRoutes.ts in wiki worker via IPC
• Uses tidgi:// custom protocol for RESTful-like API

Key Functions:

• saveTiddler(): Send tiddler to backend via IPC → putTiddler in ipcServerRoutes.ts
• loadTiddler(): Request tiddler from backend via IPC → getTiddler in ipcServerRoutes.ts
• deleteTiddler(): Request deletion via IPC → deleteTiddler in ipcServerRoutes.ts
• setupSSE(): Subscribe to file change events from backend (via IPC Observable, not real SSE)
• getUpdatedTiddlers(): Provide list of changed tiddlers to syncer

Echo Prevention Strategy:

Frontend plugin does NOT implement echo detection because:

1. Cannot distinguish between "save to fs and watch fs echo back" and "external user text edit with unchanged 'modified' timestamp metadata"
2. Echo prevention is handled by backend at two levels:
   • IPC level: ipcServerRoutes.ts filters out change events from IPC saves using recentlySavedTiddlers Set
   • Watch-FS level: watch-filesystem-adaptor temporarily excludes files during save operations

Backend: watch-filesystem-adaptor

Purpose: Monitor file system and maintain wiki state

Key Functions:

• saveTiddler(): Write to file system with temporary exclusion
• deleteTiddler(): Remove file with temporary exclusion
• initializeFileWatching(): Setup nsfw watcher for main wiki and sub-wikis
• handleFileAddOrChange(): Load changed files into wiki
• handleFileDelete(): Remove deleted tiddlers from wiki

Two-Layer Echo Prevention:

Layer 1: IPC Level (in ipcServerRoutes.ts)

async putTiddler(title, fields) {
  // Mark as recently saved to prevent echo
  this.recentlySavedTiddlers.add(title);
  
  // This triggers 'change' event synchronously
  this.wikiInstance.wiki.addTiddler(new Tiddler(fields));
  
  // Change event handler filters it out and removes the mark
}

// Change event handler
wiki.addEventListener('change', (changes) => {
  const filteredChanges = {};
  for (const title in changes) {
    if (this.recentlySavedTiddlers.has(title)) {
      // Filter out echo from IPC save
      this.recentlySavedTiddlers.delete(title);
      continue;
    }
    filteredChanges[title] = changes[title];
  }
  // Only send filtered changes to frontend
  observer.next(filteredChanges);
});

Layer 2: Watch-FS Level (in WatchFileSystemAdaptor.ts)

async saveTiddler(tiddler) {
  const filepath = await this.getTiddlerFileInfo(tiddler);
  
  // 1. Exclude file BEFORE saving
  await this.excludeFile(filepath);
  
  // 2. Save to file system (calls parent saveTiddler)
  await super.saveTiddler(tiddler);
  
  // 3. Re-include after delay
  this.scheduleFileInclusion(filepath);
  // After 200ms: includeFile(filepath) removes exclusion
}

File Change Flow:

handleNsfwEvents(events) {
  events.forEach(event => {
    const filepath = path.join(event.directory, event.file);
    
    // Skip if file is excluded (being saved)
    if (this.excludedFiles.has(filepath)) return;
    
    // Load changed file into wiki
    const tiddler = $tw.loadTiddlersFromFile(filepath);
    $tw.syncadaptor.wiki.addTiddler(tiddler);
    
    // Wiki change event fires → SSE sends to frontend
  });
}

Data Flow Examples

Example 1: User Edits in Frontend

1. User clicks save in browser
   ├─► Frontend: saveTiddler() called
   │
2. IPC call to backend
   ├─► Backend: receives putTiddler request
   │
3. Backend IPC layer: Mark tiddler in recentlySavedTiddlers
   ├─► ipcServerRoutes.ts adds title to Set
   │
4. Backend: wiki.addTiddler(tiddler)
   ├─► Triggers 'change' event
   ├─► Event handler checks recentlySavedTiddlers
   ├─► Filters out this change (IPC echo prevention)
   ├─► Removes title from recentlySavedTiddlers
   │
5. Backend Watch-FS layer: excludeFile(filepath)
   ├─► File added to excludedFiles set
   │
6. Backend: write to file system
   ├─► File content updated on disk
   │
7. nsfw detects file change
   ├─► But file is in excludedFiles
   ├─► Change event ignored (Watch-FS echo prevention)
   │
8. After 200ms delay
   ├─► includeFile(filepath) removes exclusion

Example 2: External Editor Modifies File

1. User edits file in VSCode/Vim
   ├─► File content changes on disk
   │
2. nsfw detects file change
   ├─► File NOT in excludedFiles (not being saved)
   │
3. handleFileAddOrChange() called
   ├─► Load tiddler from file
   ├─► wiki.addTiddler(tiddler)
   │
4. Wiki fires 'change' event
   ├─► ipcServerRoutes.ts checks recentlySavedTiddlers
   ├─► Title NOT in Set (external edit, not IPC save)
   ├─► Change passes through filter
   ├─► IPC Observable sends event to frontend (via ipc-cat)
   │
5. Frontend receives change event
   ├─► Adds to updatedTiddlers.modifications
   ├─► Triggers syncFromServer()
   │
6. Frontend: loadTiddler() via IPC
   ├─► Gets latest tiddler from backend
   ├─► Updates frontend wiki
   ├─► UI re-renders with new content

Example 3: Sub-Wiki Synchronization

1. Main wiki has sub-wiki folder linked by tag
   ├─► watch-fs detects sub-wiki in tiddlywiki.info
   │
2. watch-fs starts additional watcher
   ├─► Monitors SubWiki/ folder
   │
3. User saves tiddler with tag in frontend
   ├─► Backend determines file should go to SubWiki/
   ├─► Saves to SubWiki/Tiddler.tid
   │
4. Sub-wiki watcher detects new file
   ├─► (Excluded during save, so no echo)
   │
5. External edit in SubWiki/Tiddler.tid
   ├─► Sub-wiki watcher detects change
   ├─► Updates main wiki's in-memory store
   ├─► IPC Observable notifies frontend
   ├─► Frontend syncs and displays update

Key Configuration

File Exclusion

• FILE_EXCLUSION_CLEANUP_DELAY_MS = 200: Time to keep file excluded after save
• Prevents echo while allowing quick re-detection of external changes

SSE-like Debouncing (IPC Observable)

• debounce(syncFromServer, 500): Batch multiple file changes
• Reduces unnecessary sync operations

Syncer Polling

• pollTimerInterval = 2_147_483_647: Effectively disable polling
• All updates come via IPC Observable (event-driven, not polling)

Troubleshooting

Changes Not Appearing in Frontend

1. Check IPC Observable connection: Look for [test-id-SSE_READY] in logs
2. Verify watch-fs is running: Look for [test-id-WATCH_FS_STABILIZED]
3. Check file exclusion: Should see [WATCH_FS_EXCLUDE] and [WATCH_FS_INCLUDE]

Echo/Duplicate Updates

1. Verify exclusion timing: 200ms should be sufficient
2. Check for multiple watchers on same path
3. Ensure frontend isn't doing its own timestamp-based filtering

Sub-Wiki Not Syncing

1. Check sub-wiki detection: Look for [WATCH_FS_SUBWIKI] logs
2. Verify tiddlywiki.info has correct configuration
3. Check workspace subWikiFolders setting

Not Recommended

• ❌ Timestamp-based echo detection (already tried, unreliable)
• ❌ Frontend-side file watching (duplicates backend effort)
• ❌ Polling-based synchronization (SSE is better)