TidGi-Desktop/docs/internal/IPCSyncAdaptorAndFSAdaptor.md

# Sync Architecture: IPC and Watch-FS Plugins

This document describes how the `tidgi-ipc-syncadaptor` (frontend) and `watch-filesystem-adaptor` (backend) plugins work together to provide real-time bidirectional synchronization between the TiddlyWiki in-memory store and the file system.

## Architecture Overview

```
Frontend (Browser)                    Backend (Node.js Worker)
┌─────────────────────────────┐      ┌─────────────────────────────────────┐
│ TiddlyWiki                  │      │ TiddlyWiki (Server)                 │
│ In-Memory Store             │      │ In-Memory Store                     │
└──────────┬──────────────────┘      └──────────┬──────────────────────────┘
           │                                    │
           │ TidGiIPCSyncAdaptor                │ WatchFileSystemAdaptor
           │ (syncadaptor)                      │ (syncadaptor)
           │                                    │
           │                                    ├── FileSystemWatcher
           │                                    │   (monitors files via nsfw)
           │                                    │
           ├─── Save via IPC ──────────────────►│
           │                                    │
           │◄────── IPC Observable ─────────────┤
           │        (Change Events)             │
           │                                    │
           │                                    ├── FileSystemAdaptor
           │                                    │   (read/write files)
           │                                    │
           │                                    ▼
           │                                 File System
           └────────────────────────────────────┘
```

## Key Design Principles

### 1. Single Source of Truth: File System

- Backend watch-fs monitors the file system using `nsfw` library
- All file changes (external edits, saves from frontend) flow through file system
- Backend wiki state reflects file system state

### 2. Syncer-Driven Updates (Refactored Architecture)

**Previous Approach (Problematic):**

- FileSystemWatcher directly called `wiki.addTiddler()` when files changed
- Led to echo problems and complex edge case handling

**Current Approach (Syncer-Driven):**

- FileSystemWatcher only collects changes into `updatedTiddlers` list
- Triggers `$tw.syncer.syncFromServer()` to let TiddlyWiki's syncer handle updates
- Syncer calls `getUpdatedTiddlers()` to get change list
- Syncer calls `loadTiddler()` for each modified tiddler
- Syncer uses `storeTiddler()` which properly updates changeCount to prevent echo

Benefits:

- Leverages TiddlyWiki's built-in sync queue and throttling
- Proper handling of batch changes (git checkout)
- Eliminates echo loops via syncer's changeCount tracking

### 3. Two-Layer Echo Prevention

**IPC Layer (First Defense)**:

- `ipcServerRoutes.ts` tracks recently saved tiddlers in `recentlySavedTiddlers` Set
- When `wiki.addTiddler()` triggers change event, filter out tiddlers in the Set
- Prevents frontend from receiving its own save operations as change notifications

**Watch-FS Layer (Second Defense)**:

- When saving/deleting, watch-fs temporarily excludes file from monitoring
- Prevents watcher from detecting the file write operation
- Re-includes file after operation completes (with delay for nsfw debounce)

### 4. IPC Observable for Change Notification

- Backend sends change events to frontend via IPC Observable pattern (via `ipc-cat`)
- Frontend subscribes to `getWikiChangeObserver$` observable from `window.observables.wiki`
- Change events trigger frontend's `syncFromServer()` to pull updates

## Module Responsibilities

### FileSystemWatcher (Backend - New)

**Purpose:** Monitor file system changes without directly modifying wiki state

**Key Features:**

- Uses `nsfw` library for native file system watching
- Maintains `updatedTiddlers` list for pending changes
- Implements `getUpdatedTiddlers()` for syncer integration
- Implements `loadTiddler()` for lazy loading from file system
- Handles git revert/checkout via delayed deletion processing
- Manages file exclusion list for echo prevention

**Key Methods:**

- `getUpdatedTiddlers(syncer, callback)`: Returns collected changes
- `loadTiddler(title, callback)`: Loads tiddler content from file
- `excludeFile(path)`: Temporarily exclude file from watching
- `scheduleFileInclusion(path)`: Re-include file after delay

### WatchFileSystemAdaptor (Backend)

**Purpose:** Coordinate between FileSystemWatcher and syncer, implement syncadaptor interface

**Key Features:**

- Extends FileSystemAdaptor for file save/delete operations
- Delegates file watching to FileSystemWatcher
- Implements full syncadaptor interface for Node.js syncer
- Coordinates file exclusion during save/delete operations

**Key Methods:**

- `getUpdatedTiddlers()`: Delegates to FileSystemWatcher
- `loadTiddler()`: Delegates to FileSystemWatcher
- `saveTiddler()`: Saves to file with exclusion handling
- `deleteTiddler()`: Deletes file with exclusion handling

### FileSystemAdaptor (Backend - Base Class)

**Purpose:** Handle tiddler file save/delete operations with sub-wiki routing

**Key Features:**

- Routes tiddlers to sub-wikis based on tags
- Generates file paths using TiddlyWiki's FileSystemPaths
- Handles external attachment file movement
- Provides retry logic for file lock errors

### TidGiIPCSyncAdaptor (Frontend)

**Purpose:** Bridge between frontend TiddlyWiki and backend file system

**Key Features:**

- Communicates via IPC using `tidgi://` custom protocol
- Subscribes to change events via IPC Observable
- Maintains `updatedTiddlers` list from IPC events
- Implements full syncadaptor interface for browser syncer

## Data Flow Examples

### Example 1: User Edits in Frontend

```
1. User clicks save in browser
   ├─► Frontend syncer calls saveTiddler()
   │
2. TidGiIPCSyncAdaptor.saveTiddler()
   ├─► IPC call to putTiddler in ipcServerRoutes.ts
   │
3. ipcServerRoutes.putTiddler()
   ├─► Marks tiddler in recentlySavedTiddlers (IPC echo prevention)
   ├─► Calls wiki.addTiddler() (triggers change event)
   ├─► Change event filtered by recentlySavedTiddlers
   │
4. Backend syncer detects change
   ├─► Calls WatchFileSystemAdaptor.saveTiddler()
   │
5. WatchFileSystemAdaptor.saveTiddler()
   ├─► Excludes file path from watching
   ├─► Calls FileSystemAdaptor.saveTiddler()
   ├─► Writes file to disk
   ├─► Schedules file re-inclusion after delay
   │
6. nsfw might detect file change
   ├─► FileSystemWatcher checks exclusion list
   ├─► File is excluded, event ignored
```

### Example 2: External Editor Modifies File

```
1. User edits file in VSCode/Vim
   ├─► File content changes on disk
   │
2. nsfw detects file change
   ├─► File NOT in excludedFiles
   │
3. FileSystemWatcher.handleFileAddOrChange()
   ├─► Adds title to updatedTiddlers.modifications
   ├─► Stores file info in pendingFileLoads
   ├─► Schedules syncer trigger (debounced 200ms)
   │
4. $tw.syncer.syncFromServer() called
   ├─► Creates SyncFromServerTask
   │
5. SyncFromServerTask.run()
   ├─► Calls getUpdatedTiddlers()
   ├─► Gets modifications/deletions list
   ├─► Adds titles to titlesToBeLoaded
   │
6. For each title to load:
   ├─► LoadTiddlerTask.run()
   ├─► Calls loadTiddler(title)
   ├─► FileSystemWatcher loads from file
   ├─► syncer.storeTiddler() updates wiki
   ├─► Properly sets changeCount (prevents echo save)
   │
7. wiki.addTiddler() triggers change event
   ├─► getWikiChangeObserver sends to frontend
   │
8. Frontend receives change
   ├─► TidGiIPCSyncAdaptor adds to updatedTiddlers
   ├─► Frontend syncer.syncFromServer()
   ├─► Frontend loadTiddler() via IPC
   ├─► Frontend wiki updated
```

### Example 3: Git Checkout (Batch Changes)

```
1. User runs git checkout
   ├─► Many files deleted/created/modified
   │
2. nsfw debounces events (100ms)
   ├─► Multiple events batched together
   │
3. FileSystemWatcher.handleNsfwEvents()
   ├─► For each DELETED file:
   │   └─► Schedule deletion with 100ms delay
   │       (handles git revert/checkout pattern)
   ├─► For each CREATED/MODIFIED file:
   │   └─► Cancel any pending deletion for same path
   │   └─► Add to updatedTiddlers.modifications
   │
4. Syncer trigger debounced (200ms)
   ├─► All changes collected before sync starts
   │
5. Single SyncFromServerTask processes all changes
   ├─► All modifications queued for loading
   ├─► All deletions processed (wiki.deleteTiddler)
   │
6. LoadTiddlerTasks process sequentially
   ├─► Each tiddler loaded from file
   ├─► Frontend notified via IPC Observable
```

## Key Configuration

### Timing Constants

```typescript
// FileSystemWatcher.ts
FILE_DELETION_DELAY_MS = 100    // Delay before processing DELETE events
FILE_INCLUSION_DELAY_MS = 150   // Delay before re-including file after save
GIT_NOTIFICATION_DELAY_MS = 1000 // Debounce for git status notification
SYNCER_TRIGGER_DELAY_MS = 200   // Debounce for syncer trigger
```

### Syncer Configuration

- Frontend: `pollTimerInterval = 2_147_483_647` (effectively disabled)
- All updates come via IPC Observable (event-driven)

## Troubleshooting

### Changes Not Appearing in Frontend

1. Check IPC Observable connection: Look for `[test-id-SSE_READY]` in logs
2. Verify watch-fs is running: Look for `[test-id-WATCH_FS_STABILIZED]`
3. Check file exclusion: Should see file being excluded then included

### Echo/Duplicate Updates

1. Check `recentlySavedTiddlers` filtering in ipcServerRoutes.ts
2. Verify file exclusion during save/delete operations
3. Check syncer's changeCount tracking

### Git Checkout Issues

1. Ensure FILE_DELETION_DELAY_MS is working (files not prematurely deleted)
2. Check that SYNCER_TRIGGER_DELAY_MS allows batch collection
3. Verify syncer processes all changes in single SyncFromServerTask

### Sub-Wiki Not Syncing

1. Check sub-wiki watcher initialization: Look for `[WATCH_FS_SUBWIKI]` logs
2. Verify tiddlywiki.info has correct configuration
3. Check workspace `subWikiFolders` setting

## Design Decisions

### Why Syncer-Driven Instead of Direct Updates?

1. **Echo Prevention**: Syncer's `storeTiddler()` properly updates changeCount, preventing save loops
2. **Batch Handling**: Syncer queues all changes and processes them sequentially
3. **Throttling**: Built-in throttle prevents rapid-fire saves
4. **Error Recovery**: Syncer has built-in retry logic

### Why Two-Layer Echo Prevention?

1. **IPC Layer**: Prevents frontend from seeing its own saves via IPC
2. **Watch-FS Layer**: Prevents file watcher from seeing our own file writes
3. **Both needed**: IPC prevents wiki→wiki echo, Watch-FS prevents file→wiki echo

### Why Delay DELETE Events?

Git operations often delete then recreate files quickly. The delay allows:

1. CREATE event to arrive and cancel pending DELETE
2. Treat as modification instead of delete+create
3. Prevents "missing tiddler" errors during git operations