Feat/allow watch fs change on git sync

* feat: Skip restart if file system watch is enabled - the watcher will handle file changes automatically * fix: sometimes change sync interval not working fixes #310 * fix: Return false on sync failure - no successful changes were made fixes #558 * fix: step that is wrong * feat: monitoring subwiki * AI added waitForSSEReady * Revert "AI added waitForSSEReady" This reverts commit 983b1c623c. * fix: error on frontend loading worker thread * fix * Update wiki.ts * auto reload view and click subwiki icon * Refactor sync echo prevention and improve logging Removed frontend-side echo prevention logic in ipcSyncAdaptor, relying solely on backend file exclusion for echo prevention. Improved console log wrappers to preserve native behavior and added a log statement to setupSSE. Updated test steps and file modification logic to better simulate external edits without modifying timestamps. Added internal documentation on sync architecture. * feat: deboucne and prevent data race when write file * Update watch-filesystem-adaptor.ts * rename camelcase * Update filesystemPlugin.feature * Fix sync interval timezone handling and add tests Refactored syncDebounceInterval logic in Sync.tsx to be timezone-independent, ensuring correct interval storage and display across all timezones. Added comprehensive tests in Sync.timezone.test.ts to verify correct behavior and document previous timezone-related bugs. fixes #310 * i18n for notification * Update index.tsx * fix: potential symlinks problem of subwiki * Update Sync.timezone.test.ts * lint * Implement backoff for file existence check Refactor file existence check to use backoff strategy and add directory tree retrieval for error reporting. * Update BACKOFF_OPTIONS with new configuration * Update wiki.ts * remove log * Update wiki.ts * fix: draft not move to sub * Update filesystemPlugin.feature * fix: routing tw logger to file * Update filesystemPlugin.feature * test: use id to check view load and sse load * Optimize test steps and screenshot logic Removed unnecessary short waits in filesystemPlugin.feature and increased wait time for tiddler state to settle. Updated application.ts to skip screenshots for wait steps, reducing redundant screenshots during test execution. * Check if the WebContents is actually loaded and remove fake webContentsViewHelper.new.ts created by AI * Update view.ts * fix: prevent echo by exclude title * test: Then file "Draft of '新条目'.tid" should not exist in "{tmpDir}/wiki/tiddlers" * Revert "fix: prevent echo by exclude title" This reverts commit 86aa838d24. * fix: when move file to subwiki, delete old file * fix: prevent ipc echo change back to frontend * test: view might take longer to load * fix: minor issues * test: fix cleanup timeout * Update cleanup.ts * feat: capture webview screenshot * Update filesystemPlugin.feature * Update SyncArchitecture.md * rename * test: add some time to easy failed steps * Separate logs by test scenario for easier debugging * Update selectors for add and confirm buttons in tests Changed the CSS selectors for the add tiddler and confirm buttons in the filesystem plugin feature tests to use :has() with icon classes. This improves selector robustness and aligns with UI changes. * Ensure window has focus and is ready * Update window.ts * fix: webview screenshot capture prevent mini window to close * fix: Failed to take screenshot: Error: ENAMETOOLONG: name too long, open '/home/runner/work/TidGi-Desktop/TidGi-Desktop/userData-test/logs/screenshots/Agent workflow - Create notes- update embeddings- then search/2025-10-30T11-46-28-891Z-I type -在 wiki 工作区创建一个名为 AI Agent Guide 的笔记-内容是-智能体是一种可以执行任务的AI系统-它可以使用工具-搜索信息并与用户交互- in -chat input- element with selec-PASSED-page.png' * Update window.ts * feat: remove deprecated symlink subwiki approach * Update wiki.ts * fix: remove AI buggy bring window to front cause mini window test to fail * lint * Adjust wait time for draft saving in filesystemPlugin Increased wait time for file system plugin to save draft. * Adjust wait time for tiddler state stabilization Increased wait time to ensure tiddler state settles properly. * Refactor release workflow to simplify dependency installation Removed installation steps for x64 and arm64 dependencies, and adjusted the build process for plugins and native modules. * Enhance wait for IPC in filesystemPlugin feature Added a wait time to improve reliability of content update verification in CI.
2025-12-15 15:10:31 -08:00 · 2025-10-31 02:00:40 +08:00 · 2025-10-31 02:00:40 +08:00 · 7f5e1aa0cc
commit 7f5e1aa0cc
parent 7473612cec
54 changed files with 1787 additions and 778 deletions
--- a/docs/ErrorDuringStart.md
+++ b/docs/ErrorDuringStart.md
@ -93,6 +93,25 @@ Solution:
 node_modules/.bin/electron-rebuild -f -w better-sqlite3
 ```

+## Error: The module '/Users/linonetwo/Desktop/repo/TidGi-Desktop/node_modules/opencv4nodejs-prebuilt/build/Release/opencv4nodejs.node'
+
+```log
+was compiled against a different Node.js version using
+NODE_MODULE_VERSION 127. This version of Node.js requires
+NODE_MODULE_VERSION 135. Please try re-compiling or re-installing
+the module (for instance, using `npm rebuild` or `npm install`).
+```
+
+(The number above is smaller)
+
+Don't use `npm rebuild` or `npm install`, it doesn't works, it will still build for nodejs. We need to build with electron:
+
+```sh
+./node_modules/.bin/electron-rebuild
+```
+
+See <https://github.com/justadudewhohacks/opencv4nodejs/issues/401#issuecomment-463434713> if you still have problem rebuild opencv for @nut-tree/nut-js
+
 ## During test, The module 'node_modules\better-sqlite3\build\Release\better_sqlite3.node' was compiled against a different Node.js version using

 ```log
@ -113,25 +132,6 @@ cross-env ELECTRON_RUN_AS_NODE=true ./node_modules/.bin/electron ./node_modules/

 救急可以用 `chcp 65001 && pnpm run test:unit`，如果有空重启电脑，则在时区设置里找到「系统区域设置」里勾选「Unicode Beta版」，重启即可。

-## Error: The module '/Users/linonetwo/Desktop/repo/TidGi-Desktop/node_modules/opencv4nodejs-prebuilt/build/Release/opencv4nodejs.node'
-
-```log
-was compiled against a different Node.js version using
-NODE_MODULE_VERSION 127. This version of Node.js requires
-NODE_MODULE_VERSION 135. Please try re-compiling or re-installing
-the module (for instance, using `npm rebuild` or `npm install`).
-```
-
-(The number above is smaller)
-
-Don't use `npm rebuild` or `npm install`, it doesn't works, it will still build for nodejs. We need to build with electron:
-
-```sh
-./node_modules/.bin/electron-rebuild
-```
-
-See <https://github.com/justadudewhohacks/opencv4nodejs/issues/401#issuecomment-463434713> if you still have problem rebuild opencv for @nut-tree/nut-js
-
 ## Command failed with exit code 1

 When you see an error like:
--- a/docs/Testing.md
+++ b/docs/Testing.md
@ -302,7 +302,7 @@ When AI is fixing issues, you can let it add more logs for troubleshooting, and
 If you want to send frontend log to the log file, you can't directly use `import { logger } from '@services/libs/log';` you need to use `void window.service.native.log('error', 'Renderer: xxx', { ...additionalMetadata });`.
 Otherwise you will get [Can't resolve 'os' error](./ErrorDuringStart.md)

-Only use VSCode tool to read file. Don't ever use shell command to read file.
+Only use VSCode tool to read file. Don't ever use shell command to read file. Use shell command to read file may be immediately refused by user, because he don't want to manually approve shell commands.

 ## User profile

--- a/docs/internal/IPCSyncAdaptorAndFSAdaptor.md
+++ b/docs/internal/IPCSyncAdaptorAndFSAdaptor.md
@ -0,0 +1,292 @@
+# Sync Architecture: IPC and Watch-FS Plugins
+
+This document describes how the `tidgi-ipc-syncadaptor` (frontend) and `watch-filesystem-adaptor` (backend) plugins work together to provide real-time bidirectional synchronization between the TiddlyWiki in-memory store and the file system.
+
+## Architecture Overview
+
+```chart
+Frontend (Browser)          Backend (Node.js Worker)
+┌─────────────────┐        ┌──────────────────────┐
+│ TiddlyWiki      │        │ TiddlyWiki (Server)  │
+│ In-Memory Store │◄──────►│ File System Adaptor  │
+└────────┬────────┘        └──────────┬───────────┘
+         │                            │
+         │ IPC Sync                   │ Watch-FS
+         │ Adaptor                    │ Adaptor
+         │                            │
+         ├─ Save to FS ──────────────►│
+         │                            │
+         │◄───── SSE Events ──────────┤
+         │     (File Changes)         │
+         │                            │
+         │                            ├─ nsfw Watcher
+         │                            │  (File System)
+         └────────────────────────────┘
+```
+
+## Key Design Principles
+
+### 1. Single Source of Truth: File System
+
+- Backend watch-fs monitors the file system using `nsfw` library
+- All file changes (external edits, saves from frontend) flow through file system
+- Backend wiki state reflects file system state
+
+### 2. Two-Layer Echo Prevention
+
+**IPC Layer (First Defense)**:
+
+- `ipcServerRoutes.ts` tracks recently saved tiddlers in `recentlySavedTiddlers` Set
+- When `wiki.addTiddler()` triggers change event, filter out tiddlers in the Set
+- Prevents frontend from receiving its own save operations as change notifications
+
+**Watch-FS Layer (Second Defense)**:
+
+- When saving, watch-fs temporarily excludes file from monitoring for 200ms
+- Prevents watcher from detecting the file write operation
+- Combined with IPC filtering, ensures no echo even for rapid successive saves
+
+### 3. SSE-like Change Notification (IPC Observable)
+
+- Backend sends change events to frontend via IPC (not real SSE, but Observable pattern via `ipc-cat`)
+- Frontend subscribes to `getWikiChangeObserver$` observable from `window.observables.wiki`
+- Change events trigger tw's `syncFromServer()` to pull updates from backend, it will in return call our `getUpdatedTiddlers`
+
+## Plugin Responsibilities
+
+### Frontend: `tidgi-ipc-syncadaptor`
+
+Purpose: Bridge between frontend TiddlyWiki and backend file system
+
+IPC Communication Flow:
+
+- Frontend plugin calls `window.service.wiki` methods (provided by preload script)
+- Preload script uses `setupIpcServerRoutesHandlers.ts` (runs in view context) to set up IPC protocol handlers
+- These handlers forward requests to `ipcServerRoutes.ts` in wiki worker via IPC
+- Uses `tidgi://` custom protocol for RESTful-like API
+
+Key Functions:
+
+- `saveTiddler()`: Send tiddler to backend via IPC → `putTiddler` in `ipcServerRoutes.ts`
+- `loadTiddler()`: Request tiddler from backend via IPC → `getTiddler` in `ipcServerRoutes.ts`
+- `deleteTiddler()`: Request deletion via IPC → `deleteTiddler` in `ipcServerRoutes.ts`
+- `setupSSE()`: Subscribe to file change events from backend (via IPC Observable, not real SSE)
+- `getUpdatedTiddlers()`: Provide list of changed tiddlers to syncer
+
+Echo Prevention Strategy:
+
+Frontend plugin does NOT implement echo detection because:
+
+1. Cannot distinguish between "save to fs and watch fs echo back" and "external user text edit with unchanged 'modified' timestamp metadata"
+2. Echo prevention is handled by backend at two levels:
+   - **IPC level**: `ipcServerRoutes.ts` filters out change events from IPC saves using `recentlySavedTiddlers` Set
+   - **Watch-FS level**: `watch-filesystem-adaptor` temporarily excludes files during save operations
+
+### Backend: `watch-filesystem-adaptor`
+
+Purpose: Monitor file system and maintain wiki state
+
+Key Functions:
+
+- `saveTiddler()`: Write to file system with temporary exclusion
+- `deleteTiddler()`: Remove file with temporary exclusion
+- `initializeFileWatching()`: Setup `nsfw` watcher for main wiki and sub-wikis
+- `handleFileAddOrChange()`: Load changed files into wiki
+- `handleFileDelete()`: Remove deleted tiddlers from wiki
+
+Two-Layer Echo Prevention:
+
+#### Layer 1: IPC Level (in ipcServerRoutes.ts)
+
+```typescript
+async putTiddler(title, fields) {
+  // Mark as recently saved to prevent echo
+  this.recentlySavedTiddlers.add(title);
+  
+  // This triggers 'change' event synchronously
+  this.wikiInstance.wiki.addTiddler(new Tiddler(fields));
+  
+  // Change event handler filters it out and removes the mark
+}
+
+// Change event handler
+wiki.addEventListener('change', (changes) => {
+  const filteredChanges = {};
+  for (const title in changes) {
+    if (this.recentlySavedTiddlers.has(title)) {
+      // Filter out echo from IPC save
+      this.recentlySavedTiddlers.delete(title);
+      continue;
+    }
+    filteredChanges[title] = changes[title];
+  }
+  // Only send filtered changes to frontend
+  observer.next(filteredChanges);
+});
+```
+
+#### Layer 2: Watch-FS Level (in WatchFileSystemAdaptor.ts)
+
+```typescript
+async saveTiddler(tiddler) {
+  const filepath = await this.getTiddlerFileInfo(tiddler);
+  
+  // 1. Exclude file BEFORE saving
+  await this.excludeFile(filepath);
+  
+  // 2. Save to file system (calls parent saveTiddler)
+  await super.saveTiddler(tiddler);
+  
+  // 3. Re-include after delay
+  this.scheduleFileInclusion(filepath);
+  // After 200ms: includeFile(filepath) removes exclusion
+}
+```
+
+File Change Flow:
+
+```typescript
+handleNsfwEvents(events) {
+  events.forEach(event => {
+    const filepath = path.join(event.directory, event.file);
+    
+    // Skip if file is excluded (being saved)
+    if (this.excludedFiles.has(filepath)) return;
+    
+    // Load changed file into wiki
+    const tiddler = $tw.loadTiddlersFromFile(filepath);
+    $tw.syncadaptor.wiki.addTiddler(tiddler);
+    
+    // Wiki change event fires → SSE sends to frontend
+  });
+}
+```
+
+## Data Flow Examples
+
+### Example 1: User Edits in Frontend
+
+```text
+1. User clicks save in browser
+   ├─► Frontend: saveTiddler() called
+   │
+2. IPC call to backend
+   ├─► Backend: receives putTiddler request
+   │
+3. Backend IPC layer: Mark tiddler in recentlySavedTiddlers
+   ├─► ipcServerRoutes.ts adds title to Set
+   │
+4. Backend: wiki.addTiddler(tiddler)
+   ├─► Triggers 'change' event
+   ├─► Event handler checks recentlySavedTiddlers
+   ├─► Filters out this change (IPC echo prevention)
+   ├─► Removes title from recentlySavedTiddlers
+   │
+5. Backend Watch-FS layer: excludeFile(filepath)
+   ├─► File added to excludedFiles set
+   │
+6. Backend: write to file system
+   ├─► File content updated on disk
+   │
+7. nsfw detects file change
+   ├─► But file is in excludedFiles
+   ├─► Change event ignored (Watch-FS echo prevention)
+   │
+8. After 200ms delay
+   ├─► includeFile(filepath) removes exclusion
+```
+
+### Example 2: External Editor Modifies File
+
+```text
+1. User edits file in VSCode/Vim
+   ├─► File content changes on disk
+   │
+2. nsfw detects file change
+   ├─► File NOT in excludedFiles (not being saved)
+   │
+3. handleFileAddOrChange() called
+   ├─► Load tiddler from file
+   ├─► wiki.addTiddler(tiddler)
+   │
+4. Wiki fires 'change' event
+   ├─► ipcServerRoutes.ts checks recentlySavedTiddlers
+   ├─► Title NOT in Set (external edit, not IPC save)
+   ├─► Change passes through filter
+   ├─► IPC Observable sends event to frontend (via ipc-cat)
+   │
+5. Frontend receives change event
+   ├─► Adds to updatedTiddlers.modifications
+   ├─► Triggers syncFromServer()
+   │
+6. Frontend: loadTiddler() via IPC
+   ├─► Gets latest tiddler from backend
+   ├─► Updates frontend wiki
+   ├─► UI re-renders with new content
+```
+
+### Example 3: Sub-Wiki Synchronization
+
+```text
+1. Main wiki has sub-wiki folder linked by tag
+   ├─► watch-fs detects sub-wiki in tiddlywiki.info
+   │
+2. watch-fs starts additional watcher
+   ├─► Monitors SubWiki/ folder
+   │
+3. User saves tiddler with tag in frontend
+   ├─► Backend determines file should go to SubWiki/
+   ├─► Saves to SubWiki/Tiddler.tid
+   │
+4. Sub-wiki watcher detects new file
+   ├─► (Excluded during save, so no echo)
+   │
+5. External edit in SubWiki/Tiddler.tid
+   ├─► Sub-wiki watcher detects change
+   ├─► Updates main wiki's in-memory store
+   ├─► IPC Observable notifies frontend
+   ├─► Frontend syncs and displays update
+```
+
+## Key Configuration
+
+### File Exclusion
+
+- `FILE_EXCLUSION_CLEANUP_DELAY_MS = 200`: Time to keep file excluded after save
+- Prevents echo while allowing quick re-detection of external changes
+
+### SSE-like Debouncing (IPC Observable)
+
+- `debounce(syncFromServer, 500)`: Batch multiple file changes
+- Reduces unnecessary sync operations
+
+### Syncer Polling
+
+- `pollTimerInterval = 2_147_483_647`: Effectively disable polling
+- All updates come via IPC Observable (event-driven, not polling)
+
+## Troubleshooting
+
+### Changes Not Appearing in Frontend
+
+1. Check IPC Observable connection: Look for `[test-id-SSE_READY]` in logs
+2. Verify watch-fs is running: Look for `[test-id-WATCH_FS_STABILIZED]`
+3. Check file exclusion: Should see `[WATCH_FS_EXCLUDE]` and `[WATCH_FS_INCLUDE]`
+
+### Echo/Duplicate Updates
+
+1. Verify exclusion timing: 200ms should be sufficient
+2. Check for multiple watchers on same path
+3. Ensure frontend isn't doing its own timestamp-based filtering
+
+### Sub-Wiki Not Syncing
+
+1. Check sub-wiki detection: Look for `[WATCH_FS_SUBWIKI]` logs
+2. Verify tiddlywiki.info has correct configuration
+3. Check workspace `subWikiFolders` setting
+
+### Not Recommended
+
+- ❌ Timestamp-based echo detection (already tried, unreliable)
+- ❌ Frontend-side file watching (duplicates backend effort)
+- ❌ Polling-based synchronization (SSE is better)