stable-diffusion-webui/docs/serena.md

Serena — Refactor Program Ledger

Program name: Serena
Source repo: AUTOMATIC1111/stable-diffusion-webui
Fork workspace: m-cahill/serena (origin)
Source of truth: This document
Posture: Behavior-preserving by default, audit-first, milestone-governed

---

1. Project Identity

Serena is a governed refactor program for AUTOMATIC1111/stable-diffusion-webui. The goal is to transform the codebase from its current state (audit score ~2.4–2.6/5) to a 5/5 architecture with clear separation of concerns, testable runtime, and stable extension API.

Serena Refactor Principles

This refactor program follows strict behavior-preserving governance.

Core principles:

1. Behavior preservation by default Existing runtime behavior must remain stable unless explicitly changed.

2. Small milestones Each milestone introduces minimal surface change.

3. Runtime seams before architecture changes Isolation is introduced before structural refactors.

4. Extension compatibility The extension ecosystem must remain functional unless explicitly versioned.

5. Evidence-based closeout Each milestone must end with verifiable CI evidence.

Source-of-truth hierarchy:

1. docs/serena.md — This ledger (phases, milestones, invariants)
2. docs/sdwebuirefactoraudit.md — Baseline audit (architecture, modularity, refactor strategy)
3. docs/sdwebuiaudit.md — Supplementary audit (DX, phased plan)
4. Milestone docs under docs/milestones/MNN/

---

2. Current Baseline

Item	Value
Audited baseline SHA	82a973c04367123ae98bd9abdf80d9eda9b910e2
Baseline tag	baseline-pre-refactor (annotated, immutable)
Initial audit score	2.4 / 5 (sdwebuirefactoraudit)
Upstream	https://github.com/AUTOMATIC1111/stable-diffusion-webui.git

Top architectural problems (from audit):

• Global state hub: shared.opts, shared.state, shared.sd_model used by dozens of modules
• No test tiers or coverage gate
• God modules: processing.py (~1793 LOC), ui.py (~984 LOC), api/api.py (~750 LOC)
• Dependency and CI hygiene: mixed pinning, lockfile gitignored, actions use tags not SHA

---

3. Proposed Phase Map (Provisional)

Can evolve after M00 evidence.

Phase I — Baseline & Guardrails (M00–M04)

Milestone	Title
M00	Program kickoff, baseline freeze, phase map, E2E verification
M01	CI truthfulness, SHA pinning, smoke path
M02	Local dev guardrails, CONTRIBUTING, repeatable verification
M03	Test architecture (smoke / quality / nightly)
M04	Coverage/security/reproducibility guardrails

Phase II — Runtime Seam Preparation (M05–M09)

Milestone	Title
M05	Override isolation / temporary opts seam
M06	Prompt/seed prep extraction
M07	Opts snapshot introduction
M08	process_images_inner snapshot threading
M09	Execution context/state seam

Phase III — Runner & Service Boundary (M10–M15)

Milestone	Title
M10	ProcessingRunner skeleton
M11	Runner lifecycle surface (prepare / execute / finalize)
M12	Runtime instrumentation hooks
M13	txt2img path through runner
M14	API integration
M15	background/queue runner preparation

Phase IV — Runtime Extraction (M16–M20)

Milestone	Title
M16	Runtime module extraction
M17	Sampler runner extraction
M18	Decode/save separation
M19	Model provider interface
M20	Runtime tests with mockable boundaries

Phase V — UI & Extension Stabilization (M21–M25)

Milestone	Title
M21	UI tab registry
M22	txt2img/img2img tab modularization
M23	Settings/extensions modularization
M24	Extension API version/contract
M25	Deprecation/compatibility scaffolding

Phase VI — Hardening & Reproducibility (M26–M30)

Milestone	Title
M26	Locked manifests / npm ci / CI env stabilization
M27	Coverage and complexity gates
M28	Security/supply-chain evidence
M29	Health/perf verification
M30	QA/evidence publishing

Phase VII — Release Lock / 5.0 Closure (M31–M33)

Milestone	Title
M31	Architecture lock
M32	Evidence/audit closure
M33	Release-ready 5/5 close

---

4. Milestone Ledger

Milestone	Title	Status	Branch	PR	Commit	CI Run(s)	Audit Score / Notes	Completed At
M00	Program kickoff, baseline freeze, phase map, E2E verification	Completed	m00-kickoff-baseline-e2e	—	cdfe1285	Linter 22794525690 ✓; Tests 22794525698 ✗ (pre-existing CLIP/pkg_resources)	Baseline 2.4/5	2025-03-07
M01	CI truthfulness, SHA pinning, smoke path	Completed	m01-ci-truthfulness	—	2f664049	Linter 22814396752 ✓; Tests 22814850488 (server ✓, 17 pass, img2img/txt2img 500)	4.7 / 5	2026-03-08
M02	API CI truthfulness, local dev guardrails	Completed	m02-api-ci-truthfulness	—	7484170d	Linter 22831756517 ✓; Tests 22831756504 ✓ (33/33 pass)	4.9 / 5	2026-03-08
M03	Test architecture (smoke / quality / nightly)	Completed	m03-test-architecture	#2	975dda4b	Linter ✓; Smoke 22834384359 ✓; Quality 22834861040 ✓	5.0 / 5	2026-03-09
M04	Coverage/security/reproducibility guardrails	Completed	m04-coverage-guardrails	#4	47439cac	Quality 22871471473 ✓ (coverage 40%, pip-audit, verify_pinned_deps)	5.0 / 5	2026-03-09
M05	Override isolation / temporary opts seam	Completed	m05-override-isolation	#18 (+ #19 fix)	ae161cbb	Quality 22888808682 ✓	5.0 / 5	2026-03-10
M06	Prompt/seed prep extraction	Completed	m06-prompt-seed-prep	#20	6744152a	Quality 22890285319 ✓	5.0 / 5	2026-03-10
M07	Opts snapshot introduction	Completed	m07-opts-snapshot	#22	8ea50d35	Quality 22983583947 ✓	5.0 / 5	2026-03-12
M08	Opts snapshot threading	Completed	m08-snapshot-threading	#24	710a0abd	Quality 22984445599 ✓	5.0 / 5	2026-03-12
M09	Execution context introduction	Completed	m09-execution-context	#26	2c6a2510	Quality 22986731960 ✓	5.0 / 5	2026-03-12
M10	ProcessingRunner skeleton	Completed	m10-processing-runner	#27 (+ #28 fix)	0d11b587	Quality 22988627838 ✓	5.0 / 5	2026-03-12
M11	Runner lifecycle surface	Completed	m11-runner-lifecycle	#30	08ac1c0e	Quality 22989978348 ✓	5.0 / 5	2026-03-12
M12	Runtime instrumentation hooks	Completed	m12-runner-instrumentation	—	46cf6d1c	Quality 23037656379 ✓	5.0 / 5	2026-03-13
M13	txt2img path through runner	Completed	m13-txt2img-runner	#31	4dd04999	Smoke 23038170275 ✓; Linter 23072709504 ✓; Quality 23072709479 ✓	5.0 / 5	2026-03-13

M05: Introduced temporary_opts() context manager — first Phase II runtime seam. Isolates override_settings mutation from global shared.opts; preserves behavior (opts.set, setattr restore, k in opts.data). Model/VAE reload and token merging remain in process_images. Enables future opts snapshot injection (M07).

M06: Extracted prepare_prompt_seed_state(p) into modules/prompt_seed_prep.py. Second Phase II runtime seam. Populates p.all_seeds and p.all_subseeds; setup_prompts and fill_fields_from_opts unchanged. Enables M07 opts snapshot and M09 execution context.

M07: Introduced create_opts_snapshot(opts) in modules/opts_snapshot.py. Third Phase II runtime seam. Captures deterministic snapshot of opts.data in process_images_inner after prepare_prompt_seed_state; stored on p.opts_snapshot. Write-only in M07; enables M08 snapshot threading.

M08: Threaded p.opts_snapshot into process_images_inner for save-related reads. Fourth Phase II runtime seam. Migrated 12 opts (save_images_before_face_restoration, samples_format, grid_save, etc.) from shared.opts to p.opts_snapshot. save_samples(), sample_hr_pass(), metadata unchanged. Enables M09 execution context.

M09: Introduced RuntimeContext in modules/runtime_context.py. Fifth Phase II runtime seam. Attached p.runtime_context in process_images_inner() after opts_snapshot (model, opts_snapshot, device, state, cmd_opts). Write-only in M09; no migration of shared.* reads yet. Completes Phase II — Runtime Seam Preparation. Enables Phase III ProcessingRunner.

M10: Introduced ProcessingRunner in modules/runtime/runner.py. First Phase III execution boundary. process_images delegates through runner; ProcessingRequest wraps StableDiffusionProcessing. Zero blast radius; all callers unchanged. Phase III roadmap updated (M11 lifecycle, M12 instrumentation, M13 txt2img, M14 API, M15 queue). Enables M11 Runner lifecycle surface.

M11: Introduced lifecycle surface on ProcessingRunner: prepare → execute → finalize. run() delegates through stages; pass-through behavior; identical outputs. test_runner_lifecycle_order verifies lifecycle structure. Stable execution surface enables M12 instrumentation, progress hooks, cancellation, queue runners.

M12: Introduced optional instrumentation hooks on ProcessingRunner: on_prepare, on_execute, on_finalize. Hooks no-op by default; lifecycle order prepare → on_prepare → execute → on_execute → finalize → on_finalize. test_runner_hooks_called verifies hook invocation. Enables M13+ progress, cancellation, queue runners.

M13: Verification milestone. Confirmed txt2img path flows through process_images → ProcessingRunner (no routing changes; M10 already delegates). Added test_txt2img_path_uses_runner contract test. Runner boundary proven with real consumer. Enables M14 API integration.

---

5. Standing Invariants

Repo-wide non-negotiables for this program:

• No silent behavior drift — All changes must preserve or explicitly document behavior
• No CI weakening — Do not relax checks, thresholds, or truthfulness
• Preserve extension behavior — Unless intentionally versioned in a milestone
• Preserve API/UI semantics — Unless milestone explicitly approves change
• Evidence-first closeout — Every milestone closes with documented evidence

---

6. Invariant Registry

These invariants must remain stable throughout the Serena refactor program unless explicitly revised by a milestone.

This registry provides cross-milestone contract stability.

Surface	Description	Verification
CLI	Command flags and output behavior remain stable	Snapshot tests
API	JSON response schemas remain compatible	Contract tests
File formats	Serialized artifacts and saved images remain compatible	Schema validation
Public modules	Import surfaces remain available unless versioned	API compatibility tests
Extension API	Extension loading behavior and hooks remain stable	Extension integration tests
Generation semantics	txt2img / img2img parameter behavior preserved	E2E smoke tests

Notes:

• Any invariant modification must be documented in the milestone plan.
• Regression verification must be automated where possible.