Technical deep-dive into the architecture decisions behind CaptureOCR

CaptureOCR is a Windows desktop application that transforms screenshots into structured data (Markdown tables, CSV) using local OCR. Built with Python, PySide6, and ONNX Runtime, it processes everything on-device with zero cloud dependencies. Product link: CaptureOCR on Microsoft Store — $2.99 one-time, no subscription. High-Level Architecture ┌─────────────────────────────────────────────────────────────┐ │ UI Layer (PySide6) │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ │ │ Full-screen │ │ Region │ │ Canvas/Annotation │ │ │ │ Selection │ │ Drawer │ │ Toolbar │ │ │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ └────────────────────────────┬────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Application Services │ │ ┌─────────────────┐ ┌─────────────────────────────────┐ │ │ │ Capture Manager │ │ OCR Workflow Orchestrator │ │ │ │ - mss grab │ │ - Queue management │ │ │ │ - DPI mapping │ │ - Model tier switching │ │ │ │ - Multi-monitor │ │ - Progress reporting │ │ │ └─────────────────┘ └─────────────────────────────────┘ │ │ │ │ │ ┌────────▼────────┐ │ │ │ Export Service │ │ │ │ - Markdown/CSV │ │ │ │ - Session (.snipx)│ │ │ └─────────────────┘ │ └────────────────────────────┬────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Infrastructure Layer │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ │ │ OCR Backend │ │ Tiling │ │ Scroll Capture │ │ │ │ (RapidOCR + │ │ Engine │ │ (Row signatures) │ │ │ │ ONNX Runtime)│ │ │ │ │ │ │ └──────────────┘ └──────────────┘ └──────────────────┘ │ └─────────────────────────────────────────────────────────────┘ Key Design Decisions 1. Single OCR Worker Thread Decision: All model inference runs on a dedicated background thread. Rationale: ONNX Runtime's InferenceSession is not guaranteed thread-safe Avoids complex locking around model loads/unloads For a desktop tool processing ~10 regions, serial throughput is sufficient Predictable memory profile (no concurrent model instances) # Simplified worker pattern class OcrWorker(QThread): result_ready = pyqtSignal(OcrResult) def __init__(self): super().__init__() self.queue = Queue() self.session = None self.current_model = None def run(self): while True: task = self.queue.get() if task is None: break result = self._process(task) self.result_ready.emit(result) 2. Two-Tier Model Strategy Tier Model Size Behavior Fast PP-OCRv6 tiny 6.6 MB Bundled in MSIX, preloaded at startup, stays resident Accurate PP-OCRv6 small 29.8 MB Downloaded on demand (SHA-256 verified), unloads after 5s idle Implementation: class ModelManager: def __init__(self): self.tiny_session = self._load_tiny() # Always loaded self.small_session = None self.last_small_use = 0 def get_session(self, tier: str): if tier == "fast": return self.tiny_session if self.small_session is None: self.small_session = self._download_and_load_small() self.last_small_use = time.time() return self.small_session def maybe_unload_small(self): if self.small_session and time.time() - self.last_small_use > 5: self.small_session = None gc.collect() 3. DPI-Aware Multi-Monitor Capture Windows captures in physical pixels; Qt works in logical coordinates. On 125-150% scaling (and mixed-DPI setups), this mismatch causes blurry captures or offset selections. Solution: Per-monitor ScreenInfo with both logical and physical rects + device pixel ratio. @dataclass class ScreenInfo: index: int logical_rect: QRect # Qt coordinates physical_rect: QRect # Screen pixels dpr: float # Device pixel ratio def logical_to_physical(screen: ScreenInfo, logical_rect: QRect) -> QRect: """Convert Qt logical rect to physical pixels for mss capture.""" x = int((logical_rect.x() - screen.logical_rect.x()) * screen.dpr) y = int((logical_rect.y() - screen.logical_rect.y()) * screen.dpr) w = int(logical_rect.width() * screen.dpr) h = int(logical_rect.height() * screen.dpr) return QRect(x, y, w, h) 4. Tiled OCR for Long Images Feeding a 20,000px tall screenshot into a 1600px OCR cap scales text to ~2px — unreadable. Instead, tile vertically: Tile parameters: - Target height: 1600px - Overlap: 64px (≈2 lines at 1080p) - Core bands: [0, 1536), [1536, 3072), ... - Each tile extends 32px above/below core for context - Line assigned to tile where box center Y falls in core band This ensures every line is processed at native resolution, claimed exactly once. 5. Row-Signature Scroll Stitching Template matching fails for scrolling (neither frame contains the other). Instead: Convert frame to grayscale, ignore right 8% (scrollbars) Compress each row to 32 column-bucket means → (H, 32) signature For each candidate shift s, compute MAE between prev[s:] and cur[:H-s] Accept shift only if best score < threshold AND clearly better than median Sticky headers: retry skipping 8%/16%/25% of top rows Adaptive wheel step: measure first scroll, target ~45% of selection height Performance Characteristics Metric Tiny (Fast) Small (Accurate) Model size 6.61 MB 29.79 MB Load time ~1.35s ~1.06s Avg latency (23-img set) 78.7 ms 257.5 ms P95 latency 144 ms 471 ms Char accuracy (CER) 99.24% 99.69% Test set: 23 mixed Chinese/English screenshots, 1600px cap, local CPU. Not a public benchmark — treat as order of magnitude. Distribution Format: MSIX (runFullTrust), distributed via Microsoft Store Size: ~126 MB (includes tiny model, ONNX Runtime, Python runtime) Dependencies: Zero — fully self-contained Auto-updates: Handled by Store Lessons Learned Detection config matters more than model choice: Switching from limit_type="min" (upscales short side) to limit_type="max" (only downscale long side) was a bigger win than model tuning. Disable classification for screenshots: use_cls=False — the 0°/180° classifier adds latency with no signal for upright captures. DPI bugs are subtle: Logical/physical coordinate mismatch only appears on scaled displays. Test on 125%, 150%, and mixed-DPI multi-monitor. Serialization format for reproducibility: The .snipx session package (ZIP with session.json + original PNGs + Markdown/CSV) enables lossless round-tripping and collaboration. Resources GitHub: github.com/lijiannan1241/CaptureOCR — product page & issue tracker Microsoft Store: CaptureOCR Source availability: Application is closed-source; repo hosts product page and issue tracker only Disclosure: I'm the developer of CaptureOCR. This is a paid product ($6.99 one-time). The application source code is not public — the repository serves as the product page and issue tracker.