Code/UI Interconversion System

Overview

OpenHCS provides a bidirectional code/UI interconversion system that enables editing between the TUI interface and generated Python code. This system addresses the limitation in scientific computing tools where users must choose between GUI convenience or code flexibility.

The system implements bidirectional conversion with round-trip fidelity preservation between representations.

Core Functionality

The system enables three critical workflows:

TUI → Code: Generate complete, executable Python code from TUI state
Code → TUI: Parse edited Python code back into TUI interface state
Round-trip Integrity: Maintain perfect consistency between representations

This allows researchers to: - Use the TUI for rapid prototyping and visual feedback - Switch to code editing for complex parameter tuning and bulk modifications - Leverage external editors, version control, and collaborative development tools - Maintain a single source of truth that can be represented in either form

Three-Tier Generation Architecture

The code generation system follows OpenHCS’s encapsulation pattern with three hierarchical tiers:

Function Patterns (Tier 1)
       ↓ (encapsulates imports)
Pipeline Steps (Tier 2)
       ↓ (encapsulates all pattern imports)
Orchestrator Config (Tier 3)
       ↓ (encapsulates all pipeline imports)
Complete Executable Script

Tier 1: Function Pattern Generation

# Generated from FunctionListEditor code button
from openhcs.processing.backends.filters.gaussian_filter import gaussian_filter
from openhcs.constants.constants import VariableComponents

pattern = gaussian_filter(sigma=2.0, preserve_dtype=True)

Tier 2: Pipeline Step Generation

# Generated from PipelineEditor code button
from openhcs.core.steps.function_step import FunctionStep
from openhcs.constants.constants import VariableComponents
# Function imports from encapsulated patterns
from openhcs.processing.backends.filters.gaussian_filter import gaussian_filter
from openhcs.processing.backends.analysis.cell_counting import count_cells

pipeline_steps = []

step_1 = FunctionStep(
    func=(gaussian_filter, {'sigma': 2.0, 'preserve_dtype': True}),
    name="gaussian_filter",
    variable_components=[VariableComponents.PLATE]
)
pipeline_steps.append(step_1)

step_2 = FunctionStep(
    func=(count_cells, {'method': 'watershed'}),
    name="count_cells",
    variable_components=[VariableComponents.PLATE]
)
pipeline_steps.append(step_2)

Tier 3: Orchestrator Configuration Generation

# Generated from PlateManager code button
from openhcs.core.steps.function_step import FunctionStep
from openhcs.constants.constants import VariableComponents
from openhcs.core.config import GlobalPipelineConfig, PathPlanningConfig
# Function imports from all encapsulated pipelines
from openhcs.processing.backends.filters.gaussian_filter import gaussian_filter
from openhcs.processing.backends.analysis.cell_counting import count_cells

plate_paths = ['/path/to/plate1', '/path/to/plate2']

global_config = GlobalPipelineConfig(
    num_workers=16,
    path_planning=PathPlanningConfig(
        output_dir_suffix="_processed"
    )
)

pipeline_data = {}
# Steps for each plate...

Encapsulation Pattern

The system implements a strict upward import encapsulation pattern:

Function Pattern:
├── Imports: [gaussian_filter]
└── Code: pattern = gaussian_filter(...)

Pipeline Steps:
├── Imports: [FunctionStep, VariableComponents] + [gaussian_filter, count_cells]
└── Code: pipeline_steps = [step_1, step_2, ...]

Orchestrator Config:
├── Imports: [GlobalPipelineConfig, ...] + [all pipeline imports]
└── Code: plate_paths, global_config, pipeline_data

Bidirectional Conversion Workflow

The complete round-trip workflow ensures perfect fidelity:

Code Generation (TUI → Code)

User clicks "Code" button in TUI widget
Widget extracts current state (functions, parameters, configuration)
generate_python_source(...) called with the appropriate Assignment/CodeBlock
Import collection system traverses data structures
Complete Python code generated with all imports
TerminalLauncher creates temporary file with code
User's $EDITOR launched for editing

Code Parsing (Code → TUI)

User saves and exits editor
TerminalLauncher detects completion via signal file
Edited code read from temporary file
Code executed in isolated namespace: exec(edited_code, namespace)
Expected variables extracted from namespace:
   - pattern (for function patterns)
   - pipeline_steps (for pipeline steps)
   - plate_paths, pipeline_data, global_config (for orchestrator)
Widget state updated with parsed data
TUI interface refreshes to reflect changes
Temporary files cleaned up

Error Handling and Validation

The system provides comprehensive error handling at each conversion step:

Syntax Validation: Python syntax errors caught and reported with line numbers
Variable Validation: Missing expected variables detected and reported
Type Validation: Incorrect data types validated against expected structures
Import Resolution: Missing imports detected during execution
State Consistency: TUI state validated after updates

Terminal Integration Architecture

The TerminalLauncher service manages the external editor integration:

File-Based Communication Pattern

TUI Process                    Editor Process
     │                              │
     ├─ Create temp file            │
     ├─ Launch $EDITOR ────────────→│
     ├─ Start polling               │
     │                              ├─ Edit file
     │                              ├─ Save & exit
     │                              └─ Create signal file
     ├─ Detect signal file          │
     ├─ Read edited content         │
     ├─ Execute callback            │
     └─ Cleanup temp files          │

Asynchronous Polling System

The system uses asynchronous polling to detect editor completion without blocking the TUI:

async def poll_for_completion():
    while True:
        if os.path.exists(signal_file):
            # Editor completed, process changes
            with open(file_path, 'r') as f:
                content = f.read()
            callback(content)  # Update TUI state
            break
        await asyncio.sleep(0.1)  # Non-blocking poll

Editor Integration

The system respects user preferences and environment:

Environment Variable: Uses $EDITOR or defaults to nano
Terminal Compatibility: Works with vim, emacs, nano, micro
SSH Support: Full functionality over SSH connections
Unicode Support: Proper handling of special characters and encoding

Shared Update Utility: CodeEditorFormUpdater

The CodeEditorFormUpdater module provides centralized utilities for handling code editor form updates across all editing contexts.

Purpose and Design

This utility eliminates code duplication between different editors (step editor, config window, future editors) by providing:

Explicit Field Extraction: Parse edited code to identify which fields were explicitly set
Lazy Constructor Patching: Ensure exec()-created instances preserve None vs concrete value distinction
Form Update Logic: Recursively update form managers with new values while preserving lazy resolution
Nested Dataclass Handling: Properly handle nested dataclass updates without triggering unwanted lazy resolution

Core Methods

class CodeEditorFormUpdater:
    @staticmethod
    def extract_explicitly_set_fields(code: str, class_name: str, variable_name: str) -> Set[str]:
        """Parse code to extract which fields were explicitly set."""
        # Returns set of field names that appear in constructor call

    @staticmethod
    def update_form_from_instance(form_manager, new_instance, explicitly_set_fields, broadcast_callback=None):
        """Update form manager with values from new instance."""
        # Only updates explicitly set fields
        # Handles nested dataclasses recursively
        # Triggers placeholder refresh and cross-window updates

    @staticmethod
    @contextmanager
    def patch_lazy_constructors():
        """Context manager that patches lazy dataclass constructors."""
        # Ensures exec()-created instances only set explicitly provided kwargs
        # Allows unspecified fields to remain None

Lazy Constructor Patching

The patching system ensures proper None vs concrete value distinction during code execution:

# Without patching - lazy resolution happens during exec
exec("config = GlobalPipelineConfig(num_workers=8)")
# Result: All unspecified fields get default values from lazy resolution

# With patching - unspecified fields remain None
with CodeEditorFormUpdater.patch_lazy_constructors():
    exec("config = GlobalPipelineConfig(num_workers=8)")
# Result: num_workers=8, all other fields=None (can inherit from context)

Integration Pattern

All code editing contexts follow this pattern:

def _handle_edited_code(self, edited_code: str):
    # 1. Extract explicitly set fields
    explicitly_set_fields = CodeEditorFormUpdater.extract_explicitly_set_fields(
        edited_code,
        class_name=self.config_class.__name__,
        variable_name='config'
    )

    # 2. Execute code with lazy constructor patching
    namespace = {}
    with CodeEditorFormUpdater.patch_lazy_constructors():
        exec(edited_code, namespace)

    new_instance = namespace.get('config')

    # 3. Update form using shared utility
    CodeEditorFormUpdater.update_form_from_instance(
        self.form_manager,
        new_instance,
        explicitly_set_fields,
        broadcast_callback=self._broadcast_config_changed
    )

Benefits

Consistency: All editors behave identically when handling code updates
Maintainability: Single source of truth for complex update logic
Correctness: Centralized handling of lazy resolution edge cases
Extensibility: New editors can reuse existing logic without reimplementation

GlobalPipelineConfig Context Management

The configuration window implements special handling for GlobalPipelineConfig to ensure thread-local context stays synchronized with form edits.

Context Synchronization

When editing GlobalPipelineConfig, the system maintains synchronization between:

Form State: Current values in the UI form
Thread-Local Context: Global context used for lazy placeholder resolution
Other Windows: All open windows that depend on global config (steps, pipelines, etc.)

Per-Field Synchronization

Each field change triggers immediate context update:

def _on_global_config_field_changed(self, param_name: str, value: Any):
    """Keep thread-local global config context in sync with live edits."""
    if self._saving or self._suppress_global_context_sync:
        return
    self._sync_global_context_with_current_values(param_name)

This ensures that:

Placeholder values in other windows update immediately
Lazy resolution uses current global config values
Cross-window consistency is maintained

Bulk Update Optimization

During code editor saves, per-field sync is suppressed to avoid redundant updates:

# Suppress per-field sync during bulk update
self._suppress_global_context_sync = True
try:
    # Update all fields from code
    CodeEditorFormUpdater.update_form_from_instance(...)
    # Single sync after all updates
    self._sync_global_context_with_current_values()
finally:
    self._suppress_global_context_sync = False

Cancel Restoration

When user cancels config editing, the original context is restored:

def reject(self):
    """Handle dialog rejection (Cancel button)."""
    if self._global_context_dirty:
        # Restore original snapshot
        set_global_config_for_editing(
            GlobalPipelineConfig,
            copy.deepcopy(self._original_global_config_snapshot)
        )
        # Trigger cross-window refresh
        ParameterFormManager.trigger_global_cross_window_refresh()

This prevents context pollution when users:

Make experimental changes and cancel
Accidentally modify global config
Want to revert to original state

Benefits

Immediate Feedback: Changes visible in all windows instantly
Consistency: No stale values across windows
Undo Support: Cancel properly reverts all changes
Performance: Bulk updates optimized to avoid redundant syncs

Widget Integration Pattern

All TUI widgets implement a consistent code button pattern:

Standard Implementation

async def _handle_code_button(self):
    """Standard code button handler pattern."""
    try:
        # 1. Generate complete code with imports
        python_code = generate_complete_*_code(self.data, clean_mode=False)

        # 2. Launch editor with callback
        launcher = TerminalLauncher(self.app)
        await launcher.launch_editor_for_file(
            file_content=python_code,
            file_extension='.py',
            on_save_callback=self._handle_edited_code
        )
    except Exception as e:
        self.app.show_error("Code Generation Error", str(e))

def _handle_edited_code(self, edited_code: str):
    """Standard callback for handling edited code."""
    try:
        # 3. Parse edited code
        namespace = {}
        exec(edited_code, namespace)

        # 4. Extract expected variables
        if 'expected_variable' in namespace:
            new_data = namespace['expected_variable']
            self._apply_changes(new_data)
        else:
            self.app.show_error("Parse Error", "Expected variable not found")

    except SyntaxError as e:
        self.app.show_error("Syntax Error", f"Invalid Python syntax: {e}")
    except Exception as e:
        self.app.show_error("Edit Error", f"Failed to parse code: {e}")

Widget-Specific Variables

Each widget expects specific variables in the edited code:

FunctionListEditor: pattern = ... (function pattern)
PipelineEditor: pipeline_steps = [...] (list of FunctionStep objects)
PlateManager: plate_paths, pipeline_data, global_config (orchestrator config)

Performance and Scalability

The system is designed for efficiency with large pipelines:

Import Collection Optimization

Recursive Traversal: Efficient depth-first search of data structures
Deduplication: Set-based import collection prevents duplicates
Module Filtering: Only OpenHCS modules included in generated imports
Lazy Evaluation: Imports collected only when code generation requested

Memory Management

Temporary Files: Automatic cleanup prevents disk space leaks
Namespace Isolation: Code execution in isolated namespace prevents pollution
Callback Cleanup: Automatic cleanup of callback references

Debouncing and Batching

The dual editor window implements debouncing for function editor synchronization:

Batched Updates: Rapid parameter changes batched into single sync operation
QTimer-Based: Uses Qt’s timer system for efficient debouncing
Performance Impact: Reduces UI lag during rapid editing by 80-90%

Scalability Metrics

Function Patterns: <1ms generation time for typical patterns
Pipeline Steps: <10ms for pipelines with 20+ steps
Individual Steps: <5ms for step code generation with nested configs
Orchestrator Config: <100ms for multi-plate configurations with complex pipelines
Memory Usage: <5MB additional memory during code generation
Debounce Latency: <10ms for batched function editor sync