Microscope Handler Integration

OpenHCS achieves microscope-agnostic processing through a handler system that abstracts the unique characteristics of different imaging platforms while providing a unified interface to the pipeline system.

Why Microscope Abstraction Matters

High-content screening involves diverse microscope platforms (Opera Phenix, ImageXpress, etc.), each with distinct:

• Directory structures: Flat vs hierarchical organization

• Filename patterns: Different field, well, and channel encoding schemes

• Metadata formats: XML, proprietary formats, embedded TIFF tags

• File organization: Single files vs multi-file series

Without abstraction, pipelines would need platform-specific logic throughout, making them brittle and hard to maintain. The handler system isolates these differences behind a clean interface.

Architecture: Composition Over Inheritance

The handler system uses composition rather than monolithic inheritance, separating concerns into specialized components:

class MicroscopeHandler(ABC, metaclass=MicroscopeHandlerMeta):
    """Composed class for handling microscope-specific functionality."""

    def __init__(self, parser: Optional[FilenameParser], metadata_handler: MetadataHandler):
        """Initialize with parser and metadata handler instances."""
        self.parser = parser
        self.metadata_handler = metadata_handler
        self.plate_folder: Optional[Path] = None

    @property
    @abstractmethod
    def root_dir(self) -> str:
        """Root directory where virtual workspace preparation starts."""
        pass

    @property
    @abstractmethod
    def compatible_backends(self) -> List[Backend]:
        """Storage backends this handler is compatible with, in priority order."""
        pass

This design enables:

• Independent evolution: Parser and metadata logic can change separately

• Testability: Each component can be tested in isolation

• Reusability: Common parsing logic can be shared across similar formats

• Extensibility: New microscope formats require minimal code

Filename Parsers and Metadata Handlers

The core of microscope abstraction lies in two critical components that handle format-specific details:

Filename Parser Implementation

Each microscope format has unique filename conventions. Parsers extract semantic information from these patterns using the Parser Metaprogramming System for dynamic interface generation:

class ImageXpressFilenameParser(FilenameParser):
    """Parser for ImageXpress filename format with centralized component configuration."""

    # FILENAME_COMPONENTS is automatically set to AllComponents + ['extension']
    # by the FilenameParser.__init__() → GenericFilenameParser.__init__() chain

    # Actual regex pattern from codebase - supports placeholders and optional components
    # Supports: well, site, channel, z_index, timepoint
    _pattern = re.compile(r'(?:.*?_)?([A-Z]\d+)(?:_s(\d+|\{[^\}]*\}))?(?:_w(\d+|\{[^\}]*\}))?(?:_z(\d+|\{[^\}]*\}))?(?:_t(\d+|\{[^\}]*\}))?(?:_.*?)?(\.\w+)?$')

    def parse_filename(self, filename: str) -> Optional[Dict[str, Any]]:
        """Parse ImageXpress filename, handling placeholders like {iii}."""
        basename = Path(str(filename)).name
        match = self._pattern.match(basename)

        if match:
            well, site_str, channel_str, z_str, t_str, ext = match.groups()

            # Helper to parse components or return None for placeholders
            parse_comp = lambda s: None if not s or '{' in s else int(s)

            return {
                'well': well,
                'site': parse_comp(site_str),
                'channel': parse_comp(channel_str),
                'z_index': parse_comp(z_str),
                'timepoint': parse_comp(t_str),
                'extension': ext if ext else '.tif'
            }
        return None

# Component-specific methods are automatically generated at runtime:
# - self.validate_well(), self.validate_site(), etc.
# - self.extract_well(), self.extract_site(), etc.
# - All based on AllComponents configuration

class OperaPhenixFilenameParser(FilenameParser):
    """Parser for Opera Phenix format with centralized component configuration."""

    # FILENAME_COMPONENTS is automatically set to AllComponents + ['extension']
    # by the FilenameParser.__init__() → GenericFilenameParser.__init__() chain

    # Actual regex pattern - supports row, col, site (field), z_index (plane), channel, timepoint (sk)
    # sk = stack/timepoint, fk = field stack, fl = focal level (optional)
    _pattern = re.compile(r"r(\d{1,2})c(\d{1,2})f(\d+|\{[^\}]*\})p(\d+|\{[^\}]*\})-ch(\d+|\{[^\}]*\})(?:sk(\d+|\{[^\}]*\}))?(?:fk\d+)?(?:fl\d+)?(?:_.*?)?(\.\w+)$", re.I)

    def parse_filename(self, filename: str) -> Optional[Dict[str, Any]]:
        """Parse Opera Phenix filename with row/col to well conversion."""
        basename = os.path.basename(filename)
        match = self._pattern.match(basename)

        if match:
            row, col, site_str, z_str, channel_str, sk_str, ext = match.groups()

            # Helper function for placeholder handling
            def parse_comp(s):
                if not s or '{' in s:
                    return None
                return int(s)

            # Convert row/col to well format (R01C01)
            well = f"R{int(row):02d}C{int(col):02d}"

            return {
                'well': well,
                'site': parse_comp(site_str),
                'channel': parse_comp(channel_str),
                'wavelength': parse_comp(channel_str),  # Backward compatibility
                'z_index': parse_comp(z_str),
                'timepoint': parse_comp(sk_str),  # sk = stack/timepoint
                'extension': ext if ext else '.tif'
            }
        return None

# Component-specific methods are automatically generated at runtime:
# - self.validate_well(), self.validate_site(), etc.
# - self.extract_well(), self.extract_site(), etc.
# - All based on AllComponents configuration

Metadata Handler Implementation

Metadata handlers extract acquisition parameters and plate layout information. All handlers implement the MetadataHandler ABC:

class MetadataHandler(ABC):
    """Abstract base class for handling microscope metadata."""

    @abstractmethod
    def find_metadata_file(self, plate_path: Union[str, Path]) -> Path:
        """Find the metadata file for a plate (e.g., .HTD, .xml)."""
        pass

    @abstractmethod
    def get_grid_dimensions(self, plate_path: Union[str, Path]) -> Tuple[int, int]:
        """Get grid dimensions for stitching from metadata."""
        pass

    @abstractmethod
    def get_pixel_size(self, plate_path: Union[str, Path]) -> float:
        """Get the pixel size from metadata in micrometers."""
        pass

    @abstractmethod
    def get_channel_values(self, plate_path: Union[str, Path]) -> Optional[Dict[str, Optional[str]]]:
        """Get channel key→name mapping from metadata."""
        pass

    @abstractmethod
    def get_well_values(self, plate_path: Union[str, Path]) -> Optional[Dict[str, Optional[str]]]:
        """Get well key→name mapping from metadata."""
        pass

    @abstractmethod
    def get_site_values(self, plate_path: Union[str, Path]) -> Optional[Dict[str, Optional[str]]]:
        """Get site key→name mapping from metadata."""
        pass

    @abstractmethod
    def get_z_index_values(self, plate_path: Union[str, Path]) -> Optional[Dict[str, Optional[str]]]:
        """Get z_index key→name mapping from metadata."""
        pass

    def get_image_files(self, plate_path: Union[str, Path]) -> list[str]:
        """Get list of image files from OpenHCS metadata (default implementation)."""
        pass

    def parse_metadata(self, plate_path: Union[str, Path]) -> Dict[str, Dict[str, Optional[str]]]:
        """Parse all metadata using dynamic method resolution (default implementation)."""
        pass

class ImageXpressMetadataHandler(MetadataHandler):
    """Handles ImageXpress .HTD and .MES files."""

    def find_metadata_file(self, plate_path: Union[str, Path]) -> Path:
        """Find .HTD file for the plate."""
        plate_path = Path(plate_path)
        htd_file = plate_path / f"{plate_path.name}.HTD"
        if not htd_file.exists():
            raise FileNotFoundError(f"HTD file not found: {htd_file}")
        return htd_file

    def get_pixel_size(self, plate_path: Union[str, Path]) -> float:
        """Extract pixel size from HTD metadata."""
        # Parses HTD file and returns pixel size in micrometers
        pass

    def get_channel_values(self, plate_path: Union[str, Path]) -> Optional[Dict[str, Optional[str]]]:
        """Extract channel names from HTD metadata."""
        # Returns mapping like {"1": "HOECHST 33342", "2": "Calcein"}
        pass

class OperaPhenixMetadataHandler(MetadataHandler):
    """Handles Opera Phenix XML metadata files."""

    def find_metadata_file(self, plate_path: Union[str, Path]) -> Path:
        """Find XML metadata file (usually Index.idx.xml)."""
        plate_path = Path(plate_path)
        xml_files = list(plate_path.glob("*.xml"))
        if not xml_files:
            raise FileNotFoundError("No XML metadata files found")
        return xml_files[0]

    def get_grid_dimensions(self, plate_path: Union[str, Path]) -> Tuple[int, int]:
        """Extract grid dimensions from XML metadata."""
        # Parses XML and returns (grid_x, grid_y) for stitching
        pass

    def get_pixel_size(self, plate_path: Union[str, Path]) -> float:
        """Extract pixel size from XML metadata."""
        pass

Key Architectural Components

Workspace Preparation

Each microscope format requires different workspace preparation to normalize directory structures for pipeline processing. The key method is _build_virtual_mapping() which creates a virtual mapping dict (plate-relative paths) and saves it to openhcs_metadata.json:

class ImageXpressHandler(MicroscopeHandler):
    @property
    def root_dir(self) -> str:
        """Root directory where virtual workspace preparation starts.

        Returns "." (plate root) because ImageXpress TimePoint/ZStep folders
        are flattened starting from the plate root, and virtual paths have no prefix.
        """
        return "."

    def _build_virtual_mapping(self, plate_path: Path, filemanager: FileManager) -> Path:
        """Build virtual workspace mapping by flattening nested folder structures.

        Creates plate-relative mappings for TimePoint and Z-step folders.
        No physical file operations - all virtual.
        """
        workspace_mapping = {}

        # Flatten TimePoint and ZStep folders virtually
        self._flatten_timepoints(plate_path, filemanager, workspace_mapping, plate_path)
        self._flatten_zsteps(plate_path, filemanager, workspace_mapping, plate_path)

        # Save to metadata using root_dir as subdirectory key
        metadata_path = plate_path / "openhcs_metadata.json"
        writer = AtomicMetadataWriter()
        writer.merge_subdirectory_metadata(metadata_path, {
            self.root_dir: {
                "workspace_mapping": workspace_mapping,  # Plate-relative paths
                "available_backends": {"disk": True, "virtual_workspace": True}
            }
        })

        return plate_path

class OperaPhenixHandler(MicroscopeHandler):
    @property
    def root_dir(self) -> str:
        """Root directory for Opera Phenix virtual workspace preparation.

        Returns "Images" because field remapping is applied to images
        in the Images/ subdirectory.
        """
        return "Images"

    def _build_virtual_mapping(self, plate_path: Path, filemanager: FileManager) -> Path:
        """Build virtual workspace mapping with field remapping.

        Remaps field IDs to follow top-left to bottom-right pattern
        based on spatial layout from Index.xml.
        """
        image_dir = plate_path / self.root_dir
        workspace_mapping = {}

        # Load field mapping from Index.xml
        xml_parser = OperaPhenixXmlParser(image_dir / "Index.xml")
        field_mapping = xml_parser.get_field_id_mapping(exclude_orphans=True)

        # Build virtual mapping with field remapping
        for real_file in filemanager.list_files(image_dir, Backend.DISK.value):
            parsed = self.parser.parse_filename(real_file)
            if not parsed or parsed['site'] is None:
                continue

            original_field = parsed['site']
            if original_field in field_mapping:
                virtual_field = field_mapping[original_field]
                parsed['site'] = virtual_field
                virtual_filename = self.parser.construct_filename(**parsed)

                # Build plate-relative paths
                virtual_path = (Path(self.root_dir) / virtual_filename).as_posix()
                real_path = (Path(self.root_dir) / real_file).as_posix()
                workspace_mapping[virtual_path] = real_path

        # Save to metadata
        metadata_path = plate_path / "openhcs_metadata.json"
        writer = AtomicMetadataWriter()
        writer.merge_subdirectory_metadata(metadata_path, {
            self.root_dir: {
                "workspace_mapping": workspace_mapping,
                "available_backends": {"disk": True, "virtual_workspace": True}
            }
        })

        return image_dir

This workspace preparation ensures pipelines always see a consistent flat structure regardless of the original microscope organization.

Unified Image File Discovery

All microscope handlers use a unified approach to discover image files by reading from OpenHCS metadata:

class MetadataHandler(ABC):
    def get_image_files(self, plate_path: Union[str, Path]) -> list[str]:
        """Get list of image files from OpenHCS metadata.

        Default implementation reads from openhcs_metadata.json after virtual workspace preparation.
        Derives image list from workspace_mapping keys if available, otherwise from image_files list.
        """
        # Read from OpenHCS metadata (unified approach for all microscopes)
        from openhcs.microscopes.openhcs import OpenHCSMetadataHandler
        openhcs_handler = OpenHCSMetadataHandler(self.filemanager)

        metadata = openhcs_handler._load_metadata_dict(plate_path)
        subdirs = metadata.get("subdirectories", {})

        # Find main subdirectory (marked with "main": true)
        main_subdir_key = next((key for key, data in subdirs.items() if data.get("main")), None)
        if not main_subdir_key:
            main_subdir_key = next(iter(subdirs.keys()))

        subdir_data = subdirs[main_subdir_key]

        # Prefer workspace_mapping keys (virtual paths) if available
        if workspace_mapping := subdir_data.get("workspace_mapping"):
            return list(workspace_mapping.keys())

        # Otherwise use image_files list
        return subdir_data.get("image_files", [])

Key Design Points:

• Single Source of Truth: Metadata is authoritative, not filesystem

• No Filesystem Searching: Eliminates defensive directory detection logic

• Unified API: workspace_mapping keys and image_files use same format (subdirectory/filename)

• Fail-Loud: No fallback logic - if metadata doesn’t exist, return empty list

Image Path Format:

• ImageXpress: "A01_s001_w1_z001_t001.tif" (no prefix, root_dir is ".")

• OperaPhenix: "Images/remapped_file.tif" (includes Images/ prefix, root_dir is "Images")

• Zarr: "zarr/A01_s001_w1_z001_t001.tif" (includes zarr/ prefix)

Pattern Detection and File Discovery

Handlers delegate pattern detection to the PatternDiscoveryEngine, which uses the parser to identify and group image files:

class MicroscopeHandler(ABC):
    def auto_detect_patterns(self, folder_path: Union[str, Path], filemanager: FileManager, backend: str,
                            extensions=None, group_by=None, variable_components=None, **kwargs):
        """Detect all image patterns in a folder.

        Args:
            folder_path: Path to folder containing images
            filemanager: FileManager instance for file operations
            backend: Backend to use (e.g., 'disk', 'virtual_workspace')
            extensions: Optional list of file extensions to include
            group_by: GroupBy enum to group patterns by (e.g., GroupBy.CHANNEL)
            variable_components: List of components that can vary (e.g., ['site', 'z_index'])
            **kwargs: Dynamic filter parameters (e.g., well_filter=['A01', 'A02'])

        Returns:
            Dict mapping well IDs to lists of pattern strings
        """
        from openhcs.formats.pattern.pattern_discovery import PatternDiscoveryEngine
        pattern_engine = PatternDiscoveryEngine(self.parser, filemanager)

        return pattern_engine.auto_detect_patterns(
            folder_path,
            extensions=extensions,
            group_by=group_by,
            variable_components=variable_components,
            backend=backend,
            **kwargs
        )

    def path_list_from_pattern(self, directory: Union[str, Path], pattern: str,
                              filemanager: FileManager, backend: str,
                              variable_components: Optional[List[str]] = None) -> List[str]:
        """Generate file paths matching a specific pattern.

        Args:
            directory: Directory to search
            pattern: Pattern string with optional {iii} placeholders
            filemanager: FileManager instance for file operations
            backend: Backend to use for file operations
            variable_components: List of components that can vary

        Returns:
            List of matching filenames
        """
        from openhcs.formats.pattern.pattern_discovery import PatternDiscoveryEngine
        pattern_engine = PatternDiscoveryEngine(self.parser, filemanager)

        return pattern_engine.path_list_from_pattern(
            directory, pattern, backend=backend,
            variable_components=variable_components
        )

This abstraction allows pipelines to discover images without knowing the underlying filename conventions or directory structures.

Parser Metaprogramming System Integration

The microscope handler system integrates with the Parser Metaprogramming System to provide dynamic interface generation for filename parsers.

Dynamic Interface Generation

Each parser automatically generates component-specific interfaces using the DynamicParserMeta metaclass:

# Parser defines its component structure
class CustomFilenameParser(GenericFilenameParser):
    FILENAME_COMPONENTS = ['well', 'site', 'channel', 'timepoint']

    def parse_filename(self, filename: str) -> Optional[Dict[str, Any]]:
        # Parser implementation...
        pass

# Interface automatically generated with component-specific methods
CustomInterface = DynamicParserMeta.create_interface(
    CustomFilenameParser,
    interface_name="CustomInterface"
)

# Generated interface provides:
# - get_well_keys()
# - get_site_keys()
# - get_channel_keys()
# - get_timepoint_keys()
# - construct_filename(well=..., site=..., channel=..., timepoint=...)

Integration Benefits:

1. Component-Agnostic Design: Parsers work with any component configuration

2. Dynamic Method Generation: Interface methods generated based on FILENAME_COMPONENTS

3. Type Safety: Generated methods provide proper type hints and validation

4. Consistent API: All parsers expose the same interface pattern regardless of components

Generic Parser Base Class

The GenericFilenameParser provides the foundation for all microscope-specific parsers:

class GenericFilenameParser(ABC):
    """Base class for all filename parsers with centralized component configuration."""

    def __init__(self, component_enum: Type[T]):
        """Initialize with component enum - FILENAME_COMPONENTS set automatically."""
        self.component_enum = component_enum
        # FILENAME_COMPONENTS automatically set to all component values + extension
        self.FILENAME_COMPONENTS = [c.value for c in component_enum] + ['extension']
        self.PLACEHOLDER_PATTERN = '{iii}'
        self._generate_dynamic_methods()

    @abstractmethod
    def parse_filename(self, filename: str) -> Optional[Dict[str, Any]]:
        """Parse filename and return component dictionary."""
        pass

    def construct_filename(self, **kwargs) -> str:
        """Construct filename from component values."""
        # Generic implementation using component configuration
        pass

    def get_component_keys(self, component: str, filenames: List[str]) -> List[str]:
        """Extract unique values for a specific component."""
        # Generic implementation that works with any component
        pass

Generic Design Benefits:

1. Extensibility: New parsers only need to implement parse_filename()

2. Consistency: All parsers inherit common functionality

3. Component Independence: Base class works with any component structure

4. Interface Compatibility: Automatic compatibility with dynamic interface generation

Integration with Pipeline System

Handler Factory and Selection

OpenHCS provides a factory function that creates the appropriate handler based on explicit type or automatic detection:

def create_microscope_handler(microscope_type: str = 'auto',
                              plate_folder: Optional[Union[str, Path]] = None,
                              filemanager: Optional[FileManager] = None,
                              pattern_format: Optional[str] = None,
                              allowed_auto_types: Optional[List[str]] = None) -> MicroscopeHandler:
    """
    Factory function to create a microscope handler.

    Enforces explicit dependency injection by requiring a FileManager instance.

    Args:
        microscope_type: 'auto', 'imagexpress', 'opera_phenix', 'openhcs'
        plate_folder: Required for 'auto' detection
        filemanager: FileManager instance (required, no fallback)
        pattern_format: Optional pattern format name
        allowed_auto_types: For 'auto' mode, limit detection to these types

    Returns:
        Initialized MicroscopeHandler instance

    Raises:
        ValueError: If filemanager is None or microscope_type cannot be determined
    """
    if filemanager is None:
        raise ValueError("FileManager must be provided to create_microscope_handler")

    # Auto-detect microscope type if needed
    if microscope_type == 'auto':
        if not plate_folder:
            raise ValueError("plate_folder is required for auto-detection")

        microscope_type = _auto_detect_microscope_type(plate_folder, filemanager,
                                                      allowed_types=allowed_auto_types)

    # Get handler class from registry
    handler_class = MICROSCOPE_HANDLERS.get(microscope_type.lower())
    if not handler_class:
        raise ValueError(f"Unsupported microscope type: {microscope_type}")

    # Create and return handler
    return handler_class(filemanager, pattern_format=pattern_format)

FileManager Integration

Handlers work seamlessly with OpenHCS’s VFS system, supporting both disk and memory backends:

• Workspace preparation operates through FileManager abstraction

• Pattern detection works across different storage backends

• File discovery respects backend-specific optimizations

Metaclass Registration System

OpenHCS uses a metaclass-based registration system that automatically registers new handler classes:

class MicroscopeHandlerMeta(ABCMeta):
    """Metaclass that automatically registers handler classes."""

    _registry: Dict[str, Type[MicroscopeHandler]] = {}

    def __new__(mcs, name, bases, namespace, **kwargs):
        # Create the class
        cls = super().__new__(mcs, name, bases, namespace, **kwargs)

        # Register non-abstract handlers
        if not getattr(cls, '__abstractmethods__', None):
            # Extract handler type from class name (e.g., "ImageXpress" from "ImageXpressHandler")
            handler_type = name.replace('Handler', '').lower()
            mcs._registry[handler_type] = cls
            print(f"Registered microscope handler: {handler_type} -> {cls}")

        return cls

    @classmethod
    def get_handler_class(mcs, handler_type: str) -> Type[MicroscopeHandler]:
        """Get handler class by type name."""
        return mcs._registry.get(handler_type.lower())

    @classmethod
    def list_available_handlers(mcs) -> List[str]:
        """List all registered handler types."""
        return list(mcs._registry.keys())

class MicroscopeHandler(ABC, metaclass=MicroscopeHandlerMeta):
    """Base class with automatic registration."""

The metaclass automatically:

• Registers handlers upon class definition (no manual registration needed)

• Validates implementation of required abstract methods

• Maintains handler registry for factory pattern selection

• Enables automatic detection based on handler capabilities

This design ensures that new microscope formats are automatically available to the system once their handler class is defined.

OpenHCS Native Handler

The OpenHCS handler represents a special case that leverages existing handler components while using OpenHCS-specific metadata:

class OpenHCSMicroscopeHandler(MicroscopeHandler):
    """Handler for OpenHCS pre-processed format with JSON metadata."""

    def __init__(self, filemanager: FileManager, pattern_format: Optional[str] = None):
        self.filemanager = filemanager
        self.metadata_handler = OpenHCSMetadataHandler(filemanager)
        self._parser: Optional[FilenameParser] = None
        self.plate_folder: Optional[Path] = None
        self.pattern_format = pattern_format

        # Parser is loaded dynamically based on metadata
        super().__init__(parser=None, metadata_handler=self.metadata_handler)

    @property
    def parser(self) -> FilenameParser:
        """Dynamically load parser based on metadata."""
        if self._parser is None:
            parser_name = self.metadata_handler.get_source_filename_parser_name(self.plate_folder)
            available_parsers = _get_available_filename_parsers()
            ParserClass = available_parsers.get(parser_name)

            if not ParserClass:
                raise ValueError(f"Unknown parser '{parser_name}' in metadata")

            self._parser = ParserClass(pattern_format=self.pattern_format)

        return self._parser

    def _prepare_workspace(self, workspace_path: Path, filemanager: FileManager) -> Path:
        """OpenHCS format is already normalized, no preparation needed."""
        # Ensure plate_folder is set for dynamic parser loading
        if self.plate_folder is None:
            self.plate_folder = Path(workspace_path)
        return workspace_path

class OpenHCSMetadataHandler(MetadataHandler):
    """Handles OpenHCS JSON metadata format."""

    METADATA_FILENAME = "openhcs_metadata.json"

    def get_source_filename_parser_name(self, plate_path: Path) -> str:
        """Get the original filename parser used for this plate."""
        metadata = self._load_metadata(plate_path)
        return metadata.get("source_filename_parser_name")

    def determine_main_subdirectory(self, plate_path: Path) -> str:
        """Determine which subdirectory contains the main input images."""
        metadata_dict = self._load_metadata_dict(plate_path)

        # Handle subdirectory-keyed format
        if subdirs := metadata_dict.get("subdirectories"):
            # Find subdirectory marked as main, or use first available
            for subdir, subdir_metadata in subdirs.items():
                if subdir_metadata.get("main", False):
                    return subdir
            return next(iter(subdirs.keys()))  # Fallback to first

        # Legacy format fallback
        return "images"

Key Architectural Features:

• Component reuse: Leverages existing parser and metadata handler infrastructure

• JSON-based metadata: Uses openhcsmetadata.json instead of microscope-specific formats

• Structured metadata: Standardized JSON schema for plate layout, acquisition parameters, and file organization

• Self-describing datasets: Datasets carry their own metadata, making them portable and self-contained

OpenHCS Metadata Structure: The openhcs_metadata.json file uses a subdirectory-keyed format to organize metadata by processing step:

{
  "subdirectories": {
    "images": {
      "microscope_handler_name": "imagexpress",
      "source_filename_parser_name": "ImageXpressFilenameParser",
      "grid_dimensions": [2048, 2048],
      "pixel_size": 0.325,
      "image_files": [
        "images/A01_s1_w1.tif",
        "images/A01_s1_w2.tif",
        "images/A01_s2_w1.tif"
      ],
      "channels": {"1": "DAPI", "2": "GFP"},
      "wells": {"A01": "Control", "A02": "Treatment"},
      "sites": {"1": "Site1", "2": "Site2"},
      "z_indexes": null,
      "available_backends": {"disk": true},
      "main": true
    },
    "processed": {
      "microscope_handler_name": "imagexpress",
      "source_filename_parser_name": "ImageXpressFilenameParser",
      "grid_dimensions": [2048, 2048],
      "pixel_size": 0.325,
      "image_files": [
        "processed/A01_s1_w1_filtered.tif",
        "processed/A01_s1_w2_filtered.tif"
      ],
      "channels": {"1": "DAPI", "2": "GFP"},
      "wells": {"A01": "Control"},
      "sites": {"1": "Site1"},
      "z_indexes": null,
      "available_backends": {"disk": true},
      "main": false
    }
  }
}

This approach enables OpenHCS to create fully self-describing datasets that can be processed consistently regardless of the original microscope platform.

Extensibility: Adding New Microscope Formats

The handler architecture makes adding support for new microscope formats straightforward:

1. Implement the ABC Contract

Create a new handler class implementing the required abstract methods:

class NewMicroscopeHandler(MicroscopeHandler):
    _microscope_type = 'new_format'
    _metadata_handler_class = None  # Set after class definition

    def __init__(self, filemanager: FileManager, pattern_format: Optional[str] = None):
        self.parser = NewMicroscopeParser(filemanager, pattern_format)
        self.metadata_handler = NewMicroscopeMetadataHandler(filemanager)
        super().__init__(parser=self.parser, metadata_handler=self.metadata_handler)

    @property
    def root_dir(self) -> str:
        """Root directory for virtual workspace preparation."""
        return "."  # or "Images" or other subdirectory

    @property
    def microscope_type(self) -> str:
        return 'new_format'

    @property
    def metadata_handler_class(self) -> Type[MetadataHandler]:
        return NewMicroscopeMetadataHandler

    @property
    def compatible_backends(self) -> List[Backend]:
        return [Backend.DISK]  # or [Backend.ZARR, Backend.DISK]

2. Define Format-Specific Logic

• Directory structure: What directories indicate this format?

• Workspace preparation: What transformations are needed?

• Filename patterns: How are wells, fields, channels encoded?

• Metadata sources: XML files, embedded TIFF tags, etc.?

3. Register with Factory

The handler factory automatically detects and uses new handlers based on directory structure patterns.

Design Benefits

Separation of Concerns - Parser: Handles filename pattern extraction and construction - Metadata Handler: Manages acquisition parameters and plate layout - Workspace Preparation: Normalizes directory structures - Handler: Orchestrates components and provides unified interface

Testability and Maintainability - Each component can be tested independently - Format-specific logic is isolated and contained - Changes to one microscope format don’t affect others - Common functionality can be shared across similar formats

Pipeline Integration - Pipelines remain microscope-agnostic - Automatic format detection reduces user configuration - Consistent interface regardless of underlying complexity - Seamless integration with VFS and memory management systems

This architecture enables OpenHCS to process data from any supported microscope platform through a single, consistent pipeline interface, while handling the complex format-specific details transparently.

See Also

• Parser Metaprogramming System - Dynamic interface generation for filename parsers

• Component Configuration Framework - Generic component configuration system

• Component Validation System - Component validation and constraint checking

• API Reference - API reference (autogenerated from source code)