multicolor DMD-SIM

This repository contains code for designing, analyzing, and carrying out multicolor structured illumination microscopy experiments based on a digital micromirror device (DMD-SIM), including DMD simulation code, DMD pattern generation, SIM reconstruction and instrument control. It also includes a number of useful utilities for simulating the resulting diffraction pattern given certain DMD patterns, determining system point-spread functions and optical transfer functions, and determining the affine transformation between the DMD coordinates and the imaging space coordinates. The various files are described in more detail below.

Published work using this code

• This DMD simulation tools and SIM reconstruction code were originally created for our Biomedical Optics Express paper Multicolor structured illumination microscopy and quantitative control of polychromatic light with a digital micromirror device (bioRxiv). The repository state at the time of publication is archived here, or available as release on GitHub.
• For a tutorial on SIM reconstruction, see our I2K 2022 talks.
• Resolution doubling in light-sheet microscopy via oblique plane structured illumination (bioRxiv). This version is archived here or available as a release
• Spatial wavefront shaping with a multipolar-resonant metasurface for structured illumination microscopy (arXiv).
• A Structured Illumination Microscopy Framework with Spatial-Domain Noise Propagation (bioRxiv). This version is archived here or available as release v1.5.0.
• Fourier synthesis optical diffraction tomography for kilohertz rate volumetric imaging (arXiv).

Installation

The best way to use this python package is to install it with pip

git clone https://github.com/QI2lab/mcSIM.git
cd mcSIM
pip install .

If you would like to edit the code, then install using the -e option,

git clone https://github.com/QI2lab/mcSIM.git
cd mcSIM
pip install -e .

The dependencies for the experimental control code are not installed by default because the DMD control code relies on the Windows specific pywinusb. To install these dependencies run

git clone https://github.com/QI2lab/mcSIM.git
cd mcSIM
pip install .[expt_ctrl]

Some functions can be optionally run on a GPU. This functionality has been tested with CUDA 11.2 and 11.8. If GPU support is desired, first install CuPy manually, and then install mcSIM with pip. For example, if you are using a conda environment and CUDA toolkit 11.8

conda install -c conda-forge cudatoolkit=11.8
git clone https://github.com/QI2lab/mcSIM.git
cd mcSIM
pip install .[gpu]

This will install the appropriate version of the scikit-image portion of cuCIM. Note that the entire package cannot be installed on Windows, but the scikit-image portion can. This portion of cuCIM can be installed manually using

pip install "git+https://github.com/rapidsai/cucim.git@v22.12.00#egg=cucim&subdirectory=python/cucim"

or the tag v22.12.00 can be replaced with the most recent version

SIM reconstruction code

sim_reconstruction.py

Code for reconstructing SIM images from raw data using a Gustafsson/Heintzmann Wiener filter style reconstruction. Several reconstruction options are available, largely following either the approach of openSIM or fairSIM. To get started with reconstructing SIM data, see the example scripts

• reconstruct_sim_simulated_single.py
• reconstruction_sim_simulated_timelapse.py
• reconstruct_sim_experiment_multichannel.py

DMD simulation code

simulate_dmd.py

Code for simulating the diffraction patterns produced by the DMD. Various scripts illustrating the usage of this code can be found in the examples directory. This simulation code has many useful features, including an analytic solution for the joint blaze/diffraction condition, pattern simulation tools, tools for extracting the intensity pattern in the Fourier plane of a collecting lens, etc. For example usage, see

• solve_multicolor_blaze_condition.py
• simulate_dmd_diffraction_pattern.py
• simulate_multicolor_sim_patterns_1d.py
• simulate_dmd_diffraction_fourier_plane.py
• dmd_solution_explorer_gui.py

dmd_patterns.py

This file can be used to generate multicolor SIM patterns and other useful calibrations patterns for the DMD. It also contains many tools for working with the basis vector/unit cell representation of DMD patterns. This allows a complete enumeration of DMD diffraction orders in a compact and computationally efficient form.

Utility code

fit_dmd_affine.py

Code to fit the affine transformation between the DMD coordinates and camera coordinates using imaging data from a DMD pattern consisting of many small points. These code relies on tools for working with affine transformations found here. For example usage, see

• calibrate_affine_xform.py

otf_tools.py

Code for extracting optical transfer function from measurement of the strength of various Fourier peaks for a given SIM DMD pattern.

analysis_tools.py

Miscellaneous image processing tools, primarily for working with Fourier transforms

localize-psf

Useful tools for automatically localizing sparse fluorescent beads and performing both 2D and 3D PSF fitting using various PSF models. Also provides useful statistics and figures summarizing the results. This code has now been split out into a separate repository. For more information about these tools, see the readme pip installing the mcsim repository as described above will also pull in these dependencies.

Examples

Scripts illustrated examples of different DMD simulations and analysis are stored in examples. Associated data necessary to run some of these examples can be downloaded from Zenodo. These scripts assume this data has been placed in a directory called examples/data

Hardware control code

Hardware control is based around MicroManager2.0. Currently, we control the instrument using a fork of the napari-micromanager project which controls the MicroManager core using pymmcore-plus. Our fork of this project relies on MicroManager device drives to control cameras and stages, and on python code to control the DMD and DAQ. The "device adapters" for the DMD and DAQ are found below

expt_ctrl/dlp6500.py

Code for controlling the DLP6500 DMD over USB on Windows. This code was initially based on the approaches of Lightcrafter6500DMDControl and Pycrafter6500. Extension to other operating systems has not been implemented but should be straightforward.

This file also includes functions used to define pattern sequences on the DMD using patterns have either been previously loaded onto the firmware using the Texas Instruments DLP6500 and DLP9000 GUI or which are loaded "on-the-fly". There is a low-level interface for running these patterns based directly on their index in the firmware. There is also a higher-level interface which supports defining "channels" and "modes" which can be saved in a json configuration file.

When run as a script, this file provides a command line interface to programming DMD pattern sequences.

expt_ctrl/daq.py

Code for controlling a national instruments DAQ through PyDAQmx

expt_ctrl/daq_config.json

DAQ configuration file describing "modes" and "channels" for the DAQ. This file is used by nidaq instances created with daq.py

program_sim_odt.py

This file is used to create DAQ sequences for SIM and ODT experiments

expt_ctrl/*.cfg

MicroManager configuration files describing the equipment and settings used in the experiment.

useful script files

Example scripts which are useful for controlling various instruments during testing include load_dmd_pattern.py, setup_optotune_mre2.py, and set_dmd_odt_pattern.py

Instrument design

A parts list is available here. Mechanical drawings and 3D models of custom parts such as the DMD mount are available on Zenodo. For a more complete description of the optical path, see our BOE paper linked above.

Documentation

Documentation is generated from function docstrings and built with Sphinx. e.g. navigate to docs and run

make html

Then open docs/_build/html/index.html in your browser

Acknowlegements

Thank you to Songyun Gu for extending the DMD control code to work with the DLP9000