Fourier lightfield multiview stereoscope for large field-of-view 3D imaging in microsurgical settings
Advanced Photonics Nexus (2025)
1Department of Biomedical Engineering, Duke University, Durham NC, USA., 2Ramona Optics Inc., 1000 W Main St., Durham NC, USA., 3Department of Biomedical Engineering, University of Michigan, Ann Arbor MI, USA., 4Department of Electrical Engineering, Duke University, Durham NC, USA
Abstract
We present the Fourier lightfield multiview stereoscope (FiLM-Scope). This imaging device combines concepts from Fourier lightfield microscopy and multiview stereo imaging to capture high-resolution 3D videos over large fields of view. The FiLM-Scope optical hardware consists of a multicamera array, with 48 individual microcameras, placed behind a high-throughput primary lens. This allows the FiLM-Scope to simultaneously capture 48 unique 12.8 megapixel images of a 28 × 37 mm field-of-view, from unique angular perspectives over a 21 deg × 29 deg range, with down to 22 μm lateral resolution. We also describe a self-supervised algorithm to reconstruct 3D height maps from these images. Our approach demonstrates height accuracy down to 11 μm. To showcase the utility of our system, we perform tool tracking over the surface of an ex vivo rat skull and visualize the 3D deformation in stretching human skin, with videos captured at up to 100 frames per second. The FiLM-Scope has the potential to improve 3D visualization in a range of microsurgical settings.
Overview
For more than a century, surgeons have depended on stereoscopes to provide depth perception during microsurgery. These systems, whether analog or digital, work by splitting light from the surgical field into two separate paths, creating slightly different images for each eye. This setup gives surgeons a strong, intuitive sense of depth, which is essential for delicate tasks like stitching tiny blood vessels or placing implants in the brain. While today’s operating microscopes have become more sophisticated with digital displays and video recording, the fundamental optical approach remains the same: two perspectives, two images, and depth perception generated by the human brain.
However, this method has a key limitation. While it offers excellent qualitative depth perception, it is difficult to extract quantitative measurements, such as the exact position or shape of tissues and tools, from just two images. Current computer algorithms struggle to generate complete, precise 3D reconstructions from stereo pairs, especially in complex surgical environments that may include uneven lighting, harsh reflections, and obstructions from tools. This shortcoming restricts the use of automation and robotics in surgery and limits the feedback that can be provided to surgeons using handheld tools.
Research in both microscopic and macroscopic imaging has shown that capturing many multi-perspective views—up to dozens instead of just two—can dramatically improve the quality and completeness of 3D reconstructions. To bring these advances into the operating room, this work presents a new type of surgical microscope: the Fourier Light Field Multi-view Stereoscope (FiLM-Scope). The FiLM-Scope consists of 48 micro-cameras arranged in a compact grid behind a large, high-throughput lens. Each camera captures the entire surgical area from a slightly different angle, resulting in 48 unique, high-resolution (12.5 MP) images of the same scene. This setup covers a field of view of 28 x 37 millimeters with a resolution down to 22 microns and can stream video at up to 120 frames per second.
The multi-perspective images can be converted into 3D information using a custom algorithm. This approach is self-supervised and does not require pre-existing training data or ground-truth models, making it flexible and adaptable to new situations. The system can reconstruct surfaces over a depth range of up to 1 centimeter with precision down to 11 microns. Because the FiLM-Scope captures the entire field in each frame, users can digitally zoom and pan across the scene without moving the device, streamlining workflows. By providing quantitative 3D visualization over a large field of view, this technology can enhance both manual and robotic-assisted surgery, as well as other fields that require high-precision 3D imaging.
Results
In this work, we demonstrate the utility of the FiLM-Scope in several scenarios, including visualizing surface structures on human skin, quantitative tool tracking over the surface of an ex-vivo rat skull, and monitoring deformation of soft tissue.
See our explainer video for more information!