Computational 3D Topographic Microscopy from Terabytes of Data Per Sample

Kevin C. Zhou^1,2,4

Mark Harfouche²

Maxwell Zheng²

Joakim Jönsson¹

Kyung Chul Lee^1,3

Ron Appel²

Paul Reamey²

Thomas Doman²

Veton Saliu²

Gregor Horstmeyer²

Seung Ah Lee³

Roarke Horstmeyer^1,2

Journal of Big Data (2024)

¹Department of Biomedical Engineering, Duke University, Durham NC, USA., ²Ramona Optics Inc., 1000 W Main St., Durham, NC 27701, USA., ³School of Electrical and Electronic Engineering, Yonsei University, Seoul, 03722, South Korea., ⁴Current affiliation: Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA 94720, USA.

Paper Code

Fig 1 — Fig.: Overview of STARCAM. (a, b) Data acquisition involves scanning a 3D object in three dimensions and synchronously capturing ~9.4-MP, high-resolution images across an array of 9×6 cameras. Lateral scanning (8×8) is necessary to fill in the gaps between sensors, due to the high magnification, as well as provide stereo information for 3D estimation. z-stacking (65 steps) enables 3D topographic estimation of thicker samples using sharpness measures. (c) Our computational 3D reconstruction algorithm generates a 6-GP, all-in-focus gigamosaic along with a coregistered 3D height map

Abstract

We present a large-scale computational 3D topographic microscope that enables 6-gigapixel profilometric 3D imaging at micron-scale resolution across >110 cm2 areas over multi-millimeter axial ranges. Our computational microscope, termed STARCAM (Scanning Topographic All-in-focus Reconstruction with a Computational Array Microscope), features a parallelized, 54-camera architecture with 3-axis translation to capture, for each sample of interest, a multi-dimensional, 2.1-terabyte (TB) dataset, consisting of a total of 224,640 9.4-megapixel images. We developed a self-supervised neural network-based algorithm for 3D reconstruction and stitching that jointly estimates an all-in-focus photometric composite and 3D height map across the entire field of view, using multi-view stereo information and image sharpness as a focal metric. The memory-efficient, compressed differentiable representation offered by the neural network effectively enables joint participation of the entire multi-TB dataset during the reconstruction process. To demonstrate the broad utility of our new computational microscope, we applied STARCAM to a variety of decimeter-scale objects, with applications ranging from cultural heritage to industrial inspection.