电镜拍不到——AI“脑补照”

在透射电镜断层成像技术中，由于电子对材料的穿透十分有限，材料在高角度的投影数据通常无法获取。而这些缺失的投影数据会在最终的三维成像结果中引发楔形失真。

Fig. 1 Schematics of missing wedge artifact in the conventional electron tomography and our UsiNet workflow.

该研究提出了一种基于深度学习的算法，利用卷积神经网络对缺失的投影数据进行补全，以消除透射电镜断层成像中的楔形失真。在同类研究中，由于透射电镜断层成像数据集的缺乏，标签的获取一直是神经网络训练的难题。

Fig. 2 Schematic of UsiNet training workflow.

来自美国伊利诺伊大学香槟分校材料科学与工程系的陈倩教授团队，设计了无监督式的神经网络训练流程来避免训练集标签获取的难题。相比于同类算法，该研究的亮点在于其完全不依赖于任何数据库以及计算机模拟数据就能完成对模型的训练和对缺失投影的补全。

Fig. 3 Unsupervised sinogram inpainting implemented on 2D images.

作者首先使用计算机模拟的纳米粒子电镜投影数据对算法进行了验证，得到了对算法性能在不同的缺失投影角度范围下，不同噪声影响下和不同纳米粒子形貌下的定量表征。

Fig. 4 Unsupervised sinogram inpainting implemented on 3D images. 

随后该算法被应用于实验上获得的真实数据并成功消除了实验数据中的楔形失真。相比于其他的传统校正算法，该算法能对楔形失真进行最彻底的去除，并还原样品中纳米粒子的真实三维形貌。

Fig. 5 Orientation-dependent missing wedge artifact and comparison between different reconstruction algorithms.

该研究是基于人工智能的图像算法在电子显微镜中的一次成功应用，也是纳米材料三维表征技术的重大进展。相关论文近期发布于npj Computational Materials 10: 28 (2024)。

Fig. 6 Comparison of 3D reconstructions of experimentally synthesized NPs with and without inpainting.

Editorial Summary

Electron microscopy can't capture it? AI comes to the "rescue"!

In electron tomography, due to the limited penetration of electrons into materials, projection data at high angles is often inaccessible. These missing projection data can cause wedge distortion in the final three-dimensional imaging result. This study proposes a deep learning-based algorithm that utilizes convolutional neural networks to complete the missing projection data, thereby eliminating wedge distortion in electron tomography. In relevant studies, due to the lack of electron tomography datasets, obtaining labels has always been a challenge for such neural network training. 

Fig. 7 Visualizing the heterogeneity of experimentally synthesized NPs.

Professor Qian Chen's team from the Department of Materials Science and Engineering at the University of Illinois at Urbana-Champaign has developed an unsupervised neural network training workflow to bypass the difficulty of obtaining training set labels. The highlight of this study, compared to other algorithms, is that it can train and apply the model without relying on any databases or computer-simulated data. The authors first validated the algorithm using computer-simulated electron microscopy projection data of nanoparticles and obtained quantitative evaluation of the algorithm performance under different ranges of missing projection angles, noise levels, and nanoparticle morphologies. The algorithm was then applied to experimentally obtained data and successfully eliminated wedge distortion in the experimental data. Compared to other traditional correction algorithms, this algorithm can thoroughly remove wedge distortion and restore the true three-dimensional morphology of nanoparticles. This study represents a successful application of artificial intelligence-based image algorithms in electron microscopy and a significant advancement in three-dimensional characterization techniques for nanomaterials.This article was recently published in npj Computational Materials 10: 28 (2024).

原文Abstract及其翻译

No ground truth needed: unsupervised sinogram inpainting for nanoparticle electron tomography (UsiNet) to correct missing wedges (利用无监督式图像修补进行的透射电镜断层成像失真校正)

Lehan Yao, Zhiheng Lyu, Jiahui Li & Qian Chen

Abstract Complex natural and synthetic materials, such as subcellular organelles, device architectures in integrated circuits, and alloys with microstructural domains, require characterization methods that can investigate the morphology and physical properties of these materials in three dimensions (3D). Electron tomography has unparalleled (sub-)nm resolution in imaging 3D morphology of a material, critical for charting a relationship among synthesis, morphology, and performance. However, electron tomography has long suffered from an experimentally unavoidable missing wedge effect, which leads to undesirable and sometimes extensive distortion in the final reconstruction. Here we develop and demonstrate Unsupervised Sinogram Inpainting for Nanoparticle Electron Tomography (UsiNet) to correct missing wedges. UsiNet is the first sinogram inpainting method that can be realistically used for experimental electron tomography by circumventing the need for ground truth. We quantify its high performance using simulated electron tomography of nanoparticles (NPs). We then apply UsiNet to experimental tomographs, where >100 decahedral NPs and vastly different byproduct NPs are simultaneously reconstructed without missing wedge distortion. The reconstructed NPs are sorted based on their 3D shapes to understand the growth mechanism. Our work presents UsiNet as a potent tool to advance electron tomography, especially for heterogeneous samples and tomography datasets with large missing wedges, e.g. collected for beam sensitive materials or during temporally-resolved in-situ imaging.

摘要 复杂的自然或合成材料，如亚细胞器、集成电路中的器件结构和具有微结构域的合金，需要三维的表征方法来研究其形态和物理性质。透射电镜断层成像技术在表征材料的三维形态方面具有超高的纳米级分辨率，这对于建立材料合成、材料形态与材料性能三者之间的关系至关重要。然而，透射电镜断层成像技术长期以来一直受到实验上不可避免的缺失楔形效应的困扰，这导致最终的三维重建中经常出现图像失真。本研究中我们开发了基于深度学习图像修补进行的、用于纳米粒子的透射电镜断层成像失真校正算法，且本研究是相关领域第一个无需训练标签就能进行的基于正弦图补全的失真校正算法。本文中我们首先使用计算机模拟的纳米粒子断层成像数据对算法进行性能验证，然后将此算法应用于实验中获得的成像数据，并取得了良好结果。我们使用本算法在对上百个十面体纳米粒子进行的三维重建中成功避免了楔形失真，并使用这些三维表征数据对纳米粒子的形貌进行了分析，从而取得了对其生长机理的了解。本工作是对透射电镜断层成像技术的一项重大推进，也是对异质材料、电子束敏感材料和原位成像等特殊样品的三维表征的解决方案。