EyeDiT: Anatomy-Grounded Multi-Modal Eye Generation and Simulation

Yonghan Lee1*,2, Zachary Willms1, Kun Liu1, Pushkar Anand1, Ali Behrooz1, Dmitri Model1, Dinesh Manocha2, Yatong An1

1Meta Reality Labs 2University of Maryland, College Park

(* Work done during an internship at Meta)

Overview
EyeDiT learns a joint 3D model of near-eye images, OCT volumes and anatomy point clouds, and generates new eyes under pupil and identity control.

Abstract

Synthetic eye data are critical for rapid development of eye tracking (ET) in AR/VR applications. However, existing synthetic eye generators are largely image-centric and lack a high-quality underlying 3D anatomy of the eye, which limits their control mainly to the gaze and provides no anatomical labels. To address this problem, we propose EyeDiT, an anatomy-grounded, multi-modal 3D eye generation and simulation model trained with optical coherence tomography (OCT), labeled anatomy point clouds and near-eye images. Our pipeline first trains a Multi-Modal Anatomy Variational Autoencoder (VAE), which encodes an anatomy-aware latent representation from all input modalities. Then, our Eye Latent DiT is trained to generate latent samples conditioned on the optical axis, the pupil position and radius, and the subject identity. Together with Gaussian decoders for novel-view simulation, our model enables control over the pupil, subject identity and viewpoint for data synthesis. We show that our model outperforms previous eye image synthesis models in novel-view synthesis (24.9 against 21.8 dB PSNR from a single view), and that a public ET neural network trained on our synthetic data shows a smaller sim-to-real gap than when trained on data from other generators (3.5° vs. 8.4° gaze error), approaches a network trained on real data, and further improves ET performance when combined with real data (2.3° vs. 2.6°).

Video

Method

Method
Overview of EyeDiT. (Stage 1) Our multi-modal anatomy VAE learns a shared latent space from OCT volumes, near-eye images and anatomy point clouds. (Stage 2) Our eye latent DiT is then trained on this latent space to simulate various gaze, pupil, and identity conditions. Our pipeline generates multi-modal outputs for a learned identity or for a new subject given by a context image.

Multi-Modal Generation Demo

Multi-modal outputs generated for each dataset: the near-eye image, the anatomy point cloud and an OCT B-scan sweep.

Pupil Simulation

The optical axis and pupil radius are varied while the camera orbits the generated eye, and the pupil position follows the eye rotation. The anatomy and OCT follow each change.

Subject Simulation

The subject identity is varied while the pupil state is held fixed.

Novel-View Rendering

Captured view, VAE reconstruction and DiT generation from one context image, with the corresponding point cloud and OCT B-scans.

Multi-Modal Reconstruction

ImageOCTPCDBiometrics
DatasetPSNR (dB) ↑LPIPS ↓PSNR (dB) ↑LPIPS ↓Acc. (mm) ↓Comp. (mm) ↓Pos. (mm) ↓Rad. (mm) ↓OA (°) ↓Cornea front R (mm) ↓Iris thick. (mm) ↓
Reconstruction (VAE)OCT-Cam-2V25.820.10125.080.1910.0810.1760.1020.0690.930.1180.039
OCT-Cam-4V24.720.13524.400.1780.1020.1980.1390.0981.160.1820.030
Generation (DiT)OCT-Cam-2V25.400.10621.640.2470.2230.2590.1390.1391.110.1480.040
OCT-Cam-4V23.800.14319.640.2510.2720.3210.1320.1380.520.3640.043

Reconstruction and generation quality on test-unseen subjects of OCT-Cam-2V and OCT-Cam-4V. Image metrics are averaged over all views. The conditioned DiT generates samples close to the VAE reconstructions in quality, suggesting that little fidelity is lost by generating from the conditions.

Inputs (GT) and reconstructions (Ours) of our multi-modal anatomy VAE on OCT-Cam-2V (top two rows) and OCT-Cam-4V (bottom two rows). The columns show two views (nasal, temporal), two OCT B-scan positions, the projected OCT volume (en-face) and the PCD. Each row is a subject either seen or unseen during training.

Novel-View Synthesis

OCT-Cam-2VOCT-Cam-4VLightDome
1 view1 view3 views3 views
MethodSourcePSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓
GazePrior [2]OCT-Cam20.230.6940.46827.130.7840.50227.540.7880.49917.270.4930.554
Lin et al. [1]Static Fit14.150.2670.59511.770.3750.66515.850.4800.58214.220.4050.588
ControllableGaze [3]OCT-Cam21.810.7020.39719.240.7260.46318.930.7250.466–––
MapAnything [4] (+GS Head)Pretrained (+OCT-Cam)19.620.6530.35513.660.5750.50014.500.6810.48112.320.4510.758
MoGe-2 [5] (+GS Head)Pretrained (+OCT-Cam)19.100.6300.39515.350.5760.51617.950.6850.46810.330.3980.790
DA-V2 [6] (+GS Head)Pretrained (+OCT-Cam)18.820.6340.43914.080.3500.67715.100.4880.66512.380.4130.682
MVSplat [7]Pretrained××××××14.340.4900.62110.680.1020.746
DepthSplat [8]Pretrained××××××14.560.4060.66412.150.4170.767
TRELLIS [9]Pretrained15.200.5140.58113.380.2290.65315.740.5250.55515.070.5210.718
ViewCrafter [10]Pretrained15.560.5010.5148.070.4390.71113.140.5140.67610.900.3230.775
OursOCT-Cam24.890.7860.12829.230.8910.13729.420.8940.13019.300.5200.324

Novel-view rendering on OCT-Cam-2V, OCT-Cam-4V and LightDome [1]. The top three methods are baselines for eye modeling, while the others are general 3D reconstruction and generation methods. Our model, with a prior learned from multi-view eye data, shows the best novel-view synthesis performance. ×: the method needs at least two context views. –: ControllableGaze is not available on LightDome, which has no mesh or OCT data.

Novel-view rendering of our method and the baselines on OCT-Cam-4V. Three context views (left of the dashed line) are used as inputs to render the target view (GT). Pixels outside the white lines in the GT column are occluded by our capture device and excluded from novel-view evaluation. Our method renders high-fidelity novel views even from only three input views, while ControllableGaze [3] and GazePrior [2] show significantly blurrier results in this few-shot setting.

References

  1. Esther Y. H. Lin, Yimin Ding, Jogendra Kundu, Yatong An, Mohamed El-Haddad, and Alexander Fix. Digitally prototype your eye tracker: Simulating hardware performance using 3D synthetic data. In Proceedings of the 2026 Symposium on Eye Tracking Research and Applications, pages 1–7, 2026.
  2. Corentin Dumery, David Colmenares, Alexander Fix, Pascal Fua, Ali Behrooz, and Jogendra Kundu. GazePrior: Zero-shot AR/VR eye tracking via learned 3D gaze reconstruction. arXiv preprint arXiv:2605.22359, 2026.
  3. Tingwei Li, Jun Bao, Zhenzhong Kuang, and Buyu Liu. What we need is explicit controllability: Training 3D gaze estimator using only facial images. In ICCV, pages 11414–11424, 2025.
  4. Nikhil Keetha, Norman Müller, Johannes Schönberger, Lorenzo Porzi, Yuchen Zhang, Tobias Fischer, Arno Knapitsch, Duncan Zauss, Ethan Weber, Nelson Antunes, Jonathon Luiten, Manuel Lopez-Antequera, Samuel Rota Bulò, Christian Richardt, Deva Ramanan, Sebastian Scherer, and Peter Kontschieder. MapAnything: Universal feed-forward metric 3D reconstruction. In International Conference on 3D Vision (3DV), pages 499–509, 2026.
  5. Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, and Jiaolong Yang. MoGe-2: Accurate monocular geometry with metric scale and sharp details. In NeurIPS, pages 35928–35959, 2025.
  6. Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything V2. In NeurIPS, pages 21875–21911, 2024.
  7. Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. MVSplat: Efficient 3D Gaussian splatting from sparse multi-view images. In ECCV, pages 370–386, 2024.
  8. Haofei Xu, Songyou Peng, Fangjinhua Wang, Hermann Blum, Daniel Barath, Andreas Geiger, and Marc Pollefeys. DepthSplat: Connecting Gaussian splatting and depth. In CVPR, pages 16453–16463, 2025.
  9. Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3D latents for scalable and versatile 3D generation. In CVPR, pages 21469–21480, 2025.
  10. Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-Tsin Wong, Ying Shan, and Yonghong Tian. ViewCrafter: Taming video diffusion models for high-fidelity novel view synthesis. IEEE TPAMI, 2025.

BibTeX

@article{lee2026eyedit,
  title   = {EyeDiT: Anatomy-Grounded Multi-Modal Eye Generation and Simulation},
  author  = {Lee, Yonghan and Willms, Zachary and Liu, Kun and Anand, Pushkar and
             Behrooz, Ali and Model, Dmitri and Manocha, Dinesh and An, Yatong},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026}
}