(* Work done during an internship at Meta)
Synthetic eye data are critical for rapid development of eye tracking (ET) in AR/VR applications. However, existing synthetic eye generators are largely image-centric and lack a high-quality underlying 3D anatomy of the eye, which limits their control mainly to the gaze and provides no anatomical labels. To address this problem, we propose EyeDiT, an anatomy-grounded, multi-modal 3D eye generation and simulation model trained with optical coherence tomography (OCT), labeled anatomy point clouds and near-eye images. Our pipeline first trains a Multi-Modal Anatomy Variational Autoencoder (VAE), which encodes an anatomy-aware latent representation from all input modalities. Then, our Eye Latent DiT is trained to generate latent samples conditioned on the optical axis, the pupil position and radius, and the subject identity. Together with Gaussian decoders for novel-view simulation, our model enables control over the pupil, subject identity and viewpoint for data synthesis. We show that our model outperforms previous eye image synthesis models in novel-view synthesis (24.9 against 21.8 dB PSNR from a single view), and that a public ET neural network trained on our synthetic data shows a smaller sim-to-real gap than when trained on data from other generators (3.5° vs. 8.4° gaze error), approaches a network trained on real data, and further improves ET performance when combined with real data (2.3° vs. 2.6°).
Multi-modal outputs generated for each dataset: the near-eye image, the anatomy point cloud and an OCT B-scan sweep.
The optical axis and pupil radius are varied while the camera orbits the generated eye, and the pupil position follows the eye rotation. The anatomy and OCT follow each change.
The subject identity is varied while the pupil state is held fixed.
Captured view, VAE reconstruction and DiT generation from one context image, with the corresponding point cloud and OCT B-scans.
| Image | OCT | PCD | Biometrics | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | PSNR (dB) ↑ | LPIPS ↓ | PSNR (dB) ↑ | LPIPS ↓ | Acc. (mm) ↓ | Comp. (mm) ↓ | Pos. (mm) ↓ | Rad. (mm) ↓ | OA (°) ↓ | Cornea front R (mm) ↓ | Iris thick. (mm) ↓ | |
| Reconstruction (VAE) | OCT-Cam-2V | 25.82 | 0.101 | 25.08 | 0.191 | 0.081 | 0.176 | 0.102 | 0.069 | 0.93 | 0.118 | 0.039 |
| OCT-Cam-4V | 24.72 | 0.135 | 24.40 | 0.178 | 0.102 | 0.198 | 0.139 | 0.098 | 1.16 | 0.182 | 0.030 | |
| Generation (DiT) | OCT-Cam-2V | 25.40 | 0.106 | 21.64 | 0.247 | 0.223 | 0.259 | 0.139 | 0.139 | 1.11 | 0.148 | 0.040 |
| OCT-Cam-4V | 23.80 | 0.143 | 19.64 | 0.251 | 0.272 | 0.321 | 0.132 | 0.138 | 0.52 | 0.364 | 0.043 | |
Reconstruction and generation quality on test-unseen subjects of OCT-Cam-2V and OCT-Cam-4V. Image metrics are averaged over all views. The conditioned DiT generates samples close to the VAE reconstructions in quality, suggesting that little fidelity is lost by generating from the conditions.
| OCT-Cam-2V | OCT-Cam-4V | LightDome | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 view | 1 view | 3 views | 3 views | ||||||||||
| Method | Source | PSNR ↑ | SSIM ↑ | LPIPS ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
| GazePrior [2] | OCT-Cam | 20.23 | 0.694 | 0.468 | 27.13 | 0.784 | 0.502 | 27.54 | 0.788 | 0.499 | 17.27 | 0.493 | 0.554 |
| Lin et al. [1] | Static Fit | 14.15 | 0.267 | 0.595 | 11.77 | 0.375 | 0.665 | 15.85 | 0.480 | 0.582 | 14.22 | 0.405 | 0.588 |
| ControllableGaze [3] | OCT-Cam | 21.81 | 0.702 | 0.397 | 19.24 | 0.726 | 0.463 | 18.93 | 0.725 | 0.466 | – | – | – |
| MapAnything [4] (+GS Head) | Pretrained (+OCT-Cam) | 19.62 | 0.653 | 0.355 | 13.66 | 0.575 | 0.500 | 14.50 | 0.681 | 0.481 | 12.32 | 0.451 | 0.758 |
| MoGe-2 [5] (+GS Head) | Pretrained (+OCT-Cam) | 19.10 | 0.630 | 0.395 | 15.35 | 0.576 | 0.516 | 17.95 | 0.685 | 0.468 | 10.33 | 0.398 | 0.790 |
| DA-V2 [6] (+GS Head) | Pretrained (+OCT-Cam) | 18.82 | 0.634 | 0.439 | 14.08 | 0.350 | 0.677 | 15.10 | 0.488 | 0.665 | 12.38 | 0.413 | 0.682 |
| MVSplat [7] | Pretrained | × | × | × | × | × | × | 14.34 | 0.490 | 0.621 | 10.68 | 0.102 | 0.746 |
| DepthSplat [8] | Pretrained | × | × | × | × | × | × | 14.56 | 0.406 | 0.664 | 12.15 | 0.417 | 0.767 |
| TRELLIS [9] | Pretrained | 15.20 | 0.514 | 0.581 | 13.38 | 0.229 | 0.653 | 15.74 | 0.525 | 0.555 | 15.07 | 0.521 | 0.718 |
| ViewCrafter [10] | Pretrained | 15.56 | 0.501 | 0.514 | 8.07 | 0.439 | 0.711 | 13.14 | 0.514 | 0.676 | 10.90 | 0.323 | 0.775 |
| Ours | OCT-Cam | 24.89 | 0.786 | 0.128 | 29.23 | 0.891 | 0.137 | 29.42 | 0.894 | 0.130 | 19.30 | 0.520 | 0.324 |
Novel-view rendering on OCT-Cam-2V, OCT-Cam-4V and LightDome [1]. The top three methods are baselines for eye modeling, while the others are general 3D reconstruction and generation methods. Our model, with a prior learned from multi-view eye data, shows the best novel-view synthesis performance. ×: the method needs at least two context views. –: ControllableGaze is not available on LightDome, which has no mesh or OCT data.
@article{lee2026eyedit,
title = {EyeDiT: Anatomy-Grounded Multi-Modal Eye Generation and Simulation},
author = {Lee, Yonghan and Willms, Zachary and Liu, Kun and Anand, Pushkar and
Behrooz, Ali and Model, Dmitri and Manocha, Dinesh and An, Yatong},
journal = {arXiv preprint arXiv:XXXX.XXXXX},
year = {2026}
}