This repository contains the implementation of the paper

MagicDrive: Street View Generation with Diverse 3D Geometry Control
Ruiyuan Gao^1*, Kai Chen^2*, Enze Xie^{3^}, Lanqing Hong³, Zhenguo Li³, Dit-Yan Yeung², Qiang Xu^{1^
1}CUHK ²HKUST ³Huawei Noah's Ark Lab
^*Equal Contribution ^{^}Corresponding Authors

Recent advancements in diffusion models have significantly enhanced the data synthesis with 2D control. Yet, precise 3D control in street view generation, crucial for 3D perception tasks, remains elusive. Specifically, utilizing Bird’s-Eye View (BEV) as the primary condition often leads to challenges in geometry control (e.g., height), affecting the representation of object shapes, occlusion patterns, and road surface elevations, all of which are essential to perception data synthesis, especially for 3D object detection tasks. In this paper, we introduce MAGICDRIVE, a novel street view generation framework offering diverse 3D geometry controls, including camera poses, road maps, and 3D bounding boxes, together with textual descriptions, achieved through tailored encoding strategies. Besides, our design incorporates a cross-view attention module, ensuring consistency across multiple camera views. With MAGICDRIVE, we achieve high-fidelity street-view synthesis that captures nuanced 3D geometry and various scene descriptions, enhancing tasks like BEV segmentation and 3D object detection.

In MagicDrive, we employ two strategies (cross-attention and additive encoder branch) to inject text prompt, camera pose, object boxes, and road maps as conditions for generation. We also propose cross-view attention module for multiview consistency.

Coming soon.

Coming soon.

Compare MagicDrive with other methods for generation quality:

Training support with images generated from MagicDrive:

More results can be found in the main paper.

More results can be found in the main paper.

@article{gao2023magicdrive,
  title={MagicDrive: Street View Generation with Diverse 3D Geometry Control},
  author={Gao, Ruiyuan and Chen, Kai and Xie, Enze and Hong, Lanqing and Li, Zhenguo and Yeung, Dit-Yan and Xu, Qiang},
  journal={arXiv preprint arXiv:2310.02601},
  year={2023}
}