faisalshahbaz / mobile-seed Goto Github PK

This is the official PyTorch implementation of the following publication:

Mobile-Seed: Joint Semantic Segmentation and Boundary Detection for Mobile Robots
Youqi Liao, Shuhao Kang, Jianping Li, Yang Liu, Yun Liu, Zhen Dong, Bisheng Yang,Xieyuanli Chen,
ArXiv 2023
Paper | Project-page | Video

TL;DR: Mobile-Seed is an online framework for simultaneous semantic segmentation and boundary detection on compact robots.

Abstract: Precise and rapid delineation of sharp boundaries and robust semantics is essential for numerous downstream robotic tasks, such as robot grasping and manipulation, realtime semantic mapping, and online sensor calibration performed on edge computing units. Although boundary detection and semantic segmentation are complementary tasks, most studies focus on lightweight models for semantic segmentation but overlook the critical role of boundary detection. In this work, we introduce Mobile-Seed, a lightweight, dual-task framework tailored for simultaneous semantic segmentation and boundary detection. Our framework features a two-stream encoder, an active fusion decoder (AFD) and a dual-task regularization approach. The encoder is divided into two pathways: one captures category-aware semantic information, while the other discerns boundaries from multi-scale features. The AFD module dynamically adapts the fusion of semantic and boundary information by learning channel-wise relationships, allowing for precise weight assignment of each channel. Furthermore, we introduce a regularization loss to mitigate the conflicts in dual-task learning and deep diversity supervision. Compared to existing methods, the proposed Mobile-Seed offers a lightweight framework to simultaneously improve semantic segmentation performance and accurately locate object boundaries. Experiments on the Cityscapes dataset have shown that Mobile-Seed achieves notable improvement over the state-of-the-art (SOTA) baseline by 2.2 percentage points (pp) in mIoU and 4.2 pp in mF-score, while maintaining an online inference speed of 23.9 frames-per-second (FPS) with 1024×2048 resolution input on an RTX 2080 Ti GPU. Additional experiments on CamVid and PASCAL Context datasets confirm our method’s generalizability.

• 2023-11-22: [Project page] (with introduction video) is aviliable! 🎉
• 2023-11-22: [Preprint paper] is aviliable! 🎉

If you find this repo helpful, please give us a star~.Please consider citing FreeReg if this program benefits your project

@article{liao2023mobileseed,
  title={Mobile-Seed: Joint Semantic Segmentation and Boundary Detection for Mobile Robots},
  author={Youqi Liao and Shuhao Kang and Jianping Li and Yang Liu and Yun Liu and Zhen Dong and Bisheng Yang and Xieyuanli Chen},
  journal={arXiv preprint arXiv:2311.12651},
  year={2023}
}

We sincerely thank the excellent projects:

• AFFormer for head-free Transformer;
• SeaFormer for Squeeze-enhanced axial Transformer;