新闻动态

成果发表于2024年5月30日《自然》封面(Nature )

传统视觉感知芯片由于受到“功耗墙”和“带宽墙”的限制，导致其核心指标速度、精度和动态范围之间相互矛盾，不能同时实现高性能，使得应对这些场景时往往面临失真、失效或高延迟的问题，严重影响了系统的稳定性和安全性。

为  了克服这些挑战，项目团队聚焦类脑视觉感知芯片技术，提出了一种基于视觉原语的互补双通路类脑视觉感知新范式。该范式借鉴了人类视觉系统的基本原理，将开放世界的视觉信息拆解为基于视觉原语的信息表示，并通过有机组合这些原语，模仿人视觉系统的特征，形成两条优势互补、信息完备的视觉感知通路。

基于这一新范式，团队进一步研制出了世界首款类脑互补视觉芯片“天眸芯”（图2.2），在极低的带宽（降低90%）和功耗代价下，实现了每秒10000帧的高速、10bit的高精度、130dB的高动态范围的视觉信息采集。它不仅突破了传统视觉感知范式的性能瓶颈，而且能够高效应对各种极端场景，确保系统的稳定性和安全性。基于“天眸芯”，团队还自主研发了高性能软件和算法，并在开放环境车载平台上进行了性能验证（图2.3）。在多种极端场景下，该系统实现了低延迟、高性能的实时感知推理，展现了其在智能无人系统领域的巨大应用潜力。

A vision chip with complementary pathways for open-world sensing

Zheyu Yang, Taoyi Wang, Yihan Lin, Yuguo Chen, Hui Zeng, Jing Pei, Jiazheng Wang, Xue Liu, Yichun Zhou, Jianqiang Zhang, Xin Wang, Xinhao Lv, Rong Zhao & Luping Shi

Abstract：Image sensors face substantial challenges when dealing with dynamic, diverse and unpredictable scenes in open-world applications. However, the development of image sensors towards high speed, high resolution, large dynamic range and high precision is limited by power and bandwidth. Here we present a complementary sensing paradigm inspired by the human visual system that involves parsing visual information into primitive-based representations and assembling these primitives to form two complementary vision pathways: a cognition-oriented pathway for accurate cognition and an action-oriented pathway for rapid response. To realize this paradigm, a vision chip called Tianmouc is developed, incorporating a hybrid pixel array and a parallel-and-heterogeneous readout architecture. Leveraging the characteristics of the complementary vision pathway, Tianmouc achieves high-speed sensing of up to 10,000 fps, a dynamic range of 130 dB and an advanced figure of merit in terms of spatial resolution, speed and dynamic range. Furthermore, it adaptively reduces bandwidth by 90%. We demonstrate the integration of a Tianmouc chip into an autonomous driving system, showcasing its abilities to enable accurate, fast and robust perception, even in challenging corner cases on open roads. The primitive-based complementary sensing paradigm helps in overcoming fundamental limitations in developing vision systems for diverse open-world applications.