News & Events

This research achievement was featured as the cover story of Nature, published on May 30, 2024.

Constrained by the "power wall" and "bandwidth wall", conventional vision sensing chips suffer trade-offs among core metrics including speed, precision and dynamic range, and cannot deliver high performance simultaneously. When deployed in complex scenarios, such chips frequently encounter signal distortion, functional failure or excessive latency, which severely undermine the stability and safety of the overall system.

To address these bottlenecks, the project team focused on brain-inspired vision sensing chip technology and proposed a novel complementary dual-pathway brain-inspired vision sensing paradigm based on visual primitives. Drawing on fundamental mechanisms of the human visual system, this paradigm decomposes visual information from open-world scenes into representations built upon visual primitives. By organically combining these primitives to mimic the characteristics of human vision, two mutually complementary and information-complete visual sensing pathways are constructed.Guided by this new paradigm, the team developed Tianmouc, the world’s first brain-inspired complementary vision chip. 

A vision chip with complementary pathways for open-world sensing

Zheyu Yang, Taoyi Wang, Yihan Lin, Yuguo Chen, Hui Zeng, Jing Pei, Jiazheng Wang, Xue Liu, Yichun Zhou, Jianqiang Zhang, Xin Wang, Xinhao Lv, Rong Zhao & Luping Shi

Abstract：Image sensors face substantial challenges when dealing with dynamic, diverse and unpredictable scenes in open-world applications. However, the development of image sensors towards high speed, high resolution, large dynamic range and high precision is limited by power and bandwidth. Here we present a complementary sensing paradigm inspired by the human visual system that involves parsing visual information into primitive-based representations and assembling these primitives to form two complementary vision pathways: a cognition-oriented pathway for accurate cognition and an action-oriented pathway for rapid response. To realize this paradigm, a vision chip called Tianmouc is developed, incorporating a hybrid pixel array and a parallel-and-heterogeneous readout architecture. Leveraging the characteristics of the complementary vision pathway, Tianmouc achieves high-speed sensing of up to 10,000 fps, a dynamic range of 130 dB and an advanced figure of merit in terms of spatial resolution, speed and dynamic range. Furthermore, it adaptively reduces bandwidth by 90%. We demonstrate the integration of a Tianmouc chip into an autonomous driving system, showcasing its abilities to enable accurate, fast and robust perception, even in challenging corner cases on open roads. The primitive-based complementary sensing paradigm helps in overcoming fundamental limitations in developing vision systems for diverse open-world applications.