Nvidia used this year’s NeurIPS conference to reveal new AI that it has hoped will help accelerate progress toward widespread self-driving vehicles.
At the event in San Diego, the company presented Alpamayo-R1 (AR1), which it described as the world’s first industry-scale open reasoning vision language action (VLA) model for autonomous driving.
VLA models can process text and images together, meaning vehicle sensors can translate what they “see” into descriptions that use natural language.
Nvidia’s software — named after a mountain in the Peruvian Andes considered challenging to scale — combines chain of thought AI reasoning with path planning. This allows it to better process complex situations than previous iterations of self-driving software by breaking down a scenario and considering all possible options, just as a human would do, before proceeding.
This ability, Nvidia said, will be “critical” in helping to achieve Level 4 automation — defined by the Society of Automotive Engineers as when a car is in complete control of the driving process in specific circumstances.
In a blog post published to coincide with the unveiling of Alpamayo-R1, Bryan Catanzaro, Nvidia vice president of applied deep learning research, provided an example of how it would work.
Catanzaro said: “By tapping into the chain-of-thought reasoning enabled by AR1, an AV [autonomous vehicle] driving in a pedestrian-heavy area next to a bike lane could take in data from its path, incorporate reasoning traces — explanations on why it took certain actions — and use that information to plan its future trajectory, such as moving away from the bike lane or stopping for potential jaywalker.”
Other nuanced scenarios cited by Nvidia where AR1’s human-style reasoning would assist include pedestrian-heavy intersections, an upcoming lane closure or when a vehicle is double parked in a bicycle lane.
By effectively thinking aloud with its reasoning, AR1 gives engineers greater insight as to why it has made a specific decision, which obviously helps them better understand how to make vehicles safer.
The model is based on Nvidia’s Cosmos Reason, which launched earlier this year, and its open access will allow researchers to customize it for their own non-commercial use cases, either for benchmarking or building their own AVs.
AR1 is available on GitHub and Hugging Face, and according to Catanzaro, reinforcement learning post-training has proven “especially effective,” with researchers reporting “significant improvement” in reasoning capabilities.
