The post Generative AI Empowers Robots to Reason and Act with ReMEmbR appeared on BitcoinEthereumNews.com.

Lawrence Jengar Sep 24, 2024 07:06 NVIDIA’s ReMEmbR integrates generative AI, vision-language models, and retrieval-augmented generation to enhance robots’ reasoning and action capabilities over extended periods. NVIDIA has unveiled ReMEmbR, a groundbreaking project that leverages generative AI to enable robots to reason and act based on their extended observations, according to the NVIDIA Technical Blog. Innovative Vision-Language Models Vision-language models (VLMs) combine the robust language understanding of foundational large language models (LLMs) with the vision capabilities of vision transformers (ViTs). These models project text and images into the same embedding space, allowing them to handle unstructured multimodal data, reason over it, and return structured outputs. By building on extensive pretraining, VLMs can be adapted for various vision-related tasks with new prompts or parameter-efficient fine-tuning. ReMEmbR: Enhancing Robot Perception and Autonomy ReMEmbR integrates LLMs, VLMs, and retrieval-augmented generation (RAG) to enable robots to reason and act based on what they observe over extended periods, ranging from hours to days. The system is designed to address challenges such as handling large contexts, reasoning over spatial memory, and building prompt-based agents to query additional data until a user’s question is answered. The project’s memory-building phase uses VLMs and vector databases to create a long-horizon semantic memory. During the querying phase, an LLM agent reasons over this memory. ReMEmbR is fully open-source and operates on-device, making it accessible for various applications. Practical Applications and Demonstrations To demonstrate ReMEmbR’s capabilities, NVIDIA developed a practical example using Nova Carter and NVIDIA Isaac ROS. The robot, equipped with ReMEmbR, can answer questions and guide individuals within an office environment. This demonstration highlights the system’s ability to build an occupancy grid map, run the memory builder, and operate the ReMEmbR agent. In the demo, the robot uses a monocular camera and global location information to create a vector…

---

Filed under: News - @ September 24, 2024 7:19 am