LLMs In Robotics

LLMs in robotics are language-model capabilities used around robots for planning, coding, instruction following, human interaction, tool use, and higher-level reasoning.

Key points

• Back to Engineering's PiDog and microcontroller-code videos show a practical early pattern: use LLMs to write or assist robot-control code, then test the result against real hardware ^[src-076].
• Running LLMs locally on robot-adjacent hardware is constrained by memory, compute, latency, and power; cloud LLMs add privacy and reliability questions when robots perceive homes or people ^[src-076].
• LLMs can help with instructions and behaviour, but robot usefulness still depends on reliable sensors, actuators, control code, and task-specific data ^[src-076].
• Gemini Robotics-ER illustrates a more specialised model-side pattern: high-level embodied reasoning can call tools, VLA models, or user functions while reasoning about physical scenes ^[src-039].
• The important distinction is between a chat model that talks about a robot and a robot stack where model output is grounded in perception, constraints, and executable actions ^[src-076].
• Fan argues that language-heavy VLA models are not enough for the robotics endgame because they are better at nouns and knowledge than at physics and verbs ^[src-082].
• His proposed replacement does not discard language, but demotes it from the center: world/action models make vision and action first-class, with language prompting used to steer open-ended tasks ^[src-082].

Related entities

• Back to Engineering
• Raspberry Pi
• Gemini Robotics-ER
• Jim Fan
• NVIDIA

Related concepts

• Physical AI
• Edge Robotics
• Embodied Reasoning
• Agentic AI
• Robotics Data Loop
• Physical Safety Constraints for Robots
• World Action Models
• Sensorized Human Robotics Data

Source references

• ^[src-039] Laura Graesser and Peng Xu – "Gemini Robotics-ER 1.6: Powering real-world robotics tasks through enhanced embodied reasoning" (2026-04-14)
• ^[src-076] Back to Engineering (iulia) – physical AI, robotics, and data science cluster (41 videos, 2018-12-16 to 2026-05-10)
• ^[src-082] Sequoia Capital — "Robotics' End Game: Nvidia's Jim Fan" (2026-04-30)