Why AI planning and real-time robot control belong in separate layers
Foundation models can interpret open-ended tasks; robot execution still depends on bounded timing, constraints and defined failure behavior. Reliable Physical AI systems give each layer a different job.
The short answer: language and vision models are useful precisely because they can work with ambiguity. A manipulator or mobile base, however, needs measurable deadlines, predictable execution and explicit behavior when something goes wrong. Separating those concerns preserves AI flexibility without placing open-ended inference inside a high-frequency control loop.
One instruction becomes at least four engineering layers
| Layer | Primary input | Output | Question it must answer |
|---|---|---|---|
| Intent | Language and task context | Structured goal and constraints | What outcome is requested, and what must not be violated? |
| Perception | Cameras, state and workpiece data | Object pose and scene state | Where is the target, and is the evidence sufficient to act? |
| Task and motion planning | Goal, robot model and scene | Reviewable stages and trajectories | Is the motion reachable, collision-aware and within constraints? |
| Real-time execution | Trajectory, feedback and safety state | Joint/base commands and feedback | Do cycle time, jitter and fault handling meet the application need? |
MoveIt Task Constructor models a complex task as interdependent stages and passes state between them. Its PlanningScene represents both robot state and the surrounding world. That makes a request such as “pick up the cable” a sequence of inspectable steps—not a single model response. The official MTC documentation also keeps planning and execution as distinct operations.
Low latency is not the definition of real-time
The ROS 2 real-time design material frames the problem around deadlines, predictability and the failure mode after a missed deadline. A call that is fast on average can still be unsuitable for deterministic control if it occasionally blocks. The design guidance discusses separating real-time and non-real-time work and avoiding sources of nondeterminism such as dynamic allocation, page faults and lower-priority work blocking a real-time thread.
The planning layer should produce a constrained task—not motor commands
- Translate language into a structured goal. Name the object, destination, allowed contact, speed limits and stopping conditions.
- Validate the goal against the current scene. Check perception confidence, robot state, collision geometry and tool state.
- Generate reviewable stages and trajectories. Check reachability, joint limits, collision constraints and time parameterization before execution.
- Hand execution to the real-time layer. Read feedback, issue commands and monitor faults on the required cycle.
- Return outcome and exceptions upstream. Success, deviation, grasp failure and safety events become inputs to the next planning decision.
Industrial networking solves a different deterministic problem. EtherCAT Technology Group describes on-the-fly frame processing and distributed clocks for predictable transfer and synchronization. That supports the execution layer; it does not replace task or motion planning.
Architecture review checklist
- Does language output pass through a structured task and constraint check?
- Do perception results carry confidence, timestamps and failure handling?
- Does planning use current robot state and an updated collision scene?
- Are cycle time, jitter, bus synchronization and load conditions measured?
- Are emergency stop, protective stop and limits independent from open-ended AI?
- Does the end-to-end demonstration include recovery, not only one successful run?
Matrix Dimension perspective
The useful product boundary is a clear contract between layers. AI proposes intent and candidate plans; the planning layer produces a trajectory that respects robot and scene constraints; the real-time layer executes only confirmed commands and returns state continuously. Product claims should still be published from tests on a named robot, load, network and task—not inferred from architecture alone.
Safety boundary: ISO 10218-1:2025 addresses safety requirements for the industrial robot, while complete applications and integration also involve ISO 10218-2:2025. This article explains architecture; it does not replace an application risk assessment, functional-safety design or conformity work.
Frequently asked questions
Can a foundation model directly command robot joints?
Open-ended model output should not become joint commands without constraints. A safer architecture converts intent into a structured goal, checks scene, collision, limits and safety state, then hands execution to a real-time layer.
Does real-time simply mean faster?
No. Real-time is about completing work predictably before a defined deadline and specifying what happens if that deadline is missed.
Why does a dual-arm system make layering more important?
Two arms add synchronization, mutual collision, shared-object and contact-force constraints. Planning handles coordination while the execution layer maintains deterministic feedback and synchronization.
Sources
These primary sources support the material facts and engineering boundaries discussed above.
Evaluating robot control, bimanual manipulation or a mobile platform?
Talk to Matrix Dimension →