¶ AI-based Robotic Task Setup: Natural-Language Programming of Industrial Robots
A generated pick-and-place task running in NVIDIA Isaac Sim — specified by the user in plain English and implemented end-to-end by the AI system.
AI-based Robotic Task Setup is an AI-powered programming environment that turns a brief natural-language description of a pick-and-place task into a fully tested, deployable robot control program. It integrates an AI coding agent into an EDI-developed robotics framework with reusable building blocks and automated simulation-based verification, reducing the time to set up a new robot task from days or weeks of expert engineering to under two hours of natural-language interaction. The generated code runs predictably and offline on the production robot, with no run-time dependency on cloud AI services.
¶ Tool description
An end-user — not necessarily a programmer — describes the task in a few sentences. An AI coding agent, operating inside the robotics framework, generates the simulation configuration, the robot's control program and automated success-verification checks. The generated code is validated in three tiers — a fast offline check (CPU, under one second), a rapid visual preview, and full physics simulation in NVIDIA Isaac Sim — with task instances replayed under randomised initial conditions. If a verification check fails, the diagnostic output is fed back to the agent, which iteratively repairs the code until it passes, often without further user intervention. As the library of previously implemented tasks grows, new task implementations automatically reuse framework enhancements generated for previous ones, and can also be specified more easily by referencing existing tasks.
On a benchmark of 20 generated pick-and-place tasks across four complexity categories — basic sequential pick-and-place, attribute-based routing, sorting with stacking, and complex multi-feature scenarios — the prototype system achieved a 100% success rate in automated sanity-check tests, 97% in rapid visual preview, and 80% in full-physics simulation. The failures are mostly attributable to the absence of collision-aware motion planning, which is on the roadmap.
High-level architecture of the system.
Key features:
- Natural-language task specification — a one- or two-sentence description can be enough; no robotics coding required.
- End-to-end code generation — the coding agent produces the simulation setup, the robot control program and the success-verification checks together.
- Three-tier simulation verification — fast offline test (< 1m /run), rapid visual preview, and full physics in NVIDIA Isaac Sim.
- Closed-loop self-debugging — automated success checks feed failures back to the coding agent, which fixes the code in a test-analyze-edit loop.
- Randomised robustness testing — each task is validated across multiple seeds and randomised initial conditions.
- Task re-use — previously implemented tasks are recombined to build new variants with little additional work.
- Offline deployment — generated code is standalone and runs on the production robot controller with no run-time AI-service dependency.
¶ Workflow
The system operates in the following stages:
- The user enters a short natural-language description of the task.
- The AI coding agent generates the simulation setup, the control program and the automated success checks inside the robotics framework.
- The program is checked in fast offline mode (CPU, < 1 min/run).
- It is then run in a rapid visual preview, giving the user a chance to validate the task setup and end-state.
- Finally it is verified in full physics simulation in NVIDIA Isaac Sim, replayed under multiple randomised initial conditions.
- On any failure, diagnostic output, including image snapshots of failing picks or placements, is fed back to the coding agent, which iteratively refines the code until the task passes — without user intervention.
¶ Intended uses
This tool is intended for:
- Setting up industrial pick-and-place tasks — attribute-based routing, sorting, stacking and container targeting — for small-batch and high-mix manufacturing.
- Enabling production engineers without robotics-coding expertise to program and re-program robots.
- Rapidly producing new task variants by reference to an existing library of implemented tasks.
Within the AIMS5.0 context, the technology addresses the dominant cost driver in flexible Industry 5.0 automation: not the robot itself, but the time and expertise needed to re-program it for each new task variation. By combining usability, fast turnaround, measurable reliability and offline deployability, it lowers the barrier to robotic automation for SMEs that cannot maintain a permanent robotics-integration team.
¶ Limitations
- The current task class is limited to pick-and-place with attribute-based routing or sorting, stacking and container targets.
- Validated in a simulated industrial environment (NVIDIA Isaac Sim, simulated Universal Robots UR10 arm), TRL 4–5; demonstration on a physical robot is planned within AIMS5.0.
- Full-physics success is currently 80% (depending on task specifics; simpler tasks are > 95%) primarily due to incomplete functionality for collision-aware motion planning; adding this is on the roadmap.
- Task generation requires online access to a frontier-class AI coding agent (Claude Code), but deployment does not — the produced control code runs fully offline on the robot controller.
¶ Citation information
A paper describing the architecture and results is in preparation, but has not yet been published.
Developed within AIMS5.0 — Advancing Integrated Manufacturing Systems (Industry 5.0), supported by the Chips Joint Undertaking and its members, with top-up funding from National Funding Authorities of participating countries.