This use case develops a modular and reconfigurable AI-based robotic system for automated conveyor feeding of randomly distributed bottles of different sizes, weights, and shapes. The system combines deep learning-based perception, adaptive reinforcement-learning planning, context-aware task execution, high-level instructions, synthetic data, sim-to-real transfer, and reusable robotic hardware. The objective is to replace repetitive manual work and inflexible dedicated sorting machines with a smaller, more adaptable robotic line that can be quickly reconfigured for new objects, tasks, pickup locations, and production layouts. The expected impact is higher productivity, 24/7 operation, better use of floor space, improved circular reuse of robotic components, and more meaningful work for human employees.
| # |
Technology Name |
Description |
| 1 |
Object detection and segmentation |
Detects all bottles in the pile and provides pixel masks for them. |
| 2 |
Single object 6DoF pose estimation |
Detects the location and orientation of a bottle based on the 3D model of the object. |
| 3 |
Grasp affordance estimation |
Calculates the grasp pose for a given bottle. |
| 4 |
Grasp planning and execution |
Calculates the robot trajectory required to grasp and pick up the bottle. |
| 5 |
Socket detection |
Detects the socket position and classifies the socket as filled or empty. |
| 6 |
Robot arm feedback control |
Implements the robot trajectory control policy for placing the bottle into the socket. |
| 7 |
AI-based robot control script synthesis |
Generates high-level robot control logic based on examples or natural language task descriptions. |
| KPI # |
Related Technology |
KPI Target |
| 1 |
Object detection and segmentation |
Calculation time below 200 ms. |
| 2 |
Object detection and segmentation |
mAP@0.5 above 70%. |
| 3 |
Object detection and segmentation |
Reconfigurability time: 3 hours for retraining. |
| 4 |
Single object 6DoF pose estimation |
Calculation time below 100 ms. |
| 5 |
Single object 6DoF pose estimation |
Accuracy at 1.5 m: angle accuracy below 10 degrees and position accuracy below 5 mm. Accuracy at 20 cm: angle accuracy below 3 degrees and position accuracy below 3 mm. |
| 6 |
Single object 6DoF pose estimation |
Works without reconfiguration when a new object 3D model is provided. |
| 7 |
Grasp affordance estimation |
Calculation time below 200 ms. |
| 8 |
Grasp planning and execution |
Planning time below 200 ms. |
| 9 |
Grasp planning and execution |
Unpicked object ratio below 1/100. |
| 10 |
Socket detection |
Calculation time below 100 ms. |
| 11 |
Socket detection |
At 50 cm distance: mAP@0.5 above 70%, position accuracy below 2 mm, and filled/empty socket classification accuracy above 99%. |
| 12 |
Socket detection |
Reconfigurability time for a new socket or camera angle: 3 hours. |
| 13 |
Robot arm feedback control |
Unplaced object rate below 2%. |
| 14 |
Robot arm feedback control |
Policy step evaluation time below 10 ms. |
| 15 |
Robot arm feedback control |
Reconfiguration time below 2 days. |
| 16 |
AI-based robot control script synthesis |
Synthesized script correctness above 90%. |
| 17 |
AI-based robot control script synthesis |
Synthesis time below 1 hour. |
| KPI # |
Baseline |
| 1–17 |
The baseline is manual bottle placement on the conveyor belt by human workers. Technologies for pose estimation and object detection existed at EDI before the project, but their performance metrics were not evaluated in comparable conditions. |
| KPI # |
Validation Method |
| 1 |
Evaluate calculation time on images of bottles in a pile, using 1024x768 Intel RealSense camera images and a desktop computer with a GPU card. |
| 2 |
Evaluate mAP on images of bottles in a pile, using 1024x768 Intel RealSense camera images and a desktop computer with a GPU card. |
| 3 |
Measure the time required to fine-tune the model for a new object using images of the new object on a desktop computer with a GPU card. |
| 4 |
Evaluate calculation time on images and point clouds of bottles in a pile from an Intel RealSense camera, using a desktop computer with a GPU card. |
| 5 |
Evaluate 6DoF pose estimation accuracy at 20 cm and 1.5 m using point clouds and bottle images from an Intel RealSense camera on a desktop computer with a GPU card. |
| 6 |
Validate reconfigurability by providing a new object 3D model and checking whether the algorithm can use the new model automatically. |
| 7 |
Evaluate calculation time on images and point clouds of bottles in a pile from an Intel RealSense camera, using a desktop computer with a GPU card. |
| 8 |
Evaluate planning time on images and point clouds of bottles in a pile from an Intel RealSense camera, using a desktop computer with a GPU card. |
| 9 |
Evaluate the unpicked object ratio by repeatedly grasping objects from a pile and measuring how many objects cannot be picked up. |
| 10 |
Evaluate calculation time on 1024x768 Intel RealSense camera images, using a desktop computer with a GPU card. |
| 11 |
Evaluate accuracy at 50 cm distance using 1024x768 socket images and a desktop computer with a GPU card. |
| 12 |
Measure the time required to fine-tune the model for a new camera angle or socket type using images of the new setup on a desktop computer with a GPU card. |
| 13 |
Evaluate the unplaced object ratio by repeatedly placing a bottle into the socket with the bottle already in the robot gripper, using a UR5e robot. |
| 14 |
Execute several trajectories and measure the time needed to calculate one policy action for one trajectory step on a desktop computer with a GPU card. |
| 15 |
Measure the setup time required to retrain the neural network policy for a different robot, object type, or conveyor configuration on a desktop computer with a GPU card. |
| 16 |
Evaluate AI-synthesized scripts in simulation against the desired task and perform final manual code inspection. |
| 17 |
Evaluate synthesis time on a desktop computer with a GPU card, including API calls to remote AI services, automated simulation testing, and repeated improvements. |
- 20% improvement in prediction precision.
- 25% improvement in classification precision.
- Reduced re-configuration time of the AI-based system by approximately 30–50%.
- Improved overall productivity of the manufacturing factory by approximately 20%.
- Self-adjusted/adapted manufacturing processes above 15%.
- 70% reduction of time for developing and integrating a ML model in the digitalized industrial domain.
- Building AI Toolbox.
- Targeted guideline documents for integration of use-case-specific methods.
The use case contributes to improved prediction precision through object detection and segmentation, 6DoF pose estimation, socket detection, and robot arm feedback control. Object detection, segmentation, and socket detection also contribute to improved classification precision by enabling precise object and socket classification. The use case supports reduced reconfiguration time through quickly reconfigurable technology modules, AI-based robot control script synthesis for adaptation to new tasks, and 6DoF pose estimation that allows robots to manipulate objects in previously unknown locations. Productivity improvement is supported by robot arm feedback control in dynamic environments and by 6DoF pose estimation for handling objects with unknown poses. Self-adjusted and adapted manufacturing processes are supported by AI-based robot control script synthesis, feedback control, and pose estimation. The use case also contributes to reducing ML development and integration time through AI-based robot control script synthesis. In addition, six technologies are planned for inclusion in the AI Toolbox, with guideline documentation in the AI Toolbox Wiki and descriptions planned in WP2 book chapters and scientific publications.
| AI Toolbox Tool |
Use within the Use Case |
| Object detection tool |
Used to detect bottles in randomly distributed piles and support the perception pipeline for automated conveyor feeding. It provided object-level detection capabilities needed before grasping and placement. |
| Object detection with YoloV8 |
Used for real-time bottle and socket detection tasks where fast inference was required. It supported the detection targets related to mAP@0.5 and sub-200 ms calculation time. |
| Image segmentation detection with BiRefNet |
Used to generate pixel-level masks for bottles in piles, supporting object segmentation before pose estimation and grasp planning. |
| Mobile Segment Anything |
Used to support lightweight segmentation of objects in robotic perception scenarios, especially where fast object-mask generation was needed for reconfigurable handling tasks. |
| 6DoF Pose Estimation Tool |
Used to estimate the position and orientation of individual bottles from camera and point-cloud data, supporting the single-object 6DoF pose estimation technology. |
| Foundation Pose |
Used to estimate and track the 6D pose of novel objects based on their 3D models, supporting reconfiguration when a new bottle model is introduced. |
| Image Classification Tool |
Used to classify sockets as filled or empty, supporting the socket detection and placement-validation part of the robotic feeding process. |
| LLM Tool |
Used to synthesize high-level robot control scripts from examples or natural language task descriptions, supporting faster adaptation to new robotic tasks and layouts. |
| Prompt Context Injection Tool |
Used to provide task-specific robot, object, and environment context to the LLM-based script synthesis process, improving the correctness of generated robot control logic. |
| Socket Detection Tool |
Used by a robotic system to determine whether a bottle should be inserted into the socket and to support synchronization between perception, conveyor motion and robot actions. |
| Grasp Pose Estimation Tool |
Used by a robotic system to estimate a grasp pose to pick up bottles from structured and unstructured environments. |