•Dataset Collection

We take three tasks as an example to simulate the application scene of making coffee, involving pouring water into a cup of coffee powder (task 1), taking a spoon (task 2) and stirring coffee with the spoon (task 3) which is illustrated in Fig.1. These tasks are different and need robots to extract valid features to distinguish the slight different of input and reason out the appropriate task to cope with. We assemble a 6-DoF robotic arm which is light weight and low price. To collect demonstrations, we control the robotic arm to complete the task through manual programming. We take images of three tasks with a RGB camera from the third-person perspective.

Fig.1. From left to right, it represents task 1(pouring water into a cup of coffee powder) , task 2(taking a spoon) and task 3(stirring coffee with the spoon) .

The collected demonstrationsconsist of 640*480 RGB images and the joint angles as labels.is the joint angles of robotic arms andis the observations of the environment. We collect 420 demonstrations for three tasks in total. Each task includes 100 demonstrations in simple scenes and 40 ones in complex scenes. The complex scene demonstrations are illustrated in Fig.2. In order to improve the understanding of the scene and prevent the model from overfitting, we use five cups of different colors, sizes and height as target containers and two different colored spoons. The rim of the cups is about 9 to 13 centimeters in diameter, which is difficult for robots to accurately pour water into it.

Fig.2. This line represents demonstrations in the complex scenes.

 

•Access to the complete Dataset

Researchers who want to obtain the complete dataset can:

1.Fillout the form Click to download the application form

2.Send the form to myyou@tongji.edu.cn.

3.Cite the paper "Simultaneous Multi-task Determination And Manipulation With Vision-based Self-reasoning Network”.