Robot Learning on SO-ARM101

So much data collecting today. Wasn't even by choice, but stuff kept getting disconnected and cameras stopped working. Around episode 30 of attempt three, my linux computer ran out of storage and the laptop camera stopped recording and I didn't realize until after, so I started using my Macbook to run everything. Finally all 50 episodes were recorded and hopefully the data is better quality when I train tomorrow.

I trained an ACT policy today with imitation learning on Google Colab. After paying Google to use the NVIDIA GPUs, it actually trained quite quickly. Now the hard part came when I started evaluating this model with rollouts. I started with in distribution locations, so I wrote a script to run rollouts 50 times with the 5 positions I trained the model on.

The success rates of these 5 locations range from 100% to 10%. I suspected the issue to be low quality data, and indeed as I was writing down notes on why each rollout failed, I saw many similarities in what the robot struggled with to what I also struggle with when training. The robot consistently failed to bend its wrist enough or move the gripper far enough forward before grasping. Coincidentally, I also tend to undershoot those motions when controlling the arm myself! This is sarcasm, I know I'm the problem. The positions I personally find awkward to pick up were also the positions the policy struggled with most.

Perhaps naive of me, but I didn't think I'd have to train myself to pick up a block and place it in a bin before training the robot to do so as well.

Another interesting thing was how consistent the failures were. The robot often failed in almost the exact same way every rollout. Slightly wrong wrist angles or grasp positions would push the block out of place, and once the robot drifted away from states it saw during training, the rollout usually collapsed pretty quickly.

Tomorrow will involve more practice with teleop and then recording a new dataset. I haven't tested out of distribution block positions, but I don't think there's a point until I get in distribution positions working well.

So nothing was corrupted today! I recorded 50 episodes of picking up a green block and placing it into a blue bin in 5 different block positions with 10 episodes each. However, looking through the dataset visualization on HuggingFace, there seems to be a slight delay in the gripper camera and laptop camera. We will find out if that's a problem tomorrow.

I started recording episodes today. Started by recording five episodes only to find that the parquet file kept on getting corrupted. Then for some reason it just started not getting corrupted? Happy to announce I was not able to reproduce the issue, so if after the 50 episodes I plan to record tomorrow end with a corrupted file, then wow that would be unfortunate.

After recording 7 times and green-block4, green-block5, and green-block6 showing no corruption, I decided to call it a day.

Set up robot environment.

• Lighting
• Block and container
• Cameras (gripper USB camera and laptop)

Calibrated follower and leader arms and got teleop working.