Looks like great work. I was sad to learn that the entire team that designed and built those robots was let go a few weeks ago. Well, maybe some of them will get an internal transfer. But the project was canceled. I designed the sensor used in the palm of that robot to detect objects in the gripper [1][2], and worked as a hardware test engineer writing software to test hardware assemblies as the robots were being built. I’m really glad to see transformers used for multi modal datasets for robotics! You can imagine that a robot has a whole “experience” in the world and you want it to then have a new experience and predict the most logical actions to take, just like GPT models predict the next word. I suspect there are limits to this approach but it’s an important step.
I hope google research is able to continue working with these robots. There were over 50 of them when I left 4 years ago, and I suspect they ended up building a lot more. There are limits to the commercial appeal of a one armed wheeled mobile manipulator with a two finger pinch gripper, but they are wonderful machines for machine learning and robotics research.
It’s a very nice arm design. All injection molded custom gearboxes and stuff. The team previously designed a really strong metal arm that used a pulley system to eliminate backlash [3]. That could have been useful in manufacturing. It was really upsetting how they just canceled stuff and buried it. I’m so glad I work in open source robotics now.
Incidentally if you want a kind of stripped down version of this mobile manipulator, Aaron who used to be the head of X robotics during the design of [3] left to start Hello Robot [4] and I have heard people getting use out of them.
Speaking of applications, I could see something like this being helpful for people in old age, cooking or stocking small shelves. Was there talk of the final form of this tech?
There was talk of these sorts of things. At one point they talked seriously about the robot cleaning people's homes, which seemed absurd to me. It can't go up a step, it can't grab a spray bottle and use it, and they are very expensive.
I was long gone by the time the project shuttered but I suspect Tesla's entrance in to the robotics space may have been the nail in the coffin for this project. Tesla can do a manufacturing-first design approach with serious DFM the whole way through. I am still not sure if Tesla's humanoid will every come to light (right now beyond the demos we have seen my guess is that it will not become a useful product), but Tesla has the advantage that they can produce high volumes immediately with a modest cost due to their manufacturing integration, something you just cannot get with engineers in Mountain View sending designs to China.
That said the commercial prospects for this project were always limited, and they canned basically all speculative R&D while I was there 4 years ago. My in palm grip sensor was the last thing output by the small R&D subgroup I was hired in to before that group was dissolved, and that is when I moved to hardware test engineering. Lots of people wanted to develop better grippers, etc, but they wanted to try to narrow in on the existing design and commercialize it, and I just don't think this form factor has wide enough utility. It is also fairly expensive and the software side of things is very difficult. That is another area where Tesla may be able to succeed, though as much as their stack has promise, we have seen them struggle even in structured environments like roadways.
I hope google research is able to continue working with these robots. There were over 50 of them when I left 4 years ago, and I suspect they ended up building a lot more. There are limits to the commercial appeal of a one armed wheeled mobile manipulator with a two finger pinch gripper, but they are wonderful machines for machine learning and robotics research.
It’s a very nice arm design. All injection molded custom gearboxes and stuff. The team previously designed a really strong metal arm that used a pulley system to eliminate backlash [3]. That could have been useful in manufacturing. It was really upsetting how they just canceled stuff and buried it. I’m so glad I work in open source robotics now.
Incidentally if you want a kind of stripped down version of this mobile manipulator, Aaron who used to be the head of X robotics during the design of [3] left to start Hello Robot [4] and I have heard people getting use out of them.
[1] https://patents.google.com/patent/US20200391378A1/en?invento...
[2] https://patents.google.com/patent/US11407125B2/en?q=(Robot)&...
[3] https://youtu.be/ZhsEKTo7V04
[4] https://hello-robot.com/