Cars without manual controls are closer than you think. The Waymo cars do just fine without a human driver in the seat, and before they got suspended, Cruise was just starting a pilot program with vehicles without manual controls called the Cruise Origin. Sale of such a vehicle to the general public is a ways off but for a taxi service we're pretty close.
Cruise reportedly had human interventions every 4-5 miles. I haven’t seen a similar figure for Waymo, who are generally believed to be considerably better.