Google does the same with its Nexus devices[1] in that you can build a completely open-source version of Android except for certain hardware-specific binary blobs.
The hard driver parts are for things like LTE, GPS, GSM, wifi, camera, etc. which the device manufacturers haven't chosen to open-source. I don't see how it's relevant to Android itself, any other OS would have the same problems with that hardware.
[1] https://source.android.com/source/running.html