Symbols are a godsend when dealing with difficult crash reports. Have you ever tried walking through the process of analyzing one with an expert? It's a great way to level up your diagnosis skills.
Frankly, it’s not been much of an issue. I tend to be super-anal about Quality, and post-release crashes are rare as hen’s teeth; which makes the ones that do happen, more important to diagnose.
I usually get a general idea of where the crash happens, and scanning the source, has always shown me the issue.
Symbolicating a crash report is annoying but mostly mechanical (fwiw you can open them in Xcode and it will pull in all the dSYMs for you). Reading a crash report is an art.
You probably know this, but relying on your local machine and storage for this is a bad idea for production apps.
There are a ton of services that handle all the “mechanical” stuff for you (e.g. AppCenter, ex-HockeyApp), and you can just look at symbolicated crash in your browser.
For production apps I would centralize symbols, yes. But I have a strong disdain for crash reporting services, which are typically authored by people who are designing for aggregating crashes and showing pretty charts rather than really getting deep into the actual log itself. Most have pitifully less information than an actual crash log would–often missing thread state, exception codes, and memory mappings.