I think I may have suffered somewhat from selection bias when writing that, in that a lot of my friends/colleagues work at smaller shops, where they initially started as technology infrastructure developers and worked very hard to move over to the research side.

For instance, these were the guys -running- the data infrastructure so they were looking at it all day, every day. After a while it was probably straightforward to test out intuition on patterns they may have seen.

Thanks for pointing out that the difficulties in doing so at a larger firm.