You have to distinguish the sign from signs on trucks (indicating their maximum speed), reflections of signs in windows, you have to recognize the sub-signs which might limit the scope of the main sign (speed limit might only apply to exit or under certain weather conditions), then there are the usual adverse conditions like rain, reflections, dirt, plants, shadows etc.. Just search for "german traffic sign recognition benchmark" to see what the state of the art is (ConvNet).