Sure, one compiler can't be egcs 2.95 and the other clang 8 (at least not in most cases) but the target compiler system doesn't even have to have include files, libs, libfoo-devel packages installed or they can be, but from old versions.
As a proof of concept I once had a linux with gcc4.2.1 make c->.o files that later ran on OpenBSD. Would I trust that resulting binary? Sure not.
But in cases of "lets spin up 10 compile boxes with <same ubuntu within a week of updating>" or have something along the lines of Xcode which allowed you and your colleagues to help out each other with small work units in order to make a single compile run quite much faster, that is a definite possibility.
For that final release build, you might consider running it on one of those 10 VMs in the example above and have 'only' 9 others help out with the sub-parts in order to get some kind of .. guarantee.
If it takes a minute for a full build on one box, getting a hint after 10 seconds that you misspelled something and that it won't ever link decently is worth something too if you value your time as a developer.
Not saying it's perfect, only chipped in on the "must be 100% or it can't ever work", hopefully without needing to go into the "coach says we need to give 110% this game, and 120% if it's the finals" nitpicking.
This doesn't work for C++, as there is no standard ABI and thus no safe way to link object files from different compiler versions into the same binary.
