DonM Physics/geometry would clearly show that a ball, hit offline at the same angle, would travel farther off a mythical center line just by virtue of going farther.
So if what you are saying is true, it implies that people who hit driver farther, by and large, start the ball off on a straighter line. It may be the case. Probably people who hit driver farther are generally also more skilled at playing the game.
But if you take a guy who hits it 10 degrees offline and then give him 15 more yards without changing anything else, he is definitely going to miss more fairways.