If the solar system is expanding, how much closer would Mars be to Earth 500 million years ago?
There is a misconception here. The answer, if there is a difference, is not due to expansion of the solar system per se as the whole system is gravitationally bound.
Expansion of space as a result of the Big Bang doesn’t operate on these sort of scales. Even the galaxy doesn’t show this effect, due to gravity overpowering any expansion.
The solar system is expanding microscopically due to tidal forces “robbing” the sun of it’s angular momentum and transferring it to the planets. A much larger effect is seen in the moon-earth system (where tidal forces are much stronger) but still only accounts for an increase in orbital radius of some 38mm per year. For the sun-earth or sun-mars system it is of the order of nanometers per year I think, so truly insignificant.