We don't want to leak the extra migration path to other railties tests.

(cherry picked from commit a0e05057)