Ref: https://github.com/rails/rails/pull/50284

While having the inverse association configured it generally positive
as it avoid some extra queries etc, infering it may break legecy code,
as evidenced by how it broke `ActiveStorage::Blob` in https://github.com/rails/rails/pull/50800

As such we can't just enable this behavior immediately, we need to provide
and upgrade path for users.