Since #31575, `set_inverse_instance` replaces the foreign key by the
current owner immediately to make it happen when a record is added to
collection association.

But `set_inverse_instance` is not only called when a record is added,
but also when a record is loaded from queries. And also, that loaded
records are not always associated records for some reason (using `or`,
`unscope`, `rewhere`, etc).

It is hard to distinguish whether or not we should invoke
`set_inverse_instance`, but at least we should avoid the undesired
side-effect which was brought from #31575.

Fixes #34108.