`has_many_inversing` adds records to a has_many association. It does so
with destroyed records, too. So if a child was destroyed with a `touch:
true` association on the parent *and* the parent was not loaded, it
tried to load the parent to touch it. While loading the parent it added
the child record to the parent's has_many association. The logic doing
this always set the child's parent id – even if it was correct/the same
already. But since the child is destroyed, it resulted in a
`FrozenError`.

This commit prevents doing the unnecessary setting of the identical id
and therefore fixes this error.

Fixes #40943 by not doing an unneeded attribute set.

Closes #40969.

[Markus Doits + Rafael Mendonça França]