On belongs_to with `touch: true` association, unassigned object is
caused touching, but assigned object is not touched.

And also, if primary key is customized, it will touch against the wrong
target looked up by the customized key as primary key.

This change ensures correctly touching consistently between assigning
and unassigning.