That issues are caused by using only the model's cast types on the
relation.
To fix that issues, use the attribute's type caster takes precedence
over the model's cast types on the relation.

Fixes #35232.
Fixes #36042.
Fixes #37484.