In answer to this question I want to propose that to is in fact always contentful—that it is never mere Case marking, strictly speaking—but that in certain contexts (namely, in V's headed by Dative-Shifting verbs) its grammatical contribution effectively "reduces" to Case marking and therefore can be suppressed under Passive.
Source: wiktionary