Cameron's notoriety as a spoilsman who had been in turn a Democrat, a Know-Nothing, and had flirted with Whiggery, militated against his candidacy among delegates concerned that the party should appear to be as pure as Caesar's wife.
Source: wiktionary