Whittle down to size

If he starts playing the big, tough hombre on the southside, those boys are just liable to whittle him down to size."

She seldom suffered fools. With one withering glance, she could whittle my father, who was a big man, down to size.

And also, they're scared that the girls are going to realize what a hottie you are (which they will, if you ever talk to 'em), so they're trying to whittle you down to size before you steal all the women.