Americans would rather talk about almost anything other than class. Even today as income inequality and racial and gender disparities dominate the headlines, even among the tell-all Millennials, class is quarantined from the arena of respectable conversation.

The latest example of this stubborn truism emerged after a video showing the catcalling torment young women face on city streets was posted on YouTube. Many viewers noticed that in the two-minute video by the activist group Hollaback!, condensed from a single woman’s 10 hour walk around New York City, all of the offending men were black and Hispanic. They chastised the creators for creating the appearance that white men don’t do this sort of thing, which led ultimately to an apology from Rob Bliss the video producer.

To say that Bliss was being disingenuous is putting it mildly. In a little noticed item on the website of the alternative magazine Mass Appeal Chris Moore did a little sleuthing and found that more than half of the shots used in the film appeared to be shot on 125th Street in Harlem, a predominately poor, black neighborhood. Much of the rest were shot near Times Square and Canal Street, neither area what anyone would call genteel. You have to give Bliss this much credit: He knew what he and very few other people are willing to admit. Street harassment is largely a class thing. In New York, at any rate, that means it’s also a race thing.

Now before anyone tweets about the pasty skinned guys from Morgan Stanley who whisper nasty somethings in her ear when she’s on line at the Financial District Starbucks, let’s get the obvious on the table: Street harassers can come in all colors and sport all kinds of pedigrees. Without a doubt, there are white guys in Brooks Brothers or Zegna who will ruin a girl’s morning with an unwelcome suggestion for where on her body he would like to deposit his bodily fluids.

Still. These bespoke brutes may not be a rare breed, but they’re just not common enough to spoil a good 10 hour walk along the Upper East Side. Young women who tense up as they approach a construction site know full well that walking past the guys who drive the fork lift will almost surely result in some unwanted attention; walking past the architects who are pouring over the blueprints probably won’t.

The catcalling gap will make sense to anyone who has noticed that middle class men and women tend to have a different physical and sexual presentation than their less privileged peers. Psychologists have long known that there are marked class differences in child rearing that can explain this. Preparing their children for office and stable domestic life, middle class parents have always nudged their children to display what was once known as “bourgeois propriety.” The term doesn’t seem to fit an America where, as the “Advice Goddess” Amy Alkon has said, even “nice people say f–k.” But middle class homes continue to encourage their children to use their “inside voices,” to demonstrate bodily self-discipline (one reason obesity has become a class marker), to play nice, and to soften the rough edges of male physicality. They ban toy guns from their homes and petition schools to prohibit dodgeball and other “human target” games.

Lower income parents tend to be less “proper” in their childrearing, dispensing more physical punishment and shrugging off rough and tumble play. The difference shows up in school where lower income kids, particularly boys, have more trouble sitting still, paying attention, and keeping quiet; educators consistently report they have more behavior problems. It should come as no surprise that these same boys grow up to become men who are more blatantly, and for middle class women especially, more obnoxiously, interested in every passing young thing. In rare but important instances this goes well beyond obnoxious; lower income men (and women) are also more likely than middle class to be involved in domestic violence disputes.

The catcalling gap creates some cognitive dissonance for promoters of the idea of “white male privilege.” If men of color and working class dudes are the biggest offenders, then middle class (mostly white) are the good — or at least the less bad — guys. Middle class men may no longer open doors for women or help them carry heavy suitcases, but most of them would be mortified to hear a friend shout “Hey baby; shake that thing!” to passing strangers as they rush to take their poli-sci class or make their 10 a.m. project meeting.

That raises the question of how the disproportionately white campus has become the site where so many men behave badly. That’s easy to answer. Put middle class men in a frat house with flowing kegs, and their manners melt into a boozy puddle. That’s exactly the point of the whole exercise. People —men and women — drink because it feels good to shed their inhibitions, to say the sorts of things of which their parents might not approve and do things their daytime, classroom selves may wonder at.

Ironically, then, Hollaback!’s video suggests that privilege belongs to white middle class women as much as their male classmates. For all of the myriad problems they face in a college sex scene drenched in alcohol, women students can walk the ivy paths with minimum of hassle (unless they pass by the guys building the new student center with yoga studio and state of the art fitness center). When they move to a chaotic, multicultural city, however, especially if they venture into Harlem and Times Square, they find themselves bumping up against all types — blue collar and poor men, immigrants and children of immigrants, men whose parents may not have raised them to treat women with the sort of restraint their own brothers and fathers do. And they don’t like it one bit.

Even Rob Bliss — especially Rob Bliss! — has to know there’s not much anyone can do about it. Unless they’re willing to see a lot more minority men hassled by the police.

Kay S. Hymowitz is the William E. Simon Fellow at the Manhattan Institute and a contributing editor of City Journal. She is the author of four books, including Marriage and Caste in America: Separate and Unequal Families in a Post-Marital Age and Manning Up: How the Rise of Women Is Turning Men Into Boys.

Contact us at letters@time.com.