As statues come down everywhere and right-of-centre pundits waggle their fingers and say they told us so, it is important to try to draw distinctions. After all, when you topple George Washington from his pedestal — which many argued would never happen — you are making a bold claim about who deserves to be dishonoured.

I warned five years ago that we are rushing down this road with only the haziest notion of where we will wind up. Now that we have a better idea — Cervantes? Who hates Cervantes? — it is less clear than ever that the road is worth taking.

But if this is to be our journey, we would do better to travel it democratically. Decisions about which historical figures to — literally — de-platform should be taken within, not outside of, the ordinary processes of political debate.

With this caveat in mind, let me suggest two simple rules to guide our conversations.

Rule No 1: No statue is entitled to continue to exist merely because it exists now. The activists are right about this. Change in our decisions over whom to commemorate is often sensible.

Everything else is up for grabs.

Rule No 2: As we will see in a moment, this rule matters most. But it requires a preamble. Much of the social-media debate has been over whether an historical figure who took a morally objectionable position must be understood as “a man of his time”.

The response of activists has often been along the lines of “Hitler was a man of his time, too”.

Here both sides are mistaken. Hitler wasn’t a man of his time. The most devastating war in human history was fought to prove otherwise. On the other hand, we shouldn’t excuse anyone just because of — to continue the gendered tone of the debate — the times in which he lived. The right question is where he stood with respect to the moral understandings of his time.

In particular, what we should ask is whether the views of the person to whom objection is now being raised were above or below the median position held by people of the era.

And if they were below, how far below did they lie? For a simple example, look at the Washington professional football team’s removal of the statue honouring longtime owner George Preston Marshall. He was the last National Football League owner to integrate.

He hired his first black players, with ill grace, only under severe pressure from the Kennedy Administration. (Proud personal confession: my father was involved in the decision to bring that particular bit of pressure.)

Similarly, the Confederate generals who have become objectionable were not just on the wrong side of history.

They were willing to kill and die for that wrong side’s victory, at a time when a genuine battle line had been drawn. (And, no, I don’t exempt Robert E. Lee because of his occasional public disapproval of slavery. Quite the contrary.)

Yes, diehards insist that the South was fighting for independence not slavery, but this trope ignores the historical and economic reality. The southern states said they would leave the Union if Abraham Lincoln, who they considered an abolitionist, was elected; and after he was elected, they left.

True, much of the country also supported the racial hierarchy of the day, and as the historian David Blight reminds us, the determination to bind up the war’s wounds healed only the breaches between whites. (For that matter, the very term “civil war” arguably de-emphasises the role of slavery in the conflict.)

But there is a difference between, say, Ulysses Grant — his statue shamefully toppled in San Francisco — who whatever else he did pursued the Union cause with a vengeance, and the men who fought for the actual goal of preserving white supremacy.

As for George Washington and Thomas Jefferson, the case for leaving their statues in place is strong, because although they held slaves, the views they expressed about the practice tended to be in advance of the position of the nation at the time. (This doesn’t excuse either Washington’s brutal treatment of his captive workers or the viciousness of Jefferson’s comments on black people in Notes on the State of Virginia.)

But let’s jump forward into the more recent past. Even before today’s protests, Theodore Roosevelt and Woodrow Wilson had become the objects of ire.

Wilson deserves it more. Not only did he refuse to admit black applicants while serving as president of Princeton University, but he instituted racial segregation in much of federal employment — and stuck to his decision in the teeth of black protest against the practice. As for Teddy, I certainly understand the objections to honouring him. But his cousin was worse. Liberal hero Franklin D. Roosevelt was the one who presided over the Japanese internment programme.

FDR also refused repeated entreaties from black leaders during the Second World War to integrate the armed forces. His White House stated falsely that the National Association for the Advancement of Coloured People supported the segregation policy. Let’s recall, too, that he turned a deaf ear to the entreaties from black newspaper editors and his own wife to integrate the White House press corps.

He used the n-word in private conversations, and penned it in the margin of a political speech. And he infamously authored a college paper in which he referred to black people as “semi-beasts”.

Don’t get me wrong. Although I understand and often share the anguish and pain of the demonstrators, I’m not endorsing the removal of any particular bit of iconography. I’m endorsing serious conversation about which should go. Racial flat-earthers who insist on defending them all don’t have much of a case; but neither do those who think the decision should rest on anything less than proper democratic debate.

• Stephen L. Carter is a Bloomberg Opinion columnist. He is a professor of law at Yale University and was a clerk to US Supreme Court Justice Thurgood Marshall. His novels include The Emperor of Ocean Park, and his latest non-fiction book is Invisible: The Forgotten Story of the Black Woman Lawyer Who Took Down America’s Most Powerful Mobster