On Saturday, Charlottesville removed two equestrian statues of Robert E. Lee and Stonewall Jackson from the city’s public squares. That removal will not “erase history”, as the statues’ defenders have repeatedly charged. It instead allows us a clearer view of the complex Southern past.

Erected in the 1920s, the Lee and Jackson statues have long served as a screen, designed to block out any narrative that might challenge White supremacy. Rendered in bronze atop imposing pedestals, the two Confederate generals embodied the Lost Cause memory tradition — a mix of myth and dogma claiming that the Confederacy was righteous and united; that postwar Reconstruction was a travesty; and that the racial caste system should persist.

With the statues no longer looming over the landscape, casting so much of it in shadow, we can perceive Southern history and the history of Charlottesville more clearly. The region is rich with stories, previously obscured, of Southerners who fought against the Confederacy and rejected the Lost Cause mythology. Their stories reveal that the Lee and Jackson monuments represent a particular White supremacist and mythologised version of Southern heritage — and have never accurately reflected the complex history of the region.

Consider the story of James T.S. Taylor, one of the 256 Black men from Albemarle County, Virginia, who fought in the Union Army. Taylor undertook a perilous journey from Charlottesville to the Union lines in Arlington, Virginia, to enlist. He then chronicled his wartime service with the 2nd Regiment of the United States Coloured Troops in letters to the New York-based Black newspaper the Anglo-African. The men of the 2nd sought to extend “liberty and freedom to the oppressed of our native land”, Taylor explained. Alluding to the Confederate practice of giving no quarter to Black troops, Taylor voiced the USCT men's determination “never to be taken prisoner by the rebels and butchered in cold blood by them”. He wrote: “If it should be inquired about us in after times, it shall be said that we died upon the battlefield.”

With the Union's victory in 1865, Taylor’s thoughts turned to his native Virginia. During Reconstruction, he returned to Albemarle County and was elected to Virginia’s state constitutional convention, where he made the case for racial equality. On April 17, 1868, he voted to approve a new Constitution, which provided for universal manhood suffrage, a statewide public-school system and the democratisation of local government. Calling the right to vote “the palladium of American liberty”, and pointing out that “the disloyal men of Virginia are seeking, by every means in their power, to prevent the free exercise of the elective franchise”, Taylor argued that Virginia ought to use paper ballots instead of voice voting to ensure free and fair elections.

The sacrifices that Black families made for the causes of freedom and Union are starkly visible in the pension application of Frances W. Evans, a free Black washerwoman in Charlottesville. Like Taylor, Evans was part of Albemarle County’s small but resilient community of free Blacks — roughly 4 per cent of the African-American population in the region — living in the midst of slavery. Evans sought pension payments because her son, William, died in combat during the war.

As Evans related to the Pension Bureau, she had sent her son to live with his grandmother in Ohio before the war, and in 1863, young William, at age 16, joined the Union Army, enlisting in Company E of the 5th USCT in Chillicothe, Ohio. The decision to enlist carried William back to Virginia, where he was killed in action at the Battle of New Market Heights on September 29, 1864 — a Union victory in which Black troops launched a crucial assault on Robert E. Lee’s Richmond fortifications.

In 1867, Frances Evans travelled from Charlottesville to Washington to file her pension request. For more than a decade she would submit and resubmit her application, only to have it denied because she did not have sufficient documentation proving that she had depended on her son financially before and during the war. Such documentation might have included letters in which a soldier sent his paycheque to his family. But as Evans explained, with palpable frustration, because she spent the war in Confederate territory, in Charlottesville, her son had no way of contacting her while he was in the army.

The Pension Bureau conceded in its finding that Evans was caught in a sort of Catch-22: her failure to prove dependence sprang from her son being so young when he enlisted. When he died in battle, he had not yet had the chance to support his mother.

The courage and resilience of Black women in Central Virginia is further exemplified by community leader Isabella Gibbons. Gibbons was born into slavery in the 1830s and was owned by a professor at the University of Virginia. In defiance of laws criminalising Black literacy, she learnt to read and write and taught her children to do so; despite the denial of legally sanctioned marriages to the enslaved, she married William Gibbons, who, too, was held in bondage by a faculty member. After the war, as her husband became a prominent Baptist minister, Gibbons embarked on an inspirational career as an educator in Charlottesville's Freedmen's school and then in its segregated public schools.

In her deeds and words, Gibbons refuted the core tenet of the Lost Cause — its defence of slavery. Gibbons’s searing remembrance of the Old South is inscribed on to University of Virginia’s new Memorial to Enslaved Labourers: “Can we forget the crack of the whip, cowhide, whipping-post, the auction-block, the handcuffs, the spaniels, the iron collar, the Negro-trader tearing the young child from its mother’s breast as a whelp from the lioness? Have we forgotten that by those horrible cruelties, hundreds of our race have been killed? No, we have not, nor ever will.”

The claim that the Lee and Jackson statues stood for the “heritage” of the Charlottesville region has always been a falsehood. Albemarle County had a Black majority on the eve of the Civil War. That majority not only welcomed Union victory, but was instrumental in bringing it about.

With the statues gone, we can better understand the era in which Lee and Jackson, and the unsung Southerners who sought their defeat, lived. Isabella Gibbons’s eyes, engraved in granite on the Memorial to Enslaved Labourers, ask us to look unflinchingly at Southern history, in its totality, and to see it anew.

• Elizabeth R. Varon is Langbourne M. Williams professor of American history at the University of Virginia and author of Armies of Deliverance: A New History of the Civil War