On Wednesday night, the United States Court of Appeals for the 11th Circuit issued a partial stay of District Judge Aileen Cannon’s ruling granting Donald Trump a special master to review the 11,000 documents seized from his Mar-a-Lago home by the Federal Bureau of Investigation. The ruling exempts classified documents from Special Master Raymond Dearie’s review for now, and allows a criminal investigation pertaining to them to proceed.

The legal community, including many on the Right, has been baffled by Cannon’s rulings in the case. Even Trump’s former attorney-general, William P. Barr, called them “deeply flawed”. On the Left, many have questioned Cannon’s independence because Trump appointed her, and she is presiding over a case that he is actively trying to discredit.

If history is precedent, however, all of this legal wrangling may be moot because there is another component to the FBI’s investigation of Trump’s storage of documents: a counterintelligence operation, which is a whole different matter. The FBI’s Counterintelligence Programme is responsible for “exposing, preventing, and investigating intelligence activities in the US” that threaten “the nation’s critical assets”, and keeping “weapons of mass destruction from falling into the wrong hands”. Historically, this mission has been of utmost importance and justified the FBI using highly aggressive, secretive and sometimes unlawful tactics.

This history indicates that it is the counterintelligence element of the FBI’s investigation into Trump that could pose the most risk for the former president, especially because it is the part where the courts will be least likely to know what the bureau is doing and least likely to limit what it can do.

J. Edgar Hoover, the first FBI director and the man who led the bureau for nearly a half-century, epitomised the way the FBI deployed espionage-like tactics in its counterintelligence programme. Hoover learnt the art of such tactics in conjunction with fascist European police forces before the Second World War.

In the 1930s, President Franklin D. Roosevelt hitched the New Deal to Hoover’s celebrity reputation to rebuff conservative critics. The FBI could chase bank robbers and kidnappers across state lines, it was equipped with fast cars, fast guns and radios. It had links to Hollywood studios and New York publishers. Hunting outlaws such as “Baby Face” Nelson, “Machine Gun” Kelly and “Pretty Boy” Floyd mesmerised audiences about the power of Roosevelt, Hoover and the New Deal.

Meanwhile, below the radar, Hoover forged relationships with anti-communist police societies such as the International Criminal Police Commission, which was taken over by Nazi Party officials in 1935. Hoover sent FBI delegates to Berlin in 1939 to learn what the ICPC described as “repressive and preventive measures against actions preparatory to crimes and other dangerous conduct showing criminal intentions”.

Roosevelt authorised Hoover to transform the FBI into a counterintelligence agency — but he presumed Hoover was anti-fascist and would target Nazi and Japanese sympathisers. Instead, the conservative Hoover wanted to stop American “communists”, a label he applied broadly to the liberals crusading for an expanded federal government that offered a social safety net.

Hoover hid his disdain for the New Deal under the popular Roosevelt, sensing a battle he couldn’t win. But after Roosevelt died in 1945, Hoover began attacking its ideology, institutions, workforce and beneficiaries. Hoover and the new president, Harry S. Truman, already had an adversarial relationship, and whispers soon swirled around Washington that Truman was going to downsize the bureau and fire Hoover.

Hoover retorted that any attempt to remove him from office was an international communist conspiracy to weaken US defences as part of a Soviet attack. In the burgeoning Cold War climate, this was a potent claim — one that protected Hoover’s job.

The contention fit with Hoover’s broader tactics in this era. In 1947, Congress passed the National Security Act, which made the FBI responsible for protecting nuclear secrets and empowered the bureau to remain a counterintelligence agency even with the Second World War in the past. Hoover used this authority to help foment the “Red Scare”, which mixed the actual problem of nuclear espionage with fearmongering that the New Deal sheltered traitors who wanted to weaken the FBI.

Beginning in 1956, Hoover went even further tactically, launching an operation called CointelPro (Counterintelligence Programme) that combined vast amounts of surveillance, blackmail, propaganda and violence. He valued dirty secrets about his adversaries and their loved ones that he could use to silence dissent.

Assistant director William Sullivan oversaw CointelPro and later wrote in his memoirs that the moment Hoover got dirt on a senator, he sent an emissary to Capitol Hill to make sure the senator knew that the bureau had “by chance happened to come up with this data on your daughter”. Hoover wanted the senator to know this, because “from that time on, the senator’s right in his pocket”.

When illegal surveillance discovered criminal activity, agents passed it to the Department of Justice’s criminal division for prosecution. Other times, however, Hoover’s agents resorted to manufacturing such evidence.

CointelPro targeted not only politicians and activists, such as the Reverend Martin Luther King Jr, but also the organisations and social movements to which they belonged. Over the 15 years of its existence, it focused on the Women’s Liberation Movement, Communist Party USA, anti-Vietnam War organisations, environmentalist and animal rights organisations, the American-Indian Movement, Chicano and Mexican-American groups, Puerto Rican independence groups and the civil rights and Black Power movements, especially their leadership. While CointelPro mostly focused on groups on the Left, it did also investigate far-right groups such as the Ku Klux Klan and the National States’ Rights Party.

CointelPro could have a devastating impact on organisations. FBI records for “CointelPro — Black Hate” directed agents to “disrupt, misdirect, discredit or otherwise neutralise the activities of Black nationalist hate-type organisations”. Accordingly, the FBI planted inflammatory material on organisational members to sow division. The FBI also used coercive tactics to turn friends and allies against each other. Other common CointelPro tactics included perjury, witness harassment, witness intimidation and the withholding of exculpatory evidence.

CointelPro was the state’s response to the civil rights movement and accelerated with the tumult of the 1960s as Cold War fears about subversives continued to swirl. The programme was exposed in 1971 by a group of concerned citizens calling themselves the Citizens’ Commission to Investigate the FBI. The commission burgled an FBI field office in Media, Pennsylvania, and passed secret documents to news agencies. Sordid revelations about the programme caused an uproar, prompting its demise. But in its 15 years, it had a significant impact on American politics and society, exposing how far secretive counterintelligence operations could go without arousing scrutiny.

Counterintelligence programmes peak at times of heightened concern about nuclear espionage and nuclear war. The recent deterioration in the relationship between Russia and the United States, and the tensions between China and the US, as well as the unusual activities at Mar-a-Lago, are reminiscent of conditions that gave rise to CointelPro.

Trump’s efforts to delay and discredit the DOJ’s criminal case may slow the work of federal prosecutors. Cannon’s rulings may lend him support. But this probably will not slow the counterintelligence work being done to ascertain what damage — if any — Trump’s storage of documents posed. If the FBI has evidence of espionage, US politics may be on the cusp of transformation. While times have changed, one thing remains true: Americans and politicians are willing to countenance aggressive tactics in the name of national security in times of fear.

• Stephen M. Underhill is professor and chairman of the department of communication studies at Marshall University. He is author of The Manufacture of Consent: J. Edgar Hoover and the Rhetorical Rise of the FBI (MSU Press, 2020)