With its withdrawal from Afghanistan complete, the United States is left trying to remember why it invaded the country in 2001, and to wonder what two decades of war accomplished. Ostensibly the key to the post-9/11 “war on terror”, the US occupation of Afghanistan ultimately became, among other things, a state-building venture and a protracted struggle against corruption and graft.

Yet this wasn’t the first long war the West fought on Afghan soil. The last protracted conflict between Afghanistan and a global superpower, the Soviet-Afghan War in the 1980s, was a proxy for the conflict between communism and the “free world”, and presaged the end of the Soviet Union. But it is the conflicts of the 19th century that define the region’s relationship to Western interests.

The British Empire spent nearly all of the 19th century embroiled in Afghanistan, pursuing a combination of diplomacy, espionage and war that ended in a messy retreat that echoes the events of the past month. History shows us that such entanglements aren’t geopolitics gone awry or unfortunate accidents. They are instead a feature of Afghan-Western relations, a performance of global dominance undertaken by successive world powers on a strategically important region that has served for two centuries as a buffer zone between competing interests.

The story begins with British colonialism. When the British East India Co began consolidating power on the Indian sub-continent in the 18th century, the economic and political landscape of the broader region shifted, pulling Central Asia, and the lands and peoples that would become Afghanistan in particular, into a global struggle over the wealth, land and power Britain had captured in Asia.

From Britain's perspective, the biggest threat came from the Russian Empire. While Britain seized control of India, Russian frontiers were rapidly expanding to the south and east as the Russian Empire conquered much of Central Asia. By the first decade of the 19th century, the Russian Empire had incorporated much of what is today Kazakhstan. Beginning in the 1830s, the Russians began their conquest of the ancient Central Asian cities of Khiva, Bukhara and Kokand — a process that would be complete by the 1880s.

Afghanistan lay between Russian frontiers and the British sphere of influence in India. Throughout the 19th century, British politicians watched Russia's advancing frontiers with growing alarm. They were keenly aware of the threat: if Russian armies could find an overland route through their Central Asian territories, they could invade British India from the north through Afghanistan.

No one knew in the 19th century whether an overland route between Russian Central Asia and British India could support an advancing army. There had been no European survey of Afghanistan, and European scholars knew relatively little about the languages and cultures of the region. Beginning in the early 19th century, both empires sent their spies — men who had studied regional languages and were adept riders who could handle Afghanistan’s mountainous terrain — to see what could be learnt about Afghanistan and, crucially, about the rival empire’s activities there.

This contest for information about and influence in Afghanistan would occupy the British and Russian empires for much of the 19th century. What began as an extended diplomatic feud, more concerned with displays of imperial prowess than with the question of whether either empire wanted to govern Afghanistan, grew into a sort of early cold war, which eventually became known in British popular culture as the Great Game.

“You’ve a great game, a noble game, before you,” wrote the explorer and intelligence agent Arthur Conolly to a colleague who was appointed in 1840 to a political post in Afghanistan by the British East India Co. The goal would be to arrange regional politics to Britain’s advantage — distracting, harrying and subverting the Russians and keeping beneficial alliances and trade routes intact. The British public loved everything the Great Game connoted, including the stories of roguish adventurers who made their way from foreign dispatches onto the page of novels such as Rudyard Kipling’s Kim.

Afghan rulers were just as aware as the British and Russians that the global balance of power depended on their territories, and they were just as keen to use the opportunity to bolster their own power. The ruler of Afghan lands when the British first made contact, Shah Shuja Durrani, had fled to British India when he was deposed in 1809. He hoped to use the force of Britain’s influence in the region to retake his throne. He lived on a stipend from the East India Co, whose officials thought an Afghan ally might prove useful some day.

By 1823, a new family had come to power in Afghanistan, headed by the politically cunning Dost Mohammad Khan Barakzai. He also hoped the British could help him strengthen his own position, but he was well aware of the Russo-British rivalry. When British diplomats dallied in offering their support, Dost Mohammad Khan courted the Russians.

The situation came to a head in 1839. Dost Mohammad Khan's new friendship with the Russians worried the British, but even more worrisome was his political prowess. The area under his control was, at its greatest extent, larger than the boundaries of Afghanistan today, and he expanded his regional influence throughout his rule. In addition to his contact with Russia, Dost Mohammad Khan allied with Iran in an attempt to take control of Punjab, an area firmly in the British sphere. The British understood that these imperial ambitions threatened to destabilise the region even without Russian interference.

The British took the threat so seriously that George Eden, the governor-general of India, eventually committed to a full-force military invasion of Afghanistan, with the intention of removing Dost Mohammad Khan from power and installing Shah Shuja who, they hoped, would be more compliant with British interests.

Shah Shuja had not enjoyed much popular support during his rule, and when the British tried to reinstate him in 1839, it was clear that he would not be able to serve as the puppet king that Eden had hoped he might become.

Dost Mohammad Khan, on the other hand, only saw his popularity grow after the British invasion. He led a powerful resistance movement throughout Afghan lands and ultimately, to Britain’s great surprise, defeated the British army in the winter of 1842.

That was hardly the end of the Anglo-Afghan story, however. In the months after the British retreat from Afghanistan, the governor-general of India raised another army to exact revenge on the Afghan resistance, razing villages and destroying harvests on its return. Two more wars and near-constant negotiations between Britain and Afghanistan would occupy nearly another century.

Afghanistan’s position as a buffer zone had at first been the informal result of Britain and Russia’s contest for influence. In 1893, it became international law when Britain and Afghanistan signed a treaty formalising the border between their respective spheres of influence. In 1919, after a third war, Britain formally recognised Afghanistan as an independent nation.

In the end, Russia never invaded India through Afghanistan, and there’s no evidence that it ever seriously planned to do so. What Britain fought in Afghanistan throughout the 19th century was not so much a specific threat of invasion, but rather the spectre of shifting power balances in the region, and in the whole imperial world. Afghanistan — its people and its land — bore the consequences of that fear.

The United States fought a spectre, too — terrorism and Islamist extremism, problems that were never linked exclusively to Afghanistan, nor ever fully defeated. Just as Russia’s economic dominance on the other side of Central Asia threatened the British East India Co.'s total sway over the region in the 19th century, terrorism and extremist political movements pose threats both real and symbolic to US geopolitical dominance today. The United States tried for 20 years to make Afghanistan another kind of buffer, this time between the Islamic world and “the West”. Once again, the people of Afghanistan bore the consequences.

• Emily Laskin is a postdoctoral fellow at the NYU Jordan Centre for the Advanced Study of Russia, where she focuses on the Russian conquest of Central Asia in the 19th century