I first felt a tug.

It came from behind as I waited at the bus stop, and my initial thought was that it was a somewhat crude attempt to attract my attention and ask me the time. When I turned round I saw just how wrong I was. Two men, heads covered by balaclavas. It was happening at last. I had been living in Rio de Janeiro for over six years and, finally, I was having my first experience of being mugged.

I can recall every microsecond of the following moments. Time slowed down. As they pulled me back I shouted for help. The other people who were close to the bus stop fled – they made the right decision. They had seen something I had yet to see. And then the pair of them stopped trying to tug me back and instead pushed me forward. It was the direction I was trying to go anyway, and it forced me to the floor. As I turned my head round I got a crack on the nose and then I saw it – a huge gun pointed at my head. “Vai morrer,” said one. “You’re going to die.”

My instant thought was that I was going to be OK. Just keep calm and hand it over. The other, the one who had given me the punch, was already beginning to move away. The game had finished less than an hour before. There were still people around, still a police presence in the area. Just keep calm, hand it over. First came the watch, then the wallet, then the bag I was carrying. The mobile phone, still something of a rarity at the time, stayed defiantly in my pocket. A risk, of course, but one that in the circumstances I thought I could get away with. “Stand up,” he said, “don’t look back, cross the road.” I did as I was told.

And wandered down the Avenida Brasil as the reality seeped in. Life would never be the same. I had lost a kind of virginity. I had looked into the eyes of the beast and seen a deeply disturbing truth. He hadn’t had to kill me. But if the need had arisen, he could have pulled the trigger and, come the end of the day, put his head on the pillow and slept without giving me a second thought.

I have been back many times since that day late in 2000. But never with the same jaunty stride that I had employed up to that minute. I tread more cautiously, keep my wits about me and walk with respect when I visit Vasco da Gama’s São Januário stadium.

It is a ground of massive historical importance, perhaps the foremost concrete symbol of how Brazil became kings of the global game and in their golden age raised the sport to levels of joyful artistry it has seldom, if ever, seen before or since.

That is because in order for Brazil to conquer the world, first the game had to spread downwards. The roots of Brazilian football are firmly in the elite. The evidence is all there a few miles across town, at Fluminense’s Laranjeiras stadium. The team have not played there for 20 years, but the ground remains a jewel, with an English-style pavilion. The function room has stunning stained-glass windows. Everything has an air of wealth and distinction – and that is the way that those who ran Brazilian football intended to keep it.

São Januário, on the other hand, is a rough diamond. Vasco da Gama emerged from the city’s north zone and, in their debut campaign in Rio’s first division, won the title in 1923 with a team which included poor white and even black players. In semi-feudal Brazil, where slavery had only been abolished 35 years before, this was a seismic shock. The established elite clubs, Flamengo and Botafogo as well as Fluminense, all from the wealthy south zone, did everything in their power to try to head off the barbarians at the gates. One way to exclude Vasco from the league was to insist that clubs had their own stadiums. Vasco did not – but their members responded by clubbing together to build São Januário in 1927.

It was briefly the biggest stadium in South America and until the Maracanã was constructed 23 years later it was the most important ground in Rio. It was there that the 1949 version of the Copa América was played. The inaugural game was supposed to feature Wanderers of Uruguay – an illustration of the importance and prestige of Uruguayan football at that time. They could not make it and were replaced by Santos – the club which, less than 30 years later, would unleash on an unsuspecting world Pelé, who grew up as a Vasco supporter.

São Januário stands in the heart of old, traditional working class Rio. In the hugely successful 1937 samba song “O bonde de São Januário” (“The tram to São Januário”), the neighbourhood is used as a symbol of the home of the happy worker in the relatively benign tropical fascism of the regime of Getulio Vargas. Indeed, when Vargas announced labour legislation, with unprecedented worker protection, he chose São Januário stadium to make the announcement.

Over the years my many trips to the ground have always carried me back in time. You can wander around São Januário and imagine an age where all the men sported a little moustache. For those who love football and social history, it is a rich experience.

The problem, though, is that the present can be more disconcerting than the past. Once a working-class area, São Januário now has an under-class feel about it. At the time when the stadium was being built, club supporters would hold picnics on the hill overlooking the ground, turning the construction into a tourist attraction. That hill has long since become a favela, the notorious Barreira do Vasco. The happy worker is a myth that can be sustained no more. Down at street level the tight, narrow streets are full of pokey bars and evangelical churches, doling out the consolation of either beer or religion to the desperate.

Vasco have fought for the right to stage big local derby matches in their own stadium rather than across the park at the Maracanã, but it always looked like a silly campaign – and so it predictably proved when they hosted their arch-rivals Flamengo in July.

I stayed away this time, a sensible choice. I watched the match in a café and when Flamengo took the lead I feared the worst. A voice behind me commented that São Januário was a case of either win or die, and at the final whistle the home fans rioted, an overflowing of the latent violence which is such a feature of contemporary big city Brazil: attempted pitch invasions, tear gas, pepper spray, bottles and bombs and at the end of the fray, a dead young man.

For the time being, then, no one will be going to São Januário. At the time of writing the stadium lies under a ban. I fear for its future – I fear for Brazil’s future – but I’m thankful for the history and for the memories, even those that have left a psychological scar.