Mel Johnson took his tea, without milk or sugar, and sat with his back to the wall at the circular table closest to the door. His eyes moved quickly, constantly, as he completed his risk assessment. He was friendly and attentive, but there were 83 other scouts in the first-floor room at Staines Town. It was his business to know their business.

A wry grin. He noticed Bullshit Pete was sitting alone, chasing a piece of chicken around his plate. Buffet Billy was working his way through chips, rice and a curry which had the colour and consistency of melted caramel. The Ayatollah was in a conspiratorial huddle with his followers, whose synchronised glances offered clues about the nature and direction of their discussion.

Agents circulated, seeking the convertible currency of casual gossip and inside information. One, nondescript in appearance apart from a lilac roll-neck sweater, was identified by Johnson as Billy Jennings, a striker who helped West Ham win the FA Cup in 1975. His bottle-blond mullet, a feature of my schoolboy scrapbook, had receded to a monkish semi-circle.

Emissaries from the international game — Brian Eastick, England’s Under-20 coach, and Mark Wotte, Scotland’s performance director — held court. The room hummed with conjecture and cautious conversation. It had the feel of a bookmaker’s on an urban side street. The men, largely middle-aged and uniformly watchful, had the pallor of too many such midwinter nights on the road.

I had invited myself into their world, a place of light and shadow. They were the Nowhere Men, ubiquitous yet anonymous, members of football’s hidden tribe. The elders, like Johnson, had paid their dues at the biggest clubs. Initiates were paid 40 pence a mile and informed decisions on players worth £10 million or more. All were under threat from technology and the new religion of analytics.

Such men are central to the mythology of modern football. Scouts may be marginalised, professionally, but they possess the power of dreams. There is no textbook for them to follow, no diploma they can receive for their appreciation of the alchemy involved in the creation of a successful player. Their scrutiny is intimate, intense and highly individual. They must balance nuances of character with aspects of pre-programmed ability and fit them to the profile and culture of the clubs they represent.

Everyone knows what scouts do, but no one truly understands why they do it and no one knows who they are. Their anonymity is anathema to the modern game, a gaudy global carousel which invites examination on its own, highly lucrative terms. Scouts supply the star system, but remain resistant to it. At best, they are indistinct figures, judged on false impressions like old-school golf caddies, who were assumed to drink heavily and sleep in hedges. They are an enclosed order, by circumstance rather than philosophy.

Anyone who has ever stood shivering on a touchline or sat shouting from the back of a stand thinks they can do their job. Whether the average football fan would want to commit to such a disconnected lifestyle is another matter entirely. Being paid for watching up to ten games a week might be the sort of fantasy which sustains a fourth former during an afternoon of algebra, but the hours are long and family-unfriendly. Scouts eat on the run, live on their nerves and receive a relative pittance.

This may seem counterintuitive, but I resolved to study them and to share their experiences, because I believed in the essential romanticism of their role. Over the course of more than a year on the road, I was seduced by the process of discovery. I saw boys whose precocity caused time to freeze, and young men, in straitened circumstances, who seized a second chance to shine. I was there when the seeds of natural talent began to germinate; this may occur in a park, or on a non-league gluepot. It may be destined to flower in one of the cathedrals of the game, but it beckons only those who comprehend its potential.

Scouts are nowhere, and everywhere. They are rivals, but mix closely. They offered me an insight into the idiosyncrasies and insecurities of an incestuous community. Their judgements are severe, occasionally unkind, but they must be judged in the context of a game which has an unerring habit of brutalising its participants. What makes scouts different? Time and again Paul Newman’s line, delivered in his role as Butch Cassidy, in the film Butch Cassidy and the Sundance Kid, came to mind: “I got vision, and the rest of the world wears bifocals.”

On this particular Wednesday night, 15 February 2012, Chelsea were playing West Ham in the third round of the FA Youth Cup. Teenagers, courted and cosseted from adolescence, were accustomed to indulgences, such as being transported to Wheatsheaf Road in the first team’s luxury coach. They filed, with studied indifference, through the tiny car park, past the picture window of a gymnasium which framed suburban wage slaves purging themselves on treadmills.

It was a big night for the Conference South club, even if its patron, the TV astrologer Russell Grant, could not attend due to unforeseen circumstances. The surrounding streets were clogged with traffic, and the Staines Massive, the club’s inevitably entitled support base, swelled beyond 1500. The following night’s fund-raising quiz would struggle to make a quorum.

Chelsea were under pressure to justify an academy which had consumed in excess of £60 million without producing a first-team regular in its first eight years. The annual operational budget, £8 million, dwarfed that of many Football League clubs. A single youth-team player, the Brazilian striker Lucas Piazon, represented a £10 million investment. West Ham were quietly confident of nurturing a new golden generation, attuned to cherished principles and philosophies, but there were no guarantees.

For some, this Youth Cup run would be a career highlight. Around 10,000 boys are in the academy system. In the region of 1% will make a living out of the game. Two-thirds of those given a professional contract at 18 are out of professional football by the time they are 21. The scouts were there to scavenge; it was their job to be in a position to take advantage of a coach’s lack of foresight or a club’s lack of patience. They were looking for signs, hints of undervalued talent that others might miss.

Johnson, Liverpool’s principal scout in the South of England, was reassessing players with development potential. His son Jamie, Millwall’s chief scout, was one of many representatives of Football League clubs looking at long-term loan targets. Dean Austin, newly employed on a part-time basis by Bolton Wanderers, sought youngsters with resale value.

Austin, the former Spurs full-back, was working through his angst at the interruption to his coaching career, which had surprisingly stalled. He managed in the Conference and combined coaching with player recruitment at Southend before becoming Brendan Rodgers’s assistant manager at Watford and Reading. He had been seeking a manager’s role since leaving Crystal Palace in May 2011 and was prepared to keep head-butting a glass ceiling. “He’s my angry young man,” said Johnson, affectionately. 

“I just love getting out there, on that training field’ said Austin, with convincing force.

Mark Anderson arrived from another dawn shift as a senior site manager for a construction company. Football was his release, the Liver Bird on his quilted jacket a badge of honour, even if wearing it on duty defied convention. He was proud of his association with a club of Liverpool’s stature, as a youth scout, and radiated unfulfilled ambition. He, too, was determined to get back into the game on a full-time basis.

Steve Gritt was in his first year as Bournemouth’s chief scout. While Alan Curbishley, his former managerial partner, endured the purgatory of punditry, he was sustaining a career defined by spells in charge at Charlton, Brighton and Millwall. His most recent post, as director of Charlton’s Academy, had relevance, but he remained in culture shock.

“When I first started in management with Curbs, 20 years ago, the managers went out scouting,” he said, tightening the drawstrings on the fur-lined hood of his quilted coat. ‘That’s not the case these days. So many rely on the judgements of their scouts. Funny, that, because they are the ones who pay if those judgements are wrong. This job is a bit of an eye-opener, to be honest. Even managers don’t really realise what scouts do, how hard they work. They are out in all weathers, at all hours. They don’t get the credit they deserve.”

The scouts sat where they could, in a 300-seater main stand. They didn’t share the laughter as balls disappeared into adjoining gardens. They were blind to the idiosyncrasies of the setting. Closely planted leylandii, evergreen symbols of Middle England, stood guard behind one goal. Newly built houses, as neat and symmetrical as loaves of bread on a baker’s shelf, were vulnerable to stray shots at the other end.

During lulls in play, Anderson delivered despatches from football’s dirty war. Scouts were having their car tyres slashed on suspicion of poaching young players from smaller clubs, who were aggrieved by what was perceived to be the institutionalised greed of the Premier League’s Elite Player Performance Plan. “There’s a lot of aggro out there,” he reported. “It’s getting naughty. A lot of clubs won’t have us on the premises.”

Representatives of six Premier League teams, including Arsenal, Manchester United and Chelsea, drafted the plan, which created four categories of academy. It was, essentially, a self-selecting process since those in the top category were required to underwrite an annual budget of £2.3m and employ at least 18 full-time staff. Wealthier clubs were freed from the need to recruit young players within a 90-minute radius, in terms of travel time, from their bases.

Naked self-interest is excused when an outstanding 14-year-old has a bounty of up to £2 million on his head. EP3, as the plan is known in the trade, was reportedly forced through when the Premier League threatened to withdraw £5.4 million in so-called Solidarity Funding from Football League clubs if they did not sanction its adoption. Their resistance, understandable because it gave an open invitation for scouts to pilfer boys as young as nine for small change, was futile.

Young footballers are on the menu and the price list is set. Instead of paying seven-figure sums, the biggest clubs need only to pay £3,000 for every year a newly recruited boy has spent at another club’s academy between the ages of 9 and 11. As little as £12,000 compensation is required for every year he has been nurtured — elsewhere — between the ages of 12 and 16. Premier League clubs, who had accumulated debts of £361 million from a collective income of £2.3 billion, had a vested interest in portraying anarchy as opportunity. 

Johnson, a man of ritual and restraint, busied himself with his paperwork. He had folded a sheet of A4 into quarters and recorded the players’ names and numbers, in formation, in blue and red ink. It would be stored, like thousands of others, in a series of suitcases in his garage. Occasionally, in shorthand designed to offer a mental image, he wrote “looks like” next to a name. Thus Taylor Miles, scorer of West Ham’s 43rd-minute equaliser, was linked in perpetuity to Craig Bellamy. He lacked the Liverpool player’s default mode of an enraged ferret, but shared his energy and eye for goal.

Whenever Johnson spoke, he instinctively held the official team sheet over his mouth with his right hand, to prevent strangers overhearing. His voice was soft, yet insistent. “You’re very much on your own in this job,” he said. “It can be very lonely. You don’t really have friends, you have your fellow scout acquaintances, but they’re not really friends. It’s a bit of a secret world. We try not to tell each other who we’re watching, but most of the time it’s quite obvious.”

Two young men behind us, one apparently preoccupied with his iPad, another saucer-eyed from  texting on his smart phone, caught his attention. “You can tell the agents,” he said, with a barely decipherable flick of the head. “They’re the ones who are on the phone all the time. They don’t watch the game at all. They’re here to work the rooms, to see and be seen.”

Jamie was initially impressed by John Swift, Chelsea’s elegant, straight-backed central midfield player. He made good angles, picked clever passes and was sufficiently technically adept  to be comfortable on the ball in tight areas. “A young one, a Gary Gardner type, Dad,” he said, referring to another midfield tyro, at Aston Villa. “Good footballer but an academy player,” came the reply. “Ask yourself the question: will he keep me in a job if I take him for a Championship or League One club? No. He’ll get you the sack.” Swift duly faded into insignificance.

The older man was examining body shape, the probabilities of genetic inheritance. “Look at Elliot Lee — Rob’s son. Chip off the old block, isn’t he? But a big arse. Not an athlete.” Few words were wasted and judgements were harsh: “Look at the goalkeepers. One’s a great size, but a coward. The other’s a great shot stopper but too small. Their mistakes prey on your mind.”

I was struck by Chelsea’s Todd Kane, a full-back in the modern idiom. He was strong, adventurous and aggressive and his delivery from wide areas caused problems. “He could have a career, him. My first thought is Brentford. He’s a Nicky Shorey type. He does what it says on the tin. Problem is his size — can’t see him defending at the far post at top level. He’ll be a proper pro, though.”

There was logic to Johnson’s caution, borne out of 27 years’ experience. “The window of opportunity isn’t open for long, and they’re out there flicking and farting around. It is a cruel world. They have only one chance to impress. There are too many games, too many players, to spend long on them. When you are working for Liverpool, a lot of the time you are crossing names off your list.”

Islam Feruz, a small support striker blessed with extreme pace, earned a reprieve by scoring a goal of sublime quality. He picked up the ball midway in the opposition half, surged past four challenges into the heart of the penalty area and dinked a shot over the advancing West Ham goalkeeper. “Blimey!” exclaimed Johnson. “Didn’t see that coming. That was Diego Maradona.”

Feruz was a child of his times. The only son in a family of Somalian refugees that relocated to Glasgow after fleeing to London from Tanzania, he was saved from deportation at the age of 12 by the advocacy of Celtic’s youth coach, the late Tommy Burns. He made his first-team debut at the age of 14, in a memorial match for Burns, a man of immense integrity in a game of shallow expedience.

Within 18 months, Chelsea had taken advantage of a loophole in the system to spirit him south. Conscious of competition from Manchester City, they installed the family in a flat near their Cobham training ground. The boy was reportedly being paid £10,000 a month and had his own website, which proclaimed, “Islam Feruz will be famous.” Wotte, who might have been expected to be a little more circumspect, promptly compared him to Romario.

With five minutes remaining and the scores level at 2–2, most of the scouts had seen enough. Only Anderson stayed to witness Chelsea’s win, on penalties, after the game had ended 3–3 after extra time. “What have I got to go home to?” he said with a mischievous smile. “I’ll be here helping them sweep up.” He would make himself busy, networking with agents, parents and coaches. He could talk for England, but, crucially, he was a good listener. He also worked a room, like a bee collecting pollen.

Johnson scurried to his car in the company of Steve McCall, Ipswich’s chief scout. His small talk — “that Nat Chalobah, he’s got Chelsea-itis. Got all the tools, but a laid-back Larry” — was tellingly deceptive. It was several months before he revealed he had logged the defender’s speed of thought, intelligent movement and ease on the ball. He recommended him as the holding midfield player Liverpool were seeking.

Johnson had been taken to Anfield by Damien Comolli, with whom he worked as chief scout for Tottenham. He recruited Gareth Bale from Southampton, but was a victim of regime change under Harry Redknapp. It was the first time he had been “moved on” since he began scouting, as a self-confessed “football fanatic”, in 1985. The following year, on Good Friday, he recommended Norwich City sign an 11-year-old midfield player he had spotted playing for Ridgeway Rovers in the Canaries Cup.

David Beckham was duly invited for trials at Carrow Road, but joined Tottenham’s School of Excellence before Manchester United and corporate canonisation beckoned. Since Leyton Orient, the boy’s local club, were also unfulfilled suitors at that time, there was an appropriate symmetry to Johnson’s next tutorial, an Under-19 international between England and the Czech Republic at Brisbane Road.

Johnson parked in the terraced streets surrounding the ground and popped into a newsagent to buy a local paper. “Everyone canes me for it, even Damien,” he said, with a self-deprecating chuckle. “But I always buy one for the titbits. You never know what you’ll find out.” He returned to his car and studied the Czech squad on his iPad for an hour before he entered the Olympic Suite, 35 minutes from kick-off.

The scouts were devouring ham and mustard sandwiches, with the obligatory chips, as they retold tall tales of ducking and diving. My favourite revealed the ingenuity and duplicity of one solid citizen who monitored youth football for Portsmouth, did first-team match assessments for Newcastle United and covered non-League football for Wolverhampton Wanderers. All three clubs were ignorant of his involvement with the others.

Johnson preferred the company of the Tottenham coach Clive Allen. “He was good to me at Spurs,” he explained. “He kept phoning to see how I was after they outed me. You don’t forget things like that.” They discussed the striker Harry Kane. He was excelling on loan at Millwall, whose manager Kenny Jackett had worked with Johnson at Watford and QPR. The one doubt, about his pace at the highest level, was neutralised by memories of Teddy Sheringham, a player whose game intelligence compensated for a slight lack of speed.

“I love this place,” Johnson reflected, as we looked out on to a museum piece, the deserted old main stand. “The fans are the funniest around. I was here once when they started chanting, ‘We can see you washing up,’ at the inhabitants of the flats in the corner. It’s a proper club, with some great people.” Memories of the old John Chiedozie tea-bar and the fabled eccentricity of the former manager John Sitton, stirred a smile.

Stuart Pearce, who was to make a cameo appearance as England’s caretaker manager against Holland at Wembley the following night, nodded as he bustled past with his retinue. Johnson had talked football with him the previous week, in Jackett’s office at the Den, but his perspective shifted suddenly, as the teams and his sheet of A4 came out. “We know the England boys so well,” he said, “this is my chance to look at the Czechs. They’ve beaten some top sides.” He quickly concentrated on the goalkeeper Lukas Zima, a tall, slightly built fashion victim in tangerine kit and predominantly orange boots. All that remained was for him to address the error of my ways.

“Don’t look at the game, look at the man,” Johnson instructed. “You are following play just like the coaches who come out with me. Scouts study their man. I blank the other players out, although if the ball is at the other end of the pitch I’ll watch out of the corner of my eye, just in case I get asked for an opinion. You cannot follow the ball in this job.”

I felt self-conscious at first, but it was simple and startlingly effective. Watching Zima so intensely had a strange intimacy. He morphed from an unknown name on a team sheet to a definable human being. His mannerisms became familiar and the complexities of his character emerged. He unwittingly evoked sympathy and understanding. Johnson was enthused: “Look at him. Good concentration. Keeps communicating. Attention to detail. He’s alert, thinking. A good size. I like him. I like his bravery. He’s been out at people’s feet a couple of times. I know he punches, but all the foreign ones do, especially at youth level.”

As he spoke, England broke quickly from an ill-judged Czech attack. Harry Kane chested the ball down inside his own half, fed Ross Barkley on the right and sprinted to receive a return ball on the edge of the penalty area before scoring with a low shot into the corner. Johnson spoke with proprietorial authority and concern: “There was nothing the keeper could do. He’s got nothing in front of him. That Celtic boy, the left-sided centre-half, is struggling for his life. It’s not the keeper’s fault they keep getting caught out by balls over the top.”

The consensus at half-time was that the Czechs were “crap”. Lil Fuccillo, Luton’s technical director, was telling anyone who would listen, “It’s Barça this, Barça that. I’m getting sick of it. We’ve got to play to our strengths in this country.” Rather than enlist in the Bedfordshire branch of the Flat Earth Society, Johnson gravitated towards Dave Holden, the veteran Arsenal scout, who was instrumental in the recruitment of Alex Oxlade-Chamberlain.

There was an easy rapport between the pair — “not a lot out there, is there?” – and each knew not to read too much into the small talk. They strayed beyond the immediacy of the match to engage in a discussion about the best culture to instil in young players. Holden, a former school teacher who had retained a broad Geordie accent, insisted, “Players recognise the best players. They’ll have a young one in the group if they see something in him. Players also challenge coaches. They know if the coach isn’t good enough, the outstanding player in the group will regress to the level of the others.”

The gossip was global. Jaap Stam was making a positive impact as Manchester United’s Brazilian scout and was pushing hard for Dedé, the Vasco da Gama central defender. Barcelona had taken out first options on 39 young players at Boca Juniors. Glenn Roeder, the former Newcastle manager, had been added to Aston Villa’s scouting staff. By the time the litany of opportunity was complete, the second half was underway.

We were joined by Anderson, Liverpool’s youth scout, who exclaimed, “There’s our boy again!” when Todd Kane delivered a cross on the run that enabled Chelsea colleague Patrick Bamford to extend England’s lead with a stooping header. Johnson’s other duty involved monitoring winger Nathan Redmond, who was introduced as a substitute midway through the half. “Watch him,” he counselled. “He won’t really be trying. He’ll be more worried about playing for Birmingham on Saturday and in the Cup replay against Chelsea next week.” Sure enough, Redmond was measured, to the point of indolence. He contributed little, apart from a languid flick and brief bursts of pace in insignificant areas. Another early departure was entirely excusable.

Again, Johnson had rationed his intelligence. No one had detected his interest in the goalkeeper. “The trick in this game is never to let people know what you are thinking or how you are working. There are plenty out there happy to feed off your knowledge.” His report on Zima, who had been signed by Genoa from Slavia Prague, was on the Anfield system by 2 am, with a recommendation that he be watched by Liverpool’s Italian scout.

It was one of 200 such reports submitted that week. Whether it would receive the attention it deserved was another matter entirely.

This is an edited extract from Mike Calvin’s book The Nowhere Men, published by Century in September.