Skip to content
TalentTECH home

Implementation guide

3 AI tools for scouting and player evaluation

This chapter shows how to set up an AI-supported scouting tool, read its outputs and use them alongside your scouts' own judgement. It applies to tools that rank or score players from data, and to the data you feed them.

Chapter 3 of 9

Also on Zenodo, with the other three videos.

3.1 Overview: what AI tools can and cannot do

An AI scouting tool learns patterns from past data, for example which players an academy selected in earlier seasons, and applies them to new players. Its output is usually a score, a ranking or a shortlist. For the background, see Module 1 of D2.1 and D2.2, and Cases 1, 4 and 5 in D2.3.

Two limits matter most. First, even a well-built model predicts only moderately well: in a professional German academy, the best model reached an AUC (area under the curve) of 0.69, where 0.5 is chance and 1.0 is perfect (Altmann et al., 2024). Second, most published talent-identification models have not been tested on outside data, and many have a high risk of bias (Tang et al., 2026). A tool trained on past selections can learn the biases in those selections.

3.2 Technical setup: step by step

Once, before the first season

  1. Ask the supplier three questions in writing.

    Why: How was the model validated? On how many players? On data from another club or season? Be cautious if a supplier reports very high accuracy and cannot answer these (Tang et al., 2026).

    Check: you have the answers on file, in writing.

  2. Check what the tool actually does.

    Why: If it uses face recognition, emotion recognition or other biometric functions, stricter rules apply under the GDPR and the AI Act. Ask your DPO or data protection contact before you buy (section 2.5).

  3. Give the staff who will use it a short AI-literacy briefing (about 30 minutes).

    Why: The AI Act asks deployers to support AI literacy (Regulation (EU) 2024/1689, Art. 4). Cover what the score is, what it is not, and who decides.

  4. Set up player records with a code, date of birth and, where you can, a maturity estimate.

    Why: Without birth date and maturity data, you cannot check the tool for relative age and maturation bias (section 3.3). Use codes instead of names in the tool where possible (section 2.5).

  5. Write a one-line definition for every data field you enter.

    Why: A GPS sprint speed and a stopwatch sprint speed are different numbers, so the tool cannot compare them.

  6. Record the tool version and the date in your log.

    Why: When the supplier updates the model, scores can shift. A dated note explains it.

Every scouting cycle

  1. Enter or import the new assessments.

    Why: Enter data soon after each observation, the same way every time.

  2. Check data completeness before you look at any score.

    Why: A high score on a player with half the fields empty is not comparable with a high score on a complete record (Figure 3.1, callout 2).

  3. Generate the shortlist and send it to a named scout for review.

    Why: The tool only screens. A person reviews every flagged case and takes the decision (Tang et al., 2026).

    Check: each player on the shortlist has a recorded human decision (Figure 3.1, callout 4).

  4. Once a season, compare the shortlist by birth quarter and maturity status.

    Why: See section 3.3. If early-born or early-maturing players dominate, the tool may be reproducing bias.

Illustrative AI scouting tool: shortlist screen A schematic software window showing a ranked shortlist of players with an AI score, a data completeness column, the player's birth quarter and a human review status column. Callouts: 1 score is one input, 2 missing data warning, 3 birth quarter shown, 4 human decision recorded. Scouting assistant: U15 shortlist FILTERS Season 2026/27 Age: U15 Position: all PlayerAI scoreData completeBirth quarterScout review Player A-010.84100%Q1Reviewed: invite to trial Player A-070.7861% ⚠Q4Pending Player A-120.7195%Q1Reviewed: monitor Player A-190.6288%Q3Pending MODEL INFO Trained on: this academy, seasons 2019-2025; Tested on other clubs: no; Last check by birth quarter: 2026-09 1 2 3 4 1 The score is one input, not a decision; 2 Low data completeness lowers trust in the score; 3 Check scores by birth quarter 4 A named scout records the human decision for every flagged player
Figure 3.1 A shortlist screen in a typical AI scouting tool. Illustrative interface, not a real product. The columns to look for are data completeness, birth quarter and the human review status.

3.3 Interpreting AI outputs

  1. A score is one input. Combine it with live observation and the scout's judgement (Altmann et al., 2024; Tang et al., 2026). The D2.3 practitioner cases (Cases 4, 5 and 7) reached the same conclusion: the data narrows the choice and people decide.
  2. Missing data is a hidden bias. Before trusting a ranking, check whose data is absent. D2.3 (Cases 4 and 7) warns that data supplied by interested parties, such as agents, can distort a shortlist.
  3. Check for the relative age effect. Players born early in the selection year are more likely to be selected, especially in adolescence; a 2023 meta-analysis of 90 football studies confirms the effect is strongest in adolescence and at higher levels (Cobley et al., 2009; Bozděch et al., 2023). An AI tool trained on past selections will learn this.
  4. Maturation is a separate bias. Relative age and biological maturity are two different effects, and a model can learn both. Check outputs by birth quarter and by maturity status (Hill et al., 2020). Approaches such as bio-banding exist, but evidence on their effects is still developing.
  5. Ask about calibration. A score of 0.8 should mean that about 8 in 10 such players were selected in the test data. If the supplier cannot explain how the probabilities were calibrated, treat the number as a rank, not a probability (Tang et al., 2026).

3.4 Using AI tools in the scouting and training routine

  • Use the tool to build shortlists. Let it narrow 60 names to 15, then watch those 15 live.
  • Fix a review slot. One weekly shortlist review (about 20 minutes) where the named scout records a decision for every flagged player. Table 7.1 places it on MD-2 (two days before the match; see section 7.1), separate from the planning meeting.
  • Feed back to coaches in one line per player. "Flagged by the tool; watched twice; invite to trial" is enough. D2.3 (Cases 5, 6 and 8) found that players and coaches use short outputs and ignore long ones.
  • Keep the structured template running alongside. Your own consistent assessments are the data the tool needs, and your fallback if you stop using it.
  • Tell players and parents what the tool is. In simple words: it suggests who to watch, and the scout decides (section 2.5).

3.5 Troubleshooting

Table 3.1 Common AI tool problems and what to do
ProblemLikely causeWhat to do
Two scouts disagree with the tool on most playersThe tool was trained on a different level, country or age groupAsk the supplier what the model was trained on (step 1). Use the tool only for the group it was built for.
Shortlist is almost all early-born playersRelative age effect in the training dataRun the birth-quarter check (step 10). Review late-born players manually. Raise it with the supplier.
Scores changed for everyone overnightModel or software updateCheck your log (step 6). Do not compare scores across the update.
A player with little data has a high scoreMissing fields filled by default valuesCheck completeness (step 8). Do not act on scores from incomplete records.
Coaches ignore the shortlistOutput too long, or arrives too lateSend one line per player, before the next training session (section 3.4).
Parents ask what the "score" meansNo explanation given at the startUse the one-sentence explanation in section 3.4 and the information points in Annex C.

3.6 Quick-start checklist by role

Coach

  • Read the one-line scout note for each flagged player.
  • Give the scout your own observation after each session.
  • Never cut or promote a player on a score alone.

Scout

  • Record a decision for every flagged player, every week.
  • Watch late-born and late-maturing players on purpose.
  • Keep the structured template going.

Performance staff

  • Supply birth dates and maturity estimates to the data owner.
  • Define each physical field once, in writing.
  • Flag data that comes from a different system.

Academy manager

  • Get the supplier's validation answers in writing.
  • Confirm the DPIA question with the DPO (Annex C).
  • Run the 30-minute AI-literacy briefing for staff.

The implementation guide is published under a CC BY-SA 4.0 licence.