Also on Zenodo, with the other three videos.
3.1 Overview: what AI tools can and cannot do
An AI scouting tool learns patterns from past data, for example which players an academy selected in earlier seasons, and applies them to new players. Its output is usually a score, a ranking or a shortlist. For the background, see Module 1 of D2.1 and D2.2, and Cases 1, 4 and 5 in D2.3.
Two limits matter most. First, even a well-built model predicts only moderately well: in a professional German academy, the best model reached an AUC (area under the curve) of 0.69, where 0.5 is chance and 1.0 is perfect (Altmann et al., 2024). Second, most published talent-identification models have not been tested on outside data, and many have a high risk of bias (Tang et al., 2026). A tool trained on past selections can learn the biases in those selections.
3.2 Technical setup: step by step
Once, before the first season
Ask the supplier three questions in writing.
Why: How was the model validated? On how many players? On data from another club or season? Be cautious if a supplier reports very high accuracy and cannot answer these (Tang et al., 2026).
Check: you have the answers on file, in writing.
Check what the tool actually does.
Why: If it uses face recognition, emotion recognition or other biometric functions, stricter rules apply under the GDPR and the AI Act. Ask your DPO or data protection contact before you buy (section 2.5).
Give the staff who will use it a short AI-literacy briefing (about 30 minutes).
Why: The AI Act asks deployers to support AI literacy (Regulation (EU) 2024/1689, Art. 4). Cover what the score is, what it is not, and who decides.
Set up player records with a code, date of birth and, where you can, a maturity estimate.
Why: Without birth date and maturity data, you cannot check the tool for relative age and maturation bias (section 3.3). Use codes instead of names in the tool where possible (section 2.5).
Write a one-line definition for every data field you enter.
Why: A GPS sprint speed and a stopwatch sprint speed are different numbers, so the tool cannot compare them.
Record the tool version and the date in your log.
Why: When the supplier updates the model, scores can shift. A dated note explains it.
Every scouting cycle
Enter or import the new assessments.
Why: Enter data soon after each observation, the same way every time.
Check data completeness before you look at any score.
Why: A high score on a player with half the fields empty is not comparable with a high score on a complete record (Figure 3.1, callout 2).
Generate the shortlist and send it to a named scout for review.
Why: The tool only screens. A person reviews every flagged case and takes the decision (Tang et al., 2026).
Check: each player on the shortlist has a recorded human decision (Figure 3.1, callout 4).
Once a season, compare the shortlist by birth quarter and maturity status.
Why: See section 3.3. If early-born or early-maturing players dominate, the tool may be reproducing bias.
3.3 Interpreting AI outputs
- A score is one input. Combine it with live observation and the scout's judgement (Altmann et al., 2024; Tang et al., 2026). The D2.3 practitioner cases (Cases 4, 5 and 7) reached the same conclusion: the data narrows the choice and people decide.
- Missing data is a hidden bias. Before trusting a ranking, check whose data is absent. D2.3 (Cases 4 and 7) warns that data supplied by interested parties, such as agents, can distort a shortlist.
- Check for the relative age effect. Players born early in the selection year are more likely to be selected, especially in adolescence; a 2023 meta-analysis of 90 football studies confirms the effect is strongest in adolescence and at higher levels (Cobley et al., 2009; Bozděch et al., 2023). An AI tool trained on past selections will learn this.
- Maturation is a separate bias. Relative age and biological maturity are two different effects, and a model can learn both. Check outputs by birth quarter and by maturity status (Hill et al., 2020). Approaches such as bio-banding exist, but evidence on their effects is still developing.
- Ask about calibration. A score of 0.8 should mean that about 8 in 10 such players were selected in the test data. If the supplier cannot explain how the probabilities were calibrated, treat the number as a rank, not a probability (Tang et al., 2026).
3.4 Using AI tools in the scouting and training routine
- Use the tool to build shortlists. Let it narrow 60 names to 15, then watch those 15 live.
- Fix a review slot. One weekly shortlist review (about 20 minutes) where the named scout records a decision for every flagged player. Table 7.1 places it on MD-2 (two days before the match; see section 7.1), separate from the planning meeting.
- Feed back to coaches in one line per player. "Flagged by the tool; watched twice; invite to trial" is enough. D2.3 (Cases 5, 6 and 8) found that players and coaches use short outputs and ignore long ones.
- Keep the structured template running alongside. Your own consistent assessments are the data the tool needs, and your fallback if you stop using it.
- Tell players and parents what the tool is. In simple words: it suggests who to watch, and the scout decides (section 2.5).
3.5 Troubleshooting
| Problem | Likely cause | What to do |
|---|---|---|
| Two scouts disagree with the tool on most players | The tool was trained on a different level, country or age group | Ask the supplier what the model was trained on (step 1). Use the tool only for the group it was built for. |
| Shortlist is almost all early-born players | Relative age effect in the training data | Run the birth-quarter check (step 10). Review late-born players manually. Raise it with the supplier. |
| Scores changed for everyone overnight | Model or software update | Check your log (step 6). Do not compare scores across the update. |
| A player with little data has a high score | Missing fields filled by default values | Check completeness (step 8). Do not act on scores from incomplete records. |
| Coaches ignore the shortlist | Output too long, or arrives too late | Send one line per player, before the next training session (section 3.4). |
| Parents ask what the "score" means | No explanation given at the start | Use the one-sentence explanation in section 3.4 and the information points in Annex C. |
3.6 Quick-start checklist by role
Coach
- Read the one-line scout note for each flagged player.
- Give the scout your own observation after each session.
- Never cut or promote a player on a score alone.
Scout
- Record a decision for every flagged player, every week.
- Watch late-born and late-maturing players on purpose.
- Keep the structured template going.
Performance staff
- Supply birth dates and maturity estimates to the data owner.
- Define each physical field once, in writing.
- Flag data that comes from a different system.
Academy manager
- Get the supplier's validation answers in writing.
- Confirm the DPIA question with the DPO (Annex C).
- Run the 30-minute AI-literacy briefing for staff.