Exercise NLP Evaluation Metrics (BLEU, ROUGE, LLM-as-a-judge)

exercise_NLP_evaluation_metrics.md

NLP Evaluation Metrics

1. Intro to BLEU (machine translation)

Watch this video:

• "What is the BLEU metric?" (5 minutes): https://www.youtube.com/watch?v=M05L1DhFqcw

2. Implementing BLEU

Follow and understand this notebook:

• https://github.com/ironhack-ai-eng-june2026/demos_ai_eng/blob/main/15_Evaluation/1_bleu-evaluation.ipynb

Note:

• While you follow this notebook, make sure to adapt it for your own mother tongue (you'll need to update the variable reference_translations, and a parameter tgt_lang)

3. Advantages and limitations of BLEU

Discuss and/or Research:

• What are the advantages of BLEU?
• What are the limitations?
• Do some research about common alternatives (for each of them, consider its advantages and limitations)

4. Implement a solution with LLM-as-a-judge

• Implement a solution to evaluate the model facebook/nllb-200-distilled-600M, with LLM-as-a-judge

  • Note: if you want, you can do this vibe coding, but make sure to understand in detail how it works.

• Run the tests, and analyze the results.

Bonus: understanding ROUGE

• Watch this video: https://www.youtube.com/watch?v=TMshhnrEXlg
• Follow this notebook: https://github.com/ironhack-ai-eng-june2026/demos_ai_eng/blob/main/15_Evaluation/2_rouge-evaluation-untrained-vs-trained-llm.ipynb