Metadata-Version: 2.4
Name: ScandEval
Version: 16.5.0
Summary: The robust European language model benchmark.
Project-URL: Repository, https://github.com/EuroEval/EuroEval
Project-URL: Issues, https://github.com/EuroEval/EuroEval/issues
Author-email: Dan Saattrup Smart <dan.smart@alexandra.dk>
Maintainer-email: Dan Saattrup Smart <dan.smart@alexandra.dk>
License: MIT License
        
        Copyright (c) 2022-2025 Dan Saattrup Smart
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Requires-Python: <4.0,>=3.11
Requires-Dist: accelerate>=1.9.0
Requires-Dist: bert-score>=0.3.13
Requires-Dist: click>=8.1.3
Requires-Dist: cloudpickle>=3.1.1
Requires-Dist: datasets>=3.5.0
Requires-Dist: demjson3>=3.0.6
Requires-Dist: evaluate>=0.4.1
Requires-Dist: huggingface-hub>=0.30.1
Requires-Dist: levenshtein>=0.24.0
Requires-Dist: litellm>=1.75.6
Requires-Dist: more-itertools>=10.5.0
Requires-Dist: numpy>=2.0.0
Requires-Dist: ollama>=0.5.1
Requires-Dist: pandas>=2.2.0
Requires-Dist: peft>=0.15.0
Requires-Dist: protobuf>=2.0.0
Requires-Dist: pydantic>=2.6.0
Requires-Dist: pyinfer>=0.0.3
Requires-Dist: python-dotenv>=1.0.1
Requires-Dist: rouge-score>=0.1.2
Requires-Dist: sacremoses>=0.1.1
Requires-Dist: scikit-learn==1.6.1
Requires-Dist: sentencepiece>=0.1.96
Requires-Dist: seqeval>=1.2.2
Requires-Dist: setuptools>=75.8.2
Requires-Dist: tenacity>=9.0.0
Requires-Dist: termcolor>=2.0.0
Requires-Dist: torch>=2.6.0
Requires-Dist: transformers[mistral-common]>=4.56.0
Provides-Extra: all
Requires-Dist: bitsandbytes>=0.43.1; (platform_system == 'Linux') and extra == 'all'
Requires-Dist: fbgemm-gpu>=1.0.0; (platform_system == 'Linux') and extra == 'all'
Requires-Dist: timm>=1.0.19; extra == 'all'
Requires-Dist: vllm[flashinfer]>=0.11.0; (platform_system == 'Linux') and extra == 'all'
Provides-Extra: generative
Requires-Dist: bitsandbytes>=0.43.1; (platform_system == 'Linux') and extra == 'generative'
Requires-Dist: fbgemm-gpu>=1.0.0; (platform_system == 'Linux') and extra == 'generative'
Requires-Dist: timm>=1.0.19; extra == 'generative'
Requires-Dist: vllm[flashinfer]>=0.11.0; (platform_system == 'Linux') and extra == 'generative'
Description-Content-Type: text/markdown

<!-- This disables the requirement that the first line is a top-level heading -->
<!-- markdownlint-configure-file { "MD041": false } -->

<div align='center'>
<img
    src="https://raw.githubusercontent.com/EuroEval/EuroEval/main/gfx/euroeval.png"
    height="500"
    width="372"
>
</div>

### The robust European language model benchmark

(formerly known as ScandEval)

______________________________________________________________________
[![Documentation](https://img.shields.io/badge/docs-passing-green)](https://euroeval.com)
[![PyPI Status](https://badge.fury.io/py/euroeval.svg)](https://pypi.org/project/euroeval/)
[![First paper](https://img.shields.io/badge/arXiv-2304.00906-b31b1b.svg)](https://arxiv.org/abs/2304.00906)
[![Second paper](https://img.shields.io/badge/arXiv-2406.13469-b31b1b.svg)](https://arxiv.org/abs/2406.13469)
[![License](https://img.shields.io/github/license/EuroEval/EuroEval)](https://github.com/EuroEval/EuroEval/blob/main/LICENSE)
[![LastCommit](https://img.shields.io/github/last-commit/EuroEval/EuroEval)](https://github.com/EuroEval/EuroEval/commits/main)
[![Code Coverage](https://img.shields.io/badge/Coverage-76%25-yellowgreen.svg)](https://github.com/EuroEval/EuroEval/tree/main/tests)
[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.0-4baaaa.svg)](https://github.com/EuroEval/EuroEval/blob/main/CODE_OF_CONDUCT.md)

## Maintainer

- Dan Saattrup Smart ([@saattrupdan](https://github.com/saattrupdan), <dan.smart@alexandra.dk>)

## Installation

To install the package simply write the following command in your favorite terminal:

```bash
pip install euroeval[all]
```

This will install the EuroEval package with all extras. You can also install the
minimal version by leaving out the `[all]`, in which case the package will let you know
when an evaluation requires a certain extra dependency, and how you install it.

## Quickstart

### Benchmarking from the command line

The easiest way to benchmark pretrained models is via the command line interface. After
having installed the package, you can benchmark your favorite model like so:

```bash
euroeval --model <model-id>
```

Here `model` is the HuggingFace model ID, which can be found on the [HuggingFace
Hub](https://huggingface.co/models). By default this will benchmark the model on all
the tasks available. If you want to benchmark on a particular task, then use the
`--task` argument:

```bash
euroeval --model <model-id> --task sentiment-classification
```

We can also narrow down which languages we would like to benchmark on. This can be done
by setting the `--language` argument. Here we thus benchmark the model on the Danish
sentiment classification task:

```bash
euroeval --model <model-id> --task sentiment-classification --language da
```

Multiple models, datasets and/or languages can be specified by just attaching multiple
arguments. Here is an example with two models:

```bash
euroeval --model <model-id1> --model <model-id2>
```

The specific model version/revision to use can also be added after the suffix '@':

```bash
euroeval --model <model-id>@<commit>
```

This can be a branch name, a tag name, or a commit id. It defaults to 'main' for latest.

See all the arguments and options available for the `euroeval` command by typing

```bash
euroeval --help
```

### Benchmarking from a script

In a script, the syntax is similar to the command line interface. You simply initialise
an object of the `Benchmarker` class, and call this benchmark object with your favorite
model:

```python
>>> from euroeval import Benchmarker
>>> benchmarker = Benchmarker()
>>> benchmarker.benchmark(model="<model-id>")
```

To benchmark on a specific task and/or language, you simply specify the `task` or
`language` arguments, shown here with same example as above:

```python
>>> benchmarker.benchmark(
...     model="<model-id>",
...     task="sentiment-classification",
...     language="da",
... )
```

If you want to benchmark a subset of all the models on the Hugging Face Hub, you can
simply leave out the `model` argument. In this example, we're benchmarking all Danish
models on the Danish sentiment classification task:

```python
>>> benchmarker.benchmark(task="sentiment-classification", language="da")
```

### Benchmarking from Docker

A Dockerfile is provided in the repo, which can be downloaded and run, without needing
to clone the repo and installing from source. This can be fetched programmatically by
running the following:

```bash
wget https://raw.githubusercontent.com/EuroEval/EuroEval/main/Dockerfile.cuda
```

Next, to be able to build the Docker image, first ensure that the NVIDIA Container
Toolkit is
[installed](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation)
and
[configured](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuring-docker).
Ensure that the the CUDA version stated at the top of the Dockerfile matches the CUDA
version installed (which you can check using `nvidia-smi`). After that, we build the
image as follows:

```bash
docker build --pull -t euroeval -f Dockerfile.cuda .
```

With the Docker image built, we can now evaluate any model as follows:

```bash
docker run -e args="<euroeval-arguments>" --gpus 1 --name euroeval --rm euroeval
```

Here `<euroeval-arguments>` consists of the arguments added to the `euroeval` CLI
argument. This could for instance be `--model <model-id> --task
sentiment-classification`.

## Benchmarking custom inference APIs

If the model you want to benchmark is hosted by a custom inference provider, such as a
[vLLM server](https://docs.vllm.ai/en/stable/), then this is also supported in EuroEval.
When benchmarking, you simply have to set the `--api-base` argument (`api_base` when
using the `Benchmarker` API) to the URL of the inference API, and optionally the
`--api-key` argument (`api_key`) to the API key, if authentication is required.

When benchmarking models hosted on a custom inference API, the model ID
(`--model`/`model`) should be the model name as registered on the inference server,
potentially with a required prefix, depending on the type of inference server used. For
instance, if the model is hosted on a vLLM server, the model ID should be prefixed with
`hosted_vllm/`, and if the model is hosted on an Ollama server, the model ID should be
prefixed with `ollama_chat/`. See the full list of possible inference providers as well
as their corresponding prefixes in the [LiteLLM
documentation](https://docs.litellm.ai/docs/providers/), as EuroEval uses LiteLLM to
handle evaluation of inference APIs in general.

## Benchmarking in an offline environment

If you need to benchmark in an offline environment, you need to download the models,
datasets and metrics beforehand. This can be done by adding the `--download-only`
argument, from the command line, or the `download_only` argument, if benchmarking from a
script. For example to download the model you want and all of the Danish sentiment
classification datasets:

```bash
euroeval --model <model-id> --task sentiment-classification --language da --download-only
```

Or from a script:

```python
>>> benchmarker.benchmark(
... model="<model-id>",
... task="sentiment-classification",
... language="da",
... download_only=True,
... )
```

Please note: Offline benchmarking of adapter models is not currently supported, meaning
that we still require an internet connection during the evaluation of these. If offline
support of adapters is important to you, please consider [opening an
issue](https://github.com/EuroEval/EuroEval/issues).

## Benchmarking custom datasets

If you want to benchmark models on your own custom dataset, this is also possible.
First, you need to set up your dataset to be compatible with EuroEval. This means
splitting up your dataset in a training, validation and test split, and ensuring that
the column names are correct. We use `text` as the column name for the input text, and
the output column name depends on the type of task:

- **Text or multiple-choice classification**: `label`
- **Token classification**: `labels`
- **Reading comprehension**: `answers`
- **Free-form text generation**: `target_text`

Text and multiple-choice classification tasks are by far the most common. Next, you
store your three dataset splits as three different CSV files with the desired two
columns. Finally, you create a file called `custom_datasets.py` script in which you
define the associated `DatasetConfig` objects for your dataset. Here is an example of a
simple text classification dataset with two classes:

```python
from euroeval import DatasetConfig, TEXT_CLASSIFICATION
from euroeval.languages import ENGLISH

MY_CONFIG = DatasetConfig(
    name="my-dataset",
    source=dict(train="train.csv", val="val.csv", test="test.csv"),
    task=TEXT_CLASSIFICATION,
    languages=[ENGLISH],
    _labels=["positive", "negative"],
)
```

You can then benchmark your custom dataset by simply running

```bash
euroeval --dataset my-dataset --model <model-id>
```

You can also run the benchmark from a Python script, by simply providing your custom
dataset configuration directly into the `benchmark` method:

```python
from euroeval import Benchmarker

benchmarker = Benchmarker()
benchmarker.benchmark(model="<model-id>", dataset=MY_CONFIG)
```

We have included three convenience tasks to make it easier to set up custom datasets:

- `TEXT_CLASSIFICATION`, which is used for text classification tasks. This requires you
  to set the `_labels` argument in the `DatasetConfig`, and requires the columns `text`
  and `label` to be present in the dataset.
- `MULTIPLE_CHOICE`, which is used for multiple-choice classification tasks. This
  also requires you to set the `_labels` argument in the `DatasetConfig`. Note that for
  multiple choice tasks, you need to set up your `text` column to also list all the
  choices, and all the samples should have the same number of choices. This requires the
  columns `text` and `label` to be present in the dataset.
- `TOKEN_CLASSIFICATION`, which is used when classifying individual tokens in a text.
  This also require you to set the `_labels` argument in the `DatasetConfig`. This
  requires the columns `tokens` and `labels` to be present in the dataset, where
  `tokens` is a list of tokens/words in the text, and `labels` is a list of the
  corresponding labels for each token (so the two lists have the same length).

On top of these three convenience tasks, there are of course also the tasks that we use
in the official benchmark, which you can use if you want to use one of these tasks with
your own bespoke dataset:

- `LA`, for linguistic acceptability datasets.
- `NER`, for named entity recognition datasets with the standard BIO tagging scheme.
- `RC`, for reading comprehension datasets in the SQuAD format.
- `SENT`, for sentiment classification datasets.
- `SUMM`, for text summarisation datasets.
- `KNOW`, for multiple-choice knowledge datasets (e.g., MMLU).
- `MCRC`, for multiple-choice reading comprehension datasets (e.g., Belebele).
- `COMMON_SENSE`, for multiple-choice common-sense reasoning datasets (e.g., HellaSwag).

These can all be imported from `euroeval.tasks` module.

### Creating your own custom task

You are of course also free to define your own task from scratch, which allows you to
customise the prompts used when evaluating generative models, for instance. Here is an
example of a custom free-form text generation task, where the goal for the model is to
generate a SQL query based on a natural language input:

```python
from euroeval import DatasetConfig
from euroeval.data_models import Task, PromptConfig
from euroeval.enums import TaskGroup, ModelType
from euroeval.languages import ENGLISH
from euroeval.metrics import rouge_l_metric

sql_generation_task = Task(
    name="sql-generation",
    task_group=TaskGroup.TEXT_TO_TEXT,
    template_dict={
        ENGLISH: PromptConfig(
            default_prompt_prefix="The following are natural language texts and their "
            "corresponding SQL queries.",
            default_prompt_template="Natural language query: {text}\nSQL query: "
            "{target_text}",
            default_instruction_prompt="Generate the SQL query for the following "
            "natural language query:\n{text!r}",
            default_prompt_label_mapping=dict(),
        ),
    },
    metrics=[rouge_l_metric],
    default_num_few_shot_examples=3,
    default_max_generated_tokens=256,
    default_allowed_model_types=[ModelType.GENERATIVE],
)

MY_SQL_DATASET = DatasetConfig(
    name="my-sql-dataset",
    source=dict(train="train.csv", val="val.csv", test="test.csv"),
    task=sql_generation_task,
    languages=[ENGLISH],
)
```

Again, with this you can benchmark your custom dataset by simply running

```bash
euroeval --dataset my-sql-dataset --model <model-id>
```

## Reproducing the evaluation datasets

All datasets used in this project are generated using the scripts located in the
[src/scripts](src/scripts) folder. To reproduce a dataset, run the corresponding script
with the following command

```bash
uv run src/scripts/<name-of-script>.py
```

Replace <name-of-script> with the specific script you wish to execute, e.g.,

```bash
uv run src/scripts/create_allocine.py
```

## Contributors :pray:

A huge thank you to all the contributors who have helped make this project a success!

<a href="https://github.com/peter-sk">
    <img
        src="https://avatars.githubusercontent.com/u/6168908"
        width=50
        alt="Contributor avatar for peter-sk"
    />
</a>
<a href="https://github.com/AJDERS">
    <img
        src="https://avatars.githubusercontent.com/u/38854604"
        width=50
        alt="Contributor avatar for AJDERS"
    />
</a>
<a href="https://github.com/oliverkinch">
    <img
        src="https://avatars.githubusercontent.com/u/71556498"
        width=50
        alt="Contributor avatar for oliverkinch"
    />
</a>
<a href="https://github.com/versae">
    <img
        src="https://avatars.githubusercontent.com/u/173537"
        width=50
        alt="Contributor avatar for versae"
    />
</a>
<a href="https://github.com/KennethEnevoldsen">
    <img
        src="https://avatars.githubusercontent.com/u/23721977"
        width=50
        alt="Contributor avatar for KennethEnevoldsen"
    />
</a>
<a href="https://github.com/viggo-gascou">
    <img
        src="https://avatars.githubusercontent.com/u/94069687"
        width=50
        alt="Contributor avatar for viggo-gascou"
    />
</a>
<a href="https://github.com/mathiasesn">
    <img
        src="https://avatars.githubusercontent.com/u/27091759"
        width=50
        alt="Contributor avatar for mathiasesn"
    />
</a>
<a href="https://github.com/Alkarex">
    <img
        src="https://avatars.githubusercontent.com/u/1008324"
        width=50
        alt="Contributor avatar for Alkarex"
    />
</a>
<a href="https://github.com/marksverdhei">
    <img
        src="https://avatars.githubusercontent.com/u/46672778"
        width=50
        alt="Contributor avatar for marksverdhei"
    />
</a>
<a href="https://github.com/Mikeriess">
    <img
        src="https://avatars.githubusercontent.com/u/19728563"
        width=50
        alt="Contributor avatar for Mikeriess"
    />
</a>
<a href="https://github.com/ThomasKluiters">
    <img
        src="https://avatars.githubusercontent.com/u/8137941"
        width=50
        alt="Contributor avatar for ThomasKluiters"
    />
</a>
<a href="https://github.com/BramVanroy">
    <img
        src="https://avatars.githubusercontent.com/u/2779410"
        width=50
        alt="Contributor avatar for BramVanroy"
    />
</a>
<a href="https://github.com/peregilk">
    <img
        src="https://avatars.githubusercontent.com/u/9079808"
        width=50
        alt="Contributor avatar for peregilk"
    />
</a>
<a href="https://github.com/Rijgersberg">
    <img
        src="https://avatars.githubusercontent.com/u/8604946"
        width=50
        alt="Contributor avatar for Rijgersberg"
    />
</a>
<a href="https://github.com/duarteocarmo">
    <img
        src="https://avatars.githubusercontent.com/u/26342344"
        width=50
        alt="Contributor avatar for duarteocarmo"
    />
</a>
<a href="https://github.com/slowwavesleep">
    <img
        src="https://avatars.githubusercontent.com/u/44175589"
        width=50
        alt="Contributor avatar for slowwavesleep"
    />
</a>
<a href="https://github.com/mrkowalski">
    <img
        src="https://avatars.githubusercontent.com/u/6357044"
        width=50
        alt="Contributor avatar for mrkowalski"
    />
</a>

### Contribute to EuroEval

We welcome contributions to EuroEval! Whether you're fixing bugs, adding features, or
contributing new datasets, your help makes this project better for everyone.

- **General contributions**: Check out our [contribution guidelines](CONTRIBUTING.md)
  for information on how to get started.
- **Adding datasets**: If you're interested in adding a new dataset to EuroEval, we have
  a [dedicated guide](NEW_DATASET_GUIDE.md) with step-by-step instructions.

### Special thanks

- Thanks to [Google](https://google.com/) for sponsoring Gemini credits as part of their
  [Google Cloud for Researchers Program](https://cloud.google.com/edu/researchers).
- Thanks [@Mikeriess](https://github.com/Mikeriess) for evaluating many of the larger
  models on the leaderboards.
- Thanks to [OpenAI](https://openai.com/) for sponsoring OpenAI credits as part of their
  [Researcher Access Program](https://openai.com/form/researcher-access-program/).
- Thanks to [UWV](https://www.uwv.nl/) and [KU
  Leuven](https://www.arts.kuleuven.be/ling/ccl) for sponsoring the Azure OpenAI
  credits used to evaluate GPT-4-turbo in Dutch.
- Thanks to [Miðeind](https://mideind.is/en) for sponsoring the OpenAI
  credits used to evaluate GPT-4-turbo in Icelandic and Faroese.
- Thanks to [CHC](https://chc.au.dk/) for sponsoring the OpenAI credits used to
  evaluate GPT-4-turbo in German.

## Citing EuroEval

If you want to cite the framework then feel free to use this:

```bibtex
@article{smart2024encoder,
  title={Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks},
  author={Smart, Dan Saattrup and Enevoldsen, Kenneth and Schneider-Kamp, Peter},
  journal={arXiv preprint arXiv:2406.13469},
  year={2024}
}
@inproceedings{smart2023scandeval,
  author = {Smart, Dan Saattrup},
  booktitle = {Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)},
  month = may,
  pages = {185--201},
  title = {{ScandEval: A Benchmark for Scandinavian Natural Language Processing}},
  year = {2023}
}
```
