Metrics is deprecated in 🌍 Datasets. To learn more about how to use metrics, take a look at the library 🌍 ! In addition to metrics, you can find more tools for evaluating models and datasets.
Metrics are important for evaluating a model’s predictions. In the tutorial, you learned how to compute a metric over an entire evaluation set. You have also seen how to load a metric.
This guide will show you how to:
Add predictions and references.
Compute metrics using different methods.
Write your own metric loading script.
Add predictions and references
When you want to add model predictions and references to a instance, you have two options:
adds a single prediction and reference.
adds a batch of predictions and references.
Use by passing it your model predictions, and the references the model predictions should be evaluated against:
Metrics accepts various input formats (Python lists, NumPy arrays, PyTorch tensors, etc.) and converts them to an appropriate format for storage and computation.
Inspect the different argument methods for computing the metric:
Copied
>>> print(metric.inputs_description)
Produces BLEU scores along with its sufficient statistics
from a source against one or more references.
Args:
predictions: The system stream (a sequence of segments).
references: A list of one or more reference streams (each a sequence of segments).
smooth_method: The smoothing method to use. (Default: 'exp').
smooth_value: The smoothing value. Only valid for 'floor' and 'add-k'. (Defaults: floor: 0.1, add-k: 1).
tokenize: Tokenization method to use for BLEU. If not provided, defaults to 'zh' for Chinese, 'ja-mecab' for Japanese and '13a' (mteval) otherwise.
lowercase: Lowercase the data. If True, enables case-insensitivity. (Default: False).
force: Insist that your tokenized input is actually detokenized.
...
Compute the metric with the floor method, and a different smooth_value:
If the files are stored locally, provide a dictionary of path(s) instead of URLs.
Metric._download_and_prepare() will take the URLs and download the metric files specified:
Copied
def _download_and_prepare(self, dl_manager):
# check that config name specifies a valid BLEURT model
if self.config_name == "default":
logger.warning(
"Using default BLEURT-Base checkpoint for sequence maximum length 128. "
"You can use a bigger model for better results with e.g.: datasets.load_metric('bleurt', 'bleurt-large-512')."
)
self.config_name = "bleurt-base-128"
if self.config_name not in CHECKPOINT_URLS.keys():
raise KeyError(
f"{self.config_name} model not found. You should supply the name of a model checkpoint for bleurt in {CHECKPOINT_URLS.keys()}"
)
# download the model checkpoint specified by self.config_name and set up the scorer
model_path = dl_manager.download_and_extract(CHECKPOINT_URLS[self.config_name])
self.scorer = score.BleurtScorer(os.path.join(model_path, self.config_name))
Compute score
Provide the functions for DatasetBuilder._compute to calculate your metric:
Create DatasetBuilder._compute with instructions for what metric to calculate for each configuration:
Copied
def _compute(self, predictions, references):
if self.config_name == "cola":
return {"matthews_correlation": matthews_corrcoef(references, predictions)}
elif self.config_name == "stsb":
return pearson_and_spearman(predictions, references)
elif self.config_name in ["mrpc", "qqp"]:
return acc_and_f1(predictions, references)
elif self.config_name in ["sst2", "mnli", "mnli_mismatched", "mnli_matched", "qnli", "rte", "wnli", "hans"]:
return {"accuracy": simple_accuracy(predictions, references)}
else:
raise KeyError(
"You should supply a configuration name selected in "
'["sst2", "mnli", "mnli_mismatched", "mnli_matched", '
'"cola", "stsb", "mrpc", "qqp", "qnli", "rte", "wnli", "hans"]'
)
Test
Once you’re finished writing your metric loading script, try to load it locally:
Copied
>>> from datasets import load_metric
>>> metric = load_metric('PATH/TO/MY/SCRIPT.py')
The most straightforward way to calculate a metric is to call . But some metrics have additional arguments that allow you to modify the metrics behavior.
Let’s load the metric, and compute it with a different smoothing method.
Write a metric loading script to use your own custom metric (or one that is not on the Hub). Then you can load it as usual with .
To help you get started, open the and follow along.
Get jump started with our metric loading script !
If your metric needs to download, or retrieve local files, you will need to use the Metric._download_and_prepare() method. For this example, let’s examine the .
DatasetBuilder._compute provides the actual instructions for how to compute a metric given the predictions and references. Now let’s take a look at the .
