Andrea Santilli teelinsan

Research Scientist in NLP & LLM

teelinsan / calculate_mcga_mnli.py

Created March 25, 2021 13:55

Simple script to calculate the "multiple_choice_grade” accuracy for BIGBench. Run the script in the main folder of the project.

	from transformers import pipeline
	from tqdm import tqdm
	import json


	def check_mnli_benchmark(benchmark_json_path, classifier, base_path='bigbench/benchmark_tasks/', use_prefix_out=True):
	"""
	Function to compute the “multiple_choice_grade” accuracy of the classifier provided.
	The script save in the path a result json with the score and the list of the results for the elements in the dataset

teelinsan / gist:1dc5c6972e6571790241c5179e1b29ef

Created March 19, 2020 16:51

Odoo available widgets