Test script for mobilebert onnx accuracy

mobilebert_accuracy_error.py

import onnx
import tensorflow as tf
import onnxruntime.backend
import transformers
import numpy as np
import argparse
import time

tokenizer = transformers.BertTokenizer.from_pretrained("google/mobilebert-uncased")
output = tokenizer("""
Anyone who examines the Zen arts is immediately struck by how modern they seem. Many of the most famous stone gardens are abstract expressionism pure and simple, created out of found objects. The ceramics of the sixteenth-century Zen artists could be interchanged with the rugged pots of our own contemporary crafts movement and few people would notice a difference. Ancient Zen calligraphies, bold and slashing, suggest the monochromes of Franz Kline or Willem de Kooning, and if the word "impressionistic" has any real meaning left, the spontaneous, intuitive, impulsive Zen painters should have first claim to it. The apparent nonsense and illogic of Zen parables established the limitations of language long before the theater of the absurd decided to ridicule our modern doublespeak; indeed, our new-found skepticism about language as a medium for communication was a commonplace to Japanese artists who created both a drama (the No) and a poetry (the Haiku) that neatly circumvent reliance on mere words for expression—and in two entirely different ways. Four-hundred-year-old Zen architecture appears to be virtually a copy of contemporary design ideas: modular sizing, exposed woods and materials, movable partitions, multifunctional rooms, bare walls and uncluttered space, indirect lighting effects, and a California marriage of house and garden. The celebrated tea ceremony might be considered an early form of Japanese group therapy, while Zen landscape gardens are nothing less than a masterful deception masquerading as the "natural" look.

If all this were not coincidence enough, consider for a moment our present-day artistic conventions and aesthetic ideals. Like much of what we consider "modern," Zen arts tend to be as simple as possible, with clean, even severe, lines. Decoration for its own sake is virtually nonexistent; Zen artists had no more taste
""")
args = [
        np.array(output["input_ids"]).reshape(1, 384).astype("int32"), 
        np.array(output["attention_mask"]).reshape(1, 384).astype("int32"), 
        np.array(output["token_type_ids"]).reshape(1, 384).astype("int32")
       ]

tflite_file = "mobile_bert_mlperf07_tflite/mobilebert_quant_384_20200602.tflite"
onnx_file = "mobilebert_quant_384_20200602_v11.onnx"

# Run with ORT
onnx_model = onnx.load(onnx_file)
rep = onnxruntime.backend.prepare(onnx_model, "CPU")
onnx_result = rep.run(args)


#Run with TFLite
interpreter = tf.lite.Interpreter(model_path=tflite_file)

interpreter.allocate_tensors()
interpreter.set_tensor(interpreter.get_input_details()[0]["index"], args[0])
interpreter.set_tensor(interpreter.get_input_details()[1]["index"], args[1])
interpreter.set_tensor(interpreter.get_input_details()[2]["index"], args[2])

interpreter.invoke()

tflite_result = [
    interpreter.get_tensor(interpreter.get_output_details()[0]['index']),
    interpreter.get_tensor(interpreter.get_output_details()[1]['index'])
]


def rmse(x, y):
    return np.sqrt(np.mean((x -y)**2))

print("onnx vs tflite output 0", rmse(onnx_result[0], tflite_result[0]))
print("onnx vs tflite output 1", rmse(onnx_result[1], tflite_result[1]))