Skip to content

Instantly share code, notes, and snippets.

@brockmanmatt

brockmanmatt/wic_selfcontextstuffingimproved_last10_stuft_examplesnv.ipynb

Created August 1, 2020 03:28
Show Gist options
  • Save brockmanmatt/deafb4dba7e4399327e44f2c8fd97b2b to your computer and use it in GitHub Desktop.
Save brockmanmatt/deafb4dba7e4399327e44f2c8fd97b2b to your computer and use it in GitHub Desktop.
Download ZIP
WiC_SelfContextStuffingImproved_Last10_stuft_examplesNV.ipynb
Display the source blob
Display the rendered blob
Raw
wic_selfcontextstuffingimproved_last10_stuft_examplesnv.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "WiC_SelfContextStuffingImproved_Last10_stuft_examplesNV.ipynb",
"provenance": [],
"collapsed_sections": [],
"authorship_tag": "ABX9TyOTzRLl8UCCekGlitvfLlpS",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/brockmanmatt/deafb4dba7e4399327e44f2c8fd97b2b/wic_selfcontextstuffingimproved_last10_stuft_examplesnv.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "code",
"metadata": {
"id": "J7wnsgT2kPut",
"colab_type": "code",
"colab": {
"resources": {
"http://localhost:8080/nbextensions/google.colab/files.js": {
"data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCkgewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwogICAgICBwZXJjZW50LnRleHRDb250ZW50ID0KICAgICAgICAgIGAke01hdGgucm91bmQoKHBvc2l0aW9uIC8gZmlsZURhdGEuYnl0ZUxlbmd0aCkgKiAxMDApfSUgZG9uZWA7CiAgICB9CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK",
"ok": true,
"headers": [
[
"content-type",
"application/javascript"
]
],
"status": 200,
"status_text": ""
}
},
"base_uri": "https://localhost:8080/",
"height": 89
},
"outputId": "7c55416f-9cdb-44fe-8c6a-45f7f905ce2e"
},
"source": [
"from google.colab import files\n",
"uploaded = files.upload()\n",
"print(\"done\")"
],
"execution_count": null,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/html": [
"\n",
" <input type=\"file\" id=\"files-be1501c8-d29d-4aa4-8643-37ffbcf1641d\" name=\"files[]\" multiple disabled\n",
" style=\"border:none\" />\n",
" <output id=\"result-be1501c8-d29d-4aa4-8643-37ffbcf1641d\">\n",
" Upload widget is only available when the cell has been executed in the\n",
" current browser session. Please rerun this cell to enable.\n",
" </output>\n",
" <script src=\"/nbextensions/google.colab/files.js\"></script> "
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {
"tags": []
}
},
{
"output_type": "stream",
"text": [
"Saving key.json to key.json\n",
"done\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WHPHrUnhpKnI",
"colab_type": "text"
},
"source": [
"I'll install the API"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "h4BX5shF5urh",
"colab_type": "text"
},
"source": [
"get wic dataset"
]
},
{
"cell_type": "code",
"metadata": {
"id": "zq0ltp2xn4yt",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 292
},
"outputId": "19e7d898-6f65-4ac8-828c-3ed9e968598d"
},
"source": [
"!pip install openai\n",
"import openai, json, pandas as pd, numpy as np, random"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Collecting openai\n",
"\u001b[?25l Downloading https://files.pythonhosted.org/packages/a8/65/c7461f4c87984534683f480ea5742777bc39bbf5721123194c2d0347dc1f/openai-0.2.4.tar.gz (157kB)\n",
"\u001b[K |████████████████████████████████| 163kB 2.5MB/s \n",
"\u001b[?25hRequirement already satisfied: requests>=2.20 in /usr/local/lib/python3.6/dist-packages (from openai) (2.23.0)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20->openai) (2020.6.20)\n",
"Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20->openai) (3.0.4)\n",
"Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20->openai) (1.24.3)\n",
"Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20->openai) (2.10)\n",
"Building wheels for collected packages: openai\n",
" Building wheel for openai (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
" Created wheel for openai: filename=openai-0.2.4-cp36-none-any.whl size=170709 sha256=92cb1c67856702ddb35769c878ef0c3382cb45eb9af852fd3e93c31956798a12\n",
" Stored in directory: /root/.cache/pip/wheels/74/96/c8/c6e170929c276b836613e1b9985343b501fe455e53d85e7d48\n",
"Successfully built openai\n",
"Installing collected packages: openai\n",
"Successfully installed openai-0.2.4\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "k67w5H0fpTkT",
"colab_type": "text"
},
"source": [
"args to pass API; one for 1 line 1 for 2 line"
]
},
{
"cell_type": "code",
"metadata": {
"id": "e1EwpqqJkTYh",
"colab_type": "code",
"colab": {}
},
"source": [
"#arguments to send the API\n",
"kwargs = {\n",
"\"engine\":\"davinci\",\n",
"\"temperature\":0,\n",
"\"max_tokens\":200,\n",
"\"stop\":\"\\n\",\n",
"}\n",
"\n",
"kwargs2 = {\n",
"\"engine\":\"davinci\",\n",
"\"temperature\":0,\n",
"\"max_tokens\":200,\n",
"\"stop\":\"\\n\\n\",\n",
"}\n",
"\n",
"kwargs2Short = {\n",
"\"engine\":\"davinci\",\n",
"\"temperature\":0,\n",
"\"max_tokens\":20,\n",
"\"stop\":\"\\n\\n\",\n",
"}\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "XPqahGflR_Tc",
"colab_type": "code",
"colab": {}
},
"source": [
"openai.api_key = json.load(open(\"key.json\", \"r\"))[\"key\"]"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "sXTDJx0An9Bl",
"colab_type": "code",
"colab": {}
},
"source": [
"def queryTwoLine(prompt, myKwargs = kwargs2):\n",
" \"\"\"\n",
" wrapper for the API (get 2 newlines)\n",
" \"\"\"\n",
" r = openai.Completion.create(prompt=prompt, **myKwargs)[\"choices\"][0][\"text\"].strip()\n",
" return r\n",
"\n",
"\n",
"def queryOneLine(prompt, myKwargs = kwargs):\n",
" \"\"\"\n",
" wrapper for the API (get 1 newlines)\n",
" \"\"\"\n",
" r = openai.Completion.create(prompt=prompt, **myKwargs)[\"choices\"][0][\"text\"].strip()\n",
" return r"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "EdFXafcJpZ3Q",
"colab_type": "text"
},
"source": [
"Test to make sure my query works"
]
},
{
"cell_type": "code",
"metadata": {
"id": "4SlyKgjyopPn",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
},
"outputId": "96900548-cab5-45b4-f5b8-83f8d8817901"
},
"source": [
"newKwargs = kwargs.copy()\n",
"newKwargs[\"stop\"] = \"\\n\"\n",
"queryOneLine(\"q: what is 1+1?\\na:\")"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'2'"
]
},
"metadata": {
"tags": []
},
"execution_count": 6
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AAR9Guv1QPZk",
"colab_type": "text"
},
"source": [
"Get the WiC dataset"
]
},
{
"cell_type": "code",
"metadata": {
"id": "XDy96ovI9hJm",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"outputId": "f53f444a-b511-475f-c7fa-436448942b22"
},
"source": [
"!wget https://pilehvar.github.io/wic/package/WiC_dataset.zip"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"--2020-08-01 01:40:25-- https://pilehvar.github.io/wic/package/WiC_dataset.zip\n",
"Resolving pilehvar.github.io (pilehvar.github.io)... 185.199.108.153, 185.199.111.153, 185.199.110.153, ...\n",
"Connecting to pilehvar.github.io (pilehvar.github.io)|185.199.108.153|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 275513 (269K) [application/zip]\n",
"Saving to: ‘WiC_dataset.zip’\n",
"\n",
"WiC_dataset.zip 100%[===================>] 269.06K --.-KB/s in 0.05s \n",
"\n",
"2020-08-01 01:40:26 (5.00 MB/s) - ‘WiC_dataset.zip’ saved [275513/275513]\n",
"\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mh8Ztw-JQRMr",
"colab_type": "text"
},
"source": [
"Unzip the dataset"
]
},
{
"cell_type": "code",
"metadata": {
"id": "Gx-SlrBpSVXJ",
"colab_type": "code",
"colab": {}
},
"source": [
"import zipfile\n",
"with zipfile.ZipFile(\"WiC_dataset.zip\",\"r\") as zip_ref:\n",
" zip_ref.extractall(\".\")\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "wOjsEzALQTXW",
"colab_type": "text"
},
"source": [
"Read in train and add the T/F label for if they're the same"
]
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "CAHaOQKx9ogG",
"colab": {}
},
"source": [
"train = pd.read_csv(\"train/train.data.txt\", sep='\\t', header=None)\n",
"train.columns = [\"target\", \"pos\", \"position\", \"context-1\", \"context-2\"]\n",
"train_gold = pd.read_csv(\"train/train.gold.txt\", sep='\\t', header=None)\n",
"train_gold.columns = [\"label\"]\n",
"train = pd.concat([train_gold,train], axis=1)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "XC1pGirRwF8Z",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"outputId": "07ed4440-d2eb-4a19-90af-f5b90a310093"
},
"source": [
"train.head()"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>label</th>\n",
" <th>target</th>\n",
" <th>pos</th>\n",
" <th>position</th>\n",
" <th>context-1</th>\n",
" <th>context-2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>F</td>\n",
" <td>carry</td>\n",
" <td>V</td>\n",
" <td>2-1</td>\n",
" <td>You must carry your camping gear .</td>\n",
" <td>Sound carries well over water .</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>F</td>\n",
" <td>go</td>\n",
" <td>V</td>\n",
" <td>2-6</td>\n",
" <td>Messages must go through diplomatic channels .</td>\n",
" <td>Do you think the sofa will go through the door ?</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>F</td>\n",
" <td>break</td>\n",
" <td>V</td>\n",
" <td>0-2</td>\n",
" <td>Break an alibi .</td>\n",
" <td>The wholesaler broke the container loads into ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>T</td>\n",
" <td>cup</td>\n",
" <td>N</td>\n",
" <td>8-4</td>\n",
" <td>He wore a jock strap with a metal cup .</td>\n",
" <td>Bees filled the waxen cups with honey .</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>F</td>\n",
" <td>academy</td>\n",
" <td>N</td>\n",
" <td>1-2</td>\n",
" <td>The Academy of Music .</td>\n",
" <td>The French Academy .</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" label ... context-2\n",
"0 F ... Sound carries well over water .\n",
"1 F ... Do you think the sofa will go through the door ?\n",
"2 F ... The wholesaler broke the container loads into ...\n",
"3 T ... Bees filled the waxen cups with honey .\n",
"4 F ... The French Academy .\n",
"\n",
"[5 rows x 6 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 10
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3neGYQj6d9zU",
"colab_type": "text"
},
"source": [
"Def to bootstrap meanings; this prompt asks it what a term means in context, then returns the question + response "
]
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "JE6GhNTi4hTT",
"colab": {}
},
"source": [
"def getContextualNounExample(content, term):\n",
" prompt = \"Tom said '{}'\\n\".format(content, term)\n",
" prompt += \"I asked Tom what '{}' specifically means in this context, he clarified it is another word for\".format(term) \n",
"\n",
" return (prompt + \" \" +queryOneLine(prompt))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "g3IO1TK64hTX"
},
"source": [
"Get response to end of training set as my fewshot examples, print to make sure it makes sense"
]
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "FD6Sh9oY4hTX",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 377
},
"outputId": "c7f9d971-3629-44d6-a2d3-3c80c2fb2e29"
},
"source": [
"fewShotsNounDefinitions = [] #I'll build this from the training set\n",
"for row in train[train.pos==\"N\"].tail(10).iterrows():\n",
" s1 = row[1][\"context-1\"]\n",
" target = row[1][\"target\"]\n",
" r = getContextualNounExample(s1, target)\n",
" print(r)\n",
" fewShotsNounDefinitions.append(\" \".join(r.split(\"\\n\"))) #add the definition to fewShotsDefinition as 1 line\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Tom said 'The cinema relies on apparent motion .'\n",
"I asked Tom what 'motion' specifically means in this context, he clarified it is another word for 'movement'.\n",
"Tom said 'It vanished into the night .'\n",
"I asked Tom what 'night' specifically means in this context, he clarified it is another word for 'darkness'.\n",
"Tom said 'He threw the ball into the air .'\n",
"I asked Tom what 'air' specifically means in this context, he clarified it is another word for 'sky'.\n",
"Tom said 'Those clouds show little sign of raining soon .'\n",
"I asked Tom what 'sign' specifically means in this context, he clarified it is another word for 'signs' or 'indications'.\n",
"Tom said 'We added a new rosebush to our rose bed .'\n",
"I asked Tom what 'bed' specifically means in this context, he clarified it is another word for 'garden' or 'yard'\n",
"Tom said 'His state of health .'\n",
"I asked Tom what 'state' specifically means in this context, he clarified it is another word for 'condition'.\n",
"Tom said 'Likes a drink before dinner .'\n",
"I asked Tom what 'drink' specifically means in this context, he clarified it is another word for 'alcoholic drink'.\n",
"Tom said 'Piecas kronas — five krona .'\n",
"I asked Tom what 'krona' specifically means in this context, he clarified it is another word for 'coin'.\n",
"Tom said 'The harder the conflict the more glorious the triumph \"-- Thomas Paine .'\n",
"I asked Tom what 'conflict' specifically means in this context, he clarified it is another word for war.\n",
"Tom said 'An invasion of bees .'\n",
"I asked Tom what 'invasion' specifically means in this context, he clarified it is another word for 'attack' or 'assault'.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "fdmQDSbRVG8D",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 207
},
"outputId": "77813412-c89a-4ffc-a654-b0e30dac8f47"
},
"source": [
"for i in range(len(fewShotsNounDefinitions)):\n",
" print(\"{}: {}\".format(i, fewShotsNounDefinitions[i]))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"0: Tom said 'The cinema relies on apparent motion .' I asked Tom what 'motion' specifically means in this context, he clarified it is another word for 'movement'.\n",
"1: Tom said 'It vanished into the night .' I asked Tom what 'night' specifically means in this context, he clarified it is another word for 'darkness'.\n",
"2: Tom said 'He threw the ball into the air .' I asked Tom what 'air' specifically means in this context, he clarified it is another word for 'sky'.\n",
"3: Tom said 'Those clouds show little sign of raining soon .' I asked Tom what 'sign' specifically means in this context, he clarified it is another word for 'signs' or 'indications'.\n",
"4: Tom said 'We added a new rosebush to our rose bed .' I asked Tom what 'bed' specifically means in this context, he clarified it is another word for 'garden' or 'yard'\n",
"5: Tom said 'His state of health .' I asked Tom what 'state' specifically means in this context, he clarified it is another word for 'condition'.\n",
"6: Tom said 'Likes a drink before dinner .' I asked Tom what 'drink' specifically means in this context, he clarified it is another word for 'alcoholic drink'.\n",
"7: Tom said 'Piecas kronas — five krona .' I asked Tom what 'krona' specifically means in this context, he clarified it is another word for 'coin'.\n",
"8: Tom said 'The harder the conflict the more glorious the triumph \"-- Thomas Paine .' I asked Tom what 'conflict' specifically means in this context, he clarified it is another word for war.\n",
"9: Tom said 'An invasion of bees .' I asked Tom what 'invasion' specifically means in this context, he clarified it is another word for 'attack' or 'assault'.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cUpQUQTQWPC7",
"colab_type": "text"
},
"source": [
"Manually discard 3 and 8 bad"
]
},
{
"cell_type": "code",
"metadata": {
"id": "eyj-anNIWLK4",
"colab_type": "code",
"colab": {}
},
"source": [
"#bad = [3,8]\n",
"bad = []\n",
"fewShotsNounPrompt = \"\\n\".join([fewShotsNounDefinitions[i] for i in range(len(fewShotsNounDefinitions)) if i not in bad])"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "_lG5CFJSTjmm",
"colab_type": "code",
"colab": {}
},
"source": [
"def getContextualVerbExample(content, term):\n",
" prompt = \"Tom said '{}'\\n\".format(content, term)\n",
" prompt += \"I asked Tom what 'to {}' specifically means in this context, he explained it was another word for 'to\".format(term) \n",
"\n",
" return (prompt + \" \" +queryOneLine(prompt))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "50zmDtcXTODp",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 377
},
"outputId": "49a323cd-92fb-4e09-d61a-461b3153db5c"
},
"source": [
"fewShotsVerbDefinitions = [] #I'll build this from the training set\n",
"for row in train[train.pos==\"V\"][-10:].iterrows():\n",
" s1 = row[1][\"context-1\"]\n",
" target = row[1][\"target\"]\n",
" r = getContextualVerbExample(s1, target)\n",
" print(r)\n",
" fewShotsVerbDefinitions.append( \" \".join(r.split(\"\\n\"))) #add the definition to fewShotsDefinition as 1 line"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Tom said 'Sanitize the language in a book .'\n",
"I asked Tom what 'to sanitize' specifically means in this context, he explained it was another word for 'to clean up' or 'to edit'.\n",
"Tom said 'You are trying my patience .'\n",
"I asked Tom what 'to try' specifically means in this context, he explained it was another word for 'to test' or 'to attempt'.\n",
"Tom said 'How would you classify these pottery shards -- are they prehistoric ?'\n",
"I asked Tom what 'to classify' specifically means in this context, he explained it was another word for 'to classify'\n",
"Tom said 'My daughter 's fancy wedding is going to break me !'\n",
"I asked Tom what 'to break' specifically means in this context, he explained it was another word for 'to bankrupt' or 'to ruin'\n",
"Tom said 'Render thanks .'\n",
"I asked Tom what 'to render' specifically means in this context, he explained it was another word for 'to give' or 'to offer' and that it was a very old word.\n",
"Tom said 'I drive to work every day .'\n",
"I asked Tom what 'to drive' specifically means in this context, he explained it was another word for 'to go'.\n",
"Tom said 'Keep open the possibility of a merger .'\n",
"I asked Tom what 'to keep' specifically means in this context, he explained it was another word for 'to consider'.\n",
"Tom said 'Cats stretch with equal ease and agility beyond the point that breaks a man on the rack .'\n",
"I asked Tom what 'to stretch' specifically means in this context, he explained it was another word for 'to reach' or 'to extend' and that it was a common word in the English language.\n",
"Tom said 'Answer the riddle .'\n",
"I asked Tom what 'to answer' specifically means in this context, he explained it was another word for 'to solve' or 'to work out'.\n",
"Tom said 'Play the casinos in Trouville .'\n",
"I asked Tom what 'to play' specifically means in this context, he explained it was another word for 'to cheat' or 'to swindle'.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "qfKnxRbXTTzz",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 207
},
"outputId": "1bfd9240-37be-47a2-b834-614dd08868c4"
},
"source": [
"for i in range(len(fewShotsVerbDefinitions)):\n",
" print(\"{}: {}\".format(i, fewShotsVerbDefinitions[i]))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"0: Tom said 'Sanitize the language in a book .' I asked Tom what 'to sanitize' specifically means in this context, he explained it was another word for 'to clean up' or 'to edit'.\n",
"1: Tom said 'You are trying my patience .' I asked Tom what 'to try' specifically means in this context, he explained it was another word for 'to test' or 'to attempt'.\n",
"2: Tom said 'How would you classify these pottery shards -- are they prehistoric ?' I asked Tom what 'to classify' specifically means in this context, he explained it was another word for 'to classify'\n",
"3: Tom said 'My daughter 's fancy wedding is going to break me !' I asked Tom what 'to break' specifically means in this context, he explained it was another word for 'to bankrupt' or 'to ruin'\n",
"4: Tom said 'Render thanks .' I asked Tom what 'to render' specifically means in this context, he explained it was another word for 'to give' or 'to offer' and that it was a very old word.\n",
"5: Tom said 'I drive to work every day .' I asked Tom what 'to drive' specifically means in this context, he explained it was another word for 'to go'.\n",
"6: Tom said 'Keep open the possibility of a merger .' I asked Tom what 'to keep' specifically means in this context, he explained it was another word for 'to consider'.\n",
"7: Tom said 'Cats stretch with equal ease and agility beyond the point that breaks a man on the rack .' I asked Tom what 'to stretch' specifically means in this context, he explained it was another word for 'to reach' or 'to extend' and that it was a common word in the English language.\n",
"8: Tom said 'Answer the riddle .' I asked Tom what 'to answer' specifically means in this context, he explained it was another word for 'to solve' or 'to work out'.\n",
"9: Tom said 'Play the casinos in Trouville .' I asked Tom what 'to play' specifically means in this context, he explained it was another word for 'to cheat' or 'to swindle'.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "yDsvI64NTTra",
"colab_type": "code",
"colab": {}
},
"source": [
"#bad = [2,7]\n",
"bad = []\n",
"fewShotsVerbPrompt = \"\\n\".join([fewShotsVerbDefinitions[i] for i in range(len(fewShotsVerbDefinitions)) if i not in bad])"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "WvZMJAhDQjgb",
"colab_type": "text"
},
"source": [
"Now use the few shots for my actual method, and allow me to context stuff the previous answer so if the meaning is the same it'll likely choose it and if they're different it'll pick up on the difference hopefully and not choose it again. Using presense_penalty to increase odds of choosing new token. \n",
"\n",
"Potentially adjusting this is a parameter to tune."
]
},
{
"cell_type": "code",
"metadata": {
"id": "BEmsnBCErBZX",
"colab_type": "code",
"colab": {}
},
"source": [
"def getContextualNounMeaning(content, term, contexts = []):\n",
" prompt = fewShotsNounPrompt\n",
" \n",
" for context in contexts:\n",
" prompt += \"\\nTom said '{}' \".format(context[\"content\"])\n",
" prompt += \" I asked Tom what '{}' specifically means in this context, he clarified it is another word for {}\".format(context[\"term\"], context[\"meaning\"]) \n",
"\n",
" prompt += \"\\nTom said '{}' \".format(content, term)\n",
" prompt += \" I asked Tom what '{}' specifically means in this context, he clarified it is another word for\".format(term) \n",
"\n",
" r = queryOneLine(prompt, myKwargs = {'engine': 'davinci', 'max_tokens': 20, 'stop': '\\n', 'temperature': 0, \"presence_penalty\":.2})\n",
" if not r.startswith(\"'\"):\n",
" r = \"'\" + r\n",
" \n",
" return r"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "L10z2Z-IZVsy",
"colab_type": "code",
"colab": {}
},
"source": [
"def getContextualVerbMeaning(content, term, contexts = []):\n",
" prompt = fewShotsVerbPrompt\n",
" \n",
" for context in contexts:\n",
" prompt += \"\\nTom said '{}'\".format(context[\"content\"])\n",
" prompt += \" I asked Tom what 'to {}' specifically means in this context, he explained it was another word for {}\".format(context[\"term\"], context[\"meaning\"]) \n",
"\n",
" prompt += \"\\nTom said '{}'\".format(content)\n",
" prompt += \" I asked Tom what 'to {}' specifically means in this context, he explained it was another word for 'to\".format(term) \n",
"\n",
" r = queryOneLine(prompt, myKwargs = {'engine': 'davinci', 'max_tokens': 20, 'stop': '\\n', 'temperature': 0, \"presence_penalty\":.2})\n",
" if not r.startswith(\"'\"):\n",
" r = \"'\" + r\n",
" \n",
" return r"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "pPWT-NZwRq5r",
"colab_type": "text"
},
"source": [
"Now I make a method to build comparison examples; takes two sentences and a label (which is the actual answer if they're true/false); it generates an example"
]
},
{
"cell_type": "code",
"metadata": {
"id": "g5E68j_bRii0",
"colab_type": "code",
"colab": {}
},
"source": [
"def generateComparisonExample(s1, s2, label):\n",
" prompt = \"Tom says that this means {} Jerry says this means {}\\n\".format(s1, s2)\n",
" prompt += \"Q: Are Tom and Jerry basically saying the same thing here?\\nA:\"\n",
" if label == \"T\":\n",
" prompt += \" Yes\"\n",
" else:\n",
" prompt += \" No\"\n",
" return prompt"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "jHBUBl2eQ5vs",
"colab_type": "text"
},
"source": [
"Now generate a list of examples from the builder method. I'll be able to select them from the list, but I'll end up just joining all of them together for the actual prompt (possibly something to use to improve)"
]
},
{
"cell_type": "code",
"metadata": {
"id": "DNSFTSUt07CB",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 153
},
"outputId": "266962c5-022c-4270-aaed-0222a82577b5"
},
"source": [
"comparisonFewShotVerbExamples = []\n",
"for row in train[train.pos==\"V\"][-20:-12].iterrows():\n",
" s1 = row[1][\"context-1\"]\n",
" s2 = row[1][\"context-2\"]\n",
" label = row[1][\"label\"]\n",
" target = row[1][\"target\"]\n",
" r1 = getContextualVerbMeaning(s1, target)\n",
" r2 = getContextualVerbMeaning(s2, target, contexts=[{\"content\":s1, \"term\":target, \"meaning\":r1}])\n",
" \n",
" if len(comparisonFewShotVerbExamples) < 2:\n",
" print(s1)\n",
" print(s2)\n",
" \n",
" print(r1)\n",
" print(r2)\n",
"\n",
" r= generateComparisonExample(r1, r2, label)\n",
" comparisonFewShotVerbExamples.append(r)\n",
" #print(r)\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"The company agrees to meet the cost of any repairs .\n",
"Does this paper meet the requirements for the degree ?\n",
"'pay' or 'to cover'.\n",
"'comply' or 'to satisfy'.\n",
"You anger too easily .\n",
"He angers easily .\n",
"'annoy' or 'to irritate'.\n",
"'annoy' or 'to irritate'.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "7P0ixd5PflJm",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 221
},
"outputId": "5e63cefd-1733-4ab5-cdd5-83901a3dfcc8"
},
"source": [
"comparisonFewShotNounExamples = []\n",
"for row in train[train.pos==\"N\"][-20:-12].iterrows():\n",
" s1 = row[1][\"context-1\"]\n",
" s2 = row[1][\"context-2\"]\n",
" label = row[1][\"label\"]\n",
" target = row[1][\"target\"]\n",
" r1 = getContextualNounMeaning(s1, target)\n",
" r2 = getContextualNounMeaning(s2, target, contexts=[{\"content\":s1, \"term\":target, \"meaning\":r1}])\n",
" if len(comparisonFewShotNounExamples) < 3:\n",
" print(s1)\n",
" print(s2)\n",
" \n",
" print(r1)\n",
" print(r2)\n",
"\n",
" r= generateComparisonExample(r1, r2, label)\n",
" comparisonFewShotNounExamples.append(r)\n",
" #print(r)\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"I do it for the fun of it .\n",
"He is fun to have around .\n",
"'pleasure'.\n",
"'funny'.\n",
"A history of France .\n",
"A critical time in the school 's history .\n",
"'past'.\n",
"'past'.\n",
"The rate of production at the factory is skyrocketing .\n",
"He works at a great rate .\n",
"'speed'.\n",
"'speed'.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Frih6lJBSDJ6",
"colab_type": "text"
},
"source": [
"Now I have my actual comparison prompt that uses the few shots from above to let it know I just want a Yes/No answer."
]
},
{
"cell_type": "code",
"metadata": {
"id": "dmtBiasT1SUP",
"colab_type": "code",
"colab": {}
},
"source": [
"def generateNounComparison(s1, s2):\n",
"\n",
" prompt = \"\\n\\n\".join(comparisonFewShotNounExamples)\n",
" prompt += \"\\n\\n\"\n",
"\n",
" prompt += \"Tom says that this means {}; Jerry says this means {}\\n\".format(s1, s2)\n",
" prompt += \"Q: Are Tom and Jerry basically saying the same thing here?\\nA:\"\n",
"\n",
"\n",
" return queryTwoLine(prompt, myKwargs = kwargs2Short)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "htMuKEzDgR9Y",
"colab_type": "code",
"colab": {}
},
"source": [
"def generateVerbComparison(s1, s2):\n",
"\n",
" prompt = \"\\n\\n\".join(comparisonFewShotVerbExamples)\n",
" prompt += \"\\n\\n\"\n",
"\n",
" prompt += \"Tom says that this means {}; Jerry says this means {}\\n\".format(s1, s2)\n",
" prompt += \"Q: Are Tom and Jerry basically saying the same thing here?\\nA:\"\n",
"\n",
"\n",
" return queryTwoLine(prompt, myKwargs = kwargs2Short)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "U-NL6c6TSLux",
"colab_type": "text"
},
"source": [
"Test to make sure it works OK"
]
},
{
"cell_type": "code",
"metadata": {
"id": "L58zdQUp2B9v",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 377
},
"outputId": "9267f79d-82ea-47be-8258-5fa27f0ba65f"
},
"source": [
"for row in train[20:25].iterrows():\n",
" s1 = row[1][\"context-1\"]\n",
" s2 = row[1][\"context-2\"]\n",
" label = row[1][\"label\"]\n",
" target = row[1][\"target\"]\n",
" if row[1][\"pos\"] == \"N\":\n",
" r1 = getContextualNounMeaning(s1, target)\n",
" r2 = getContextualNounMeaning(s2, target, contexts=[{\"content\":s1, \"term\":target, \"meaning\":r1}])\n",
" r= generateNounComparison(r1, r2)\n",
"\n",
" else:\n",
" r1 = getContextualVerbMeaning(s1, target)\n",
" r2 = getContextualVerbMeaning(s2, target, contexts=[{\"content\":s1, \"term\":target, \"meaning\":r1}])\n",
" r= generateVerbComparison(r1, r2)\n",
"\n",
" print(\"'{}' v '{}'\".format(s1, s2))\n",
" print(\"{} v {}\".format(r1, r2))\n",
" print(\"returned: {}\".format(r))\n",
" print(\"actual: {}\".format(label))\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"'This speech did n't play well with the American public .' v 'Play football .'\n",
"'perform' or 'to act'. v 'compete' or 'to engage in a contest'.\n",
"returned: No\n",
"actual: F\n",
"'What do you make of his remarks ?' v 'Make a dress .'\n",
"'create' or 'to produce'. v 'sew' or 'to stitch'.\n",
"returned: No\n",
"actual: F\n",
"'The so-called glorious experiment came to an inglorious end .' v 'The end was exciting .'\n",
"'conclusion'. v 'conclusion'.\n",
"returned: Yes\n",
"actual: F\n",
"'There was a break in the action when a player was hurt .' v 'He was up two breaks in the second set .'\n",
"'pause'. v 'game'.\n",
"returned: No\n",
"actual: F\n",
"'It was a heavy play and the actors tried in vain to give life to it .' v 'The life of this milk carton may be thousands of years in this landfill .'\n",
"'success'. v 'existence'.\n",
"returned: No\n",
"actual: F\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bpc6A5pl68RM",
"colab_type": "text"
},
"source": [
"k, so now I'll load the dev set"
]
},
{
"cell_type": "code",
"metadata": {
"id": "eru96XmC6iWt",
"colab_type": "code",
"colab": {}
},
"source": [
"dev = pd.read_csv(\"dev/dev.data.txt\", sep='\\t', header=None)\n",
"dev.columns = [\"target\", \"pos\", \"position\", \"context-1\", \"context-2\"]\n",
"dev_gold = pd.read_csv(\"dev/dev.gold.txt\", sep='\\t', header=None)\n",
"dev_gold.columns = [\"label\"]\n",
"dev = pd.concat([dev_gold,dev], axis=1)\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "v5ycY33vSQ_O",
"colab_type": "text"
},
"source": [
"Keep track of right/wrong as I go along"
]
},
{
"cell_type": "code",
"metadata": {
"id": "CY6ixlHl77U7",
"colab_type": "code",
"colab": {}
},
"source": [
"devResults = {}\n",
"correct = 0\n",
"complete = 0\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "d-PZePcLSTZ4",
"colab_type": "text"
},
"source": [
"check entire dev set"
]
},
{
"cell_type": "code",
"metadata": {
"id": "k7P4pzAL2Pxl",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"outputId": "3e7ec63c-8734-4bb2-a9ac-55ab560eb00a"
},
"source": [
"for row in dev.iterrows():\n",
" if row[0] in devResults: continue\n",
" s1 = row[1][\"context-1\"]\n",
" s2 = row[1][\"context-2\"]\n",
" label = row[1][\"label\"]\n",
" target = row[1][\"target\"]\n",
" if row[1][\"pos\"] == \"N\":\n",
" r1 = getContextualNounMeaning(s1, target)\n",
" r2 = getContextualNounMeaning(s2, target, contexts=[{\"content\":s1, \"term\":target, \"meaning\":r1}])\n",
" r= generateNounComparison(r1, r2)\n",
"\n",
" else:\n",
" r1 = getContextualVerbMeaning(s1, target)\n",
" r2 = getContextualVerbMeaning(s2, target, contexts=[{\"content\":s1, \"term\":target, \"meaning\":r1}])\n",
" r= generateVerbComparison(r1, r2)\n",
"\n",
" myResults = {}\n",
" myResults[\"s1\"] = s1\n",
" myResults[\"s2\"] = s2\n",
"\n",
" myResults[\"pos\"] = row[1][\"pos\"]\n",
"\n",
" myResults[\"target\"] = target\n",
"\n",
" myResults[\"pred\"] = r\n",
"\n",
" myResults[\"actual\"] = label\n",
"\n",
" devResults[row[0]] = myResults\n",
"\n",
" complete +=1\n",
" if label == \"T\":\n",
" if r.strip()==\"Yes\":\n",
" correct += 1\n",
" if label == \"F\":\n",
" if r.strip()==\"No\":\n",
" correct += 1\n",
"\n",
" if row[0] %10 ==0:print (\"Complete: {} Correct: {} Wrong: {}\".format(complete, correct, complete-correct))\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Complete: 1 Correct: 1 Wrong: 0\n",
"Complete: 11 Correct: 7 Wrong: 4\n",
"Complete: 21 Correct: 15 Wrong: 6\n",
"Complete: 31 Correct: 23 Wrong: 8\n",
"Complete: 41 Correct: 29 Wrong: 12\n",
"Complete: 51 Correct: 37 Wrong: 14\n",
"Complete: 61 Correct: 44 Wrong: 17\n",
"Complete: 71 Correct: 50 Wrong: 21\n",
"Complete: 81 Correct: 58 Wrong: 23\n",
"Complete: 91 Correct: 63 Wrong: 28\n",
"Complete: 101 Correct: 71 Wrong: 30\n",
"Complete: 111 Correct: 81 Wrong: 30\n",
"Complete: 121 Correct: 87 Wrong: 34\n",
"Complete: 131 Correct: 97 Wrong: 34\n",
"Complete: 141 Correct: 105 Wrong: 36\n",
"Complete: 151 Correct: 115 Wrong: 36\n",
"Complete: 161 Correct: 117 Wrong: 44\n",
"Complete: 171 Correct: 123 Wrong: 48\n",
"Complete: 181 Correct: 130 Wrong: 51\n",
"Complete: 191 Correct: 139 Wrong: 52\n",
"Complete: 201 Correct: 145 Wrong: 56\n",
"Complete: 211 Correct: 152 Wrong: 59\n",
"Complete: 221 Correct: 158 Wrong: 63\n",
"Complete: 231 Correct: 166 Wrong: 65\n",
"Complete: 241 Correct: 174 Wrong: 67\n",
"Complete: 251 Correct: 181 Wrong: 70\n",
"Complete: 261 Correct: 187 Wrong: 74\n",
"Complete: 271 Correct: 194 Wrong: 77\n",
"Complete: 281 Correct: 201 Wrong: 80\n",
"Complete: 291 Correct: 210 Wrong: 81\n",
"Complete: 301 Correct: 217 Wrong: 84\n",
"Complete: 311 Correct: 223 Wrong: 88\n",
"Complete: 321 Correct: 230 Wrong: 91\n",
"Complete: 331 Correct: 239 Wrong: 92\n",
"Complete: 341 Correct: 243 Wrong: 98\n",
"Complete: 351 Correct: 251 Wrong: 100\n",
"Complete: 361 Correct: 257 Wrong: 104\n",
"Complete: 371 Correct: 264 Wrong: 107\n",
"Complete: 381 Correct: 268 Wrong: 113\n",
"Complete: 391 Correct: 273 Wrong: 118\n",
"Complete: 401 Correct: 280 Wrong: 121\n",
"Complete: 411 Correct: 286 Wrong: 125\n",
"Complete: 421 Correct: 292 Wrong: 129\n",
"Complete: 431 Correct: 298 Wrong: 133\n",
"Complete: 441 Correct: 304 Wrong: 137\n",
"Complete: 451 Correct: 313 Wrong: 138\n",
"Complete: 461 Correct: 320 Wrong: 141\n",
"Complete: 471 Correct: 328 Wrong: 143\n",
"Complete: 481 Correct: 333 Wrong: 148\n",
"Complete: 491 Correct: 340 Wrong: 151\n",
"Complete: 501 Correct: 346 Wrong: 155\n",
"Complete: 511 Correct: 355 Wrong: 156\n",
"Complete: 521 Correct: 361 Wrong: 160\n",
"Complete: 531 Correct: 368 Wrong: 163\n",
"Complete: 541 Correct: 375 Wrong: 166\n",
"Complete: 551 Correct: 383 Wrong: 168\n",
"Complete: 561 Correct: 388 Wrong: 173\n",
"Complete: 571 Correct: 394 Wrong: 177\n",
"Complete: 581 Correct: 400 Wrong: 181\n",
"Complete: 591 Correct: 406 Wrong: 185\n",
"Complete: 601 Correct: 415 Wrong: 186\n",
"Complete: 611 Correct: 425 Wrong: 186\n",
"Complete: 621 Correct: 432 Wrong: 189\n",
"Complete: 631 Correct: 439 Wrong: 192\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vWyri1BiSWDX",
"colab_type": "text"
},
"source": [
"save dev set"
]
},
{
"cell_type": "code",
"metadata": {
"id": "55MlUCdD7sUa",
"colab_type": "code",
"colab": {}
},
"source": [
"devDf = pd.DataFrame(devResults).T"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "WcSxf4IX8C0S",
"colab_type": "code",
"colab": {}
},
"source": [
"devDf.to_pickle(\"newDevResultsSpecifically_all_8examples.pkl\")"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "DRVg4lTKHeHZ",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 17
},
"outputId": "5ee85625-4b97-443d-f75d-bd4aff8a125e"
},
"source": [
"files.download(\"newDevResultsSpecifically_all_8examples.pkl\")"
],
"execution_count": null,
"outputs": [
{
"output_type": "display_data",
"data": {
"application/javascript": [
"\n",
" async function download(id, filename, size) {\n",
" if (!google.colab.kernel.accessAllowed) {\n",
" return;\n",
" }\n",
" const div = document.createElement('div');\n",
" const label = document.createElement('label');\n",
" label.textContent = `Downloading \"${filename}\": `;\n",
" div.appendChild(label);\n",
" const progress = document.createElement('progress');\n",
" progress.max = size;\n",
" div.appendChild(progress);\n",
" document.body.appendChild(div);\n",
"\n",
" const buffers = [];\n",
" let downloaded = 0;\n",
"\n",
" const channel = await google.colab.kernel.comms.open(id);\n",
" // Send a message to notify the kernel that we're ready.\n",
" channel.send({})\n",
"\n",
" for await (const message of channel.messages) {\n",
" // Send a message to notify the kernel that we're ready.\n",
" channel.send({})\n",
" if (message.buffers) {\n",
" for (const buffer of message.buffers) {\n",
" buffers.push(buffer);\n",
" downloaded += buffer.byteLength;\n",
" progress.value = downloaded;\n",
" }\n",
" }\n",
" }\n",
" const blob = new Blob(buffers, {type: 'application/binary'});\n",
" const a = document.createElement('a');\n",
" a.href = window.URL.createObjectURL(blob);\n",
" a.download = filename;\n",
" div.appendChild(a);\n",
" a.click();\n",
" div.remove();\n",
" }\n",
" "
],
"text/plain": [
"<IPython.core.display.Javascript object>"
]
},
"metadata": {
"tags": []
}
},
{
"output_type": "display_data",
"data": {
"application/javascript": [
"download(\"download_65806ac7-0966-4eba-98e8-27bb741b5870\", \"newDevResultsSpecifically_all_8examples.pkl\", 79220)"
],
"text/plain": [
"<IPython.core.display.Javascript object>"
]
},
"metadata": {
"tags": []
}
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "M6JUOi7HHaJm",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"outputId": "b05f3df0-a9ad-489b-e530-4de92319edaa"
},
"source": [
"devDf.head()"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>s1</th>\n",
" <th>s2</th>\n",
" <th>pos</th>\n",
" <th>target</th>\n",
" <th>pred</th>\n",
" <th>actual</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Room and board .</td>\n",
" <td>He nailed boards across the windows .</td>\n",
" <td>N</td>\n",
" <td>board</td>\n",
" <td>No</td>\n",
" <td>F</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Circulate a rumor .</td>\n",
" <td>This letter is being circulated among the facu...</td>\n",
" <td>V</td>\n",
" <td>circulate</td>\n",
" <td>No</td>\n",
" <td>F</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Hook a fish .</td>\n",
" <td>He hooked a snake accidentally , and was so sc...</td>\n",
" <td>V</td>\n",
" <td>hook</td>\n",
" <td>Yes</td>\n",
" <td>T</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>For recreation he wrote poetry and solved cros...</td>\n",
" <td>Drug abuse is often regarded as a form of recr...</td>\n",
" <td>N</td>\n",
" <td>recreation</td>\n",
" <td>Yes</td>\n",
" <td>T</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Making a hobby of domesticity .</td>\n",
" <td>A royal family living in unpretentious domesti...</td>\n",
" <td>N</td>\n",
" <td>domesticity</td>\n",
" <td>Yes</td>\n",
" <td>F</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" s1 ... actual\n",
"0 Room and board . ... F\n",
"1 Circulate a rumor . ... F\n",
"2 Hook a fish . ... T\n",
"3 For recreation he wrote poetry and solved cros... ... T\n",
"4 Making a hobby of domesticity . ... F\n",
"\n",
"[5 rows x 6 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 123
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RMbeh9h_SXVy",
"colab_type": "text"
},
"source": [
"Convert labels to the WiC labels"
]
},
{
"cell_type": "code",
"metadata": {
"id": "Qkj00g8zOxuq",
"colab_type": "code",
"colab": {}
},
"source": [
"devDf[\"pred\"] = devDf[\"pred\"].apply(lambda x: \"T\" if x.strip() ==\"Yes\" else \"F\")"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "mZDo2UViSaC1",
"colab_type": "text"
},
"source": [
"67.24 on dev overall"
]
},
{
"cell_type": "code",
"metadata": {
"id": "AxnOkF0b8G2P",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "6a5591ed-673f-46d2-f4a2-40720e4949c7"
},
"source": [
"tmp = devDf.copy()\n",
"tmp[\"accurate\"] = tmp[\"actual\"] == tmp[\"pred\"]\n",
"tmp[\"accurate\"].sum()/len(tmp)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.6927899686520376"
]
},
"metadata": {
"tags": []
},
"execution_count": 125
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ryj3aNwCSb8h",
"colab_type": "text"
},
"source": [
"70 on the nouns"
]
},
{
"cell_type": "code",
"metadata": {
"id": "nsrx68C3OrQ5",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "ffb559d2-a0a7-4228-9d6d-02a423457397"
},
"source": [
"tmp = devDf[devDf.pos==\"N\"].copy()\n",
"tmp[\"accurate\"] = tmp[\"actual\"] == tmp[\"pred\"]\n",
"tmp[\"accurate\"].sum()/len(tmp)\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.7164556962025317"
]
},
"metadata": {
"tags": []
},
"execution_count": 126
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LfixNMBPSdkk",
"colab_type": "text"
},
"source": [
"62 on verbs"
]
},
{
"cell_type": "code",
"metadata": {
"id": "91CpA465OrJF",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "1b91cde0-d3fe-427e-adcb-793759ac6874"
},
"source": [
"tmp = devDf[devDf.pos==\"V\"].copy()\n",
"tmp[\"accurate\"] = tmp[\"actual\"] == tmp[\"pred\"]\n",
"tmp[\"accurate\"].sum()/len(tmp)\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.654320987654321"
]
},
"metadata": {
"tags": []
},
"execution_count": 127
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nSYCMotZSfBX",
"colab_type": "text"
},
"source": [
"59% accuracy on True match ones"
]
},
{
"cell_type": "code",
"metadata": {
"id": "Y6s-Jj1IOvg_",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "97c03661-7269-4feb-ef87-9bf8baf06583"
},
"source": [
"tmp = devDf[devDf.actual==\"T\"].copy()\n",
"tmp[\"accurate\"] = tmp[\"actual\"] == tmp[\"pred\"]\n",
"tmp[\"accurate\"].sum()/len(tmp)\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.7115987460815048"
]
},
"metadata": {
"tags": []
},
"execution_count": 128
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NFDcsJuLSjqv",
"colab_type": "text"
},
"source": [
"74% accuracy on False match ones"
]
},
{
"cell_type": "code",
"metadata": {
"id": "FU3RVeBNOxGd",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "7aec61d7-d637-43d1-8eec-2f6cbda88529"
},
"source": [
"tmp = devDf[devDf.actual==\"F\"].copy()\n",
"tmp[\"accurate\"] = tmp[\"actual\"] == tmp[\"pred\"]\n",
"tmp[\"accurate\"].sum()/len(tmp)\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.6739811912225705"
]
},
"metadata": {
"tags": []
},
"execution_count": 129
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "vGwWtYarPBi2",
"colab_type": "code",
"colab": {}
},
"source": [
""
],
"execution_count": null,
"outputs": []
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment