Skip to content

Instantly share code, notes, and snippets.

@kyo-takano

kyo-takano/rawllm-predictability-example-2.ipynb

Created August 19, 2026 06:57
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save kyo-takano/21cdf90ad34920f787535145fbb7beb4 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save kyo-takano/21cdf90ad34920f787535145fbb7beb4 to your computer and use it in GitHub Desktop.
Download ZIP
rawllm:predictability.example-2.ipynb
Display the source blob
Display the rendered blob
Raw
rawllm-predictability-example-2.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/kyo-takano/21cdf90ad34920f787535145fbb7beb4/rawllm-predictability-example-2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"# `rawllm.com/predictability` 応用例: ツイート候補を絞り込む\n",
"\n",
"[rawllm.com/predictability](rawllm.com/predictability) を発表するツイートを例として、条件となるプロンプトはなく `completion` 側を評価対象とする使用例を紹介する。\n",
"\n",
"共通のpromptを使用し、同じ意味を伝達する複数の原稿の圧縮率（BPB）を評価する。\n",
"そうすることで、情報論的に最も読みやすそうな（LLMにとって予測しやすい）文章を探すことができる。"
],
"metadata": {
"id": "Ri-kdQOULll0"
}
},
{
"cell_type": "markdown",
"metadata": {
"id": "43d947bc"
},
"source": [
"## Evaluation loop"
]
},
{
"cell_type": "code",
"source": [
"import time\n",
"import itertools\n",
"import numpy as np\n",
"import pandas as pd\n",
"import requests\n",
"from scipy import stats\n",
"\n",
"\n",
"def compute_bpb(prompt, completion) -> float:\n",
" return requests.post(\n",
" \"https://rawllm.com/api/predictability\",\n",
" headers={\"content-type\": \"application/json\"},\n",
" json={\"prompt\": prompt, \"completion\": completion},\n",
" ).json()[\"bits_per_byte\"]\n",
"\n",
"\n",
"candidates = [\n",
" # 手書きオリジナル\n",
" \"ローカルLLMを使い、「ある文脈におけるテキストの予測がどれだけ困難か？」を評価するAPIを無料公開します。\\n\\nhttps://\", # 実際のURLを正確に予測できることは重要でないため省略\n",
" \"ローカルLLMを使い、「文脈に続くテキストがどれだけ予測しやすいか？」を評価するAPIを無料公開します。\\n\\nhttps://\",\n",
" # 以下variantsはLLMで生成\n",
" \"ローカルLLMを使い、与えた文脈に続くテキストが「どの程度予測しやすいか」を評価するAPIを無料で公開します。\\n\\nhttps://\",\n",
" \"「この文脈に続く文章はどれくらい予測が難しいか？」をローカルLLMで評価できるAPIを無料公開します。\\n\\nhttps://\",\n",
" \"文脈を与えると、その続きのテキストの「予測しにくさ」をローカルLLMが数値化してくれる——そんなAPIを無料で公開します。\\n\\nhttps://\",\n",
" \"入力した文脈に対して、「その先の文章の予測の難しさ」をローカルLLMで算出するAPIを無料で公開しました。\\n\\nhttps://\",\n",
" \"ローカルLLMで「文脈に続くテキストの予測しづらさ」を測るAPIを無料公開しました。\\n\\nhttps://\",\n",
" \"文脈に続くテキストの「意外性」をローカルLLMで評価するAPIを無料公開します。\\n\\nhttps://\",\n",
" \"文脈の続きがどれだけ予測しにくいかを数値化する無料APIを公開しました（ローカルLLM使用）。\\n\\nhttps://\",\n",
" \"ローカルLLMを使った「テキスト予測の難易度」評価APIを無料で公開します。\\n\\nhttps://\",\n",
" \"「続きのテキストの予測がどれだけ難しいか」をローカルLLMが評価してくれるAPIを無料公開しました。\\n\\nhttps://\",\n",
" \"文脈に続くテキストの予測の難しさを評価するローカルLLMベースのAPIを無料で公開します。\\n\\nhttps://\",\n",
" \"与えられた文脈に対して、次のテキストがどれほど予測しにくいかをローカルLLMで測るAPIを無料公開しました。\\n\\nhttps://\",\n",
" \"ローカルLLMを使い、文脈に続くテキストの「予測のしにくさ」を評価するAPIを無料で公開しました。\\n\\nhttps://\",\n",
" \"ローカルLLMで「文脈に続くテキストの予測困難度」を算出するAPIを無料公開します。\\n\\nhttps://\",\n",
" \"文脈に続くテキストがどれだけ予測しにくいかを、ローカルLLMで評価するAPIを無料公開します。\\n\\nhttps://\",\n",
" \"ローカルLLMによって、文脈に続くテキストの予測の難しさを評価するAPIを無料公開しました。\\n\\nhttps://\",\n",
"]\n",
"records = []\n",
"\n",
"for candidate_id, candidate in enumerate(candidates):\n",
" records.append(\n",
" {\n",
" \"candidate\": candidate,\n",
" \"candidate_id\": candidate_id,\n",
" \"bpb\": compute_bpb(\"```tweet\\n\", candidate), # Evaluate the completion itself with a fixed context\n",
" }\n",
" )\n",
"\n",
"df = pd.DataFrame(records)\n",
"df"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 582
},
"id": "Nf8vewe62OLV",
"outputId": "eef86de6-96a7-45a0-daa2-b3d2f4ea8ee0"
},
"execution_count": 1,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
" candidate candidate_id bpb\n",
"0 ローカルLLMを使い、「ある文脈におけるテキストの予測がどれだけ困難か？」を評価するAPIを... 0 1.120165\n",
"1 ローカルLLMを使い、「文脈に続くテキストがどれだけ予測しやすいか？」を評価するAPIを無料... 1 1.122144\n",
"2 ローカルLLMを使い、与えた文脈に続くテキストが「どの程度予測しやすいか」を評価するAPIを... 2 1.147192\n",
"3 「この文脈に続く文章はどれくらい予測が難しいか？」をローカルLLMで評価できるAPIを無料公... 3 1.164281\n",
"4 文脈を与えると、その続きのテキストの「予測しにくさ」をローカルLLMが数値化してくれる——そ... 4 1.251490\n",
"5 入力した文脈に対して、「その先の文章の予測の難しさ」をローカルLLMで算出するAPIを無料で... 5 1.189876\n",
"6 ローカルLLMで「文脈に続くテキストの予測しづらさ」を測るAPIを無料公開しました。\\n\\n... 6 1.365482\n",
"7 文脈に続くテキストの「意外性」をローカルLLMで評価するAPIを無料公開します。\\n\\nht... 7 1.343719\n",
"8 文脈の続きがどれだけ予測しにくいかを数値化する無料APIを公開しました（ローカルLLM使用）... 8 1.310532\n",
"9 ローカルLLMを使った「テキスト予測の難易度」評価APIを無料で公開します。\\n\\nhttp... 9 1.209343\n",
"10 「続きのテキストの予測がどれだけ難しいか」をローカルLLMが評価してくれるAPIを無料公開し... 10 1.159741\n",
"11 文脈に続くテキストの予測の難しさを評価するローカルLLMベースのAPIを無料で公開します。\\... 11 1.238605\n",
"12 与えられた文脈に対して、次のテキストがどれほど予測しにくいかをローカルLLMで測るAPIを無... 12 1.215695\n",
"13 ローカルLLMを使い、文脈に続くテキストの「予測のしにくさ」を評価するAPIを無料で公開しま... 13 1.167243\n",
"14 ローカルLLMで「文脈に続くテキストの予測困難度」を算出するAPIを無料公開します。\\n\\n... 14 1.293014\n",
"15 文脈に続くテキストがどれだけ予測しにくいかを、ローカルLLMで評価するAPIを無料公開します... 15 1.204615\n",
"16 ローカルLLMによって、文脈に続くテキストの予測の難しさを評価するAPIを無料公開しました。... 16 1.181887"
],
"text/html": [
"\n",
" <div id=\"df-e604716d-0a1b-4b28-aac2-b605975a718a\" class=\"colab-df-container\">\n",
" <div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>candidate</th>\n",
" <th>candidate_id</th>\n",
" <th>bpb</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>ローカルLLMを使い、「ある文脈におけるテキストの予測がどれだけ困難か？」を評価するAPIを...</td>\n",
" <td>0</td>\n",
" <td>1.120165</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>ローカルLLMを使い、「文脈に続くテキストがどれだけ予測しやすいか？」を評価するAPIを無料...</td>\n",
" <td>1</td>\n",
" <td>1.122144</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>ローカルLLMを使い、与えた文脈に続くテキストが「どの程度予測しやすいか」を評価するAPIを...</td>\n",
" <td>2</td>\n",
" <td>1.147192</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>「この文脈に続く文章はどれくらい予測が難しいか？」をローカルLLMで評価できるAPIを無料公...</td>\n",
" <td>3</td>\n",
" <td>1.164281</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>文脈を与えると、その続きのテキストの「予測しにくさ」をローカルLLMが数値化してくれる——そ...</td>\n",
" <td>4</td>\n",
" <td>1.251490</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>入力した文脈に対して、「その先の文章の予測の難しさ」をローカルLLMで算出するAPIを無料で...</td>\n",
" <td>5</td>\n",
" <td>1.189876</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>ローカルLLMで「文脈に続くテキストの予測しづらさ」を測るAPIを無料公開しました。\\n\\n...</td>\n",
" <td>6</td>\n",
" <td>1.365482</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>文脈に続くテキストの「意外性」をローカルLLMで評価するAPIを無料公開します。\\n\\nht...</td>\n",
" <td>7</td>\n",
" <td>1.343719</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>文脈の続きがどれだけ予測しにくいかを数値化する無料APIを公開しました（ローカルLLM使用）...</td>\n",
" <td>8</td>\n",
" <td>1.310532</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>ローカルLLMを使った「テキスト予測の難易度」評価APIを無料で公開します。\\n\\nhttp...</td>\n",
" <td>9</td>\n",
" <td>1.209343</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>「続きのテキストの予測がどれだけ難しいか」をローカルLLMが評価してくれるAPIを無料公開し...</td>\n",
" <td>10</td>\n",
" <td>1.159741</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>文脈に続くテキストの予測の難しさを評価するローカルLLMベースのAPIを無料で公開します。\\...</td>\n",
" <td>11</td>\n",
" <td>1.238605</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>与えられた文脈に対して、次のテキストがどれほど予測しにくいかをローカルLLMで測るAPIを無...</td>\n",
" <td>12</td>\n",
" <td>1.215695</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>ローカルLLMを使い、文脈に続くテキストの「予測のしにくさ」を評価するAPIを無料で公開しま...</td>\n",
" <td>13</td>\n",
" <td>1.167243</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>ローカルLLMで「文脈に続くテキストの予測困難度」を算出するAPIを無料公開します。\\n\\n...</td>\n",
" <td>14</td>\n",
" <td>1.293014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>文脈に続くテキストがどれだけ予測しにくいかを、ローカルLLMで評価するAPIを無料公開します...</td>\n",
" <td>15</td>\n",
" <td>1.204615</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>ローカルLLMによって、文脈に続くテキストの予測の難しさを評価するAPIを無料公開しました。...</td>\n",
" <td>16</td>\n",
" <td>1.181887</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" <div class=\"colab-df-buttons\">\n",
"\n",
" <div class=\"colab-df-container\">\n",
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e604716d-0a1b-4b28-aac2-b605975a718a')\"\n",
" title=\"Convert this dataframe to an interactive table.\"\n",
" style=\"display:none;\">\n",
"\n",
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
" </svg>\n",
" </button>\n",
"\n",
" <style>\n",
" .colab-df-container {\n",
" display:flex;\n",
" gap: 12px;\n",
" }\n",
"\n",
" .colab-df-convert {\n",
" background-color: #E8F0FE;\n",
" border: none;\n",
" border-radius: 50%;\n",
" cursor: pointer;\n",
" display: none;\n",
" fill: #1967D2;\n",
" height: 32px;\n",
" padding: 0 0 0 0;\n",
" width: 32px;\n",
" }\n",
"\n",
" .colab-df-convert:hover {\n",
" background-color: #E2EBFA;\n",
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
" fill: #174EA6;\n",
" }\n",
"\n",
" .colab-df-buttons div {\n",
" margin-bottom: 4px;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert {\n",
" background-color: #3B4455;\n",
" fill: #D2E3FC;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert:hover {\n",
" background-color: #434B5C;\n",
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
" fill: #FFFFFF;\n",
" }\n",
" </style>\n",
"\n",
" <script>\n",
" const buttonEl =\n",
" document.querySelector('#df-e604716d-0a1b-4b28-aac2-b605975a718a button.colab-df-convert');\n",
" buttonEl.style.display =\n",
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
"\n",
" async function convertToInteractive(key) {\n",
" const element = document.querySelector('#df-e604716d-0a1b-4b28-aac2-b605975a718a');\n",
" const dataTable =\n",
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
" [key], {});\n",
" if (!dataTable) return;\n",
"\n",
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
" + ' to learn more about interactive tables.';\n",
" element.innerHTML = '';\n",
" dataTable['output_type'] = 'display_data';\n",
" await google.colab.output.renderOutput(dataTable, element);\n",
" const docLink = document.createElement('div');\n",
" docLink.innerHTML = docLinkHtml;\n",
" element.appendChild(docLink);\n",
" }\n",
" </script>\n",
" </div>\n",
"\n",
"\n",
" <div id=\"id_c0df2625-613d-48dc-96d3-a5a1b2393406\">\n",
" <style>\n",
" .colab-df-generate {\n",
" background-color: #E8F0FE;\n",
" border: none;\n",
" border-radius: 50%;\n",
" cursor: pointer;\n",
" display: none;\n",
" fill: #1967D2;\n",
" height: 32px;\n",
" padding: 0 0 0 0;\n",
" width: 32px;\n",
" }\n",
"\n",
" .colab-df-generate:hover {\n",
" background-color: #E2EBFA;\n",
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
" fill: #174EA6;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-generate {\n",
" background-color: #3B4455;\n",
" fill: #D2E3FC;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-generate:hover {\n",
" background-color: #434B5C;\n",
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
" fill: #FFFFFF;\n",
" }\n",
" </style>\n",
" <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
" title=\"Generate code using this dataframe.\"\n",
" style=\"display:none;\">\n",
"\n",
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
" width=\"24px\">\n",
" <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
" </svg>\n",
" </button>\n",
" <script>\n",
" (() => {\n",
" const buttonEl =\n",
" document.querySelector('#id_c0df2625-613d-48dc-96d3-a5a1b2393406 button.colab-df-generate');\n",
" buttonEl.style.display =\n",
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
"\n",
" buttonEl.onclick = () => {\n",
" google.colab.notebook.generateWithVariable('df');\n",
" }\n",
" })();\n",
" </script>\n",
" </div>\n",
"\n",
" </div>\n",
" </div>\n"
],
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "dataframe",
"variable_name": "df",
"summary": "{\n \"name\": \"df\",\n \"rows\": 17,\n \"fields\": [\n {\n \"column\": \"candidate\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 17,\n \"samples\": [\n \"\\u30ed\\u30fc\\u30ab\\u30ebLLM\\u3092\\u4f7f\\u3044\\u3001\\u300c\\u3042\\u308b\\u6587\\u8108\\u306b\\u304a\\u3051\\u308b\\u30c6\\u30ad\\u30b9\\u30c8\\u306e\\u4e88\\u6e2c\\u304c\\u3069\\u308c\\u3060\\u3051\\u56f0\\u96e3\\u304b\\uff1f\\u300d\\u3092\\u8a55\\u4fa1\\u3059\\u308bAPI\\u3092\\u7121\\u6599\\u516c\\u958b\\u3057\\u307e\\u3059\\u3002\\n\\nhttps://\",\n \"\\u30ed\\u30fc\\u30ab\\u30ebLLM\\u3092\\u4f7f\\u3044\\u3001\\u300c\\u6587\\u8108\\u306b\\u7d9a\\u304f\\u30c6\\u30ad\\u30b9\\u30c8\\u304c\\u3069\\u308c\\u3060\\u3051\\u4e88\\u6e2c\\u3057\\u3084\\u3059\\u3044\\u304b\\uff1f\\u300d\\u3092\\u8a55\\u4fa1\\u3059\\u308bAPI\\u3092\\u7121\\u6599\\u516c\\u958b\\u3057\\u307e\\u3059\\u3002\\n\\nhttps://\",\n \"\\u5165\\u529b\\u3057\\u305f\\u6587\\u8108\\u306b\\u5bfe\\u3057\\u3066\\u3001\\u300c\\u305d\\u306e\\u5148\\u306e\\u6587\\u7ae0\\u306e\\u4e88\\u6e2c\\u306e\\u96e3\\u3057\\u3055\\u300d\\u3092\\u30ed\\u30fc\\u30ab\\u30ebLLM\\u3067\\u7b97\\u51fa\\u3059\\u308bAPI\\u3092\\u7121\\u6599\\u3067\\u516c\\u958b\\u3057\\u307e\\u3057\\u305f\\u3002\\n\\nhttps://\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"candidate_id\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 5,\n \"min\": 0,\n \"max\": 16,\n \"num_unique_values\": 17,\n \"samples\": [\n 0,\n 1,\n 5\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"bpb\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.07426731501956302,\n \"min\": 1.1201651752154642,\n \"max\": 1.3654817099575969,\n \"num_unique_values\": 17,\n \"samples\": [\n 1.1201651752154642,\n 1.122144180919067,\n 1.18987646035425\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}"
}
},
"metadata": {},
"execution_count": 1
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ecf78b76"
},
"source": [
"## 結果の分析\n",
"\n",
"BPB（Bits Per Byte）が低いほど、そのテキストはLLMにとって予測・圧縮しやすく、情報効率が高いことを示す。\n",
"したがって、各ツイート候補のBPBを算出し、最も自然で予測しやすい（＝読みやすい可能性が高い）表現を選択する。"
]
},
{
"cell_type": "code",
"source": [
"df = df.sort_values(\"bpb\", ascending=True).reset_index(drop=True)\n",
"display(df)\n",
"\n",
"print()\n",
"print(\"best candidate:\", df.candidate[0])"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 656
},
"id": "9pEvruxS1WkB",
"outputId": "9295fbb5-0c3f-41b1-be5e-f7af97a466a8"
},
"execution_count": 2,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
" candidate candidate_id bpb\n",
"0 ローカルLLMを使い、「ある文脈におけるテキストの予測がどれだけ困難か？」を評価するAPIを... 0 1.120165\n",
"1 ローカルLLMを使い、「文脈に続くテキストがどれだけ予測しやすいか？」を評価するAPIを無料... 1 1.122144\n",
"2 ローカルLLMを使い、与えた文脈に続くテキストが「どの程度予測しやすいか」を評価するAPIを... 2 1.147192\n",
"3 「続きのテキストの予測がどれだけ難しいか」をローカルLLMが評価してくれるAPIを無料公開し... 10 1.159741\n",
"4 「この文脈に続く文章はどれくらい予測が難しいか？」をローカルLLMで評価できるAPIを無料公... 3 1.164281\n",
"5 ローカルLLMを使い、文脈に続くテキストの「予測のしにくさ」を評価するAPIを無料で公開しま... 13 1.167243\n",
"6 ローカルLLMによって、文脈に続くテキストの予測の難しさを評価するAPIを無料公開しました。... 16 1.181887\n",
"7 入力した文脈に対して、「その先の文章の予測の難しさ」をローカルLLMで算出するAPIを無料で... 5 1.189876\n",
"8 文脈に続くテキストがどれだけ予測しにくいかを、ローカルLLMで評価するAPIを無料公開します... 15 1.204615\n",
"9 ローカルLLMを使った「テキスト予測の難易度」評価APIを無料で公開します。\\n\\nhttp... 9 1.209343\n",
"10 与えられた文脈に対して、次のテキストがどれほど予測しにくいかをローカルLLMで測るAPIを無... 12 1.215695\n",
"11 文脈に続くテキストの予測の難しさを評価するローカルLLMベースのAPIを無料で公開します。\\... 11 1.238605\n",
"12 文脈を与えると、その続きのテキストの「予測しにくさ」をローカルLLMが数値化してくれる——そ... 4 1.251490\n",
"13 ローカルLLMで「文脈に続くテキストの予測困難度」を算出するAPIを無料公開します。\\n\\n... 14 1.293014\n",
"14 文脈の続きがどれだけ予測しにくいかを数値化する無料APIを公開しました（ローカルLLM使用）... 8 1.310532\n",
"15 文脈に続くテキストの「意外性」をローカルLLMで評価するAPIを無料公開します。\\n\\nht... 7 1.343719\n",
"16 ローカルLLMで「文脈に続くテキストの予測しづらさ」を測るAPIを無料公開しました。\\n\\n... 6 1.365482"
],
"text/html": [
"\n",
" <div id=\"df-8a3b4840-cd2d-4450-9378-9b6147571318\" class=\"colab-df-container\">\n",
" <div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>candidate</th>\n",
" <th>candidate_id</th>\n",
" <th>bpb</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>ローカルLLMを使い、「ある文脈におけるテキストの予測がどれだけ困難か？」を評価するAPIを...</td>\n",
" <td>0</td>\n",
" <td>1.120165</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>ローカルLLMを使い、「文脈に続くテキストがどれだけ予測しやすいか？」を評価するAPIを無料...</td>\n",
" <td>1</td>\n",
" <td>1.122144</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>ローカルLLMを使い、与えた文脈に続くテキストが「どの程度予測しやすいか」を評価するAPIを...</td>\n",
" <td>2</td>\n",
" <td>1.147192</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>「続きのテキストの予測がどれだけ難しいか」をローカルLLMが評価してくれるAPIを無料公開し...</td>\n",
" <td>10</td>\n",
" <td>1.159741</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>「この文脈に続く文章はどれくらい予測が難しいか？」をローカルLLMで評価できるAPIを無料公...</td>\n",
" <td>3</td>\n",
" <td>1.164281</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>ローカルLLMを使い、文脈に続くテキストの「予測のしにくさ」を評価するAPIを無料で公開しま...</td>\n",
" <td>13</td>\n",
" <td>1.167243</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>ローカルLLMによって、文脈に続くテキストの予測の難しさを評価するAPIを無料公開しました。...</td>\n",
" <td>16</td>\n",
" <td>1.181887</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>入力した文脈に対して、「その先の文章の予測の難しさ」をローカルLLMで算出するAPIを無料で...</td>\n",
" <td>5</td>\n",
" <td>1.189876</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>文脈に続くテキストがどれだけ予測しにくいかを、ローカルLLMで評価するAPIを無料公開します...</td>\n",
" <td>15</td>\n",
" <td>1.204615</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>ローカルLLMを使った「テキスト予測の難易度」評価APIを無料で公開します。\\n\\nhttp...</td>\n",
" <td>9</td>\n",
" <td>1.209343</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>与えられた文脈に対して、次のテキストがどれほど予測しにくいかをローカルLLMで測るAPIを無...</td>\n",
" <td>12</td>\n",
" <td>1.215695</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>文脈に続くテキストの予測の難しさを評価するローカルLLMベースのAPIを無料で公開します。\\...</td>\n",
" <td>11</td>\n",
" <td>1.238605</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>文脈を与えると、その続きのテキストの「予測しにくさ」をローカルLLMが数値化してくれる——そ...</td>\n",
" <td>4</td>\n",
" <td>1.251490</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>ローカルLLMで「文脈に続くテキストの予測困難度」を算出するAPIを無料公開します。\\n\\n...</td>\n",
" <td>14</td>\n",
" <td>1.293014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>文脈の続きがどれだけ予測しにくいかを数値化する無料APIを公開しました（ローカルLLM使用）...</td>\n",
" <td>8</td>\n",
" <td>1.310532</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>文脈に続くテキストの「意外性」をローカルLLMで評価するAPIを無料公開します。\\n\\nht...</td>\n",
" <td>7</td>\n",
" <td>1.343719</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>ローカルLLMで「文脈に続くテキストの予測しづらさ」を測るAPIを無料公開しました。\\n\\n...</td>\n",
" <td>6</td>\n",
" <td>1.365482</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" <div class=\"colab-df-buttons\">\n",
"\n",
" <div class=\"colab-df-container\">\n",
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8a3b4840-cd2d-4450-9378-9b6147571318')\"\n",
" title=\"Convert this dataframe to an interactive table.\"\n",
" style=\"display:none;\">\n",
"\n",
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
" </svg>\n",
" </button>\n",
"\n",
" <style>\n",
" .colab-df-container {\n",
" display:flex;\n",
" gap: 12px;\n",
" }\n",
"\n",
" .colab-df-convert {\n",
" background-color: #E8F0FE;\n",
" border: none;\n",
" border-radius: 50%;\n",
" cursor: pointer;\n",
" display: none;\n",
" fill: #1967D2;\n",
" height: 32px;\n",
" padding: 0 0 0 0;\n",
" width: 32px;\n",
" }\n",
"\n",
" .colab-df-convert:hover {\n",
" background-color: #E2EBFA;\n",
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
" fill: #174EA6;\n",
" }\n",
"\n",
" .colab-df-buttons div {\n",
" margin-bottom: 4px;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert {\n",
" background-color: #3B4455;\n",
" fill: #D2E3FC;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert:hover {\n",
" background-color: #434B5C;\n",
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
" fill: #FFFFFF;\n",
" }\n",
" </style>\n",
"\n",
" <script>\n",
" const buttonEl =\n",
" document.querySelector('#df-8a3b4840-cd2d-4450-9378-9b6147571318 button.colab-df-convert');\n",
" buttonEl.style.display =\n",
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
"\n",
" async function convertToInteractive(key) {\n",
" const element = document.querySelector('#df-8a3b4840-cd2d-4450-9378-9b6147571318');\n",
" const dataTable =\n",
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
" [key], {});\n",
" if (!dataTable) return;\n",
"\n",
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
" + ' to learn more about interactive tables.';\n",
" element.innerHTML = '';\n",
" dataTable['output_type'] = 'display_data';\n",
" await google.colab.output.renderOutput(dataTable, element);\n",
" const docLink = document.createElement('div');\n",
" docLink.innerHTML = docLinkHtml;\n",
" element.appendChild(docLink);\n",
" }\n",
" </script>\n",
" </div>\n",
"\n",
"\n",
" <div id=\"id_fb402c57-bd86-4012-b18f-6a474c049193\">\n",
" <style>\n",
" .colab-df-generate {\n",
" background-color: #E8F0FE;\n",
" border: none;\n",
" border-radius: 50%;\n",
" cursor: pointer;\n",
" display: none;\n",
" fill: #1967D2;\n",
" height: 32px;\n",
" padding: 0 0 0 0;\n",
" width: 32px;\n",
" }\n",
"\n",
" .colab-df-generate:hover {\n",
" background-color: #E2EBFA;\n",
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
" fill: #174EA6;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-generate {\n",
" background-color: #3B4455;\n",
" fill: #D2E3FC;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-generate:hover {\n",
" background-color: #434B5C;\n",
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
" fill: #FFFFFF;\n",
" }\n",
" </style>\n",
" <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
" title=\"Generate code using this dataframe.\"\n",
" style=\"display:none;\">\n",
"\n",
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
" width=\"24px\">\n",
" <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
" </svg>\n",
" </button>\n",
" <script>\n",
" (() => {\n",
" const buttonEl =\n",
" document.querySelector('#id_fb402c57-bd86-4012-b18f-6a474c049193 button.colab-df-generate');\n",
" buttonEl.style.display =\n",
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
"\n",
" buttonEl.onclick = () => {\n",
" google.colab.notebook.generateWithVariable('df');\n",
" }\n",
" })();\n",
" </script>\n",
" </div>\n",
"\n",
" </div>\n",
" </div>\n"
],
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "dataframe",
"variable_name": "df",
"summary": "{\n \"name\": \"df\",\n \"rows\": 17,\n \"fields\": [\n {\n \"column\": \"candidate\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 17,\n \"samples\": [\n \"\\u30ed\\u30fc\\u30ab\\u30ebLLM\\u3092\\u4f7f\\u3044\\u3001\\u300c\\u3042\\u308b\\u6587\\u8108\\u306b\\u304a\\u3051\\u308b\\u30c6\\u30ad\\u30b9\\u30c8\\u306e\\u4e88\\u6e2c\\u304c\\u3069\\u308c\\u3060\\u3051\\u56f0\\u96e3\\u304b\\uff1f\\u300d\\u3092\\u8a55\\u4fa1\\u3059\\u308bAPI\\u3092\\u7121\\u6599\\u516c\\u958b\\u3057\\u307e\\u3059\\u3002\\n\\nhttps://\",\n \"\\u30ed\\u30fc\\u30ab\\u30ebLLM\\u3092\\u4f7f\\u3044\\u3001\\u300c\\u6587\\u8108\\u306b\\u7d9a\\u304f\\u30c6\\u30ad\\u30b9\\u30c8\\u304c\\u3069\\u308c\\u3060\\u3051\\u4e88\\u6e2c\\u3057\\u3084\\u3059\\u3044\\u304b\\uff1f\\u300d\\u3092\\u8a55\\u4fa1\\u3059\\u308bAPI\\u3092\\u7121\\u6599\\u516c\\u958b\\u3057\\u307e\\u3059\\u3002\\n\\nhttps://\",\n \"\\u30ed\\u30fc\\u30ab\\u30ebLLM\\u3092\\u4f7f\\u3044\\u3001\\u6587\\u8108\\u306b\\u7d9a\\u304f\\u30c6\\u30ad\\u30b9\\u30c8\\u306e\\u300c\\u4e88\\u6e2c\\u306e\\u3057\\u306b\\u304f\\u3055\\u300d\\u3092\\u8a55\\u4fa1\\u3059\\u308bAPI\\u3092\\u7121\\u6599\\u3067\\u516c\\u958b\\u3057\\u307e\\u3057\\u305f\\u3002\\n\\nhttps://\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"candidate_id\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 5,\n \"min\": 0,\n \"max\": 16,\n \"num_unique_values\": 17,\n \"samples\": [\n 0,\n 1,\n 13\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"bpb\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.07426731501956303,\n \"min\": 1.1201651752154642,\n \"max\": 1.3654817099575969,\n \"num_unique_values\": 17,\n \"samples\": [\n 1.1201651752154642,\n 1.122144180919067,\n 1.1672425413249725\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}"
}
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"\n",
"best candidate: ローカルLLMを使い、「ある文脈におけるテキストの予測がどれだけ困難か？」を評価するAPIを無料公開します。\n",
"\n",
"https://\n"
]
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b9e1fedd"
},
"source": [
"上位3件は、いずれも「ローカルLLMを使い、」で始まる候補（オリジナルまたはそれに類似するLLM合成案）で占められた。このことは、そうした表現が表層的な「意外性」を最も抑えている一方で、LLMにとって自然かつ理解しやすい文章であることを示唆している。\n",
"\n",
"ただし、固定されているのはcompletion側ではなくprompt側であるため、この結果は各候補のバイト長の影響を受けている（一般に長い方が有利となる）。したがって、この結果が示す数値はあくまで参考程度であること留意されたい。"
]
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment