Last active
September 15, 2016 09:01
-
-
Save p-baleine/5ccdf96ded0af4965a257d087827bb9c to your computer and use it in GitHub Desktop.
Generalizing and Hybridizing Count-based and Neural Language Modelsのメモ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "# Generalizing and Hybridizing Count-based and Neural Language Modelsのメモ\n", | |
| "\n", | |
| "> ニューラル言語モデルとn-gram言語モデルを１つの統一された枠組みで定式化し、この枠組に基づいてそれぞれのモデルを改良する論文をアップしておきました： http://arxiv.org/abs/1606.00499 。言語モデルや言語処理の機械学習モデルに興味がある人はぜひご覧ください！\n", | |
| "\n", | |
| "[by @neubig](https://twitter.com/neubig/status/738620471852204034)\n", | |
| "\n", | |
| "---" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## 概要\n", | |
| "\n", | |
| "言語モデル(LM)は大きく以下に大分される:\n", | |
| "\n", | |
| "* 出現頻度に基づく$n$-gramモデル(count-based $n$-gram models): スケーラビリティとテスト可用性に優れている\n", | |
| "* ニューラルLM(neural LM): しばしば優れたパフォーマンスを達成する\n", | |
| "\n", | |
| "この論文では、これらのLMを統一された枠組みで定式化する方法と、この枠組を用いてLMを組み合わせたハイブリッドモデルを提案している。出現頻度に基づくLMとニューラルLMの長所を兼ね合わせたハイブリッドモデルが得られることが期待される。" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "collapsed": true | |
| }, | |
| "source": [ | |
| "## MODLM\n", | |
| "\n", | |
| "まずLMの定式化について。\n", | |
| "\n", | |
| "文脈${\\bf c}$を与えられ、複数の確率分布$P_k(w_i|{\\bf c})$に基いて文章$w_i$中の次の単語の確率分布:\n", | |
| "\n", | |
| "$$\n", | |
| "P(w_i|{\\bf c}) = \\sum_{k=1}^K \\lambda_k({\\bf c}) P_k(w_i|{\\bf c}) \\tag{1}\n", | |
| "$$\n", | |
| "\n", | |
| "を与えるLMを全てMODLM(_mixture of distributions LM_)と定義する。ここで$\\lambda_k({\\bf c})$は各確率分布の重みを定義する関数で、和が1になるように正規化されている($\\sum_{k=1}^K \\lambda_k({\\bf c}) = 1$)。\n", | |
| "\n", | |
| "従来上記式は複数の$n$-gram LMをモデリングするために用いられてきたが、この論文ではこの式を順伝播型や再帰型LMにも適用する。こうすることで、単一の数式で色々なLMを取り扱えるだけでなく、$n$-gramとニューラルモデルの長所を組み合わせた新しいモデルを模索することができるようになる、論文中で提案されるハイブリッドモデルは:\n", | |
| "\n", | |
| "* neurally interpolated $n$-gram LMs: $n$-gramモデルの線形補間の重みをニューラルネットワークを用いて学習する\n", | |
| "* neural/$n$-gram hybrid LMs: 出現品語に基づく$n$-gramモデルとニューラルモデルを足し合わせることで、膨大な外部データ・ソースをニューラルLMに与えることができるようになる\n", | |
| "\n", | |
| "ここで式(1)を以下の通り変形しておく:\n", | |
| "\n", | |
| "$$\n", | |
| "{\\bf p}_{\\bf c}^T = D_{\\bf c} {\\bf \\lambda}_{\\bf c}^T\n", | |
| "$$\n", | |
| "\n", | |
| "これは以下式と同じ:\n", | |
| "\n", | |
| "$$\n", | |
| "\\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " p_1 \\\\\n", | |
| " p_2 \\\\\n", | |
| " \\vdots \\\\\n", | |
| " p_J\n", | |
| " \\end{array}\n", | |
| "\\right] = \\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " d_{1, 1} & d_{1, 2} & \\cdots & d_{1, K} \\\\\n", | |
| " d_{2, 1} & d_{2, 2} & \\cdots & d_{2, K} \\\\\n", | |
| " \\vdots & \\vdots & \\ddots & \\vdots \\\\\n", | |
| " d_{J, 1} & d_{J, 2} & \\cdots & d_{J, K}\n", | |
| " \\end{array}\n", | |
| "\\right] \\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " \\lambda_1 \\\\\n", | |
| " \\lambda_2 \\\\\n", | |
| " \\vdots \\\\\n", | |
| " \\lambda_J\n", | |
| " \\end{array}\n", | |
| "\\right]\n", | |
| "$$\n", | |
| "\n", | |
| "${\\bf p}_{\\bf c}$は長さが語彙数$J$のベクトルで、要素$p_{{\\bf c}, j}$は$P(w_i = j | {\\bf c})$に対応している。${\\lambda}_{\\bf c}$は長さが$K$のベクトルで分布の混合の重みを保持している。$D_{\\bf c}$は$J$行$K$列の行列で要素$d_{{\\bf c},j,k}$は確率$P_k(w_i = j|{\\bf c})$に対応している。\n", | |
| "\n", | |
| "以下既存のLMについて上記式で表現していく。\n", | |
| "\n", | |
| "### $n$-gram LM\n", | |
| "\n", | |
| "$n$-gramモデルは次の単語をそれまでの$N-1$個の単語に基いて予測する。文脈${\\bf c}$は$w_{i-N+1}^{i-1}$で表され、次の単語$P(w_i|w_{i-N+1}^{i-1})$は最尤推定法(maximum-likelihood; ML)を用いて以下の通り表せる:\n", | |
| "\n", | |
| "$$\n", | |
| "P_{ML}(w_i | w_{i-N+1}^{i-1}) = \\frac{ c(w_{i-N+1}^{i}) } { c(w_{i-N+1}^{i-1} }\n", | |
| "$$\n", | |
| "\n", | |
| "但しここで低頻度$n$-gramの問題を考慮する必要がある。$n$-gramの頻度が0の場合$n$-gramの確率も0になってしまう。これに対応するために線形補間を用いる、最も簡単な静的な線形補間は:\n", | |
| "\n", | |
| "$$\n", | |
| "P(w_i | w_{i-N+1}^{i-1}) = \\sum_{n=1}^N \\lambda_{S,n} P_{ML}(w_i | w_{i-N+1}^{i-1})\n", | |
| "$$\n", | |
| "\n", | |
| "ここで$\\lambda_{S,n}$は分布$P_{ML}(w_i | w_{i-N+1}^{i-1})$に掛けられる重みを表している。これは以下式と同じ。\n", | |
| "\n", | |
| "$$\n", | |
| "\\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " p_1 \\\\\n", | |
| " p_2 \\\\\n", | |
| " \\vdots \\\\\n", | |
| " p_J\n", | |
| " \\end{array}\n", | |
| "\\right] = \\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " d_{1, 1} & d_{1, 2} & \\cdots & d_{1, N} \\\\\n", | |
| " d_{2, 1} & d_{2, 2} & \\cdots & d_{2, N} \\\\\n", | |
| " \\vdots & \\vdots & \\ddots & \\vdots \\\\\n", | |
| " d_{J, 1} & d_{J, 2} & \\cdots & d_{J, N}\n", | |
| " \\end{array}\n", | |
| "\\right] \\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " \\lambda_1 \\\\\n", | |
| " \\lambda_2 \\\\\n", | |
| " \\vdots \\\\\n", | |
| " \\lambda_N\n", | |
| " \\end{array}\n", | |
| "\\right]\n", | |
| "$$\n", | |
| "\n", | |
| "論文中では静的な線形補間の他に、ヒューリスティックな頻度計算の方法やModified Kneser-Neyの方法について触れられている。\n", | |
| "\n", | |
| "### ニューラルLM\n", | |
| "\n", | |
| "#### 順伝播型ニューラルネットワークLM\n", | |
| "\n", | |
| "順伝播型LMもそれまでの単語から次の単語を予測する。文脈$w_{i-N+1}^{i-1}$をベクトル表現${\\bf r}_{i-N+1}^{i-1}$に変換した上でこれを連結したベクトルを${\\bf q}$とすると、確率を表す${\\bf p}$は以下の通り表せる:\n", | |
| "\n", | |
| "$$\n", | |
| "\\begin{align}\n", | |
| "{\\bf h} &= tanh({\\bf q}W_q + {\\bf b}_q) \\\\\n", | |
| "{\\bf p} &= softmax({\\bf h}W_s + {\\bf b}_s)\n", | |
| "\\end{align}\n", | |
| "$$\n", | |
| "\n", | |
| "ここで$W_q$と${\\bf b}_q$、$W_s$、${\\bf b}_s$はネットワークのパラメーターを表している。\n", | |
| "\n", | |
| "このモデルは${\\bf p}$を直接予測していて混合重み${\\bf \\lambda}$の概念が登場してこないのでMODLMと解釈できないように考えられるが、${\\bf p}$を直接計算する代わりに重み${\\bf \\lambda} = softmax({\\bf h}W_s + {\\bf b}_s)$を計算して、また分布行列$D$の代わりに単位行列を用いることで以下の通りMODLMとして定義することができる。\n", | |
| "\n", | |
| "$$\n", | |
| "\\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " p_1 \\\\\n", | |
| " p_2 \\\\\n", | |
| " \\vdots \\\\\n", | |
| " p_J\n", | |
| " \\end{array}\n", | |
| "\\right] = \\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " 1 & 0 & \\cdots & 0 \\\\\n", | |
| " 0 & 1 & \\cdots & 0 \\\\\n", | |
| " \\vdots & \\vdots & \\ddots & \\vdots \\\\\n", | |
| " 0 & 0 & \\cdots & 1\n", | |
| " \\end{array}\n", | |
| "\\right] \\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " \\lambda_1 \\\\\n", | |
| " \\lambda_2 \\\\\n", | |
| " \\vdots \\\\\n", | |
| " \\lambda_J\n", | |
| " \\end{array}\n", | |
| "\\right]\n", | |
| "$$\n", | |
| "\n", | |
| "#### 再帰型ニューラルネットワークLM\n", | |
| "\n", | |
| "再帰型ニューラルネットワークを用いるLMはそれまでの単語に加えて、それまでの文章を要約する隠れ状態を使って次の単語を予測する。\n", | |
| "\n", | |
| "$$\n", | |
| "{\\bf h_i} = tanh({\\bf q}_i W_q + {\\bf h}_{i-1}W_h + {\\bf b}_q)\n", | |
| "$$\n", | |
| "\n", | |
| "ここで${\\bf q}_i$は現在の入力ベクトルで、${\\bf h}_{i-1}$は一つ前のタイムステップの隠れ状態を表している。再帰型ニューラルネットワークLMも順伝播型LMと同じように${\\bf p}$を直接予測する代わりに${\\bf \\lambda}$を予測することでMODLMとして表現できる。" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## MODLMのハイブリッドモデル\n", | |
| "\n", | |
| "### Neurally Interpolated $n$-gramモデル\n", | |
| "\n", | |
| "Neurally Interpolated $n$-gramモデルは以下の通り定義される。\n", | |
| "\n", | |
| "$$\n", | |
| "\\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " p_1 \\\\\n", | |
| " p_2 \\\\\n", | |
| " \\vdots \\\\\n", | |
| " p_J\n", | |
| " \\end{array}\n", | |
| "\\right] = \\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " d_{1, 1} & d_{1, 2} & \\cdots & d_{1, N} \\\\\n", | |
| " d_{2, 1} & d_{2, 2} & \\cdots & d_{2, N} \\\\\n", | |
| " \\vdots & \\vdots & \\ddots & \\vdots \\\\\n", | |
| " d_{J, 1} & d_{J, 2} & \\cdots & d_{J, N}\n", | |
| " \\end{array}\n", | |
| "\\right] \\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " \\lambda_1 \\\\\n", | |
| " \\lambda_2 \\\\\n", | |
| " \\vdots \\\\\n", | |
| " \\lambda_N\n", | |
| " \\end{array}\n", | |
| "\\right]\n", | |
| "$$\n", | |
| "\n", | |
| "ここで$D$は$n$-gram LMと同じ行列である。そして${\\bf \\lambda}({\\bf c})$をニューラルネットワークモデルで学習する。${\\bf \\lambda}({\\bf c})$はデータから学習されるため、ヒューリスティックな線形補間より賢い補完関数を学習できることが期待できる。加えてニューラルネットワークは語彙$J$の代わりに$N$の分布からのsoftmaxを計算すれば済むので、通常のニューラルネットワークLMと比較してより効率的に学習できることが期待できる。\n", | |
| "\n", | |
| "### Neural/$n$-gram Hybridモデル\n", | |
| "\n", | |
| "Neural/$n$-gram Hybridモデルは以下の通り定義される。\n", | |
| "\n", | |
| "$$\n", | |
| "\\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " p_1 \\\\\n", | |
| " p_2 \\\\\n", | |
| " \\vdots \\\\\n", | |
| " p_J\n", | |
| " \\end{array}\n", | |
| "\\right] = \\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " d_{1, 1} & d_{1, 2} & \\cdots & d_{1, N} & 1 & 0 & \\cdots & 0 \\\\\n", | |
| " d_{2, 1} & d_{2, 2} & \\cdots & d_{2, N} & 0 & 1 & \\cdots & 0 \\\\\n", | |
| " \\vdots & \\vdots & \\ddots & \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\\n", | |
| " d_{J, 1} & d_{J, 2} & \\cdots & d_{J, N} & 0 & 0 & \\cdots & 1\n", | |
| " \\end{array}\n", | |
| "\\right] \\left[\n", | |
| " \\begin{array}{rrr}\n", | |
| " \\lambda_1 \\\\\n", | |
| " \\lambda_2 \\\\\n", | |
| " \\vdots \\\\\n", | |
| " \\lambda_{J+N}\n", | |
| " \\end{array}\n", | |
| "\\right]\n", | |
| "$$\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": true | |
| }, | |
| "outputs": [], | |
| "source": [] | |
| } | |
| ], | |
| "metadata": { | |
| "kernelspec": { | |
| "display_name": "Python 3", | |
| "language": "python", | |
| "name": "python3" | |
| }, | |
| "language_info": { | |
| "codemirror_mode": { | |
| "name": "ipython", | |
| "version": 3 | |
| }, | |
| "file_extension": ".py", | |
| "mimetype": "text/x-python", | |
| "name": "python", | |
| "nbconvert_exporter": "python", | |
| "pygments_lexer": "ipython3", | |
| "version": "3.5.1" | |
| } | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 0 | |
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
まだ途中