Created
March 6, 2019 00:23
-
-
Save shotahorii/64e772fa6d5d767a1fc1bcc87ee40353 to your computer and use it in GitHub Desktop.
Multicollinearity
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "### Multicollinearity and VIF\n", | |
| "References: \n", | |
| "1. [多重共線性の問題と対策 (井上勝雄)](https://ci.nii.ac.jp/els/contentscinii_20171213203115.pdf?id=ART0000839130)\n", | |
| "2. [最小二乗法1 数学的性質](http://www2.toyo.ac.jp/~mihira/keizaitoukei2014/ols1.pdf)\n", | |
| "3. [計量経済学 参考資料 - 最小2乗パラメータ推定値の導出(3変数の場合) - 河田 正樹](http://www2.tokuyama-u.ac.jp/kawada/2013/econome/ols_parameter2.pdf)\n", | |
| "4. [(偏差)平方和 S(大文字)](http://nhxperia.hatenablog.jp/entry/2012/01/14/005142)\n", | |
| "5. [相関関係のと偏差平方和,偏差積和の関係](https://toukeigaku-jouhou.info/2015/08/26/446/)\n", | |
| "6. [Detecting Multicollinearity Using Variance Inflation Factors](https://newonlinecourses.science.psu.edu/stat501/node/347/)\n", | |
| "7. [分散拡大係数 - wikipedia](https://ja.wikipedia.org/wiki/%E5%88%86%E6%95%A3%E6%8B%A1%E5%A4%A7%E4%BF%82%E6%95%B0)\n", | |
| "\n", | |
| "**多重共線性とは？** \n", | |
| "重回帰分析において、説明変数間に強い相関関係がある(多重共線性が存在する)時、最小二乗法による推定ができない(変数間に完全な多重共線性がある場合)、あるいは推定値が不安定になる。 \n", | |
| " \n", | |
| "**完全な多重共線性があるケース** \n", | |
| "まず、変数間に完全な多重共線性がある場合に、なぜ最小二乗法による推定ができないのかを見ていく。(主にReference1)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 32, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/latex": [ | |
| "ここ(https://gist.github.com/shotahorii/037dc487ab3c70cc45af27fc9d03c9fb)で書いた通り、最小二乗法による推定量は\n", | |
| "正規方程式を解くことで得られる。<br>\n", | |
| "例えば、以下の2変数のモデルについて考える。<br><br>\n", | |
| "$$y_i = \\beta_0 + \\beta_1 x_{1i} + \\beta_2 x_{2i} + \\epsilon_i$$<br>\n", | |
| "$$i = 1,2,...,n$$<br><br>\n", | |
| "\n", | |
| "この時、正規方程式は以下である。<br><br>\n", | |
| "\n", | |
| "$$\\beta_0 \\sum_{i=1}^n 1 + \\beta_1 \\sum_{i=1}^n x_{1i} + \\beta_2 \\sum_{i=1}^n x_{2i} = \\sum_{i=1}^n y_i $$<br>\n", | |
| "$$\\beta_0 \\sum_{i=1}^n x_{1i} + \\beta_1 \\sum_{i=1}^n x_{1i}^2 + \\beta_2 \\sum_{i=1}^n x_{1i} x_{2i} = \\sum_{i=1}^n x_{1i} y_i $$<br>\n", | |
| "$$\\beta_0 \\sum_{i=1}^n x_{2i} + \\beta_1 \\sum_{i=1}^n x_{1i} x_{2i} + \\beta_2 \\sum_{i=1}^n x_{2i}^2 = \\sum_{i=1}^n x_{2i} y_i $$\n", | |
| "<br><br>\n", | |
| "\n", | |
| "ここで、説明変数$$x_{1i}$$と$$x_{2i}$$の間に完全な多重共線性が存在するとする。その場合、以下の関係が導かれる。\n", | |
| "<br><br>\n", | |
| "$$x_{2i} = \\alpha x_{1i}$$<br>\n", | |
| "$$i = 1,2,...,n$$<br><br>\n", | |
| "\n", | |
| "これを上述の正規方程式の第二式、第三式に代入すると以下のようになる。\n", | |
| "<br><br>\n", | |
| "$$\\beta_0 \\sum_{i=1}^n x_{1i} + \\beta_1 \\sum_{i=1}^n x_{1i}^2 + \\beta_2 \\alpha \\sum_{i=1}^n x_{1i}^2 = \\sum_{i=1}^n x_{1i} y_i $$<br>\n", | |
| "$$\\beta_0 \\alpha \\sum_{i=1}^n x_{1i} + \\beta_1 \\alpha \\sum_{i=1}^n x_{1i}^2 + \\beta_2 \\alpha^2 \\sum_{i=1}^n x_{1i}^2 = \\alpha \\sum_{i=1}^n x_{1i} y_i $$\n", | |
| "<br><br>\n", | |
| "下の式をαで割ると上と全く同じ式になる。つまりこの連立方程式は独立ではなく、3変数に対して2式のみとなり一意な解を得ることができない。\n", | |
| "<br><br>\n", | |
| "しかしこのようなケースは実務上稀であり、かつ実際にあったとしても容易に対処できる場合がほとんどであろう。<br>\n", | |
| "多重共線性が大きな問題となるのは次のようなケースである。" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.Latex object>" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "%%latex\n", | |
| "ここ(https://gist.github.com/shotahorii/037dc487ab3c70cc45af27fc9d03c9fb)で書いた通り、最小二乗法による推定量は\n", | |
| "正規方程式を解くことで得られる。<br>\n", | |
| "例えば、以下の2変数のモデルについて考える。<br><br>\n", | |
| "$$y_i = \\beta_0 + \\beta_1 x_{1i} + \\beta_2 x_{2i} + \\epsilon_i$$<br>\n", | |
| "$$i = 1,2,...,n$$<br><br>\n", | |
| "\n", | |
| "この時、正規方程式は以下である。<br><br>\n", | |
| "\n", | |
| "$$\\beta_0 \\sum_{i=1}^n 1 + \\beta_1 \\sum_{i=1}^n x_{1i} + \\beta_2 \\sum_{i=1}^n x_{2i} = \\sum_{i=1}^n y_i $$<br>\n", | |
| "$$\\beta_0 \\sum_{i=1}^n x_{1i} + \\beta_1 \\sum_{i=1}^n x_{1i}^2 + \\beta_2 \\sum_{i=1}^n x_{1i} x_{2i} = \\sum_{i=1}^n x_{1i} y_i $$<br>\n", | |
| "$$\\beta_0 \\sum_{i=1}^n x_{2i} + \\beta_1 \\sum_{i=1}^n x_{1i} x_{2i} + \\beta_2 \\sum_{i=1}^n x_{2i}^2 = \\sum_{i=1}^n x_{2i} y_i $$\n", | |
| "<br><br>\n", | |
| "\n", | |
| "ここで、説明変数$$x_{1i}$$と$$x_{2i}$$の間に完全な多重共線性が存在するとする。その場合、以下の関係が導かれる。\n", | |
| "<br><br>\n", | |
| "$$x_{2i} = \\alpha x_{1i}$$<br>\n", | |
| "$$i = 1,2,...,n$$<br><br>\n", | |
| "\n", | |
| "これを上述の正規方程式の第二式、第三式に代入すると以下のようになる。\n", | |
| "<br><br>\n", | |
| "$$\\beta_0 \\sum_{i=1}^n x_{1i} + \\beta_1 \\sum_{i=1}^n x_{1i}^2 + \\beta_2 \\alpha \\sum_{i=1}^n x_{1i}^2 = \\sum_{i=1}^n x_{1i} y_i $$<br>\n", | |
| "$$\\beta_0 \\alpha \\sum_{i=1}^n x_{1i} + \\beta_1 \\alpha \\sum_{i=1}^n x_{1i}^2 + \\beta_2 \\alpha^2 \\sum_{i=1}^n x_{1i}^2 = \\alpha \\sum_{i=1}^n x_{1i} y_i $$\n", | |
| "<br><br>\n", | |
| "下の式をαで割ると上と全く同じ式になる。つまりこの連立方程式は独立ではなく、3変数に対して2式のみとなり一意な解を得ることができない。\n", | |
| "<br><br>\n", | |
| "しかしこのようなケースは実務上稀であり、かつ実際にあったとしても容易に対処できる場合がほとんどであろう。<br>\n", | |
| "多重共線性が大きな問題となるのは次のようなケースである。" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "**完全な多重共線性はないが変数間に強い相関があるケース** \n", | |
| "このケースがなぜ問題になるのか、式から見ていく。なお、以下の推定量の詳しい導出は多くの資料(例えばReference1,2)で省略されているが、Reference3では細かく書かれていてわかりやすい。" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 64, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/latex": [ | |
| "完全な多重共線性があるケースと同様の2変数のモデルについて考える。正規方程式は上の例と同様以下である。\n", | |
| "<br><br>\n", | |
| "注意点として、第一式で$$\\sum_{i=1}^n 1$$を単純に$$n$$と置き換えていることと、見やすさのため$$\\sum_{i=1}^n$$を単純に\n", | |
| "$$\\sum$$と記載している。\n", | |
| "<br><br>\n", | |
| "$$n \\beta_0 + \\beta_1 \\sum x_{1i} + \\beta_2 \\sum x_{2i} = \\sum y_i $$<br>\n", | |
| "$$\\beta_0 \\sum x_{1i} + \\beta_1 \\sum x_{1i}^2 + \\beta_2 \\sum x_{1i} x_{2i} = \\sum x_{1i} y_i $$<br>\n", | |
| "$$\\beta_0 \\sum x_{2i} + \\beta_1 \\sum x_{1i} x_{2i} + \\beta_2 \\sum x_{2i}^2 = \\sum x_{2i} y_i $$\n", | |
| "<br><br>\n", | |
| "\n", | |
| "ここで、第一式を以下のように変換する。<br><br>\n", | |
| "$$\\beta_0 = \\frac{\\sum y_i}{n} - \\beta_1 \\frac{\\sum x_{1i}}{n} - \\beta_2 \\frac{\\sum x_{2i}}{n} $$\n", | |
| "<br><br>\n", | |
| "\n", | |
| "上の式を正規方程式の第二式に代入すると以下が得られる。\n", | |
| "<br><br>\n", | |
| "\n", | |
| "$$\\frac{\\sum x_{1i} \\sum y_i}{n} - \\beta_1 \\frac{(\\sum x_{1i})^2}{n} \n", | |
| "- \\beta_2 \\frac{\\sum x_{1i} \\sum x_{2i}}{n} \n", | |
| "+ \\beta_1 \\sum x_{1i}^2 + \\beta_2 \\sum x_{1i} x_{2i} = \\sum x_{1i} y_i $$\n", | |
| "<br><br>\n", | |
| "これを以下のように変換する。\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 (\\sum x_{1i}^2 - \\frac{(\\sum x_{1i})^2}{n})\n", | |
| "+ \\beta_2 (\\sum x_{1i} x_{2i} - \\frac{\\sum x_{1i} \\sum x_{2i}}{n}) \n", | |
| "= \\sum x_{1i} y_i - \\frac{\\sum x_{1i} \\sum y_i}{n}\n", | |
| "$$\n", | |
| "<br><br>\n", | |
| "\n", | |
| "同様に第三式に代入して以下の式を得る。\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 (\\sum x_{1i} x_{2i} - \\frac{\\sum x_{1i} \\sum x_{2i}}{n}) \n", | |
| "+ \\beta_2 (\\sum x_{2i}^2 - \\frac{(\\sum x_{2i})^2}{n})\n", | |
| "= \\sum x_{2i} y_i - \\frac{\\sum x_{2i} \\sum y_i}{n}\n", | |
| "$$" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.Latex object>" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "%%latex\n", | |
| "完全な多重共線性があるケースと同様の2変数のモデルについて考える。正規方程式は上の例と同様以下である。\n", | |
| "<br><br>\n", | |
| "注意点として、第一式で$$\\sum_{i=1}^n 1$$を単純に$$n$$と置き換えていることと、見やすさのため$$\\sum_{i=1}^n$$を単純に\n", | |
| "$$\\sum$$と記載している。\n", | |
| "<br><br>\n", | |
| "$$n \\beta_0 + \\beta_1 \\sum x_{1i} + \\beta_2 \\sum x_{2i} = \\sum y_i $$<br>\n", | |
| "$$\\beta_0 \\sum x_{1i} + \\beta_1 \\sum x_{1i}^2 + \\beta_2 \\sum x_{1i} x_{2i} = \\sum x_{1i} y_i $$<br>\n", | |
| "$$\\beta_0 \\sum x_{2i} + \\beta_1 \\sum x_{1i} x_{2i} + \\beta_2 \\sum x_{2i}^2 = \\sum x_{2i} y_i $$\n", | |
| "<br><br>\n", | |
| "\n", | |
| "ここで、第一式を以下のように変換する。<br><br>\n", | |
| "$$\\beta_0 = \\frac{\\sum y_i}{n} - \\beta_1 \\frac{\\sum x_{1i}}{n} - \\beta_2 \\frac{\\sum x_{2i}}{n} $$\n", | |
| "<br><br>\n", | |
| "\n", | |
| "上の式を正規方程式の第二式に代入すると以下が得られる。\n", | |
| "<br><br>\n", | |
| "\n", | |
| "$$\\frac{\\sum x_{1i} \\sum y_i}{n} - \\beta_1 \\frac{(\\sum x_{1i})^2}{n} \n", | |
| "- \\beta_2 \\frac{\\sum x_{1i} \\sum x_{2i}}{n} \n", | |
| "+ \\beta_1 \\sum x_{1i}^2 + \\beta_2 \\sum x_{1i} x_{2i} = \\sum x_{1i} y_i $$\n", | |
| "<br><br>\n", | |
| "これを以下のように変換する。\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 (\\sum x_{1i}^2 - \\frac{(\\sum x_{1i})^2}{n})\n", | |
| "+ \\beta_2 (\\sum x_{1i} x_{2i} - \\frac{\\sum x_{1i} \\sum x_{2i}}{n}) \n", | |
| "= \\sum x_{1i} y_i - \\frac{\\sum x_{1i} \\sum y_i}{n}\n", | |
| "$$\n", | |
| "<br><br>\n", | |
| "\n", | |
| "同様に第三式に代入して以下の式を得る。\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 (\\sum x_{1i} x_{2i} - \\frac{\\sum x_{1i} \\sum x_{2i}}{n}) \n", | |
| "+ \\beta_2 (\\sum x_{2i}^2 - \\frac{(\\sum x_{2i})^2}{n})\n", | |
| "= \\sum x_{2i} y_i - \\frac{\\sum x_{2i} \\sum y_i}{n}\n", | |
| "$$" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 90, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/latex": [ | |
| "ここで、一旦上の式は傍において、偏差平方和と偏差積和について考える。\n", | |
| "<br><br>\n", | |
| "偏差平方和は以下の式で表される。<br><br>\n", | |
| "$$S_x = \\sum (x_i - \\overline{x})^2$$\n", | |
| "<br><br>\n", | |
| "ここで、この式を展開していくと...\n", | |
| "<br>\n", | |
| "$$S_x = \\sum (x_i - \\overline{x})^2$$<br>\n", | |
| "$$= \\sum x_i^2 - 2 \\sum x_i \\overline{x} + \\sum \\overline{x}^2$$<br>\n", | |
| "$$= \\sum x_i^2 - 2 \\overline{x} \\sum x_i + \\overline{x}^2 \\sum 1$$<br>\n", | |
| "$$= \\sum x_i^2 - 2 \\frac{\\sum x_i}{n} \\sum x_i + n (\\frac{\\sum x_i}{n})^2$$<br>\n", | |
| "$$= \\sum x_i^2 - 2 \\frac{(\\sum x_i)^2}{n} + \\frac{(\\sum x_i)^2}{n}$$<br>\n", | |
| "$$= \\sum x_i^2 - \\frac{(\\sum x_i)^2}{n}$$<br><br>\n", | |
| "\n", | |
| "また、偏差積和は以下の式で表される。<br><br>\n", | |
| "$$S_{xy} = \\sum (x_i - \\overline{x})(y_i - \\overline{y})$$\n", | |
| "<br><br>\n", | |
| "ここで、この式を展開していくと...\n", | |
| "<br>\n", | |
| "$$S_{xy} = \\sum (x_i - \\overline{x})(y_i - \\overline{y})$$<br>\n", | |
| "$$=\\sum x_i y_i - \\sum \\overline{y} x_i - \\sum \\overline{x} y_i + \\sum \\overline{x} \\overline{y}$$<br>\n", | |
| "$$=\\sum x_i y_i - \\overline{y} \\sum x_i - \\overline{x} \\sum y_i + \\overline{x} \\overline{y} \\sum 1$$<br>\n", | |
| "$$=\\sum x_i y_i - \\frac{\\sum y_i}{n} \\sum x_i - \\frac{\\sum x_i}{n} \\sum y_i \n", | |
| "+ n \\frac{\\sum x_i}{n} \\frac{\\sum y_i}{n}$$<br>\n", | |
| "$$=\\sum x_i y_i - 2 \\frac{\\sum x_i \\sum y_i}{n} + \\frac{\\sum x_i \\sum y_i}{n}$$<br>\n", | |
| "$$=\\sum x_i y_i - \\frac{\\sum x_i \\sum y_i}{n}$$" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.Latex object>" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "%%latex\n", | |
| "ここで、一旦上の式は傍において、偏差平方和と偏差積和について考える。\n", | |
| "<br><br>\n", | |
| "偏差平方和は以下の式で表される。<br><br>\n", | |
| "$$S_x = \\sum (x_i - \\overline{x})^2$$\n", | |
| "<br><br>\n", | |
| "ここで、この式を展開していくと...\n", | |
| "<br>\n", | |
| "$$S_x = \\sum (x_i - \\overline{x})^2$$<br>\n", | |
| "$$= \\sum x_i^2 - 2 \\sum x_i \\overline{x} + \\sum \\overline{x}^2$$<br>\n", | |
| "$$= \\sum x_i^2 - 2 \\overline{x} \\sum x_i + \\overline{x}^2 \\sum 1$$<br>\n", | |
| "$$= \\sum x_i^2 - 2 \\frac{\\sum x_i}{n} \\sum x_i + n (\\frac{\\sum x_i}{n})^2$$<br>\n", | |
| "$$= \\sum x_i^2 - 2 \\frac{(\\sum x_i)^2}{n} + \\frac{(\\sum x_i)^2}{n}$$<br>\n", | |
| "$$= \\sum x_i^2 - \\frac{(\\sum x_i)^2}{n}$$<br><br>\n", | |
| "\n", | |
| "また、偏差積和は以下の式で表される。<br><br>\n", | |
| "$$S_{xy} = \\sum (x_i - \\overline{x})(y_i - \\overline{y})$$\n", | |
| "<br><br>\n", | |
| "ここで、この式を展開していくと...\n", | |
| "<br>\n", | |
| "$$S_{xy} = \\sum (x_i - \\overline{x})(y_i - \\overline{y})$$<br>\n", | |
| "$$=\\sum x_i y_i - \\sum \\overline{y} x_i - \\sum \\overline{x} y_i + \\sum \\overline{x} \\overline{y}$$<br>\n", | |
| "$$=\\sum x_i y_i - \\overline{y} \\sum x_i - \\overline{x} \\sum y_i + \\overline{x} \\overline{y} \\sum 1$$<br>\n", | |
| "$$=\\sum x_i y_i - \\frac{\\sum y_i}{n} \\sum x_i - \\frac{\\sum x_i}{n} \\sum y_i \n", | |
| "+ n \\frac{\\sum x_i}{n} \\frac{\\sum y_i}{n}$$<br>\n", | |
| "$$=\\sum x_i y_i - 2 \\frac{\\sum x_i \\sum y_i}{n} + \\frac{\\sum x_i \\sum y_i}{n}$$<br>\n", | |
| "$$=\\sum x_i y_i - \\frac{\\sum x_i \\sum y_i}{n}$$" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 114, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/latex": [ | |
| "ここで、正規方程式を変換して得た以下の2式に戻る。<br><br>\n", | |
| "$$\\beta_1 (\\sum x_{1i}^2 - \\frac{(\\sum x_{1i})^2}{n})\n", | |
| "+ \\beta_2 (\\sum x_{1i} x_{2i} - \\frac{\\sum x_{1i} \\sum x_{2i}}{n}) \n", | |
| "= \\sum x_{1i} y_i - \\frac{\\sum x_{1i} \\sum y_i}{n}\n", | |
| "$$\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 (\\sum x_{1i} x_{2i} - \\frac{\\sum x_{1i} \\sum x_{2i}}{n}) \n", | |
| "+ \\beta_2 (\\sum x_{2i}^2 - \\frac{(\\sum x_{2i})^2}{n})\n", | |
| "= \\sum x_{2i} y_i - \\frac{\\sum x_{2i} \\sum y_i}{n}\n", | |
| "$$\n", | |
| "<br><br>\n", | |
| "上の偏差平方和、偏差積和の式展開より、以下のように書き直すことができる。\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 \\sum (x_{1i} - \\overline{x_1})^2\n", | |
| "+ \\beta_2 \\sum (x_{1i} - \\overline{x_1})(x_{2i} - \\overline{x_2})\n", | |
| "= \\sum (x_{1i} - \\overline{x_1})(y_i - \\overline{y})\n", | |
| "$$\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 \\sum (x_{1i} - \\overline{x_1})(x_{2i} - \\overline{x_2})\n", | |
| "+ \\beta_2 \\sum (x_{2i} - \\overline{x_2})^2\n", | |
| "= \\sum (x_{2i} - \\overline{x_2})(y_i - \\overline{y})\n", | |
| "$$\n", | |
| "<br><br>\n", | |
| "ここで、偏差平方和と偏差積和を$$S_x, S_{xy}$$の形で書き直す。\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 S_{x_1} + \\beta_2 S_{x_1x_2} = S_{x_1y}$$<br>\n", | |
| "$$\\beta_1 S_{x_1x_2} + \\beta_2 S_{x_2} = S_{x_2y}$$\n", | |
| "<br><br>\n", | |
| "以下のように変換できる。\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 = \\frac{S_{x_2}S_{x_1y} - S_{x_1x_2}S_{x_2y}}{S_{x_1}S_{x_2} - S_{x_1x_2}^2}$$<br>\n", | |
| "$$\\beta_2 = \\frac{S_{x_1}S_{x_2y} - S_{x_1x_2}S_{x_1y}}{S_{x_1}S_{x_2} - S_{x_1x_2}^2}$$\n", | |
| "<br><br>\n", | |
| "(ちなみに、$$\\beta_0 = \\overline{y} - \\beta_1 \\overline{x_1} - \\beta_2 \\overline{x_2}$$である)" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.Latex object>" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "%%latex\n", | |
| "ここで、正規方程式を変換して得た以下の2式に戻る。<br><br>\n", | |
| "$$\\beta_1 (\\sum x_{1i}^2 - \\frac{(\\sum x_{1i})^2}{n})\n", | |
| "+ \\beta_2 (\\sum x_{1i} x_{2i} - \\frac{\\sum x_{1i} \\sum x_{2i}}{n}) \n", | |
| "= \\sum x_{1i} y_i - \\frac{\\sum x_{1i} \\sum y_i}{n}\n", | |
| "$$\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 (\\sum x_{1i} x_{2i} - \\frac{\\sum x_{1i} \\sum x_{2i}}{n}) \n", | |
| "+ \\beta_2 (\\sum x_{2i}^2 - \\frac{(\\sum x_{2i})^2}{n})\n", | |
| "= \\sum x_{2i} y_i - \\frac{\\sum x_{2i} \\sum y_i}{n}\n", | |
| "$$\n", | |
| "<br><br>\n", | |
| "上の偏差平方和、偏差積和の式展開より、以下のように書き直すことができる。\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 \\sum (x_{1i} - \\overline{x_1})^2\n", | |
| "+ \\beta_2 \\sum (x_{1i} - \\overline{x_1})(x_{2i} - \\overline{x_2})\n", | |
| "= \\sum (x_{1i} - \\overline{x_1})(y_i - \\overline{y})\n", | |
| "$$\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 \\sum (x_{1i} - \\overline{x_1})(x_{2i} - \\overline{x_2})\n", | |
| "+ \\beta_2 \\sum (x_{2i} - \\overline{x_2})^2\n", | |
| "= \\sum (x_{2i} - \\overline{x_2})(y_i - \\overline{y})\n", | |
| "$$\n", | |
| "<br><br>\n", | |
| "ここで、偏差平方和と偏差積和を$$S_x, S_{xy}$$の形で書き直す。\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 S_{x_1} + \\beta_2 S_{x_1x_2} = S_{x_1y}$$<br>\n", | |
| "$$\\beta_1 S_{x_1x_2} + \\beta_2 S_{x_2} = S_{x_2y}$$\n", | |
| "<br><br>\n", | |
| "以下のように変換できる。\n", | |
| "<br><br>\n", | |
| "$$\\beta_1 = \\frac{S_{x_2}S_{x_1y} - S_{x_1x_2}S_{x_2y}}{S_{x_1}S_{x_2} - S_{x_1x_2}^2}$$<br>\n", | |
| "$$\\beta_2 = \\frac{S_{x_1}S_{x_2y} - S_{x_1x_2}S_{x_1y}}{S_{x_1}S_{x_2} - S_{x_1x_2}^2}$$\n", | |
| "<br><br>\n", | |
| "(ちなみに、$$\\beta_0 = \\overline{y} - \\beta_1 \\overline{x_1} - \\beta_2 \\overline{x_2}$$である)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 137, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/latex": [ | |
| "ここで、偏差平方和、偏差積和と相関係数との関係は以下である。<br><br>\n", | |
| "$$r_{xy} = \\frac{S_{xy}}{\\sqrt{S_x}\\sqrt{S_y}}$$<br><br>\n", | |
| "上の$$\\beta_1$$についての式を展開して、変数間の相関係数を含む形で$$\\beta_1$$を\n", | |
| "表記してみる。<br><br>\n", | |
| "$$\\beta_1 = \\frac{S_{x_2}S_{x_1y} - S_{x_1x_2}S_{x_2y}}{S_{x_1}S_{x_2} - S_{x_1x_2}^2}$$<br>\n", | |
| "$$=\\frac{\\frac\n", | |
| " {S_{x_2}S_{x_1y} - S_{x_1x_2}S_{x_2y}}\n", | |
| " {\\sqrt{S_{x1}}\\sqrt{S_{y}}\\sqrt{S_{x2}}\\sqrt{S_{x2}}}\n", | |
| " }\n", | |
| " {\\frac\n", | |
| " {S_{x_1}S_{x_2} - S_{x_1x_2}^2}\n", | |
| " {\\sqrt{S_{x1}}\\sqrt{S_{y}}\\sqrt{S_{x2}}\\sqrt{S_{x2}}}\n", | |
| " }$$<br>\n", | |
| "$$=\\frac{r_{x_1y}-r_{x_1 x_2}r_{x_2y}}\n", | |
| " {\\frac\n", | |
| " {S_{x_1}S_{x_2} - S_{x_1x_2}^2}\n", | |
| " {\\sqrt{S_{x1}}\\sqrt{S_{y}}\\sqrt{S_{x2}}\\sqrt{S_{x2}}}\n", | |
| " }$$<br>\n", | |
| "$$=\\frac{r_{x_1y}-r_{x_1 x_2}r_{x_2y}}\n", | |
| " {\\frac\n", | |
| " {(S_{x_1}S_{x_2} - S_{x_1x_2}^2)\\sqrt{S_{x1}}}\n", | |
| " {\\sqrt{S_{x1}}\\sqrt{S_{y}}\\sqrt{S_{x2}}\\sqrt{S_{x2}}\\sqrt{S_{x1}}}\n", | |
| " }$$<br>\n", | |
| "$$=\\frac{r_{x_1y}-r_{x_1 x_2}r_{x_2y}}\n", | |
| " {(1-r_{x_1 x_2}^2)\\frac{\\sqrt{S_{x1}}}{\\sqrt{S_{y}}}}$$<br>\n", | |
| "$$=\\frac{r_{x_1y}-r_{x_1 x_2}r_{x_2y}}{1-r_{x_1 x_2}^2}\\sqrt{\\frac{S_y}{S_{x_1}}}$$<br>" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.Latex object>" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "%%latex\n", | |
| "ここで、偏差平方和、偏差積和と相関係数との関係は以下である。<br><br>\n", | |
| "$$r_{xy} = \\frac{S_{xy}}{\\sqrt{S_x}\\sqrt{S_y}}$$<br><br>\n", | |
| "上の$$\\beta_1$$についての式を展開して、変数間の相関係数を含む形で$$\\beta_1$$を\n", | |
| "表記してみる。<br><br>\n", | |
| "$$\\beta_1 = \\frac{S_{x_2}S_{x_1y} - S_{x_1x_2}S_{x_2y}}{S_{x_1}S_{x_2} - S_{x_1x_2}^2}$$<br>\n", | |
| "$$=\\frac{\\frac\n", | |
| " {S_{x_2}S_{x_1y} - S_{x_1x_2}S_{x_2y}}\n", | |
| " {\\sqrt{S_{x1}}\\sqrt{S_{y}}\\sqrt{S_{x2}}\\sqrt{S_{x2}}}\n", | |
| " }\n", | |
| " {\\frac\n", | |
| " {S_{x_1}S_{x_2} - S_{x_1x_2}^2}\n", | |
| " {\\sqrt{S_{x1}}\\sqrt{S_{y}}\\sqrt{S_{x2}}\\sqrt{S_{x2}}}\n", | |
| " }$$<br>\n", | |
| "$$=\\frac{r_{x_1y}-r_{x_1 x_2}r_{x_2y}}\n", | |
| " {\\frac\n", | |
| " {S_{x_1}S_{x_2} - S_{x_1x_2}^2}\n", | |
| " {\\sqrt{S_{x1}}\\sqrt{S_{y}}\\sqrt{S_{x2}}\\sqrt{S_{x2}}}\n", | |
| " }$$<br>\n", | |
| "$$=\\frac{r_{x_1y}-r_{x_1 x_2}r_{x_2y}}\n", | |
| " {\\frac\n", | |
| " {(S_{x_1}S_{x_2} - S_{x_1x_2}^2)\\sqrt{S_{x1}}}\n", | |
| " {\\sqrt{S_{x1}}\\sqrt{S_{y}}\\sqrt{S_{x2}}\\sqrt{S_{x2}}\\sqrt{S_{x1}}}\n", | |
| " }$$<br>\n", | |
| "$$=\\frac{r_{x_1y}-r_{x_1 x_2}r_{x_2y}}\n", | |
| " {(1-r_{x_1 x_2}^2)\\frac{\\sqrt{S_{x1}}}{\\sqrt{S_{y}}}}$$<br>\n", | |
| "$$=\\frac{r_{x_1y}-r_{x_1 x_2}r_{x_2y}}{1-r_{x_1 x_2}^2}\\sqrt{\\frac{S_y}{S_{x_1}}}$$<br>" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "上記の式より、変数x<sub>1</sub>とx<sub>2</sub>の相関が非常に大きい(例えば0.98)時、サンプルのちょっとした変化による変数x<sub>1</sub>とyの相関係数のごく小さな変化が推定量β<sub>1</sub>の値を大きく変えてしまうことがわかる。 \n", | |
| " \n", | |
| "**推定量の分散**" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 144, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/latex": [ | |
| "ここで、推定量$$\\beta_1$$の分散は以下で表される。\n", | |
| "<br><br>\n", | |
| "$$V(\\beta_1) = \\frac{\\sigma^2}{S_{x_1}}\\frac{1}{1-r_{x_1x_2}^2}$$\n", | |
| "<br><br>\n", | |
| "変数$$x_1,x_2$$の相関が0.98の時、相関が0の時と比べて$$V(\\beta_1)$$は25.25倍大きくなる。" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.Latex object>" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "%%latex\n", | |
| "ここで、推定量$$\\beta_1$$の分散は以下で表される。\n", | |
| "<br><br>\n", | |
| "$$V(\\beta_1) = \\frac{\\sigma^2}{S_{x_1}}\\frac{1}{1-r_{x_1x_2}^2}$$\n", | |
| "<br><br>\n", | |
| "変数$$x_1,x_2$$の相関が0.98の時、相関が0の時と比べて$$V(\\beta_1)$$は25.25倍大きくなる。" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 147, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/latex": [ | |
| "ここで、上の$$V(\\beta_1)$$式は2変数の最小二乗法における推定量の分散を示す式であるが、\n", | |
| "変数の数を任意として一般化すると、変数$$x_k$$の推定量$$\\beta_k$$の分散は以下のように表せる。\n", | |
| "<br><br>\n", | |
| "$$V(\\beta_k) = \\frac{\\sigma^2}{S_k}\\frac{1}{1-R_k^2}$$\n", | |
| "<br><br>\n", | |
| "ここで、$$R_k^2$$は$$x_k$$以外の説明変数を用いた$$x_k$$の回帰における決定係数\n", | |
| "である。(目的変数のyは回帰に含めない。)" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.Latex object>" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "%%latex\n", | |
| "ここで、上の$$V(\\beta_1)$$式は2変数の最小二乗法における推定量の分散を示す式であるが、\n", | |
| "変数の数を任意として一般化すると、変数$$x_k$$の推定量$$\\beta_k$$の分散は以下のように表せる。\n", | |
| "<br><br>\n", | |
| "$$V(\\beta_k) = \\frac{\\sigma^2}{S_k}\\frac{1}{1-R_k^2}$$\n", | |
| "<br><br>\n", | |
| "ここで、$$R_k^2$$は$$x_k$$以外の説明変数を用いた$$x_k$$の回帰における決定係数\n", | |
| "である。(目的変数のyは回帰に含めない。)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "#### Variance inflation factor (VIF)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 172, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/latex": [ | |
| "上記の、推定量の分散を表す式において、後ろの項$$\\frac{1}{1-R_k^2}$$をVIFと呼び、\n", | |
| "推定された回帰係数$$\\beta_k$$の分散が、多重共線性のためにどれだけ増加したかを測る\n", | |
| "指標として用いることができる。\n", | |
| "<br><br>\n", | |
| "$$VIF_k = \\frac{1}{1-R_k^2}$$\n", | |
| "<br><br>\n", | |
| "重回帰分析において、各推定量がどれだけ多重共線性の影響を受けたかを調べる方法として、\n", | |
| "以下のような手続きを用いることができる。(説明変数の数をmとする)\n", | |
| "<br><br>\n", | |
| "<b>Pseudo Code</b><br>\n", | |
| "For k in range(m):<br>\n", | |
| "STEP1: $$X_k = \\alpha_1X_1 + \\alpha_2X_2 + ... \n", | |
| "+ \\alpha_{k-1}X_{k-1} + \\alpha_{k+1}X_{k+1} + ... + \\alpha_mX_m + c + e$$<br>\n", | |
| "(cは定数、eは誤差)として回帰分析を行い、決定係数$$R_k^2$$を算出する。<br>\n", | |
| "STEP2: $$VIF_k=\\frac{1}{1-R_k^2}$$として$$\\beta_k$$に対するVIFの値を算出。\n", | |
| "このVIF値の大きさにより多重共線性の程度を判断する。経験的に、$$VIF_k>10$$であれば\n", | |
| "多重共線性の程度は大きい。" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.Latex object>" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "%%latex\n", | |
| "上記の、推定量の分散を表す式において、後ろの項$$\\frac{1}{1-R_k^2}$$をVIFと呼び、\n", | |
| "推定された回帰係数$$\\beta_k$$の分散が、多重共線性のためにどれだけ増加したかを測る\n", | |
| "指標として用いることができる。\n", | |
| "<br><br>\n", | |
| "$$VIF_k = \\frac{1}{1-R_k^2}$$\n", | |
| "<br><br>\n", | |
| "重回帰分析において、各推定量がどれだけ多重共線性の影響を受けたかを調べる方法として、\n", | |
| "以下のような手続きを用いることができる。(説明変数の数をmとする)\n", | |
| "<br><br>\n", | |
| "<b>Pseudo Code</b><br>\n", | |
| "For k in range(m):<br>\n", | |
| "STEP1: $$X_k = \\alpha_1X_1 + \\alpha_2X_2 + ... \n", | |
| "+ \\alpha_{k-1}X_{k-1} + \\alpha_{k+1}X_{k+1} + ... + \\alpha_mX_m + c + e$$<br>\n", | |
| "(cは定数、eは誤差)として回帰分析を行い、決定係数$$R_k^2$$を算出する。<br>\n", | |
| "STEP2: $$VIF_k=\\frac{1}{1-R_k^2}$$として$$\\beta_k$$に対するVIFの値を算出。\n", | |
| "このVIF値の大きさにより多重共線性の程度を判断する。経験的に、$$VIF_k>10$$であれば\n", | |
| "多重共線性の程度は大きい。" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "#### 実装" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 176, | |
| "metadata": { | |
| "collapsed": true | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "import numpy as np\n", | |
| "import pandas as pd\n", | |
| "from sklearn.datasets import load_boston\n", | |
| "import statsmodels.api as sm\n", | |
| "from statsmodels.stats.outliers_influence import variance_inflation_factor" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 186, | |
| "metadata": { | |
| "collapsed": true | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "# boston data\n", | |
| "d = load_boston()\n", | |
| "y = d.target\n", | |
| "X = d.data" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 191, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Number of variables: 13\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "print(\"Number of variables:\", X.shape[1])" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 198, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "VIF for column 0 is 2.10037281996\n", | |
| "VIF for column 1 is 2.84401326695\n", | |
| "VIF for column 2 is 14.4857577065\n", | |
| "VIF for column 3 is 1.15295185894\n", | |
| "VIF for column 4 is 73.8949465281\n", | |
| "VIF for column 5 is 77.9482830464\n", | |
| "VIF for column 6 is 21.3868504899\n", | |
| "VIF for column 7 is 14.6996523837\n", | |
| "VIF for column 8 is 15.1677248579\n", | |
| "VIF for column 9 is 61.2272740096\n", | |
| "VIF for column 10 is 85.0295473106\n", | |
| "VIF for column 11 is 20.1049426362\n", | |
| "VIF for column 12 is 11.1020247722\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "# check VIF for all variables\n", | |
| "for k in range(X.shape[1]):\n", | |
| " vif = variance_inflation_factor(X, k)\n", | |
| " print('VIF for column',k,'is',vif)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 202, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "VIF for column 0 is 2.09934509889\n", | |
| "VIF for column 1 is 2.45162393335\n", | |
| "VIF for column 2 is 14.2752834432\n", | |
| "VIF for column 3 is 1.14216676953\n", | |
| "VIF for column 4 is 73.8941709297\n", | |
| "VIF for column 5 is 60.5988463004\n", | |
| "VIF for column 6 is 21.3612344102\n", | |
| "VIF for column 7 is 12.2216048089\n", | |
| "VIF for column 8 is 15.1591624262\n", | |
| "VIF for column 9 is 59.301541455\n", | |
| "VIF for column 10 is 18.6147508012\n", | |
| "VIF for column 11 is 10.1383237154\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "# Remove column 10 as it has high VIF\n", | |
| "new_X = np.delete(X, 10, 1)\n", | |
| "# check VIF for all variables without column 10\n", | |
| "for k in range(new_X.shape[1]):\n", | |
| " vif = variance_inflation_factor(new_X, k)\n", | |
| " print('VIF for column',k,'is',vif)" | |
| ] | |
| } | |
| ], | |
| "metadata": { | |
| "kernelspec": { | |
| "display_name": "Python 3", | |
| "language": "python", | |
| "name": "python3" | |
| }, | |
| "language_info": { | |
| "codemirror_mode": { | |
| "name": "ipython", | |
| "version": 3 | |
| }, | |
| "file_extension": ".py", | |
| "mimetype": "text/x-python", | |
| "name": "python", | |
| "nbconvert_exporter": "python", | |
| "pygments_lexer": "ipython3", | |
| "version": "3.5.2" | |
| } | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 0 | |
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment