High dimensional selective

High-dimensional randomized LASSO?.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our randomized paper (say for logistic LASSO) the active KKT equations are\n",
    "$$\n",
    "X_E^Ty - \\nabla \\Lambda(\\hat{\\beta}_E)  - \\epsilon \\hat{\\beta}_E= -\\omega_E + \\lambda s_E\n",
    "$$\n",
    "\n",
    "A Taylor expansion around $\\bar{\\beta}_E$ the unpenalized estimator yields\n",
    "$$\n",
    "X_E^Ty - \\nabla \\Lambda(\\bar{\\beta}_E) - Q_E(\\hat{\\beta}_E - \\bar{\\beta}_E) - \\epsilon \\hat{\\beta}_E= -\\omega_E + \\lambda s_E\n",
    "$$\n",
    "or,\n",
    "$$\n",
    "- Q_E(\\hat{\\beta}_E - \\bar{\\beta}_E) - \\epsilon \\hat{\\beta}_E= -\\omega_E + \\lambda s_E\n",
    "$$\n",
    "or,\n",
    "$$\n",
    "\\bar{\\beta}_E - \\hat{\\beta}_E = Q_E^{-1}\\left( \\epsilon \\hat{\\beta}_E - \\omega_E + \\lambda s_E\\right) = Q_E^{-1}(X_E^Ty - \\nabla \\Lambda(\\hat{\\beta}_E)).\n",
    "$$\n",
    "\n",
    "Plug this in to the inactive block\n",
    "$$\n",
    "X_{-E}^T(y - \\nabla \\Lambda(\\hat{\\beta}_E)) = -\\omega_{-E} + u_{-E}\n",
    "$$\n",
    "yielding\n",
    "$$\n",
    "X_{-E}^T(y - \\nabla \\Lambda(\\bar{\\beta}_E)) + C_E(X_E^Ty - \\nabla \\Lambda(\\hat{\\beta}_E)) = -\\omega_{-E} + u_{-E}\n",
    "$$\n",
    "Or,\n",
    "$$\n",
    "\\omega_{-E} = - X_{-E}^T(y - \\nabla \\Lambda(\\bar{\\beta}_E)) - C_E(X_E^Ty - \\nabla \\Lambda(\\hat{\\beta}_E)) + u_{-E}.\n",
    "$$\n",
    "\n",
    "\n",
    "\n",
    "If we are willing to assume $X_{-E}^T(y - \\nabla \\Lambda(\\bar{\\beta}_E))$ is asymptotically independent of \n",
    "$\\bar{\\beta}_E$ or $X_E^Ty$ as it is in the parametric setting then maybe we can do something.\n",
    "\n",
    "How bad would it be to just condition on it?\n",
    "\n",
    "In the OLS case, if the errors are independent of $X$ then, conditional on $X$ for any $E$\n",
    "$$\n",
    "\\text{Cov}(X_{-E}^T(y - X_E \\bar{\\beta}_E), \\bar{\\beta}_E|X) = 0\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}