ECCV

eccv

{
 "metadata": {
  "name": "",
  "signature": "sha256:5233d79a19c9913a5b8655df7e591f48af81002fb35ed89c0a45d273491e93d9"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "$$\n",
      "\\DeclareMathOperator*{\\argmin}{arg\\,min}\n",
      "\\DeclareMathOperator{\\rank}{rank}\n",
      "\\DeclareMathOperator{\\mat}{mat}\n",
      "\\def\\S{\\mathbf{S}}\n",
      "\\def\\B{\\mathbf{B}}\n",
      "\\def\\U{\\mathbf{U}}\n",
      "\\def\\V{\\mathbf{V}}\n",
      "\\def\\P{\\mathbf{P}}\n",
      "\\def\\S{\\mathbf{S}}\n",
      "\\def\\E{\\mathbf{E}}\n",
      "\\def\\c{\\mathbf{c}}\n",
      "\\def\\N{\\mathbf{N}}\n",
      "\\def\\C{\\mathbf{C}}\n",
      "\\def\\l{\\mathbf{l}}\n",
      "\\def\\I{\\mathbf{I}}\n",
      "\\def\\L{\\mathbf{L}}\n",
      "\\def\\X{\\mathbf{X}}\n",
      "$$"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Problem definition\n",
      "\n",
      "Robustly recover a set of normals by performing a generalised uncalibrated photometric stereo.\n",
      "\n",
      "$f$ = # of images  \n",
      "$p$ = # of pixels  \n",
      "$k$ = # of components kept  \n",
      "\n",
      "$4$ = # of harmonic bases\n",
      "\n",
      "\n",
      "## Uncalibrated Photometric Stereo\n",
      "$\\X: p \\times f$ The matrix of stacked images, each column is an image  \n",
      "$\\B: p \\times 4$ The matrix of spherical harmonics, representing the scaled surface normals  \n",
      "$\\L: 4 \\times f$ The matrix of lighting coefficients  \n",
      "\n",
      "$$\n",
      "\\X = \\B\\L\n",
      "$$\n",
      "\n",
      "# Generalized Uncalibrated Photometric Stereo\n",
      "$\\X: p \\times f$ The matrix of stacked images, each column is an image  \n",
      "$\\B: p \\times 4k$ The matrix of spherical harmonics bases, representing the scaled surface normals bases  \n",
      "$\\P: 4k \\times f$ The combined lighting and coefficient matrix  \n",
      "$\\L: 4 \\times f$ The matrix of lighting coefficients  \n",
      "$\\C: k \\times f$ The basis coefficients  \n",
      "\n",
      "$$\n",
      "\\X = \\B \\P \\\\\n",
      "where \\; \\rank(\\mat(\\P_i)) = 1\n",
      "$$\n",
      "\n",
      "where $\\mat(\\P_i)$ denotes reshaping a single column of $\\P$ into a $k \\times 4$ matrix."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# 1: Generic Morphable Model\n",
      "Split the problem into two discrete parts:\n",
      "\n",
      " 1. Recovering the shape basis ($\\B\\P + \\U\\V + \\E$)\n",
      " 2. Recovering the lighting and shape basis coefficients"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# The Shape Basis"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Option 1: Norm on $\\P$\n",
      "**Input**: Many images of different individuals, aligned with an existing sparse facial feature point detector.\n",
      "$$\n",
      "\\begin{equation*}\n",
      "    \\begin{aligned}\n",
      "       & \\argmin_{\\P,\\V,\\B,\\U,\\E} & & \\lVert \\P \\rVert_{SP} + \\lVert \\V \\rVert_{SP} + \\lVert \\E \\rVert_{1} \\\\\n",
      "       & \\text{subject to}  & & \\X = \\B \\P + \\U \\V + \\E \\\\\n",
      "       &                    & & \\B^{\\top} \\B = \\I \\\\\n",
      "       &                    & & \\U^{\\top} \\U = \\I\n",
      "    \\end{aligned}\n",
      "\\end{equation*}\n",
      "$$"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Option 2: Explicit $\\rank$ on $\\B$\n",
      "\n",
      "$$\n",
      "\\begin{equation*}\n",
      "    \\begin{aligned}\n",
      "       & \\argmin_{\\P,\\V,\\B,\\U,\\E} & & \\lVert \\V \\rVert_{SP} + \\lVert \\E \\rVert_{1} \\\\\n",
      "       & \\text{subject to}  & & \\X = \\B \\P + \\U \\V + \\E \\\\\n",
      "       &                    & & \\rank(\\B) = 4k \\\\       \n",
      "       &                    & & \\rank(\\V) \\leq \\rank(\\U) \\\\   \n",
      "       &                    & & \\B^{\\top} \\B = \\I \\\\\n",
      "       &                    & & \\U^{\\top} \\U = \\I\n",
      "    \\end{aligned}\n",
      "\\end{equation*}\n",
      "$$"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# The coefficients from $\\P$"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Each column of $\\P$ is a the combined coefficients of a single image, $i$.  \n",
      "We denote this column by $\\P_i$.\n",
      "\n",
      "Rearrange $\\P_i$ into a $k \\times 4$ vector, $\\tilde{\\P}_i$:\n",
      "\n",
      "$$\n",
      "\\tilde{\\P}_i = \\begin{bmatrix}\n",
      "                    c_{1i} l_1 & c_{1i} l_2 & c_{1i} l_3 & c_{1i} l_4 \\\\\n",
      "                    \\vdots     & \\vdots     & \\vdots     & \\vdots     \\\\\n",
      "                    c_{ki} l_1 & c_{ki} l_2 & c_{ki} l_3 & c_{ki} l_4\n",
      "               \\end{bmatrix}\n",
      "$$\n",
      "\n",
      "and rake the $\\rank = 1$ SVD to form two vectors:\n",
      "\n",
      "$$\n",
      "\\C_i = [c_1, \\ldots, c_k]^{\\top} \\\\\n",
      "\\L_i = [l_1, l_2, l_3, l_4]^{\\top}\n",
      "$$"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# 1: Video, fixed lighting\n",
      "Split the problem into two discrete parts:\n",
      "\n",
      " 1. Recovering the shape basis ($\\B\\P + \\U\\V + \\E$)\n",
      " 2. Recovering the single light and the shape coefficients"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# The Shape Basis\n",
      "As above"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# The coefficients from $\\P$\n",
      "This is a more complex decomposition that involves another seperate optimisation problem. We need to recover the single lighting vector from the matrix $\\P$ and shape coefficients per image."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": []
    }
   ],
   "metadata": {}
  }
 ]
}