Skip to content

Instantly share code, notes, and snippets.

@tobydriscoll

tobydriscoll/TB-Lecture-02.ipynb

Created September 2, 2016 15:34
Show Gist options
  • Save tobydriscoll/090d35caea2c59dd69edcc0432daa5fc to your computer and use it in GitHub Desktop.
Save tobydriscoll/090d35caea2c59dd69edcc0432daa5fc to your computer and use it in GitHub Desktop.
Download ZIP
TB Lecture 2 in julia
Display the source blob
Display the rendered blob
Raw
TB-Lecture-02.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Lecture 2: Orthogonal vectors and matrices"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"With real vectors and matrices, the transpose operation is simple and familiar. It also happens to correspond to what we call the **adjoint** mathematically. In the complex case, one also has to conjugate the entries to keep the mathematical structure intact. We call this operator the **hermitian** of a matrix and use a star superscript for it."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"2x4 Array{Complex{Float64},2}:\n",
" 0.40468+0.619917im 0.518203+0.943943im … 0.281234+0.375899im\n",
" 0.650155+0.443388im 0.901375+0.531532im 0.41967+0.292833im"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A = rand(2,4) + 1im*rand(2,4)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"4x2 Array{Complex{Float64},2}:\n",
" 0.40468-0.619917im 0.650155-0.443388im\n",
" 0.518203-0.943943im 0.901375-0.531532im\n",
" 0.613032-0.444494im 0.981122-0.997295im\n",
" 0.281234-0.375899im 0.41967-0.292833im"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Aadjoint = A'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To get plain transpose, use a `.^` operator."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"4x2 Array{Complex{Float64},2}:\n",
" 0.40468+0.619917im 0.650155+0.443388im\n",
" 0.518203+0.943943im 0.901375+0.531532im\n",
" 0.613032+0.444494im 0.981122+0.997295im\n",
" 0.281234+0.375899im 0.41967+0.292833im"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Atrans = A.'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Inner products"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If **u** and **v** are column vectors of the same length, then their **inner product** is $\\mathbf{u}^*\\mathbf{v}$. The result is a scalar. (In Julia, though, the scalar inner product is `dot(u,v)`, whereas the matrix multiplication result is a 1-by-1 matrix that is not equivalent for all purposes.)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"dot(u,v) gives -2 - 3im\n",
"u'*v gives Complex{Int64}[-2 - 3im]\n"
]
}
],
"source": [
"u = [ 4; -1; 2+2im ]\n",
"v = [ -1; 1im; 1 ]\n",
"println(\"dot(u,v) gives \", dot(u,v))\n",
"println(\"u'*v gives \",u'*v)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The inner product has geometric significance. It is used to define length through the 2-norm,"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1-element Array{Complex{Int64},1}:\n",
" 25+0im"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"length_u_squared = u'*u"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"25.0"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sum( abs(u).^2 )"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"5.0"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"norm_u = norm(u)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It also defines the angle between two vectors as a generalization of the familiar dot product."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1-element Array{Complex{Float64},1}:\n",
" -0.23094-0.34641im"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cos_theta = (u'*v) / ( norm(u)*norm(v) )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The angle may be complex when the vectors are complex! "
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1-element Array{Complex{Float64},1}:\n",
" 1.79019+0.34786im"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"theta = acos(cos_theta)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The operations of inverse and hermitian commute."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"4x4 Array{Complex{Float64},2}:\n",
" -0.970125+3.47419im -3.04879+1.46845im … 2.83379-1.633im \n",
" 1.77428-2.22706im 3.23757-1.19382im -3.33357+1.1817im \n",
" -0.989828-0.524151im -1.42288-0.448626im 0.77963+1.09731im \n",
" 1.22113-0.524663im 1.84886+0.933936im -0.551587-0.808527im"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A = rand(4,4)+1im*rand(4,4); (inv(A))'"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"4x4 Array{Complex{Float64},2}:\n",
" -0.970125+3.47419im -3.04879+1.46845im … 2.83379-1.633im \n",
" 1.77428-2.22706im 3.23757-1.19382im -3.33357+1.1817im \n",
" -0.989828-0.524151im -1.42288-0.448626im 0.77963+1.09731im \n",
" 1.22113-0.524663im 1.84886+0.933936im -0.551587-0.808527im"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"inv(A')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So we just write $\\mathbf{A}^{-*}$ for either case. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Orthogonality"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Orthogonality, which is the multidimensional extension of perpendicularity, means that $\\cos \\theta=0$, i.e., that the inner product between vectors is zero. A collection of vectors is orthogonal if they are all pairwise orthogonal. \n",
"\n",
"Don't worry about how we are creating the vectors here for now."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"5x3 Array{Float64,2}:\n",
" -0.0837883 -0.0420715 0.960899 \n",
" -0.0200034 -0.426554 0.116046 \n",
" -0.00473062 -0.901772 -0.113508 \n",
" -0.51613 0.0539206 -0.221186 \n",
" -0.852154 -0.0135028 0.0373927"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Q = qr(rand(5,3))[1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since $\\mathbf{Q}^*\\mathbf{Q}$ is a matrix of all inner products between columns of $\\mathbf{Q}$, those columns are orthogonal if and only if that matrix is diagonal."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"3x3 Array{Float64,2}:\n",
" 1.0 1.75626e-17 -5.17635e-16\n",
" 1.75626e-17 1.0 -2.94949e-17\n",
" -5.17635e-16 -2.94949e-17 1.0 "
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"QhQ = Q'*Q"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In fact we have a stronger condition here: the columns are **orthonormal**, meaning that they are orthogonal and each has 2-norm equal to 1. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Given any other vector of length 5, we can compute its inner product with each of the columns of $\\mathbf{Q}$. "
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"3x1 Array{Float64,2}:\n",
" -0.81406 \n",
" -0.320388\n",
" 0.335601"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"u = rand(5,1); c = Q'*u"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can then use these coefficients to find a vector in the column space of $\\mathbf{Q}$."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"5x1 Array{Float64,2}:\n",
" 0.404167\n",
" 0.191892\n",
" 0.254675\n",
" 0.328655\n",
" 0.71058 "
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"v = Q*c"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As explained in the text, $\\mathbf{r} = \\mathbf{u}-\\mathbf{v}$ is orthogonal to all of the columns of $\\mathbf{Q}$."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"3x1 Array{Float64,2}:\n",
" 5.1683e-16 \n",
" -1.24441e-16\n",
" -7.61287e-16"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"r = u-v; Q'*r"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Consequently, we have decomposed $\\mathbf{u}=\\mathbf{v}+\\mathbf{r}$ into the sum of two orthogonal parts, one lying in the range of $\\mathbf{Q}$. "
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1x1 Array{Float64,2}:\n",
" -6.28355e-16"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"v'*r"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Unitary matrices"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We just saw that a matrix whose columns are orthonormal is pretty special. It becomes even more special if the matrix is also square, in which case we call it **unitary**. (In the real case, such matrices are confusingly called _orthogonal_. Ugh.) Say $\\mathbf{Q}$ is unitary and $m\\times m$. Then $\\mathbf{Q}^*\\mathbf{Q}$ is an $m\\times m$ identity matrix---that is, $\\mathbf{Q}^*=\\mathbf{Q}^{-1}$! It can't get much easier in terms of finding the inverse of a matrix. "
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"5x5 Array{Float64,2}:\n",
" 5.55112e-17 6.20634e-17 1.08389e-16 1.18572e-16 0.0 \n",
" 1.14439e-16 3.826e-16 6.20634e-17 4.00297e-16 1.17757e-16\n",
" 5.55112e-17 1.05005e-16 1.24127e-16 6.73645e-17 7.85046e-17\n",
" 2.22478e-16 4.00297e-16 5.59432e-17 1.24127e-16 5.55112e-17\n",
" 2.00148e-16 1.11886e-16 2.1138e-16 2.22045e-16 2.80661e-16"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Q = qr(rand(5,5)+1im*rand(5,5))[1]\n",
"abs( inv(Q) - Q' )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The rank of $\\mathbf{Q}$ is $m$, so continuing the discussion above, the original vector $\\mathbf{u}$ lies in its column space. Hence the remainder $\\mathbf{r}=\\boldsymbol{0}$. "
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"5x1 Array{Complex{Float64},2}:\n",
" -1.11022e-16-0.0im\n",
" -1.11022e-16-2.77556e-17im \n",
" 5.55112e-17-2.77556e-17im \n",
" -1.11022e-16-0.0im\n",
" -1.11022e-16+8.32667e-17im "
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"c = Q'*u; \n",
"v = Q*c;\n",
"r = u - v"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is another way to arrive at a fact we already knew: Multiplication by $\\mathbf{Q}^*=\\mathbf{Q}^{-1}$ changes the basis to the columns of $\\mathbf{Q}$."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Julia 0.4.3",
"language": "julia",
"name": "julia-0.4"
},
"language_info": {
"file_extension": ".jl",
"mimetype": "application/julia",
"name": "julia",
"version": "0.4.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment