mass infinity sketch

A_list_of_tasks.md

What tasks should mass_infinity accomplish?

• Take raw data records of fixed length and noise. Return basis for analyzing that data, and future data.
  • Requires cuts to select down to clean pulses from which to make the basis. This suggests a sort of self consistent cut procedure. Luckily Alpert et all describe such a procedure https://link.springer.com/article/10.1007/s10909-015-1402-y
• Take raw data records. Return subspace representation + residual std deviation.
• Take subspace representation and spectral information. Return rough calibration. J?
• Take J and ??. Return various corrected Js.
  • Baseline-J Correlation Correction
  • Subsample arrival time Correction
  • Time Drift Correction
  • any combination thereof (to evaluate if correction improves energy resolution)
• Take Corrected J and return best Calibation.
• Take PulseHeightEstimators and Sepctral Information. Return Energy Resolution achieved.
  • Often redundant with best Calibration work.
• Take Basis (with embedded noise info?) and Calibration. Return predicted energy resolution.
• Write pulse in subspace representation + residual + timestamp to disk.
• Write Recipe output+timestamp to disk.

What quality of life features should it include?

• Handle failed pixels.
  • Continue with analysis of good pixels.
  • Allow inspection of failed pixels easily.

Possible Terms

• Pulse: the raw microcalorimter data representing currentt vs time in the time domain
• Noise: event free raw microcalorimter data used to learn about noise properties of microcalorimeter data
• Basis: A set of N+1 orthogonal vectors defining the subspace in which we will represent a pulse
• psrr: A pulse in the subspace represtation, with residual value, a Vector of length N+1

mass_infinity_sketch.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Base.Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we define Channel (equivalent of a dataset in mass). SharedState plays most of the role of ChannelGroup, and we abstract out the readout."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "mutable struct Channel{T,S}\n",
    "    sharedstate::T\n",
    "    datasource::S\n",
    "    basis::Array{Float64}\n",
    "    components::Array{Float64}\n",
    "    recipes::Dict{String,Any}\n",
    "end\n",
    "npulses(c::Channel) = size(c.components,2)\n",
    "Base.getindex(c::Channel,s::String) = c.recipes[s]()\n",
    "function markbad(c::Channel, ex) \n",
    "    println(c.datasource,\" marked bad\")\n",
    "    c.recipes[\"marked bad by this exception\"]=ex\n",
    "end\n",
    "\n",
    "struct SharedState{T}\n",
    "    readout::T\n",
    "    other::Dict{String,Any}\n",
    "end\n",
    "\n",
    "struct TDMSharedInfo\n",
    "    ncol::Int\n",
    "    nrow::Int\n",
    "    frametime_s::Float64\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we define a recipe, a lazy way of describing how data products result from out data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "struct Recipe{T,S,F}\n",
    "    components::T\n",
    "    input::S\n",
    "    transform::F\n",
    "end\n",
    "function (r::Recipe)()\n",
    "    r.transform(r.input(),r.components)\n",
    "end\n",
    "struct DriftCorrect\n",
    "    a::Float64\n",
    "    b::Float64\n",
    "end\n",
    "function (dc::DriftCorrect)(input,components)\n",
    "    components[1,:]*dc.a+components[2,:]*dc.b\n",
    "end\n",
    "struct PhaseCorrect\n",
    "    a::Float64\n",
    "end\n",
    "function (pc::PhaseCorrect)(input::Vector{Float64}, components)\n",
    "    input+components[3,:]*pc.a\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here I define forall, a function that can be used to loop over all channels, while catching errors. This replaces approximatley 1000 lines in mass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "forall (generic function with 1 method)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function forall(f,channels)\n",
    "    for c in channels\n",
    "        try\n",
    "            f(c)\n",
    "        catch ex\n",
    "            markbad(c,ex)\n",
    "        end\n",
    "    end\n",
    "end    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we instantiate some variables, and check that recipes actually work. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[1m\u001b[32mTest Passed\n",
       "\u001b[39m\u001b[22m"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tdminfo = TDMSharedInfo(8,24,320e-9)\n",
    "sharedstate = SharedState(tdminfo,Dict{String,Any}(\"date\"=>\"today\"))\n",
    "\n",
    "# 1000 sample pulses, 6 element basis, 100 pulses\n",
    "channels = [Channel(sharedstate,\"ljhfilechan$i\",rand(1000,6),rand(6,100),Dict{String,Any}()) for i=1:10];\n",
    "c=channels[1]\n",
    "dc_recipe = Recipe(c.components,()->nothing,DriftCorrect(1,2))\n",
    "pc_recipe = Recipe(c.components,dc_recipe, PhaseCorrect(4))\n",
    "c.recipes[\"dc\"]=dc_recipe\n",
    "c.recipes[\"pc\"]=pc_recipe\n",
    "dc_recipe();\n",
    "pc_recipe();\n",
    "@test all(c[\"pc\"].==pc_recipe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we test the `forall` by adding recipes to each channel, and calling the \"pc\" recipe. Channel 1 will fail because it already has recipes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ljhfilechan1\n",
      "ljhfilechan1 marked bad\n",
      "ljhfilechan2\n",
      "ljhfilechan3\n",
      "ljhfilechan4\n",
      "ljhfilechan5\n",
      "ljhfilechan6\n",
      "ljhfilechan7\n",
      "ljhfilechan8\n",
      "ljhfilechan9\n",
      "ljhfilechan10\n"
     ]
    }
   ],
   "source": [
    "forall(channels) do c\n",
    "    println(c.datasource)\n",
    "    if !isempty(c.recipes)\n",
    "        error(\"already has recipes\")\n",
    "    end\n",
    "    dc_recipe = Recipe(c.components,()->nothing,DriftCorrect(rand(),rand()))\n",
    "    pc_recipe = Recipe(c.components,dc_recipe, PhaseCorrect(rand()))   \n",
    "    c.recipes[\"dc\"]=dc_recipe\n",
    "    c.recipes[\"pc\"]=pc_recipe\n",
    "    c[\"pc\"]\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets just check that the exception is in the right place. This is just a sketch, so I just threw in the dict I already had."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String,Any} with 3 entries:\n",
       "  \"pc\"                           => Recipe{Array{Float64,2},Recipe{Array{Float6…\n",
       "  \"dc\"                           => Recipe{Array{Float64,2},##3#4,DriftCorrect}…\n",
       "  \"marked bad by this exception\" => ErrorException(\"already has recipes\")"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(c.recipes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we define a simple version of the categorical cut.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "struct Selections{T}\n",
    "    v::Vector{T}\n",
    "end\n",
    "function Base.getindex(s::Selections{T},a::T) where T\n",
    "    s.v.==a\n",
    "end\n",
    "struct SelectableVector{S,T}\n",
    "    selector::Selections{S}\n",
    "    v::T\n",
    "end\n",
    "function Base.getindex(s::SelectableVector{S,T},a::S) where {S,T}\n",
    "    s.v[s.selector[a]]\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4-element Array{Array{Int64,1},1}:\n",
       " [4, 8]    \n",
       " [1, 5, 9] \n",
       " [2, 6, 10]\n",
       " [3, 7]    "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@enum Sides A B C D\n",
    "mockdata = collect(1:10)\n",
    "labels = [Sides(i%4) for i in mockdata]\n",
    "s = Selections(labels)\n",
    "selectabledata = SelectableVector(s,mockdata)\n",
    "[selectabledata[x] for x in [A,B,C,D]]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets apply the categorical cut to one of the recipes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25-element Array{Float64,1}:\n",
       " 0.558354 \n",
       " 0.412467 \n",
       " 0.544715 \n",
       " 0.71094  \n",
       " 0.893479 \n",
       " 0.231677 \n",
       " 0.308937 \n",
       " 0.838057 \n",
       " 0.71602  \n",
       " 0.346693 \n",
       " 0.975361 \n",
       " 0.502465 \n",
       " 0.68475  \n",
       " 0.490839 \n",
       " 0.0758445\n",
       " 0.304326 \n",
       " 0.235888 \n",
       " 0.440229 \n",
       " 0.619837 \n",
       " 0.615173 \n",
       " 0.454734 \n",
       " 0.485533 \n",
       " 0.752814 \n",
       " 0.70363  \n",
       " 0.379467 "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = channels[2]\n",
    "labels_c = [Sides(i%4) for i in 1:npulses(c)]\n",
    "s_c = Selections(labels_c)\n",
    "selectable_recipe_output = SelectableVector(s_c,c[\"pc\"])\n",
    "selectable_recipe_output[A]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 0.6.0",
   "language": "julia",
   "name": "julia-0.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "0.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}