Analyse data files directly from the LCO archive

Opening remote LCO FITS files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a short notebook illustrating how to open LCO data in Python, without having to download to local"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image, display\n",
    "from IPython.core.display import HTML\n",
    "from astropy.io import fits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://s3.us-west-2.amazonaws.com/archive.lcogt.net/ecbb/tfn0m410-kb23-20191014-0276-d00?versionId=oXt_GmysfvtAisA_shUdK4ptpnAiynsz&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA6FT4CXR464A32PW2%2F20191015%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20191015T080117Z&X-Amz-Expires=172800&X-Amz-SignedHeaders=host&X-Amz-Signature=accd7086bf14ff0c7ac04aaaf7bd06faf2e346a6600ef0751ab174124b92cf3c'\n",
    "hdul = fits.open(url)\n",
    "hdul.info()\n",
    "data = hdul[1].data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is what we want but its a bit faff because you need to look up the URL of each file. If only there was a way to do ths with code..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "TOKEN = \"<TOKEN>\"\n",
    "PORTAL_API = 'https://observe.lco.global/api/requestgroups/{}/'\n",
    "headers = {'Authorization': 'Token {}'.format(TOKEN)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do a quick test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "request_id = '866850'\n",
    "url = PORTAL_API.format(request_id)\n",
    "r = requests.get(url, headers=headers, timeout=20.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(r.json()['state'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the state is COMPLETED then we can go and look in the archive for the files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "request_ids = [rs['id'] for rs in r.json()['requests']]\n",
    "print(request_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to log in *again* to get the token for the data archive (which is different to your portal token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ARCHIVE_API = 'https://archive.lco.global/?REQNUM={}&start=2014-01-01'\n",
    "ar = requests.post(\n",
    "    'https://archive-api.lco.global/api-token-auth/',\n",
    "    data = {\n",
    "        'username': '<USERNAME>',\n",
    "        'password': '<PASSWORD>'\n",
    "    }\n",
    ")\n",
    "ar_token = ar.json()['token']\n",
    "archive_headers={'Authorization': 'Token {}'.format(ar_token)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we find our data files in the archive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reqnum = 1952010 # Change this to the ID of your \"sub-request\" from the Observing Portal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ar = requests.get(f'https://archive-api.lco.global/frames/?REQNUM={reqnum}&start=2014-01-01&RLEVEL=91',headers=archive_headers,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ar.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If that says `200` then everything should be fine and the authentication has worked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"You have {} data files \".format(ar.json()['count']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can go through all of our results to get thumbnails of our data and display them, along with the frame id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for req in ar.json()['results'][0:2]:\n",
    "    tb = requests.get('https://thumbnails.lco.global/{}/'.format(req['id'])).json()\n",
    "    display(Image(url= tb['url'], width=500, height=500))\n",
    "    print(req['id'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also run through the data and stream the FITS data files, doing whatever science you need. For illustrative purposes, I'll just inspect the FITS header info of each file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for req in ar.json()['results'][0:2]:\n",
    "    with fits.open(req['url']) as hdul:\n",
    "        print(hdul.info())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}