Stat Distribution Report

01 KIM - Statistical Distribution.ipynb

{
  "cells": [
    {
      "metadata": {
        "collapsed": false,
        "trusted": true
      },
      "cell_type": "code",
      "source": "%load_ext autoreload\n%reload_ext autoreload\n%autoreload 2\nimport datetime as dt\nimport sys; sys.path.append('/home/karl/notebooks/Helpers')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "%matplotlib inline\nimport graphlab as gl\nfrom IPython.display import Image\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set(color_codes=True)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Import Raw Data"
    },
    {
      "metadata": {
        "collapsed": false,
        "trusted": true
      },
      "cell_type": "code",
      "source": "from Helpers.Analysis import Transform as tf\nfrom Helpers.RawData import C1\nfrom Helpers.Analysis import Detection as dt",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "wp_sf = C1.wp_raw_recovery_dcs_csv_only()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "wp_df = wp_sf.to_dataframe()\nwp_df.index = pd.to_datetime(wp_df.timestamp)\nwp_df.sort_index(inplace=True)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "\"Total Number of minutes: {} days\".format((len(wp_df)/60)/24)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Prep Data "
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "cons_tph_tagname = '03_mfi_330_daca_pv'\nrougher_tph_tagname = 'a_b_mass_flow_daca_pv'\ntag_list = {'rougher_tph': 'a_b_mass_flow_daca_pv',\n            'cons_tph': '03_mfi_330_daca_pv'}\nrename_cols = dict(zip(tag_list.values(), tag_list.keys()))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "wp_df[[cons_tph_tagname,rougher_tph_tagname]].rename(columns=rename_cols).describe()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Observations\n* Min Values\n     * HMC TPH dips below zero\n     * Rougher TPH dips below zero\n* Max Values\n    * HMC TPH is extreme \n    * Rougher TPH is extreme\n    \n**Root Cause = Bad instrumentation and DCS Calculations e.g. Div0**"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Apply Low Limit cutoff to remove negative values"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "wp_df.loc[wp_df[cons_tph_tagname]<0, cons_tph_tagname]= 0\nwp_df.loc[wp_df[rougher_tph_tagname]<0, rougher_tph_tagname]= 0",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "wp_df[[cons_tph_tagname,rougher_tph_tagname]].rename(columns=rename_cols).describe()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Remove Outliers"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Remove Bad HMC Data"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "plt.plot(wp_df['03_mfi_330_daca_pv'].quantile(np.arange(0,1,0.01)))\nprint 'HMC Reading at 99% Percentile: {}'.format(wp_df['03_mfi_330_daca_pv'].quantile(0.99))\nprint 'HMC Reading at 99.9% Percentile: {}'.format(wp_df['03_mfi_330_daca_pv'].quantile(0.999))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Remove Values above the 99 % Rank"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "wp_df= wp_df[wp_df[cons_tph_tagname] < wp_df[cons_tph_tagname].quantile(0.99)]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "wp_df[cons_tph_tagname].describe()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Drop Bad HMC data\ni.e. Drop all values which are above 99 percentile"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "plt.plot(wp_df[rougher_tph_tagname].quantile(np.arange(0,1,0.01)))\nprint 'Rougher Reading at 99% Percentile: {}'.format(wp_df[rougher_tph_tagname].quantile(0.99))\nprint 'Rougher Reading at 99.999% Percentile: {}'.format(wp_df[rougher_tph_tagname].quantile(0.99999))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "wp_df= wp_df[wp_df[rougher_tph_tagname] < wp_df[rougher_tph_tagname].quantile(0.99)]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "wp_df[rougher_tph_tagname].describe()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "print(\"Data Ready for Analysis \\n\")\nprint (\"Total Number of minutes: {} [{} days]\".format(len(wp_df),\n                                                      (len(wp_df)/60)/24))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "wp_5min_mean_df = wp_df.resample('5T').mean()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Statistical Distribution of the following for the whole of last year "
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Question 1\n1) Spud Move times (exclude cases where fault occurs during the move)"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Question 2\n2) Centreline change times (exclude cases where fault occurs during the move)"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Question 3\n3) Anchor Move times"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Question 4  \n### Calc HMC:Rougher Ratio, while Rougher < 1000 TPH\n5 min Average Ratio of HMC/Rougher tph when total Rougher tph < 1000 tph (exclude the cases when the plant is down)"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "q4_df= wp_5min_mean_df.loc[(wp_5min_mean_df[rougher_tph_tagname]<1000) & \n                           (wp_5min_mean_df[cons_tph_tagname]>0) & \n                           (wp_5min_mean_df[rougher_tph_tagname]>0), \n                           tag_list.values()].rename(columns=rename_cols)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "q4_df['hmc_rougher_ratio'] = q4_df['cons_tph']/ q4_df['rougher_tph']\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Q4 Answer"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "sns.distplot(q4_df['hmc_rougher_ratio'] )",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "sns.violinplot(q4_df['hmc_rougher_ratio'])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "q4_df.describe()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Question 5 \n### Total Rougher Duration\n#### A) Rougher < 1000 TPH\n#### B) Rougher > 1000 TPH\n\nDuration when Rougher tph when total Rougher tph < 1000 tph (between periods of > 1000 tph)"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "q5_rougher_value_bins = [0, \n                         1000, \n                         wp_df[rougher_tph_tagname].max()]\n\nq5_rougher_labels = ['less_than_1k',\n                     'great_than_eq_1k']\nq5_df = pd.cut(wp_df[rougher_tph_tagname], \n               bins=q5_rougher_value_bins, \n               labels=q5_rougher_labels).value_counts()\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Q5) Answer"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "q5_results = {}\nfor label, total_number_of_minutes in q5_df.iteritems():\n    q5_results[label] = total_number_of_minutes\n    print \"Total Time when Rougher is {}: {} mins [{} days]\".format(label, \n                                                                    total_number_of_minutes,\n                                                                    (total_number_of_minutes/60)/24)\npercent_less_than_1k = int((q5_results['less_than_1k']/float(q5_results['great_than_eq_1k']))*100)\nprint('\\n')\nprint('Wetplant Rougher < 1000 TPN: {}%'.format(percent_less_than_1k))\nprint('\\n')\nprint('Wetplant Rougher > 1000 TPN: {}%'.format(100-percent_less_than_1k))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Question 6\n5min Average Ratio of HMC/Rougher tph when total Rougher tph >1500 tph"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "q6_df= wp_5min_mean_df.loc[(wp_5min_mean_df[rougher_tph_tagname]>1500) &\n                           (wp_5min_mean_df[cons_tph_tagname]>0) &\n                           (wp_5min_mean_df[rougher_tph_tagname]>0), \n                           tag_list.values()].rename(columns=rename_cols)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "q6_df['hmc_rougher_ratio'] = q6_df['cons_tph']/ q6_df['rougher_tph']",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "sns.distplot(q6_df['hmc_rougher_ratio'])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "sns.violinplot(q6_df['hmc_rougher_ratio'] )",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "q6_df.describe()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Question 7\nDuration when Rougher tph when total Rougher tph >1500 tph ( between periods < 1500tph )"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": false
      },
      "cell_type": "code",
      "source": "q7_rougher_bins = [0,1500,wp_df[rougher_tph_tagname].max()]\nq7_rougher_labels = ['less_than_eq_1k5','greater_than_1k5']\nq7_df = pd.cut(wp_df[rougher_tph_tagname], bins=q7_rougher_bins, labels=q7_rougher_labels).value_counts()\n\nq7_results = {}\nfor label, total_number_of_minutes in q7_df.iteritems():\n    q7_results[label] = total_number_of_minutes\n    print \"Total Time when Rougher is {}: {} mins [{} days]\".format(label, \n                                                                    total_number_of_minutes,\n                                                                    (total_number_of_minutes/60)/24)\npercent_less_than_1k5 = int((q7_results['less_than_eq_1k5']/float(q7_results['greater_than_1k5']))*100)\nprint('\\n')\nprint('Wetplant Rougher < 1500 TPN: {}%'.format(percent_less_than_1k5))\nprint('\\n')\nprint('Wetplant Rougher > 1500 TPN: {}%'.format(100-percent_less_than_1k5))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "8)Spud carriage position at the initiation of spud move (when main spud is raised, aux spud is dropped and dredge heading is zero)"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "9)Spud carriage position at the end of spud move (when aux spud is lifted, main spud is dropped and dredge heading is zero)"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": ""
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Please help us to collect the following data. We would like to use the data to justify the potential improvement from implementing an integrated dredging automation.  If possible, we would like the data within the next 3 weeks.\n\n\n\n5)Duration when Rougher tph when total Rougher tph <1000 tph (between periods of >1000tph)\n6)5min Average Ratio of HMC/Rougher tph when total Rougher tph >1500 tph\n7)Duration when Rougher tph when total Rougher tph >1500 tph (between periods <1500tph)\n8)Spud carriage position at the initiation of spud move (when main spud is raised, aux spud is dropped and dredge heading is zero)\n9)Spud carriage position at the end of spud move (when aux spud is lifted, main spud is dropped and dredge heading is zero)\n\nMining and processing performance:\n1.      Total tonnes dredged last year dredging per shift team (Pelican and Cooljarloo 1 separately)\n2.      Total HMC tonnes last year per dredging shift team (Pelican and Cooljarloo 1 separately)\n3.      Total duration over the year when Rougher tph <1000tph (sum of item 5 from the Statistical data)\n4.      Total duration over the year when Rougher tph >1500tph (sum of item 7 from the Statistical data)\n5.      Total number to cutter trip events per dredging shift team\n6.      Total number of winch overload events per dredging shift team\n7.      Total number of feed bin 1 or 2 high level events per dredging shift team\n\nOur client contact at Rio Tinto QMM for Dredge automation and WP Coordination Control Projects  is Matthew Neighbour ([email protected]).  Please feel free to get feedback from him on the Hatch Integrated Dredging Technology.  Our current Hatch Integrated Dredging Technology includes:\n1.      Import mine plan to derive executable Permissive Manoevring Region (PMRs) and corresponding anchor sets (anchor positioning are optimized to achieved the largest possible PMR) through the Hatch MineView software.\n2.      The MineView automatically sends the PMR and anchor sets the control system for execution.\n3.      The Dredge automation includes automatic slew control, automatic ladder positioning, automatic spud move and centerline change using direct GPS feedback.\n4.      The dredge mining sequence allow configuration of number face mining slews before commencing clean up sequence.  \n5.      The Sweep sequence is initiated at the end of PMR to minimize anchor moves.  However, operator can initiate any of these modes manually as well.\n6.      The system allow to operator to initiate either face mining, clean up and sweeping to start the mining sequence.  Once started, the system automatically sequence through the different mining mode until all centerlines within PMR are completed.  \n7.      During mining, if the system detects bulldozing event, it will also initiate a slump recovery sequence before returning to mining.\n8.      The system will prompt the operator to move the anchor to a proposed set of coordinates near to the completion of all the centerline within the PMR.\n9.      After the anchor move and the operator has confirm the anchor positions, then the operator can initiate the next mining sequence.\n10.   The system also allow relocation of dredge to a specified cutter location and relocation of dredge to a specified centerline.\n11.   The dredge slewing speed is automatically control to maximize throughput with the constraint wet plant bin inventory, mining hardness, pump constraint, winch torque constraint, cutter force, etc.\n12.   The Human Man Interface (HMI) provides a view of the previous, current and next PMR and its associated channel with cutter trajectories with different colors for different modes (mining, cleanup and sweep)\n13.   The MineView also provide “heatmap” showing cutter depth compliance to mine plan, average tph per mode, etc. to facilitate future optimization of mining sequence and planning.\n14.   The system also monitors the distance between the WCP and dredge to protect the feed line and prevents collision.\n\nIf needed, it would be our pleasure to schedule presentation or  demo of the Hatch Integrated Dredging Technology at our office.\n\nPlease do not hesitate to call me if you have any queries.  \n\nRegards,\n"
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python2",
      "display_name": "Python 2",
      "language": "python"
    },
    "language_info": {
      "mimetype": "text/x-python",
      "nbconvert_exporter": "python",
      "name": "python",
      "pygments_lexer": "ipython2",
      "version": "2.7.13",
      "file_extension": ".py",
      "codemirror_mode": {
        "version": 2,
        "name": "ipython"
      }
    },
    "gist": {
      "id": "",
      "data": {
        "description": "Stat Distribution Report",
        "public": true
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}