Created on Cognitive Class Labs

ML0101EN-Reg-NoneLinearRegression-py-v1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.bigdatauniversity.com\"><img src = \"https://ibm.box.com/shared/static/cw2c7r3o20w9zn8gkecaeyjhgw3xdgbj.png\" width=\"400\" align=\"center\"></a>\n",
    "\n",
    "<h1><center>Non Linear Regression Analysis</center></h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear. \n",
    "Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"importing_libraries\">Importing required libraries</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.arange(-5.0, 5.0, 0.1)\n",
    "\n",
    "##You can adjust the slope and intercept to verify the changes in the graph\n",
    "y = 2*(x) + 3\n",
    "y_noise = 2 * np.random.normal(size=x.size)\n",
    "ydata = y + y_noise\n",
    "#plt.figure(figsize=(8,6))\n",
    "plt.plot(x, ydata,  'bo')\n",
    "plt.plot(x,y, 'r') \n",
    "plt.ylabel('Dependent Variable')\n",
    "plt.xlabel('Indepdendent Variable')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$). \n",
    "\n",
    "$$ \\ y = a x^3 + b x^2 + c x + d \\ $$\n",
    "\n",
    "Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \\log(x)$$\n",
    "    \n",
    "Or even, more complicated such as :\n",
    "$$ y = \\log(a x^3 + b x^2 + c x + d)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at a cubic function's graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = np.arange(-5.0, 5.0, 0.1)\n",
    "\n",
    "##You can adjust the slope and intercept to verify the changes in the graph\n",
    "y = 1*(x**3) + 1*(x**2) + 1*x + 3\n",
    "y_noise = 20 * np.random.normal(size=x.size)\n",
    "ydata = y + y_noise\n",
    "plt.plot(x, ydata,  'bo')\n",
    "plt.plot(x,y, 'r') \n",
    "plt.ylabel('Dependent Variable')\n",
    "plt.xlabel('Indepdendent Variable')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some other types of non-linear functions are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quadratic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ Y = X^2 $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = np.arange(-5.0, 5.0, 0.1)\n",
    "\n",
    "##You can adjust the slope and intercept to verify the changes in the graph\n",
    "\n",
    "y = np.power(x,2)\n",
    "y_noise = 2 * np.random.normal(size=x.size)\n",
    "ydata = y + y_noise\n",
    "plt.plot(x, ydata,  'bo')\n",
    "plt.plot(x,y, 'r') \n",
    "plt.ylabel('Dependent Variable')\n",
    "plt.xlabel('Indepdendent Variable')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exponential"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = np.arange(-5.0, 5.0, 0.1)\n",
    "\n",
    "##You can adjust the slope and intercept to verify the changes in the graph\n",
    "\n",
    "Y= np.exp(X)\n",
    "\n",
    "plt.plot(X,Y) \n",
    "plt.ylabel('Dependent Variable')\n",
    "plt.xlabel('Indepdendent Variable')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logarithmic\n",
    "\n",
    "The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of __log()__: i.e. $$ y = \\log(x)$$\n",
    "\n",
    "Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as  \n",
    "\\begin{equation}\n",
    "y = \\log(X)\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = np.arange(-5.0, 5.0, 0.1)\n",
    "\n",
    "Y = np.log(X)\n",
    "\n",
    "plt.plot(X,Y) \n",
    "plt.ylabel('Dependent Variable')\n",
    "plt.xlabel('Indepdendent Variable')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sigmoidal/Logistic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ Y = a + \\frac{b}{1+ c^{(X-d)}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.arange(-5.0, 5.0, 0.1)\n",
    "\n",
    "\n",
    "Y = 1-4/(1+np.power(3, X-2))\n",
    "\n",
    "plt.plot(X,Y) \n",
    "plt.ylabel('Dependent Variable')\n",
    "plt.xlabel('Indepdendent Variable')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref2\"></a>\n",
    "# Non-Linear Regression example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "#downloading dataset\n",
    "!wget -nv -O china_gdp.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv\n",
    "    \n",
    "df = pd.read_csv(\"china_gdp.csv\")\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Did you know?__ When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting the Dataset ###\n",
    "This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8,5))\n",
    "x_data, y_data = (df[\"Year\"].values, df[\"Value\"].values)\n",
    "plt.plot(x_data, y_data, 'ro')\n",
    "plt.ylabel('GDP')\n",
    "plt.xlabel('Year')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choosing a model ###\n",
    "\n",
    "From an initial look at the plot, we determine that the logistic function could be a good approximation,\n",
    "since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = np.arange(-5.0, 5.0, 0.1)\n",
    "Y = 1.0 / (1.0 + np.exp(-X))\n",
    "\n",
    "plt.plot(X,Y) \n",
    "plt.ylabel('Dependent Variable')\n",
    "plt.xlabel('Indepdendent Variable')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "The formula for the logistic function is the following:\n",
    "\n",
    "$$ \\hat{Y} = \\frac1{1+e^{\\beta_1(X-\\beta_2)}}$$\n",
    "\n",
    "$\\beta_1$: Controls the curve's steepness,\n",
    "\n",
    "$\\beta_2$: Slides the curve on the x-axis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building The Model ###\n",
    "Now, let's build our regression model and initialize its parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x, Beta_1, Beta_2):\n",
    "     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))\n",
    "     return y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look at a sample sigmoid line that might fit with the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "beta_1 = 0.10\n",
    "beta_2 = 1990.0\n",
    "\n",
    "#logistic function\n",
    "Y_pred = sigmoid(x_data, beta_1 , beta_2)\n",
    "\n",
    "#plot initial prediction against datapoints\n",
    "plt.plot(x_data, Y_pred*15000000000000.)\n",
    "plt.plot(x_data, y_data, 'ro')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our task here is to find the best parameters for our model. Lets first normalize our x and y:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lets normalize our data\n",
    "xdata =x_data/max(x_data)\n",
    "ydata =y_data/max(y_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How we find the best parameters for our fit line?\n",
    "we can use __curve_fit__ which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, *popt) - ydata is minimized.\n",
    "\n",
    "popt are our optimized parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.optimize import curve_fit\n",
    "popt, pcov = curve_fit(sigmoid, xdata, ydata)\n",
    "#print the final parameters\n",
    "print(\" beta_1 = %f, beta_2 = %f\" % (popt[0], popt[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we plot our resulting regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(1960, 2015, 55)\n",
    "x = x/max(x)\n",
    "plt.figure(figsize=(8,5))\n",
    "y = sigmoid(x, *popt)\n",
    "plt.plot(xdata, ydata, 'ro', label='data')\n",
    "plt.plot(x,y, linewidth=3.0, label='fit')\n",
    "plt.legend(loc='best')\n",
    "plt.ylabel('GDP')\n",
    "plt.xlabel('Year')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practice\n",
    "Can you calculate what is the accuracy of our model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click __here__ for the solution.\n",
    "\n",
    "<!-- Your answer is below:\n",
    "    \n",
    "# split data into train/test\n",
    "msk = np.random.rand(len(df)) < 0.8\n",
    "train_x = xdata[msk]\n",
    "test_x = xdata[~msk]\n",
    "train_y = ydata[msk]\n",
    "test_y = ydata[~msk]\n",
    "\n",
    "# build the model using train set\n",
    "popt, pcov = curve_fit(sigmoid, train_x, train_y)\n",
    "\n",
    "# predict using test set\n",
    "y_hat = sigmoid(test_x, *popt)\n",
    "\n",
    "# evaluation\n",
    "print(\"Mean absolute error: %.2f\" % np.mean(np.absolute(y_hat - test_y)))\n",
    "print(\"Residual sum of squares (MSE): %.2f\" % np.mean((y_hat - test_y) ** 2))\n",
    "from sklearn.metrics import r2_score\n",
    "print(\"R2-score: %.2f\" % r2_score(y_hat , test_y) )\n",
    "\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Want to learn more?</h2>\n",
    "\n",
    "IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: <a href=\"http://cocl.us/ML0101EN-SPSSModeler\">SPSS Modeler</a>\n",
    "\n",
    "Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at <a href=\"https://cocl.us/ML0101EN_DSX\">Watson Studio</a>\n",
    "\n",
    "<h3>Thanks for completing this lesson!</h3>\n",
    "\n",
    "<h4>Author:  <a href=\"https://ca.linkedin.com/in/saeedaghabozorgi\">Saeed Aghabozorgi</a></h4>\n",
    "<p><a href=\"https://ca.linkedin.com/in/saeedaghabozorgi\">Saeed Aghabozorgi</a>, PhD is a Data Scientist in IBM with a track record of developing enterprise level applications that substantially increases clients’ ability to turn data into actionable knowledge. He is a researcher in data mining field and expert in developing advanced analytic methods like machine learning and statistical modelling on large datasets.</p>\n",
    "\n",
    "<hr>\n",
    "\n",
    "<p>Copyright &copy; 2018 <a href=\"https://cocl.us/DX0108EN_CC\">Cognitive Class</a>. This notebook and its source code are released under the terms of the <a href=\"https://bigdatauniversity.com/mit-license/\">MIT License</a>.</p>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}