{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# On importe les librairies\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 - Créer une fonction J(theta qui implémente J(theta). Quelle est la valeur de de J(4) ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def J(theta_i):\n",
    "    return (theta_i + 1)**2\n",
    "\n",
    "J(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# On génère les points dans theta\n",
    "theta = np.arange(-10.00, 10.00, 0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3\n",
    "Plot theta vs J(theta). When do we reach the minimum?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15476 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot de J(theta)\n",
    "plt.plot(theta,J(theta), label='Fitted line - closed form')\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('J(x)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "J_theta = [J(x) for x in theta]\n",
    "# Valeur de theta à laquelle on obtien le plus petit J(theta)\n",
    "round(theta[J_theta.index(min(J_theta))], 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4\n",
    "Create a function dJ_dtheta(x_i) which is computing the gradient\n",
    "?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# On calcule la fonction dérivée. D'après J(theta). Merci dcode.fr\n",
    "def dJ_theta(x_i):\n",
    "    return 2 * (x_i + 1)\n",
    "# Affichage de la valeur à 0\n",
    "dJ_theta(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fab8eb08438>]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 12760 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(theta, dJ_theta(theta))\n",
    "# dJ_t = [dJ_theta(x) for x in theta]\n",
    "# plt.plot(theta, dJ_t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5\n",
    "Create a function gradient_descent(theta_0, lr, nb_iters, df_dx, f) which returns the solution\n",
    "of argmin f (θ).  \n",
    "x_0, lr, nb_iters, df_dx, f correspond respectively to the initial value of θ, the\n",
    "θ\n",
    "learning rate, the number of iterations allowed for solving the problem, the gradient of θ and the\n",
    "function J(.).\n",
    "\n",
    "What is the solution θ̂ found by gradient descent to our problem? Note: assume that\n",
    "x_0, lr, nb_iters = -7, 0.1, 100 while debugging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(theta_0, lr, nb_iters, dJ_t, J_theta):\n",
    "    # puis on reprend les étapes vues en cours\n",
    "    theta = theta_0\n",
    "    \n",
    "    for t in range(nb_iters):    \n",
    "        # calcul de theta\n",
    "        theta = theta - lr * dJ_t(theta)\n",
    "\n",
    "    return theta\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.0000000100978041"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# On appelle la fonction \n",
    "gradient_descent(-7, 0.01, 1000, dJ_theta, J)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6\n",
    "Update your function gradient_descent for printing the optimization path (i.e. print the line between\n",
    "θ t and θ t+1 and saving the figure at the end of the optimization process)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.0000000100978041"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 11860 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def gradient_descent(theta_0, lr, nb_iters, dJ_t, J_theta):\n",
    "    # puis on reprend les étapes vues en cours\n",
    "    theta = theta_0\n",
    "    \n",
    "    # on initialise les séries pour le plot\n",
    "    x = [theta]\n",
    "    y = [J_theta(theta)]\n",
    "    \n",
    "    for t in range(nb_iters):    \n",
    "        # calcul de theta\n",
    "        theta = theta - lr * dJ_t(theta)\n",
    "        # ajout de la valeur au plot\n",
    "        x.append(theta)\n",
    "        y.append(J_theta(theta))\n",
    "        \n",
    "    # affichage du plot\n",
    "    plt.plot(x,y)\n",
    "        \n",
    "    return theta\n",
    "\n",
    "# On appelle la fonction \n",
    "gradient_descent(-7, 0.01, 1000, dJ_theta, J)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7\n",
    "Assuming that you are setting nb_iters equals to 100 what is the solution θ̂ when varying nb_iters in\n",
    "[-8, -1, 7] and lr in [0.1, 0.01, 0.001, -0.01, 0.8, 1.01]. Does the estimated solution always\n",
    "the same? What are pros and cons about these hyperparameters and GD in general?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8\n",
    "Now assume the function J(θ) = sin(2θ + 1), what is the solution θ̂ given by GD?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
