hw1-PolicyGrad.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 1\n",
    "\n",
    "### Due Date: Thursday, July 30, PST, 2020\n",
    "\n",
    "### Policy Gradient\n",
    "\n",
    "In this assignment, we will implement vanilla policy gradient algorithm (REINFORCE) covered in the lecture. You will work on i) a function approximator, ii) computing action, iii) collecting samples, iV) training the agent, V) plotting the resutls. \n",
    "\n",
    "\n",
    "***Complete the missing operations and test your implemented algorithm on the Gym environment.***\n",
    "\n",
    "***Software requirements:***\n",
    "- Python >= 3.6\n",
    "- Tensorflow version <= 1.15.3 (1.X version)\n",
    "- OpenAI Gym\n",
    "\n",
    "- Training the agent (policy) can take long time. It is recomended to start solving the problems earlier.\n",
    "\n",
    "- Save any plots you generated in this notebook. The grade will be given based on the plots you showed.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure the packages you installed meet the requirements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "gym.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Tensorflow Implementation\n",
    "\n",
    "We will be implementing policy gradient algorithm using Tensorflow 1.X., which simply updates the parameters of policy from obtaining gradient estimates. The core of policy gradient is to design a function approximator, computing actions, collecting samples, and training the policy. In the below cell, you are encouraged to fill in the components that are missing. ***Your tasks*** are \n",
    "\n",
    "1. Complete the 'create_model' method to output the mean value for diagonal Guassian policy. Covariance is already defined in the model, so focus on creating neural network model.\n",
    "\n",
    "2. Complete the 'action_op' method to calculate and return the actions for diagonal Gaussian policy. The applied action should be $\\pi(s) = \\pi_{\\text{mean}}(s) + exp(logstd) * \\mathcal{N}(0,1)$\n",
    "\n",
    "***Hints***:\n",
    "- Some useful tensorflow classes and methods include: 'tf.exp', 'tf.random_normal'\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***IF you are using MAC, please run below box***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "# MAC user only\n",
    "os.environ['KMP_DUPLICATE_LIB_OK']='True'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import os\n",
    "import ipdb\n",
    "\n",
    "\n",
    "class PolicyOpt(object):\n",
    "\n",
    "    def __init__(self, env, linear=False, stochastic=True, hidden_size=32, nonlinearity=tf.nn.relu):\n",
    "        \"\"\"Instantiate the policy iteration class.\n",
    "\n",
    "        This initializes the policy optimization with a set of trainable \n",
    "        parameters, and creates a tensorflow session.\n",
    "\n",
    "        Attributes\n",
    "        ----------\n",
    "        env : gym.Env\n",
    "            the environment that the policy will be trained on\n",
    "        linear : bool, optional\n",
    "            specifies whether to use a linear or neural network \n",
    "            policy, defaults to False (i.e. Fully-Connected-Neural-Network)\n",
    "        stochastic : bool, optional\n",
    "            specifies whether to use a stochastic or deterministic \n",
    "            policy, defaults to True\n",
    "        hidden_size : list of int, optional\n",
    "            list of hidden layers, with each value corresponding \n",
    "            to the number of nodes in that layer \n",
    "        nonlinearity : tf.nn.*\n",
    "            activation nonlinearity\n",
    "        \"\"\"\n",
    "        \n",
    "        # clear computation graph\n",
    "        tf.reset_default_graph()\n",
    "        \n",
    "        # set a random seed\n",
    "        tf.set_random_seed(1234)\n",
    "        \n",
    "        # start a tensorflow session\n",
    "        self.sess = tf.Session()\n",
    "        \n",
    "        # environment to train on\n",
    "        self.env = env\n",
    "        \n",
    "        # number of elements in the action space\n",
    "        self.ac_dim = env.action_space.shape[0]\n",
    "        \n",
    "        # number of elements in the observation space\n",
    "        self.obs_dim = env.observation_space.shape[0]\n",
    "\n",
    "        # actions placeholder\n",
    "        self.a_t_ph = tf.placeholder(dtype=tf.float32, \n",
    "                                     shape=[None, self.ac_dim])\n",
    "        # state placeholder\n",
    "        self.s_t_ph = tf.placeholder(dtype=tf.float32, \n",
    "                                     shape=[None, self.obs_dim])\n",
    "        # expected reward placeholder\n",
    "        self.rew_ph = tf.placeholder(dtype=tf.float32, \n",
    "                                     shape=[None])\n",
    "\n",
    "        # specifies whether the policy is stochastic\n",
    "        self.stochastic = stochastic\n",
    "\n",
    "        # policy that the agent executes during training/testing\n",
    "        self.policy = self.create_model(\n",
    "            args={\n",
    "                \"num_actions\": self.ac_dim,\n",
    "                \"hidden_size\": hidden_size,\n",
    "                \"linear\": linear,\n",
    "                \"nonlinearity\": nonlinearity,\n",
    "                \"stochastic\": stochastic,\n",
    "                \"scope\": \"policy\",\n",
    "            }\n",
    "        )\n",
    "        \n",
    "        # define symbolic action\n",
    "        self.symbolic_action = self.action_op()\n",
    "\n",
    "        # initialize all variables\n",
    "        self.sess.run(tf.global_variables_initializer())\n",
    "\n",
    "        # create saver to save model variables\n",
    "        self.saver = tf.train.Saver()\n",
    "\n",
    "    def create_model(self, args):\n",
    "        \"\"\"Create a model for your policy or other components.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        args : dict\n",
    "            model-specific arguments, with keys:\n",
    "              - \"stochastic\": True by default\n",
    "              - \"hidden_size\": Number of neurons in hidden layer\n",
    "              - \"num_actions\" number of output actions\n",
    "              - \"scope\": scope of the model\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        tf.Variable\n",
    "            mean actions of the policy\n",
    "        tf.Variable \n",
    "            logstd of the policy actions\n",
    "        \"\"\"\n",
    "\n",
    "#################### Build Your Neural Network Here! ####################        \n",
    "        ??\n",
    "        ??\n",
    "        \n",
    "##########################################################################        \n",
    "\n",
    "        if args[\"stochastic\"]:\n",
    "            output_logstd =  tf.get_variable(name=\"action_logstd\",shape=[self.ac_dim],trainable=True)\n",
    "        else:\n",
    "            output_logstd = None\n",
    "\n",
    "        return output_mean, output_logstd\n",
    "    \n",
    "    def action_op(self):\n",
    "        \"\"\"\n",
    "        Create a symbolic expression that will be used to compute actions from observations.\n",
    "\n",
    "        When the policy is stochastic, the action follows \n",
    "\n",
    "            a_t = output_mean + exp(output_logstd) * z; z ~ N(0,1)\n",
    "        \"\"\"\n",
    "        if self.stochastic:\n",
    "            output_mean, output_logstd = self.policy\n",
    "\n",
    "            #################### Implement a stochastic policy here ####################        \n",
    "            # Implement a stochastic version of computing actions.       #\n",
    "            #                                                            #\n",
    "            # The action in a stochastic policy represented by           #\n",
    "            # a diagonal Gaussian distribution with mean \"M\" and log     #\n",
    "            # standard deviation \"logstd\" is computed as follows:        #\n",
    "            #                                                            #\n",
    "            #     a = M + exp(logstd) * z                                #\n",
    "            #                                                            #\n",
    "            # where z is a random normal value, i.e. z ~ N(0,1)          #\n",
    "            #                                                            #\n",
    "            # In order to generate numbers from a normal distribution,   #\n",
    "            # use the `tf.random_normal` function.                       #\n",
    "            ############################################################################ \n",
    "            symbolic_action = ?\n",
    "            \n",
    "        else:\n",
    "            symbolic_action, _ = self.policy\n",
    "        \n",
    "        return symbolic_action\n",
    "\n",
    "    def compute_action(self, obs):\n",
    "        \"\"\"Returns a list of actions for a given observation.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        obs : np.ndarray\n",
    "            observations\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        np.ndarray\n",
    "            actions by the policy for a given observation\n",
    "        \"\"\"\n",
    "        return self.sess.run(self.symbolic_action,feed_dict={self.s_t_ph: obs})\n",
    "\n",
    "    def rollout(self, s_mean=None, s_std=None):\n",
    "        \"\"\"Collect samples from one rollout of the policy.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        dict\n",
    "            dictionary containing trajectory information for the rollout,\n",
    "            specifically containing keys for \"state\", \"action\", \"next_state\", \"reward\", and \"done\"\n",
    "        \"\"\"\n",
    "        states = []\n",
    "        next_states = []\n",
    "        actions = []\n",
    "        rewards = []\n",
    "        dones = []\n",
    "\n",
    "        # start a new rollout by re-setting the environment and collecting the initial state\n",
    "        state =  self.env.reset()\n",
    "\n",
    "        steps = 0\n",
    "        while True:\n",
    "            steps += 1\n",
    "\n",
    "            # compute the action given the state\n",
    "            if s_mean is not None and s_std is not None:\n",
    "                action = self.compute_action([(state - s_mean) / s_std])\n",
    "            else:\n",
    "                action = self.compute_action([state])\n",
    "            action = action[0]\n",
    "\n",
    "            # advance the environment once and collect the next state, reward, done, and info parameters from the environment\n",
    "            next_state, reward, done, info =  self.env.step(action)\n",
    "\n",
    "            # add to the samples list\n",
    "            states.append(state)\n",
    "            actions.append(action)\n",
    "            next_states.append(next_state)\n",
    "            rewards.append(reward)\n",
    "            dones.append(done)\n",
    "\n",
    "            state = next_state\n",
    "\n",
    "            # if the environment returns a True for the done parameter,\n",
    "            # end the rollout before the time horizon is met\n",
    "            if done or steps > env._max_episode_steps:\n",
    "                break\n",
    "\n",
    "        # create the output trajectory\n",
    "        trajectory = {\"state\": np.array(states, dtype=np.float32),\n",
    "                      \"reward\": np.array(rewards, dtype=np.float32),\n",
    "                      \"action\": np.array(actions, dtype=np.float32),\n",
    "                      \"next_state\": np.array(next_states, dtype=np.float32),\n",
    "                      \"done\": np.array(dones, dtype=np.float32)}\n",
    "\n",
    "        return trajectory\n",
    "\n",
    "    def train(self, args):\n",
    "        \"\"\"Abstract training method.\n",
    "\n",
    "        This method will be filled in by algorithm-specific\n",
    "        training operations in subsequent problems.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        args : dict\n",
    "            algorithm-specific hyperparameters\n",
    "        \"\"\"\n",
    "        raise NotImplementedError\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Tensorflow Interpretation\n",
    "\n",
    "In order to test your implementation of the **stochastic policy**, run the below cell. The task is to interpret the code you implemented in previous section. If you implement correctly, you can see the value_1 and value_2.\n",
    "\n",
    "***Question: How do you interpret value_1 and value_2 below cell?***\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "TEST_ENV = gym.make(\"Pendulum-v0\")\n",
    "\n",
    "alg = PolicyOpt(TEST_ENV, linear=False)\n",
    "input_1 = [[0, 1, 2]]\n",
    "value_1 = alg.sess.run(alg.policy[0], feed_dict={alg.s_t_ph: input_1})\n",
    "value_2 = alg.compute_action(input_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "value_1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "value_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Implement Policy Gradient\n",
    "\n",
    "In this section, we will implement REINFORCE algorithm presented in the lecture. As a review, the objective is optimize the parameters $\\theta$ of some policy $\\pi_\\theta$ so that the expected return\n",
    "\n",
    "\\begin{equation}\n",
    "J(\\theta) = \\mathbb{E} \\bigg\\{ \\sum_{t=0}^T \\gamma^t r(s_{t},a_{t}) \\bigg\\}\n",
    "\\end{equation}\n",
    "\n",
    "is optimized. In this algorithm, this is done by calculating the gradient $\\nabla_\\theta J$ and applying a gradient descent method to find a better policy.\n",
    "\n",
    "\\begin{equation}\n",
    "\\theta ' = \\theta + \\alpha \\nabla_\\theta J(\\theta)\n",
    "\\end{equation}\n",
    "\n",
    "In the lecture, we derive how we compute $\\nabla_{\\theta} J(\\theta)$. We can rewrite our policy gradient as:\n",
    "\n",
    "\\begin{equation}\n",
    "\\nabla_\\theta J (\\theta) \\approx \\frac{1}{N} \\sum_{i=0}^{N} \\bigg( \\sum_{t=0}^{T} \\nabla_\\theta \\log \\pi_\\theta (a_{it} | s_{it}) \\bigg) \\bigg( \\sum_{t=0}^T \\gamma^{t}r_i(t) \\bigg)\n",
    "\\end{equation}\n",
    "\n",
    "Finally, taking into account the causality principle discussed in class, we are able to simplifiy the gradient estimate such as:\n",
    "\n",
    "\\begin{equation}\n",
    "\\nabla_\\theta J (\\theta) \\approx \\frac{1}{N} \\sum_{i=0}^{N} \\sum_{t=0}^{T} \\nabla_\\theta \\log \\pi_\\theta (a_{it} | s_{it}) \\sum_{t'=t}^T \\gamma^{t'-t}r_i(t')\n",
    "\\end{equation}\n",
    "\n",
    "You will be implementing final expression in this assignment!\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The process of REINFOCE algorithm follows:\n",
    "\n",
    "1. Collect samples from current policy $\\pi_\\theta(s)$ by executing rollouts of the environment.\n",
    "2. Calculate an estimate for the expected return at state $s_t$. \n",
    "3. Compute the log-likelihood of each action that was performed by the policy at every given step.\n",
    "4. Estimate the gradient and update the parameters of policy using gradient-based technique.\n",
    "5. Repeat steps 1-4 for a number of training iterations.\n",
    "\n",
    "***Your task*** is to fill out the skeleton code for REINFORCE algorithm,\n",
    "\n",
    "1. Complete the 'log_likelihoods' method to compute gradient of policy, $\\nabla_{\\theta}\\pi_{\\theta}$ for diagonal Guassian policy. \n",
    "\n",
    "2. Complete the 'compute_expected_return' method to calculate the reward-to-go, $\\sum_{t^{\\prime}=t}^{T}$. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import time\n",
    "import tensorflow_probability as tfp\n",
    "\n",
    "class REINFORCE(PolicyOpt):\n",
    "\n",
    "    def train(self, num_iterations=1000, steps_per_iteration=1000, learning_rate=int(1e-4), gamma=0.95, \n",
    "              **kwargs):\n",
    "        \"\"\"Perform the REINFORE training operation.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        num_iterations : int\n",
    "            number of training iterations\n",
    "        steps_per_iteration : int\n",
    "            number of individual samples collected every training iteration\n",
    "        learning_rate : float\n",
    "            optimizer learning rate\n",
    "        gamma : float\n",
    "            discount rate\n",
    "        kwargs : dict\n",
    "            additional arguments\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        list of float\n",
    "            average return per iteration\n",
    "        \"\"\"\n",
    "        # set the discount as an attribute\n",
    "        self.gamma = gamma\n",
    "        \n",
    "        # set the learning rate as an attribute\n",
    "        self.learning_rate = learning_rate\n",
    "\n",
    "        # create a symbolic expression to compute the log-likelihoods \n",
    "        log_likelihoods = self.log_likelihoods()\n",
    "\n",
    "        # create a symbolic expression for updating the parameters of your policy\n",
    "        self.opt, self.opt_baseline = self.define_updates(log_likelihoods)\n",
    "        \n",
    "        # initialize all variables\n",
    "        self.sess.run(tf.global_variables_initializer())\n",
    "\n",
    "\n",
    "        # average return per training iteration\n",
    "        ret_per_iteration = []\n",
    "        \n",
    "        samples = []\n",
    "        for i in range(num_iterations):\n",
    "            \n",
    "            # collect samples from the current policy\n",
    "            samples.clear()\n",
    "            steps_so_far = 0\n",
    "            while steps_so_far < steps_per_iteration:\n",
    "                new_samples = self.rollout()\n",
    "                steps_so_far += new_samples[\"action\"].shape[0]\n",
    "                samples.append(new_samples)\n",
    "\n",
    "            # compute the expected returns\n",
    "            v_s = self.compute_expected_return(samples)\n",
    "\n",
    "            # compute and apply the gradients\n",
    "            self.call_updates(log_likelihoods, samples, v_s, **kwargs)\n",
    "\n",
    "            # compute the average cumulative return per iteration\n",
    "            average_rew = np.mean([sum(s[\"reward\"]) for s in samples])\n",
    "\n",
    "            # display iteration statistics\n",
    "            print(\"Iteration {} return: {}\".format(i, average_rew))\n",
    "            ret_per_iteration.append(average_rew)\n",
    "\n",
    "        return ret_per_iteration\n",
    "\n",
    "    def log_likelihoods(self):\n",
    "        \"\"\"Create a tensorflow operation that computes the log-likelihood \n",
    "        of each performed action.\n",
    "        \"\"\"\n",
    "        \n",
    "        output_mean, output_logstd = self.policy\n",
    "\n",
    "        ##############################################################\n",
    "        # Create a tf operation to compute the log-likelihood of     #\n",
    "        # each action that was performed by the policy               #\n",
    "        #                                                            #\n",
    "        # The log likelihood in the continuous case where the policy #\n",
    "        # is expressed by a multivariate gaussian can be computing   #\n",
    "        # using the tensorflow object:                               #\n",
    "        #                                                            #\n",
    "        #    p = tfp.distributions.MultivariateNormalDiag(           #\n",
    "        #        loc=...,                                            #\n",
    "        #        scale_diag=...,                                     #\n",
    "        #    )                                                       #\n",
    "        #                                                            #\n",
    "        # This method takes as input a mean (loc) and standard       #\n",
    "        # deviation (scale_diag), and then can be used to compute    #\n",
    "        # the log-likelihood of a variable as follows:               #\n",
    "        #                                                            #\n",
    "        #    log_likelihoods = p.log_prob(...)                       #\n",
    "        #                                                            #\n",
    "        # For this operation, you will want to use placeholders      #\n",
    "        # created in the __init__ method of problem 1.               #\n",
    "        ##############################################################\n",
    "\n",
    "        p = ?\n",
    "        log_likelihoods = ?\n",
    "\n",
    "        return log_likelihoods\n",
    "\n",
    "    def compute_expected_return(self, samples):\n",
    "        \"\"\"Compute the expected return from a given starting state.\n",
    "        This is done by using the reward-to-go method.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        rewards : list of list of float\n",
    "            a list of N trajectories, with each trajectory contain T \n",
    "            returns values (one for each step in the trajectory)\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        list of float, or np.ndarray\n",
    "            expected returns for each step in each trajectory\n",
    "        \"\"\"\n",
    "        rewards = [s[\"reward\"] for s in samples]\n",
    "\n",
    "        ##############################################################\n",
    "        # Estimate the expected return from any given starting state #\n",
    "        # using the reward-to-go method.                             #\n",
    "        #                                                            #\n",
    "        # Using this method, the reward is estimated at every step   #\n",
    "        # of the trajectory as follows:                              #\n",
    "        #                                                            #\n",
    "        #   r = sum_{t'=t}^T gamma^(t'-t) * r_{t'}                   #\n",
    "        #                                                            #\n",
    "        # where T is the time horizon at t is the index of the       #\n",
    "        # current reward in the trajectory. For example, for a given #\n",
    "        # set of rewards r = [1,1,1,1] and discount rate gamma = 1,  #\n",
    "        # the expected reward-to-go would be:                        #\n",
    "        #                                                            #\n",
    "        #   v_s = [4, 3, 2, 1]                                       #\n",
    "        #                                                            #\n",
    "        # You will be able to test this in one of the cells below!   #\n",
    "        ##############################################################\n",
    "        v_s = ?\n",
    "        \n",
    "        return v_s\n",
    "\n",
    "    def define_updates(self, log_likelihoods):\n",
    "        \"\"\"Create a tensorflow operation to update the parameters of \n",
    "        your policy.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        log_likelihoods : tf.Operation\n",
    "            the symbolic expression you created to estimate the log \n",
    "            likelihood of a set of actions\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        tf.Operation\n",
    "            a tensorflow operation for computing and applying the \n",
    "            gradients to the parameters of the policy\n",
    "        None\n",
    "            the second component is used in problem 2.b, please ignore \n",
    "            for this problem\n",
    "        \"\"\"\n",
    "\n",
    "        loss = - tf.reduce_mean(tf.multiply(log_likelihoods, self.rew_ph))\n",
    "        opt = tf.train.AdamOptimizer(self.learning_rate).minimize(loss)\n",
    "\n",
    "        return opt, None\n",
    "\n",
    "    def call_updates(self, log_likelihoods, samples, v_s, **kwargs):\n",
    "        \"\"\"Apply the gradient update methods in a tensorflow session.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        log_likelihoods: tf.Operation\n",
    "            the symbolic expression you created to estimate the log \n",
    "            likelihood of a set of actions\n",
    "        samples : list of dict\n",
    "            a list of N trajectories, with each trajectory containing \n",
    "            a dictionary of trajectory data (see self.rollout)\n",
    "        v_s : list of float, or np.ndarray\n",
    "            the estimated expected returns from your\n",
    "            `comput_expected_return` function \n",
    "        kwargs : dict\n",
    "            additional arguments (used in question 3)\n",
    "        \"\"\"\n",
    "        # concatenate the states\n",
    "        states = np.concatenate([s[\"state\"] for s in samples])\n",
    "\n",
    "        # concatenate the actions\n",
    "        actions = np.concatenate([s[\"action\"] for s in samples])\n",
    "\n",
    "        # execute the optimization step\n",
    "        self.sess.run(self.opt, feed_dict={self.s_t_ph: states,\n",
    "                                           self.a_t_ph: actions,\n",
    "                                           self.rew_ph: v_s})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check your 'log_likelihoods' method by running below cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alg = REINFORCE(TEST_ENV, stochastic=True)\n",
    "\n",
    "log_likelihoods = alg.log_likelihoods()\n",
    "\n",
    "# collect a sample output for a given input state\n",
    "input_s = [[0, 0, 0], [0, 1, 2], [1, 2, 3]]\n",
    "input_a = [[0], [1], [2]]\n",
    "\n",
    "# Check\n",
    "computed = alg.sess.run(log_likelihoods, feed_dict={alg.a_t_ph: input_a, alg.s_t_ph: input_s})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test your 'compute_expected_return' by running below cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Test the non-normalized case\n",
    "alg = REINFORCE(TEST_ENV, stochastic=True)\n",
    "alg.gamma = 1.0\n",
    "    \n",
    "input_1 = [{\"reward\": [1, 1, 1, 1]},\n",
    "           {\"reward\": [1, 1, 1, 1]}]\n",
    "vs_1 = alg.compute_expected_return(samples=input_1)\n",
    "ans_1 = np.array([4, 3, 2, 1, 4, 3, 2, 1])\n",
    "\n",
    "if np.linalg.norm(vs_1 - ans_1) < 1e-3:\n",
    "    print('Great job!')\n",
    "else:\n",
    "    print('Check your implementation (compute_expected_return)')\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4 Testing your algorithm\n",
    "\n",
    "When you are ready, test your policy gradient algorithms on the *Pendulum-v0* environment in the cell below. *Pendulum-v0* environment is similar to *off-shore wind power*, the goal here is to maintain the Pendulum is upright using control input. The best policy should get around -200 scores. ***Your task*** is to run your REINFORCE algorithm and plot the result!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# set this number as 1 for testing your algorithm, and 3 for plotting\n",
    "NUM_TRIALS = 3\n",
    "\n",
    "# ===========================================================================\n",
    "# Do not modify below line\n",
    "# ===========================================================================\n",
    "\n",
    "# we will test the algorithms on the Pendulum-v0 gym environment\n",
    "import gym\n",
    "env = gym.make(\"Pendulum-v0\")\n",
    "\n",
    "# train on the REINFORCE algorithm\n",
    "import numpy as np\n",
    "r = []\n",
    "for i in range(NUM_TRIALS):\n",
    "    print(\"\\n==== Training Run {} ====\".format(i))\n",
    "    alg = REINFORCE(env, stochastic=True)\n",
    "    res = alg.train(learning_rate=0.005, gamma=0.95, num_iterations=500, steps_per_iteration=15000)\n",
    "    r.append(np.array(res))\n",
    "    alg = None\n",
    "\n",
    "# save results\n",
    "np.savetxt(\"InvertedPendulum_results.csv\", np.array(r), delimiter=\",\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# collect saved results\n",
    "import numpy as np\n",
    "r1 = np.genfromtxt(\"InvertedPendulum_results.csv\", delimiter=\",\")\n",
    "all_results = [r1]\n",
    "labels = [\"REINFORCE\"]\n",
    "\n",
    "##############################################################\n",
    "# Plot your Policy Gradient results below\n",
    "##############################################################\n",
    "?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Policy gradient result looks like this:\n",
    "\n",
    "<img src=\"HW1_PG_Result_Sample.png\"\n",
    "     alt=\"Markdown Monster icon\"\n",
    "     style=\"float: left; margin-right: 10px;\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}