RL Notes (short)

Things to remember:

• discounted return is with respect to the future even though, pragmatically, this is usually seen as an eligibility trace. This is because of the Markov Property.

• $\gamma = 0$ implies a myopic agent – one that only cares about immediate reward.

• $\gamma = 1$ implies a farsighted agent – one that cares about optimizing future reward.

• $0 < \gamma < 1$ implies that an agent cares about the future to varying degrees. Cart-pole is an environment where we can model the reward in two different ways, which require different levels of $\gamma$ for any RL agent.

  • In one situation we set the reward to be +1. In this case any positive discount will enforce that future reward will accumulate as the agent steps through time.
  • Alternatively, we can set the reward to be -1. In this case an agent must set the discount rate to be $0 < \gamma < 1$. In the case of $\gamma = 0$ the agent fails to learn anything (all immediate rewards are 0 and the agent forgets everything by the final timestep), and at $\gamma = 1$ the agent treats all episode lengths the same. With $0 < \gamma < 1$, the -1 value will be minimized the larger we can set $t$ in $(-1)\cdot \gamma^t$.
  • in general we choose a discount rate that is closer to 1 than 0, like 0.9, otherwise the agent will be shortsighted.

• The most simple policy is a deterministic one. This is usually the first set of rules/hueristics you would find in a naive business solution. Bumping these up to stochastic policies is as simple as an MDP and as complicated as function approximation.

• Definition: $\pi\prime \geq \pi$ if and only if $v_{\pi}\prime(s) \geq v_\pi(s) \forall s \in S$. Following this we get that an optimal policy $\pi*$ is where $\pi* \geq \pi \forall \pi$

• When you talk about deterministic action-value functions for a given policy, you are asking about the optimal route at each state. This is an example of how you can compare state-value and action-value functions, but you can only do this for a deterministic policy.