深度强化学习(七)策略梯度

策略学习的目的是通过求解一个优化问题，学出最优策略函数或它的近似函数（比如策略网络）

一.策略网络

假设动作空间是离散的,，比如 A = { 左 , 右 , 上 } \cal A=\{左,右,上\} A={左,右,上}，策略函数$\pi$是个条件概率函数：
$$\pi (a\mid s)=\mathbb{P}(A=a\mid S=s)$$
与$DQN$类似，我们可以用神经网络$\pi (a\mid s;\mathbf{\theta })$去近似策略函数$\pi (a\mid s)$,$\mathbf{\theta }$是我们需要训练的神经网络的参数。

回忆动作价值函数的定义是
$$Q_{\pi }(a_t,s_t)={\mathbb{E}}_{A_{t+1},S_{t+1}\dots }[U_t\mid A_t=a_t,S_t=s_t]$$
状态价值函数的定义是
$$V_{\pi }(s_t)={\mathbb{E}}_{A_t\sim \pi (a\mid s)}[Q_{\pi }(A_t,s_t)]$$
$\text{ 状态价值既依赖于当前状态 }{s}_t\text{, 也依赖于策略网络 }\pi \text{ 的参数 }\mathbf{\theta }\text{ 。 }$

为排除状态对策略的影响，我们对状态$S_t$求期望，得出
$$J(\mathbf{\theta })={\mathbb{E}}_{S_t}[V_{\pi }(S_t)]$$
这个目标函数排除掉了状态$S$ 的因素，只依赖于策略网络$\pi$的参数$\mathbf{\theta }$；策略越好，则 $J$越大。所以策略学习可以描述为这样一个优化问题
$${\text{Max}}_{\mathbf{\theta }}J(\mathbf{\theta })$$
由于是求最大化问题，我们可利用梯度上升对$J(\mathbf{\theta })$进行更新，问题的关键是计算${\nabla }_{\mathbf{\theta }}J(\mathbf{\theta })$

二.策略梯度定理推导

Theorem:递归公式,其中$S^{\prime }$是 下一时刻的状态。
$$\frac{\partial V_{\pi }(s)}{\partial \mathbf{\theta }}={\mathbb{E}}_{A\sim \pi (\cdot \mid s;\mathbf{\theta })}\left[\frac{\partial \ln \pi (A\mid s;\mathbf{\theta })}{\partial \mathbf{\theta }}\cdot Q_{\pi }(s,A)+\gamma \cdot {\mathbb{E}}_{S^{\prime }\sim p(\cdot \mid s,A)}\left[\frac{\partial V_{\pi }\left(S^{\prime }\right)}{\partial \mathbf{\theta }}\right]\right]\text{\hspace{1em}(2.1)}$$

Proof:
$$\begin{array}{rl}\frac{\partial V_{\pi }(s)}{\partial \mathbf{\theta }}& =\frac{\partial }{\partial \mathbf{\theta }}[{\mathbb{E}}_{A\sim \pi (\cdot \mid s;\mathbf{\theta })}[Q_{\pi }(s,A)]]\\ & =\frac{\partial }{\partial \mathbf{\theta }}[\sum _A\pi (a\mid s;\mathbf{\theta })Q_{\pi }(s,a)]\\ & =\sum _A[\frac{\partial \pi (a\mid s;\mathbf{\theta })}{\partial \mathbf{\theta }}{Q}_{\pi }(s,a)+\pi (a\mid s;\mathbf{\theta })\frac{\partial Q_{\pi }(s,a)}{\partial \mathbf{\theta }}]\\ & =\sum _A[\pi (a\mid s;\mathbf{\theta })\cdot \frac{\partial \ln \pi (a\mid s;\mathbf{\theta })}{\partial \mathbf{\theta }}\cdot Q_{\pi }(s,a)+\pi (a\mid s;\mathbf{\theta })\frac{\partial Q_{\pi }(s,a)}{\partial \mathbf{\theta }}]\\ & ={\mathbb{E}}_{A\sim \pi (\cdot \mid s;\mathbf{\theta })}\left[\frac{\partial \ln \pi (A\mid s;\mathbf{\theta })}{\partial \mathbf{\theta }}\cdot Q_{\pi }(s,A)\right]+{\mathbb{E}}_{A\sim \pi (\cdot \mid s;\mathbf{\theta })}\left[\frac{\partial Q_{\pi }(s,a)}{\partial \mathbf{\theta }}\right].\\ & ={\mathbb{E}}_{A\sim \pi (\cdot \mid s;\mathbf{\theta })}[\frac{\partial \ln \pi (A\mid s;\mathbf{\theta })}{\partial \mathbf{\theta }}\cdot Q_{\pi }(s,A)+\frac{\partial Q_{\pi }(s,a)}{\partial \mathbf{\theta }}]\end{array}$$
接下来仅需证明$\frac{\partial Q_{\pi }(s,a)}{\partial \mathbf{\theta }}=\gamma {\mathbb{E}}_{S^{\prime }\sim p(\cdot \mid s,A)}[\frac{\partial V_{\pi }\left(S^{\prime }\right)}{\partial \mathbf{\theta }}]$,贝尔曼方程为
$$\begin{array}{rl}{Q}_{\pi }(s,a)& ={\mathbb{E}}_{S^{\prime }\sim p(\cdot \mid s,a)}\left[R\left(s,a,S^{\prime }\right)+\gamma \cdot V_{\pi }\left(s^{\prime }\right)\right]\\ & =\sum _{s^{\prime }\in \mathcal{S}}p\left(s^{\prime }\mid s,a\right)\cdot \left[R\left(s,a,s^{\prime }\right)+\gamma \cdot V_{\pi }\left(s^{\prime }\right)\right]\\ & =\sum _{s^{\prime }\in \mathcal{S}}p\left(s^{\prime }\mid s,a\right)\cdot R\left(s,a,s^{\prime }\right)+\gamma \cdot \sum _{s^{\prime }\in \mathcal{S}}p\left(s^{\prime }\mid s,a\right)\cdot V_{\pi }\left(s^{\prime }\right).\end{array}$$

在观测到 $s、a、s^{\prime }$ 之后, $p\left(s^{\prime }\mid s,a\right)$ 和 $R\left(s,a,s^{\prime }\right)$ 都与策略网络 $\pi$ 无关, 因此
$$\frac{\partial }{\partial \mathbf{\theta }}\left[p\left(s^{\prime }\mid s,a\right)\cdot R\left(s,a,s^{\prime }\right)\right]=0.$$

可得:
$$\begin{array}{rl}\frac{\partial Q_{\pi }(s,a)}{\partial \mathbf{\theta }}& =\sum _{s^{\prime }\in \mathcal{S}}\underset{\text{等于零 }}{\underbrace{\frac{\partial }{\partial \mathbf{\theta }}\left[p\left(s^{\prime }\mid s,a\right)\cdot R\left(s,a,s^{\prime }\right)\right]}}+\gamma \cdot \sum _{s^{\prime }\in \mathcal{S}}\frac{\partial }{\partial \mathbf{\theta }}\left[p\left(s^{\prime }\mid s,a\right)\cdot V_{\pi }\left(s^{\prime }\right)\right]\\ & =\gamma \cdot \sum _{s^{\prime }\in \mathcal{S}}p\left(s^{\prime }\mid s,a\right)\cdot \frac{\partial V_{\pi }\left(s^{\prime }\right)}{\partial \mathbf{\theta }}\\ & =\gamma \cdot {\mathbb{E}}_{S^{\prime }\sim p(\cdot \mid s,a)}\left[\frac{\partial V_{\pi }\left(S^{\prime }\right)}{\partial \mathbf{\theta }}\right].\end{array}$$

证毕

设 $\mathbf{g}(s,a;\mathbf{\theta })\triangleq Q_{\pi }(s,a)\cdot \frac{\partial \ln \pi (a\mid s;\theta )}{\partial \mathbf{\theta }}$ 。设一局游戏在第 $n$ 步之后结束。那么
$$\begin{array}{rl}\frac{\partial J(\mathbf{\theta })}{\partial \mathbf{\theta }}=& {\mathbb{E}}_{S_1,A_1}\left[\mathbf{g}\left(S_1,A_1;\mathbf{\theta }\right)\right]\\ & +\gamma \cdot {\mathbb{E}}_{S_1,A_1,S_2,A_2}\left[\mathbf{g}\left(S_2,A_2;\mathbf{\theta }\right)\right]\\ & +{\gamma }^2\cdot {\mathbb{E}}_{S_1,A_1,S_2,A_2,S_3,A_3}\left[\mathbf{g}\left(S_3,A_3;\mathbf{\theta }\right)\right]\\ & +\cdots \\ & +{\gamma }^{n-1}\cdot {\mathbb{E}}_{S_1,A_1,S_2,A_2,S_3,A_3,\cdots S_n,A_n}[\mathbf{g}\left(S_n,A_n;\mathbf{\theta }\right)]\end{array}\text{\hspace{1em}(2.2)}$$

Proof:由式$2.1$可知
$$\begin{array}{rl}{\nabla }_{\mathbf{\theta }}{V}_{\pi }(s_t)& ={\mathbb{E}}_{A_t\sim \pi (\cdot \mid s_t;\mathbf{\theta })}\left[\frac{\partial \ln \pi (A_t\mid s_t;\mathbf{\theta })}{\partial \mathbf{\theta }}\cdot Q_{\pi }(s_t,A_t)+\gamma \cdot {\mathbb{E}}_{S_{t+1}\sim p(\cdot \mid s_t,A_t)}[{\nabla }_{\mathbf{\theta }}{V}_{\pi }\left(S_{t+1}\right)]\right]\\ & ={\mathbb{E}}_{A_t\sim \pi (\cdot \mid s_t;\mathbf{\theta })}\left[\mathbf{g}(s_t,A_t;\mathbf{\theta })+\gamma \cdot {\mathbb{E}}_{S_{t+1}}[{\nabla }_{\mathbf{\theta }}{V}_{\pi }\left(S_{t+1}\right)\mid A_t,S_t=s_t]\right]\\ & ={\mathbb{E}}_{A_t}[\mathbf{g}(s_t,A_t;\mathbf{\theta })\mid S_t=s_t]+\gamma {\mathbb{E}}_{A_t}[{\mathbb{E}}_{S_{t+1}}[{\nabla }_{\mathbf{\theta }}{V}_{\pi }(S_{t+1})\mid A_t,S_t=s_t]\mid S_t=s_t]\\ & ={\mathbb{E}}_{A_t}[\mathbf{g}(s_t,A_t;\mathbf{\theta })\mid S_t=s_t]+\gamma {\mathbb{E}}_{A_t,S_{t+1}}[{\nabla }_{\mathbf{\theta }}{V}_{\pi }(S_{t+1})\mid S_t=s_t]\end{array}$$
则${\nabla }_{\mathbf{\theta }}{V}_{\pi }(S_{t+1})={\mathbb{E}}_{A_{t+1}}[\mathbf{g}(S_{t+1},A_{t+1};\mathbf{\theta })\mid S_{t+1}]+\gamma {\mathbb{E}}_{A_{t+1},S_{t+2}}[{\nabla }_{\mathbf{\theta }}{V}_{\pi }(S_{t+2})\mid S_{t+1}]$,带入上式中可得
$$\begin{array}{rl}{\nabla }_{\mathbf{\theta }}{V}_{\pi }(s_t)& ={\mathbb{E}}_{A_t}[\mathbf{g}(s_t,A_t;\mathbf{\theta })\mid S_t=s_t]+\gamma {\mathbb{E}}_{A_t,S_{t+1}}[{\nabla }_{\mathbf{\theta }}{V}_{\pi }(S_{t+1})\mid S_t=s_t]\\ & ={\mathbb{E}}_{A_t}[\mathbf{g}(s_t,A_t;\mathbf{\theta })\mid S_t=s_t]+\gamma {\mathbb{E}}_{A_t,S_{t+1}}[{\mathbb{E}}_{A_{t+1}}[\mathbf{g}(S_{t+1},A_{t+1};\mathbf{\theta })\mid S_{t+1}]+\gamma {\mathbb{E}}_{A_{t+1},S_{t+2}}[{\nabla }_{\mathbf{\theta }}{V}_{\pi }(S_{t+2})\mid S_{t+1}]\mid S_t=s_t]\\ & ={\mathbb{E}}_{A_t}[\mathbf{g}(s_t,A_t;\mathbf{\theta })\mid S_t=s_t]+\gamma {\mathbb{E}}_{A_t,S_{t+1}}[{\mathbb{E}}_{A_{t+1}}[\mathbf{g}(S_{t+1},A_{t+1};\mathbf{\theta })\mid S_{t+1},S_t=s_t,A_t]+\gamma {\mathbb{E}}_{A_{t+1},S_{t+2}}[[{\nabla }_{\mathbf{\theta }}{V}_{\pi }(S_{t+2})\mid S_{t+1}]\mid S_t=s_t]\text{马尔可可夫性}\\ & ={\mathbb{E}}_{A_t}[\mathbf{g}(s_t,A_t;\mathbf{\theta })\mid S_t=s_t]+\gamma {\mathbb{E}}_{A_t,S_{t+1},A_{t+1}}[\mathbf{g}(S_{t+1},A_{t+1};\mathbf{\theta })\mid S_t=s_t]+\gamma {\mathbb{E}}_{A_{t+1},S_{t+2}}[[{\nabla }_{\mathbf{\theta }}{V}_{\pi }(S_{t+2})\mid S_{t+1}]\mid S_t=s_t]\end{array}$$
继续利用上式反复带入，最后可得
$$\begin{array}{rl}\frac{\partial V_{\pi }\left(S_1\right)}{\partial \mathbf{\theta }}=& {\mathbb{E}}_{A_1}\left[\mathbf{g}\left(S_1,A_1;\mathbf{\theta }\right)\mid S_1\right]\\ & +\gamma \cdot {\mathbb{E}}_{A_1,S_2,A_2}\left[\mathbf{g}\left(S_2,A_2;\mathbf{\theta }\right)\mid S_1\right]\\ & +{\gamma }^2\cdot {\mathbb{E}}_{A_1,S_2,A_2,S_3,A_3}\left[\mathbf{g}\left(S_3,A_3;\mathbf{\theta }\right)\mid S_1\right]\\ & +\cdots \\ & +{\gamma }^{n-1}\cdot {\mathbb{E}}_{A_1,S_2,A_2,S_3,A_3,\cdots S_n,A_n}\left[\mathbf{g}\left(S_n,A_n;\mathbf{\theta }\right)\mid S_1\right]\\ & +{\gamma }^n\cdot {\mathbb{E}}_{A_1,S_2,A_2,S_3,A_3,\cdots S_n,A_n,S_{n+1}}[\underset{\text{等于零 }}{\underbrace{\frac{\partial V_{\pi }\left(S_{n+1}\right)}{\partial \mathbf{\theta }}}}\mid S_1]\end{array}$$
上式中最后一项等于零，原因是游戏在n时刻后结束，而$n+1$时刻之后没有奖励，所以$n+1$时刻的回报和价值都是零。最后，由上面的公式和,最后，由$J(\mathbf{\theta })$定义知
$$\frac{\partial J(\mathbf{\theta })}{\partial \mathbf{\theta }}={\mathbb{E}}_{S_1}\left[\frac{\partial V_{\pi }\left(S_1\right)}{\partial \mathbf{\theta }}\right]$$
证毕

稳态分布：想要严格证明策略梯度定理, 需要用到马尔科夫链 (Markov chain) 的稳态分布 (stationary distribution)。设状态 $S^{\prime }$ 是这样得到的: $S\to A\to S^{\prime }$ 。回忆一下, 状态转移函数 $p\left(S^{\prime }\mid S,A\right)$, 是一个概率质量函数。设 $f(S)$ 是状态 $S$ 的概率质量函数那么状态 $S^{\prime }$的边缘分布$f(S^{\prime })$是
$$\begin{array}{rl}f(S^{\prime })& ={\mathbb{E}}_{S,A}[p(S^{\prime }\mid A,S)]\\ & ={\mathbb{E}}_S[{\mathbb{E}}_A[p(S^{\prime }\mid A,S)\mid S]]\\ & ={\mathbb{E}}_S[\sum _{A}p(S^{\prime }\mid a,S)\cdot \pi (a\mid S)]\\ & =\sum _S\sum _{A}p(S^{\prime }\mid a,s)\cdot \pi (a\mid s)\cdot f(s)\end{array}$$
如果 $f(S^{\prime })$ 与 $f(S)$ 是相同的概率质量函数, 即 $f(S)=f(S’) $, 则意味着马尔科夫链达到稳态, 而 $f(S)$ 就是稳态时的概率质量函数。

Theorem:

设 $f(S)$ 是马尔科夫链稳态时的概率质量 (密度) 函数。那么对于任意函数 $G\left(S^{\prime }\right)$,
$${\mathbb{E}}_{S\sim f(\cdot )}\left[{\mathbb{E}}_{A\sim \pi (\cdot \mid S;\mathbf{\theta })}\left[{\mathbb{E}}_{S^{\prime }\sim p(\cdot \mid s,A)}\left[G\left(S^{\prime }\right)\right]\right]\right]={\mathbb{E}}_{S^{\prime }\sim f(\cdot )}\left[G\left(S^{\prime }\right)\right]\text{\hspace{1em}(2.3)}$$

Proof:
$$\begin{array}{rl}{\mathbb{E}}_{S\sim f(\cdot )}\left[{\mathbb{E}}_{A\sim \pi (\cdot \mid S;\mathbf{\theta })}\left[{\mathbb{E}}_{S^{\prime }\sim p(\cdot \mid S,A)}\left[G\left(S^{\prime }\right)\right]\right]\right]& ={\mathbb{E}}_{S\sim f(\cdot )}[{\mathbb{E}}_A[{\mathbb{E}}_{S^{\prime }}[G(S^{\prime })\mid S,A]\mid S]]\\ & ={\mathbb{E}}_{S\sim f(\cdot )}[{\mathbb{E}}_{A,S^{\prime }}[G(S^{\prime })\mid S]]\\ & ={\mathbb{E}}_{S,A,S^{\prime }}[G(S^{\prime })]\\ & ={\mathbb{E}}_{S^{\prime }}[G(S^{\prime })]\end{array}$$
又因$S,S^{\prime }$有相同的分布$f(\cdot )$,所以${\mathbb{E}}_{S^{\prime }}[G(S^{\prime })]={\mathbb{E}}_{S^{\prime }\sim f(\cdot )}\left[G\left(S^{\prime }\right)\right]$

Theorem:策略梯度定理

设目标函数为 $J(\mathbf{\theta })={\mathbb{E}}_{S\sim f(\cdot )}\left[V_{\pi }(S)\right]$, 设 $f(S)$ 为马尔科夫链稳态分布的概率质量 (密度) 函数。那么
$$\frac{\partial J(\mathbf{\theta })}{\partial \mathbf{\theta }}=\left(1+\gamma +{\gamma }^2+\cdots +{\gamma }^{n-1}\right)\cdot {\mathbb{E}}_{S\sim f(\cdot )}\left[{\mathbb{E}}_{A\sim \pi (\cdot \mid S;\mathbf{\theta })}\left[\frac{\partial \ln \pi (A\mid S;\mathbf{\theta })}{\partial \mathbf{\theta }}\cdot Q_{\pi }(S,A)\right]\right]$$

Proof:设初始状态 $S_1$ 服从马尔科夫链的稳态分布，它的概率质量函数是 $f\left(S_1\right)$ 。对于所有的 $t=1,\cdots ,n$, 动作 $A_t$ 根据策略网络抽样得到:
$$A_t\sim \pi \left(\cdot \mid S_t;\mathbf{\theta }\right)$$
对于任意函数 $G$, 反复应用式 2.3 可得:
E A 1 , … , A t − 1 , S 1 , … , S t [ G ( S t ) ] = E S 1 ∼ f { E A 1 ∼ π , S 2 ∼ p { E A 2 , S 3 , A 3 , S 4 , ⋯ , A t − 1 , S t [ G ( S t ) ] } } = E S 2 ∼ f { E A 2 , S 3 , A 3 , S 4 , ⋯ , A t − 1 , S t [ G ( S t ) ] } = E S 2 ∼ f { E A 2 ∼ π , S 3 ∼ p { E A 3 , S 4 , A 4 , S 5 , ⋯ , A t − 1 , S t [ G ( S t ) ] } } = E S 3 ∼ f { E A 3 , S 4 , A 4 , S 5 , ⋯ , A t − 1 , S t [ G ( S t ) ] } ⋮ = E S t − 1 ∼ f { E A t − 1 ∼ π , S t ∼ p { G ( S t ) } } = E S t ∼ f { G ( S t ) } . \begin{aligned} \Bbb E_{A_1,\ldots,A_{t-1},S_1,\ldots,S_{t}}[G(S_t)] & =\mathbb{E}_{S_1 \sim f}\left\{\mathbb{E}_{A_1 \sim \pi, S_2 \sim p}\left\{\mathbb{E}_{A_2, S_3, A_3, S_4, \cdots, A_{t-1}, S_t}\left[G\left(S_t\right)\right]\right\}\right\} \\ & =\mathbb{E}_{S_2 \sim f}\left\{\mathbb{E}_{A_2, S_3, A_3, S_4, \cdots, A_{t-1}, S_t}\left[G\left(S_t\right)\right]\right\} \quad \\ & =\mathbb{E}_{S_2 \sim f}\left\{\mathbb{E}_{A_2 \sim \pi, S_3 \sim p}\left\{\mathbb{E}_{A_3, S_4, A_4, S_5, \cdots, A_{t-1}, S_t}\left[G\left(S_t\right)\right]\right\}\right\} \\ & =\mathbb{E}_{S_3 \sim f}\left\{\mathbb{E}_{A_3, S_4, A_4, S_5, \cdots, A_{t-1}, S_t}\left[G\left(S_t\right)\right]\right\} \quad \\ & \vdots \\ & =\mathbb{E}_{S_{t-1} \sim f}\left\{\mathbb{E}_{A_{t-1} \sim \pi, S_t \sim p}\left\{G\left(S_t\right)\right\}\right\} \\ & =\mathbb{E}_{S_t \sim f}\left\{G\left(S_t\right)\right\} . \end{aligned} EA1​,…,At−1​,S1​,…,St​​[G(St​)]​=ES1​∼f​{EA1​∼π,S2​∼p​{EA2​,S3​,A3​,S4​,⋯,At−1​,St​​[G(St​)]}}=ES2​∼f​{EA2​,S3​,A3​,S4​,⋯,At−1​,St​​[G(St​)]}=ES2​∼f​{EA2​∼π,S3​∼p​{EA3​,S4​,A4​,S5​,⋯,At−1​,St​​[G(St​)]}}=ES3​∼f​{EA3​,S4​,A4​,S5​,⋯,At−1​,St​​[G(St​)]}⋮=ESt−1​∼f​{EAt−1​∼π,St​∼p​{G(St​)}}=ESt​∼f​{G(St​)}.​
设 $\mathbf{g}(s,a;\mathbf{\theta })\triangleq Q_{\pi }(s,a)\cdot \frac{\partial \ln \pi (a\mid s;\mathbf{\theta })}{\partial \mathbf{\theta }}$ 。设一局游戏在第 $n$ 步之后结束。由式2.2与上面的公式可得:
$$\begin{array}{rl}\frac{\partial J(\mathbf{\theta })}{\partial \mathbf{\theta }}=& {\mathbb{E}}_{S_1,A_1}\left[\mathbf{g}\left(S_1,A_1;\mathbf{\theta }\right)\right]\\ & +\gamma \cdot {\mathbb{E}}_{S_1,A_1,S_2,A_2}\left[\mathbf{g}\left(S_2,A_2;\mathbf{\theta }\right)\right]\\ & +{\gamma }^2\cdot {\mathbb{E}}_{S_1,A_1,S_2,A_2,S_3,A_3}\left[\mathbf{g}\left(S_3,A_3;\mathbf{\theta }\right)\right]\\ & +\cdots \\ & +{\gamma }^{n-1}\cdot {\mathbb{E}}_{S_1,A_1,S_2,A_2,S_3,A_3,\cdots S_n,A_n}\left[\mathbf{g}\left(S_n,A_n;\mathbf{\theta }\right)\right]]\\ =& {\mathbb{E}}_{S_1\sim f(\cdot )}\left\{{\mathbb{E}}_{A_1\sim \pi \left(\cdot \mid S_1;\mathbf{\theta }\right)}\left[\mathbf{g}\left(S_1,A_1;\mathbf{\theta }\right)\right]\right\}\\ & +\gamma \cdot {\mathbb{E}}_{S_2\sim f(\cdot )}\left\{{\mathbb{E}}_{A_2\sim \pi \left(\cdot \mid S_2;\mathbf{\theta }\right)}\left[\mathbf{g}\left(S_2,A_2;\mathbf{\theta }\right)\right]\right\}\\ & +{\gamma }^2\cdot {\mathbb{E}}_{S_3\sim f(\cdot )}\left\{{\mathbb{E}}_{A_3\sim \pi \left(\cdot \mid S_3;\mathbf{\theta }\right)}\left[\mathbf{g}\left(S_3,A_3;\mathbf{\theta }\right)\right]\right\}\\ & +\cdots \\ & +{\gamma }^{n-1}\cdot {\mathbb{E}}_{S_n\sim f(\cdot )}\left\{{\mathbb{E}}_{A_n\sim \pi \left(\cdot \mid S_n;\mathbf{\theta }\right)}\left[\mathbf{g}\left(S_n,A_n;\mathbf{\theta }\right)\right]\right\}\\ =& \left(1+\gamma +{\gamma }^2+\cdots +{\gamma }^{n-1}\right)\cdot {\mathbb{E}}_{S\sim f(\cdot )}\left\{{\mathbb{E}}_{A\sim \pi (\cdot \mid S;\mathbf{\theta })}[\mathbf{g}(S,A;\mathbf{\theta })]\right\}.\end{array}$$

证毕