无线通信：基于深度强化学习-编程知识

这里写自定义目录标题

异构蜂窝网络：用户关联和信道分配
- a stochastic game
- Multi-Agent Q-Learning Method
- Multi-Agent dueling double DQN Algorithm
分布式动态下行链路波束成形
- Limited-Information Exchange Protocol
- Distributed DRL-Based DTDE Scheme for DDBC
- - Distributed DRL-Based DTDE Scheme for DDBC
空基网络中的分布式多用户接入控制：BS和UE关联
- 基于分布式 DRL 的用户驱动接入控制算法
- 基准算法
天基网络中的LEO卫星多用户信道分配：RB分配
- 基于协作 DRL 的 LEO 卫星多用户信道分配
天基网络中的LEO地球固定小区设计：波束角度、RB分配、RB预配置子集
天空融合网络中的多维资源优化：卫星的频谱分配、UAV轨迹、UAV和UE关联

面向非地面网络的智能无线资源管理机制与算法研究
[1]曹阳. 面向非地面网络的智能无线资源管理机制与算法研究[D]. 电子科技大学, 2023. DOI: 10.27005/d.cnki.gdzku.2023.000168.

异构蜂窝网络：用户关联和信道分配

N. Zhao, Y. -C. Liang, D. Niyato, Y. Pei, M. Wu and Y. Jiang, “Deep Reinforcement Learning for User Association and Resource Allocation in Heterogeneous Cellular Networks,” in IEEE Transactions on Wireless Communications, vol. 18, no. 11, pp. 5141-5152, Nov. 2019, doi: 10.1109/TWC.2019.2933417.

Three-tier heterogeneous cellular network.
L BSs → N UEs
K orthogonal channels

the Joint user association and resource allocation Optimization Problem

variables: discrete
1.bli(t)=1: the $i$ th UE chooses to associate with the BS $l$ at time $t$
2.cik(t)=1: the $i$ th UE utilizes the channel $C k$ at time $t$
constraints
1.each UE can only choose at most one BS at any time
2.each UE can only choose at most one channel at any time
3.the SINR of the $i t h$ UE ≥

a stochastic game

state
si(t) ∈{0, 1}
si(t)=0 means that the ith UE cannot meet its the minimum QoS requirement, that is, Γi(t) < Ωi
the number of possible states is $2^N$

action
alki(t)={bli(t),cik(t)},
1.bli(t)=1: the $i$ th UE chooses to associate with the BS $l$ at time $t$
2.cik(t)=1: the $i$ th UE utilizes the channel $C k$ at time $t$
The number of possible actions of each UE is $L K$ = L种选择方式 * K种选择方式

reward
the long-term reward Φi = the weighted sum of the instantaneous rewards over a finite period T

the reward of the ith UE = the ith UE’s utility - the action-selection cost Ψi
Ψi > 0. Note that the negative reward (−Ψi) acts as a punishment.
to guarantee the minimum QoS of all UEs, this negative reward should be set big enough.

the ith UE’s utility = \rho_i * the total transmission capacity of the ith UE - the total transmission cost associated with the ith UE

Multi-Agent Q-Learning Method

At the beginning of each training episode, the network state is initialized through message passing.

Each UE is connected to the neighboring BS with the maximum received signal power.
By using a pilot signal, each UE can measure the received power from the associated BS and the randomly-selected channel.
Then, each UE reports its own current state to its current associated BS.
By the message passing among the BSs through the backhaul communication link, the global state information of all UEs is obtained.
Then, the BSs send this global state informations to all UEs.

Each episode ends when the QoS of all UEs is satisfied or when the maximum step T is reached.
The total episode reward is the accumulation of instantaneous rewards of all steps within an episode.
Multi-Agent Q-Learning Method

${Q_{i}}(s,a_{i})={Q_{i}}(s,a_{i})+ \delta \left[{ {u_{i}}(s,a_{i},{\mathcal{ \pi }}_{-i}) + \gamma \max \limits _{a_{i}' \in {\mathcal{ A}}_{i}} {Q_{i}}({s'},{a_{i}'})} {- {Q_{i}}(s,a_{i})\vphantom {\left [{ {u_{i}}(s,a_{i},{\mathcal{ \pi }}_{-i}) + \gamma \max \limits _{a_{i}' \in {\mathcal{ A}}_{i}} {Q_{i}}({s'},{a_{i}'})}\right.} }\right],$

Multi-Agent dueling double DQN Algorithm

dueling double deep Q-network (D3QN)

A NN function approximator $Q_{i}(s,a_{i};{\theta }) \approx {Q_{i}^{*}}(s,a_{i})$ with weights θ is used as an online network.
The DQN utilizes a target network alongside the online network to stabilize the overall network performance.

experience replay
During learning, instead of using only the current experience (s, ai,ui(s, ai),s′), the NN can be trained through sampling mini-batches of experiences from replay memory D uniformly at random.
By reducing the correlation among the training examples, the experience replay strategy ensures that the optimal policy cannot be driven to a local minima.

double DQN
since the same values are used to select and evaluate an action in Q-learning and DQN methods, Q-value function may be over-optimistically estimated.
Thus, double DQN (DDQN) [44] is used to mitigate the above problem

dueling architecture
The advantage function A(s, ai) describes the advantage of the action ai compared with the other possible actions.
This dueling architecture can lead to better policy evaluation.
Three hidden layers with dueling architecture used in our D3QN.
Reinforcement learning with DDQN strategy.

${L_{i}}({\theta }) = {E_{s,a_{i},u_{i}(s,a_{i}),s'}}[{(y_{i}^{DQN} - Q_{i}(s,a_{i};{\theta }))^{2}}],$
上图里面红色的y为
$y_{i}^{DDQN} = {u_{i}}(s,a_{i}) + \gamma Q_{i}\left ({s',\mathop {\arg \max }\limits _{a'_{i} \in {\mathcal{ A}}_{i}} Q_{i}(s',a'_{i};{\theta });\theta ^{-} }\right).$

Multi-Agent dueling double DQN Algorithm

分布式动态下行链路波束成形

J. Ge, Y. -C. Liang, J. Joung and S. Sun, “Deep Reinforcement Learning for Distributed Dynamic MISO Downlink-Beamforming Coordination,” in IEEE Transactions on Communications, vol. 68, no. 10, pp. 6070-6085, Oct. 2020, doi: 10.1109/TCOMM.2020.3004524.

波束成形论文代码

multi-cell MISO-IC model
a downlink cellular network of K cells
no intra-cell interference
all the BSs are equipped with a uniform linear array having $N$ ( $N \geq 1$ ) antenna elements.

$\max _{\mathbf {W}{(t)}}~\sum _{k=1}^{K}C_{k}(\mathbf {W}{(t)}) \text{8a}\\{\mathrm{ s.t.}}~0\leq \left \|{\mathbf {w}_{k}{(t)}}\right \|^{2} \leq p_{\mathrm{ max}},~\forall k \in \mathcal {K},\text{8b}$
the beamformer of BS $k$
the available maximum transmit power budget of each BS

Limited-Information Exchange Protocol

a downlink data transmission framework
The first phase (phase 1) is a preparing phase for the subsequent data transmission
the second phase (phase 2) is for the downlink data transmission.
in the centralized approaches, the cascade procedure of collecting global CSI, computing beamformers, and sending beamformers to the corresponding BSs is supposed to be carried out within phase 1.

Designed limited-information exchange protocol in time slot t.
BSs are able to share their historical measurements and other information with their interferers and interfered neighbors.

The received interference power from interferer $j \in I k (t)$ in time slot $t - 1$
, i.e., $∣∣ h † j, k (t - 1) w j (t - 1) ∣∣ 2$ .
The total interference-plus-noise power of UE $k$ in time slot $t - 1$
, i.e., $\sum l \neq = k ∣∣ h † l, k (t - 1) wl (t - 1) ∣∣ 2 + σ 2$ .
The achievable rate of direct link $k$ in time slot $t - 1$
, i.e., $C k (W (t - 1))$ .
The equivalent channel gain of direct link $k$ in time slot $t - 1$
, i.e., $∣∣ h † k, k (t - 1) w ¯ k (t - 1) ∣∣ 2$ .

Distributed DRL-Based DTDE Scheme for DDBC

distributed-training-distributed-executing (DTDE)
distributed dynamic downlink-beamforming coordination (DDBC)

each BS is an independent agent
a multi-agent reinforcement learning problem

Illustration of the proposed distributed DRL-based DTDE scheme in the considered multi-agent system.

Actions
$\mathcal {A} = \{(p, {\mathbf{c}}),~p\in \mathcal {P},~{\mathbf{c}}\in \mathcal {C}\},$
$\mathcal {P}=\left \{{0, \tfrac {1}{Q_{\mathrm{ pow}}-1}p_{\mathrm{ max}},\,\,\tfrac {2}{Q_{\mathrm{ pow}}-1}p_{\mathrm{ max}},\,\,\cdots,\,\,p_{\mathrm{ max}}}\right \}$
$\mathcal {C}=\left \{{\mathbf {c}_{0},\,\,\mathbf {c}_{1},\,\,\cdots,\,\,\mathbf {c}_{Q_{\mathrm{ code}}-1}}\right \}$
1.the transmit power of BS k in time slot t
2.code $c k (t)$
the total number of available actions is $Q = Qp o wQ co d e$
States
1.Local Information
2.Interferers’ Information
3.Interfered Neighbors’ Information
Reward
the achievable rate of agent $k$
the penalty on BS k is defined as the sum of the achievable rate losses of the interfered neighbors j∈Ok(t+1) , which are interfered by BS k , as follows:

Distributed DRL-Based DTDE Scheme for DDBC

Distributed DRL-Based DTDE Scheme for DDBC
In training step t , the prediction error
$L(\boldsymbol {\theta })= \frac {1}{2M_{b}}\sum \limits _{\langle s,a,r,s'\rangle \in \mathcal {D}}\left ({r'-q(s,a; \boldsymbol {\theta })}\right)^{2}$
the target value of reward
$\gamma \max \limits _{a'}q(s',a'; \boldsymbol {\theta }^{-})$

the optimizer returns a set of gradients shown in (22) to update the weights of the trained DQN through the back-propagation (BP) technique
$\frac {\partial L(\boldsymbol {\theta })}{\partial \boldsymbol {\theta }}= \frac {1}{M_{b}}\sum _{\langle s,a,r,s'\rangle \in \mathcal {D}}\left ({r'-q(s,a; \boldsymbol {\theta })}\right)\nabla q(s,a; \boldsymbol {\theta }).$

空基网络中的分布式多用户接入控制：BS和UE关联

Y. Cao, S. -Y. Lien and Y. -C. Liang, “Deep Reinforcement Learning For Multi-User Access Control in Non-Terrestrial Networks,” in IEEE Transactions on Communications, vol. 69, no. 3, pp. 1605-1619, March 2021, doi: 10.1109/TCOMM.2020.3041347.

基于空基平台的 NTN 实现方案
K架固定翼式UAV作为NT-BS，为特定区域内的M个移动UE提供下行传输服务

ci,j(t)表示第j个NT-BS对第i个UE在时隙t的传输速率
变量：ui,j(t)表示第i个UE在时隙t是否接入第j个NT-BS
约束：每个UE在单个时隙只能接入一个NT-BS

基于分布式 DRL 的用户驱动接入控制算法

每个用户均作为独立的智能体，利用DQN在UE侧建立本地接入决策模块，并且每个UE仅采用本地观测量自主地完成NT-BS选择。

状态空间
si(t)表示第i个UE在时隙t的状态（4K+1个元素）
1.第i个用户在时隙t−1所接入的NT-BS标号（K个元素）
2.第i个用户处在时隙t−1和时隙t来自各NT-BS的RSS（2K个元素）
3.每个NT-BS在时隙t−1的接入用户数目（K个元素）
4.第i个UE在时隙t−1实现的传输速率（1个元素）

动作空间
ai(t)表示第i个UE在时隙t的动作（K个元素）
ui,j(t)表示第i个UE在时隙t是否接入第j个NT-BS

奖励函数
第i个UE在第t个时隙内的奖励函数
ri(t)=ri−(t)−ηφi(t)
UE获得的本地奖励
该UE对接入相同NT-BS的其他UE和速率所造成的影响

第i个UE的本地奖励：由第i个UE在当前时隙的传输速率和对应的切换代价所决定
ri−(t)=ωi(t),ai(t)=ai(t−1),
ri−(t)=ωi(t)−C,ai(t) ̸=ai(t−1).

第i个UE采用特定接入决策对集合Oj(t)中其他UE造成的影响φi(t)=
在假定第i个UE没有接入第j个NT-BS的情况下第k个UE的传输速率ωk−i(t)-第k个UE的真实传输速率ωk(t)
对k求和

算法：UE 端DQN

用户驱动的智能接入控制方案

基于分布式DRL 的用户驱动接入控制算法

基准算法

RSS 算法：在每个时隙，每个UE 选择可以提供最强RSS 的NT-BS 接入。
Q 学习算法 [122]：在该算法中，每个UE 采用表格形式函数（即 Q 表格）来估计每个状态-动作对的 Q 值。在每个时隙，每个UE 根据所建立的 Q 表格做出NT-BS 选择。
UCB 算法 [123]：在UCB 算法中，UE 在每个时隙根据下列公式确定其NT-BS 选择
随机算法：在该算法中，每个UE 在每个时隙随机选择一个NT-BS 接入。
搜索算法（最优结果）：为了获得最优结果，该算法假定存在一个集中决策节点实时收集全局网络信息。

M = 40 个UE 和 K = 6 个NT-BS

天基网络中的LEO卫星多用户信道分配：RB分配

Y. Cao, S. -Y. Lien and Y. -C. Liang, “Multi-tier Collaborative Deep Reinforcement Learning for Non-terrestrial Network Empowered Vehicular Connections,” 2021 IEEE 29th International Conference on Network Protocols (ICNP), Dallas, TX, USA, 2021, pp. 1-6, doi: 10.1109/ICNP52444.2021.9651962.

多波束LEO卫星下行传输系统
每颗LEO卫星采用多波束技术在地面形成N个频率复用因子为1的波束小区，并且每个波束小区内有M个移动UE。

在第t个时隙中，第j个小区中第i个UE在第k个RB可以实现的下行传输速率可以表示为ci,j,k(t)
在第t个时隙中，第j个小区中第i个UE可以实现的下行传输速率可以表示为 ci,j(t) rate
UE满意度αi,j(t)：描述LEO卫星为UE分配的RB数目与该UE真实的RB数目需求之间的偏差
速率-满意度效用函数Ui,j(t) = wr ci,j(t) + ws αi,j(t)，其中wr+ws=1。

从时隙0到时隙T−1的平均UE间速率-满意度效用函数最小值为

最大化从时隙0到时隙T−1的平均UE间速率-满意度效用函数最小值
变量：如果在时隙t，第j个波束小区中第i个UE接入第k个RB，那么xti,j,k=1。
约束：在单个时隙内，每个RB只能分配给一个UE

基于协作 DRL 的 LEO 卫星多用户信道分配

状态空间
在每个时隙 t，LEO 卫星的状态向量 s(t)
1.上一个时隙的RB 分配决策
2.每个UE 在上一个时隙所分配RB 上经历的干扰强度
3.每个UE 在上一个时隙的满意度
4.上一个时隙多个UE 试图接入相同RB 时发生碰撞的RB 的索引

动作空间
LEO 卫星在每个时隙 t 需要为波束小区内各UE 分配RB
如果在时隙t，第j个波束小区中第i个UE接入第k个RB，那么xti,j,k=1。
将网络实际输出连续变量ˆ a(t) ∈ [0, 1] 进行离散化处理

奖励函数r(t)
UE 中速率-满意度效用最小值 min i∈Uj,j∈N ci,j(t)
UE 的当前RB 接入决策发生碰撞的惩罚项 Ci,j collision(t)
UE 满意度不满足的惩罚项 Csatisfactory αi,j(t)

LEO 卫星 DRL 分层协作架构

LEO 卫星星间和星地协作关系

算法：双延迟深度确定性策略梯度 (Twin Delayed Deep Deterministic Policy Gradient, TD3) 算法
协作DRL 算法中LEO 卫星端操作流程

协作DRL 算法中地面UE 端操作流程

在这两种方案中，多波束LEO 卫星在半径为 500 千米的圆形区域内形成 N = 2 个相同尺寸的波束小区，并在每个波束小区内采用 K = 5 个带宽为 180 kHz 的RB 服务 M = 3 个移动的UE。

基准算法
DDPG 算法
无协作TD3 算法

天基网络中的LEO地球固定小区设计：波束角度、RB分配、RB预配置子集

Y. Cao, S. -Y. Lien, Y. -C. Liang, D. Niyato and X. S. Shen, “Collaborative Deep Reinforcement Learning for Resource Optimization in Non-Terrestrial Networks,” 2023 IEEE 34th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), Toronto, ON, Canada, 2023, pp. 1-7, doi: 10.1109/PIMRC56721.2023.10294047.

多时间尺度资源配置机制下的LEO卫星地球固定小区方案

最小化UE从时隙0到时隙N−1利用的RB数目