Multi-Drone Reinforcement Learning and Planning

This project investigates Reinforcement Learning algorithm with the environments of multi-drone navigation and decision making in uncertain 3D environments.

[MultiDroneUnc Environment Documentation]

[video]

The project compares different planning and reinforcement learning approaches:

• Monte Carlo Tree Search (MCTS) [MCTS report]
• Proximal Policy Optimization (PPO)
• Deep Q Network (DQN)
• Hybrid MCTS + DQN

Overview

Multi-drone navigation is formulated as a Markov Decision Process (MDP).

Each drone operates in a bounded grid environment with:

• Static obstacles
• Stochastic action outcomes
• Multiple simultaneous agents
• Goal-reaching objectives

The main research question is:

How can learning-based methods and online planning methods improve multi-drone navigation under uncertainty?

Methods

1. MCTS

MCTS performs online planning from the current state.

At every step:

Current State -> MCTS Search -> Select Best Action -> Execute Action

Search:

1. From current state, randomly choose an unexpanded action.
2. Simulate (s, a)-> next_s, from next_s, Epsilon Greedy choose action, then get new state, then choose action again and also get new state. Repeat it until done or exceed planning time.
3. Accumulate the reward of simulation and use it as the Value Q(s,a)
4. Backpropagate to update the Q(s,a)

Advantages:

• No training required
• Can reason about future states

Limitations:

• High computational cost
• Search starts from scratch at every step

---

2. PPO

PPO learns a policy network:

pi(a|s)

Two settings are evaluated:

PPO Offline Training

Simulation -> Collect Rollouts -> PPO Update->Test Policy

The trained policy directly selects actions during evaluation.

PPO Online Adaptation

Simulation Training->Policy Update->Execute Real Step->New Observation->Repeat This evaluates continuous adaptation capability.

3. DQN

DQN learns:

Q(s,a)

Training: Simulation->store Replay Buffer-> Q Network Update -> Save Model

Testing: State-> argmax Q(s,a) -> Action

4. MCTS + DQN

A hybrid planner is proposed.

Instead of initializing MCTS nodes with:

Q(s,a)=0 the trained DQN provides prior knowledge:

      State

        |
        v

    MCTS Node

        |
        v

     DQN Q(s,a)

        |
        v

  Guided Tree Search

        |
        v

     Action

Motivation:

• DQN provides learned long-term value estimation.
• MCTS provides online search and correction.

This combines:

The environment includes:

• Static obstacles
• Collision penalty
• Goal reward
• Action uncertainty

---

Experiments

Experiment 1: PPO

Objective:

Evaluate whether a policy network can learn multi-drone navigation.

Metrics:

• Success rate
• Episode reward
• Number of steps
• Collision rate

PPO-Offline Results:

Drone num	Success Rate%	Reward	Steps
1	100	72.6	10.7
2	90.0	124.3	11.7
3	90.0	159.5	17.8
4	70.0	147.9	19.7

PPO-Online Results:

Drone num	Success Rate%	Reward	Steps
1	66.7	62.7	14.9
2	63.3	91.8	25.8
3	26.7	111.2	33.6
4	20.0	74.0	55.7

---

Experiment 2: DQN

Objective:

Evaluate value-based learning for discrete joint actions.

Metrics:

• Convergence speed
• Q-value stability
• Success rate

Results: It is hard to train for more drones (manu joint actions)

Drone num	Success Rate%	Reward	Steps
1	100.0	67.4	12.9

---

Experiment 3: MCTS

Objective:

Evaluate pure online planning.

Metrics:

• Planning time
• Search depth
• Success rate

Results:

Drone num	Success Rate%	Reward	Steps
1	93.7	65.3	13.8
2	83.3	38.8	44.7
3	73.7	123.5	30.2
4	50.0	128.7	40.9