Lunar Lander Policy vs Value

This project focuses on developing two reinforcement learning algorithms, REINFORCE and SARSA, and comparing their results in a Lunar Lander environment.

Dependencies

This project tests its simulations on: PyTorch, Gymnasium, and uses a LunarLander-v3 environment. Also uses Python 3.10+

Virtual Environment Setup

1. Setup necessary virtual environment:

   python3 -m venv venv_cpu

2. Activate virtual environment:

   source venv_cpu/bin/activate (or .\venv_cpu\Scripts\Activate.ps1)

3. Install necessary packages:

   pip install gymnasium[box2d]
   pip intall torch
   pip install matplotlib
   pip install numpy

How to Use the Code (Can follow the same steps for both REINFORCE and SARSA)

1. Using a python terminal

   python reinforce_lunar.py 

   or

   python sarsa_lunar.py

   Simulation Variables:

   • seed: 0,
   • env_name: "LunarLander-v3",
   • display: true,
   • display_episodes: 500,
   • num_episodes: 2000, (between 2000-3000)
   • max_steps: 200,
   • hidden_layers: [128, 128], (vary for experiment)
   • activation_function: "relu",
   • learning_rate: 0.0001, (vary for experiment)
   • gamma: 0.99, (vary for experiment)
   • initial_epsilon: 1.0, (vary for experiment)
   • min_epsilon: 0.01, (vary for experiment)
   • decay_rate: 0.005, (vary for experiment)
   • use_baseline: true,
   • use_boltzmann: true (vary for experiment)

2. Output

   1. Initializes the PyTorch and Gymnasium simulation environment and trains Lunar Lander for your desired num_episodes.
   2. Displays Lunar Lander every display_episodes.
   3. After training is finished, will output graphical results of 20-episode moving average reward results and total reward results per episode

Example Plots

1. REINFORCE Algorithm Results in LunarLander-v3 for 20-episode Moving Average

2. SARSA Algorithm Results in LunarLander-v3 for 20-episode Moving Average