AI & Machine Learning20235 min readAI & RL Research Developer● Open Source

Go Up Autonomous Reinforcement Learning Agent

Custom Gym-Style Reinforcement Learning Environment and Deep Q-Network (DQN) Agent Autonomously Navigating Dynamic Vertical Platforms

LanguagePython 3.9+
RL MethodDeep Q-Learning (DQN)
Deep LearningPyTorch
EnvironmentCustom Pygame Gym
StatusTrained Autonomous Policy
Observation SpaceContinuous Kinematics
Action Primitives4 Discrete Actions
Replay Buffer50,000 Transitions
Convergence3,000+ Episodes
Go Up Autonomous Reinforcement Learning Agent
System Preview
Go Up Autonomous Reinforcement Learning Agent Interface Screenshot
01. Overview & Problem Solved

System Context & Value Proposition

GoUp-RL is a reinforcement learning platform where an autonomous neural policy learns to navigate procedurally generated vertical platform stages without human intervention. Built from scratch in Python and PyTorch, the project implements a custom Gym-compatible game environment (reset, step, close), a Deep Q-Network (DQN) agent with experience replay, target network stabilization, and custom reward engineering.

Core Problem Statement:

Training an autonomous agent in a fast-paced vertical platformer requires balancing exploration vs exploitation in sparse, high-penalty reward landscapes where fatal falls terminate episodes instantly.

✕Standard RL benchmark suites (Atari, MuJoCo) lack customizable platformer mechanics with procedural stage variance.
✕Sparse reward problem: The agent only receives rewards when ascending, causing early training agents to become trapped in idle states.
✕Moving platform unpredictability causing catastrophic policy forgetting.
✕Temporal correlation between consecutive game steps destabilizing gradient descent updates.
💡 Engineered Solution:Engineered a custom Gym-style Pygame environment, formulated an informative continuous observation space, shaped reward functions penalizing idle behavior while rewarding altitude gains, and stabilized training via Deep Q-Learning.
02. Engineering Blueprint

System Architecture & Execution Flow

Architecture Data FlowASCII Topology
Custom Pygame RL Environment
      ↓ (Observation State: Player Pos, Velocity, Nearest Platforms)
Deep Q-Network Agent (PyTorch Neural Policy)
      ↓ (Q-Value Estimation for Discrete Actions)
Action Selection (Epsilon-Greedy Exploration vs Exploitation)
      ↓ (0: Idle, 1: Left, 2: Right, 3: Jump)
Game Physics Execution & Collision Check
      ↓
Reward Calculation (+Altitude Gain, -Idle Time, -Fall Penalty)
      ↓
Experience Replay Buffer (State, Action, Reward, Next State, Done)
      ↓
Target Network Parameter Updates

Multi-Tier Architectural Layers

Layer 01 • Environment Simulation Layer

Pygame, Python

Gym-style environment interface (reset, step, close) exposing player kinematics, jump timers, and relative distance vectors to upcoming platforms.

Layer 02 • Deep Q-Network Policy

PyTorch Multi-Layer Perceptron

Neural network approximating action-value Q(s, a) functions with fully connected linear layers, ReLU activations, and Mean Squared Error loss.

Layer 03 • Experience Replay Memory

Circular Deque Buffer

Stores 50,000 transition tuples and samples uncorrelated mini-batches during gradient updates to stabilize learning.

Layer 04 • Curriculum Progression

Multi-Stage Environments

Progressive environment stages (rl_env_stage_1.py to stage_4.py) introducing increasing platform spacing and moving hazards.

03. Key Capabilities

Core Engineering Modules & Workflows

✓Gym-Compatible Environment API

Custom environment exposing standardized reset() and step(action) interfaces returning numerical observation arrays and rewards.

✓Continuous Observation Vector

Encodes player position, velocity (dx, dy), jump state, nearest candidate platforms, and landing zone coordinates.

✓Discrete Action Combinations

Horizontal navigation (Stay, Left, Right) combined with binary jump execution.

✓Reward Shaping Algorithm

Penalizes idle hovering, severely penalizes fatal descents, and rewards sustained vertical altitude advancement.

✓Epsilon-Greedy Decay Policy

Balances random exploratory jumps in early episodes with learned Q-policy exploitation in mature training stages.

04. Data Modeling & Interfaces

Persistence Schemas & Execution Contracts

Database Schema & Entity Structure

Schema RepresentationValidated DDL
Observation Vector Structure:
  [player_x, player_y, player_vx, player_vy, can_jump,
   dist_plat_1_x, dist_plat_1_y, dist_plat_2_x, dist_plat_2_y,
   dist_plat_3_x, dist_plat_3_y, nearest_hazard_dist]
Trained Model Checkpoints:
  └── models/GoUp_DQN_Model.pt (Trained PyTorch Policy Tensor)

Key Endpoints & Query Interfaces

CLI Commandpython train_dqn.py
Executes training loop with live loss tracking and checkpointing.
CLI Commandpython test_agent.py --model models/GoUp_DQN_Model.pt
Runs evaluation visualizer showing real-time autonomous gameplay.
05. Implementation & Retrospective

Engineering Challenges & Specifications

Technical Obstacles Solved

⚡ Challenge:Catastrophic Policy Forgetting

Resolution: Maintained a secondary target network updated every 1,000 steps to decouple target Q-value calculation from active weight gradients.

⚡ Challenge:Over-Cautious Stationary Behavior

Resolution: Engineered negative step decay penalties forcing the agent to prioritize upward momentum rather than loitering on initial platforms.

Technology Stack Matrix

CategoryTooling & Frameworks
LanguagePython 3.9+
Deep Learning LibraryPyTorch (torch, torch.nn)
RL AlgorithmDeep Q-Learning (DQN) with Replay Buffer
Game SimulationPygame
Environment StandardGym-Style API (reset, step, close)

Local Execution Commands

TerminalBash
$git clone https://github.com/KESHAV-26-2004/GoUp-RL.git
$pip install -r requirements.txt
$python train_dqn.py (To train new policy)
$python test_agent.py (To watch trained agent navigate procedural levels)

Future Roadmap & Upcoming Milestones

→Transition from DQN to Proximal Policy Optimization (PPO) for continuous action control.
→End-to-End visual reinforcement learning trained directly on raw pixel frames using convolutional neural networks.