TechPrufer
Get started
Hugging Face
Hugging Face logoHugging Face · Feature trail

Reinforcement Learning Value-Based Methods

9 posts · May 2022 – Jan 2026

What this trail covers

Hugging Face offers a comprehensive Deep Reinforcement Learning Class, covering foundational algorithms like REINFORCE and Actor-Critic methods, with practical applications in robotics simulations. This post details the journey to unlock agentic RL training for GPT-OSS, addressing challenges in MoE log-probability mismatch and attention sink support, and introduces memory-efficient training techniques like mitigating FSDP memory blow-ups and sequence parallel with Flash Attention V3. The work establishes a stable foundation for RL objectives by eliminating train-inference mismatches. This post introduces Decision Transformers, abstracting RL as a conditional-sequence modeling problem using Transformers, and details how to train an offline Decision Transformer from scratch for tasks like making a half-cheetah run, using a custom data collator and the 🤗 Trainer.

All Hugging Face trails
  1. Unlocking Agentic RL Training for GPT-OSS: A Practical RetrospectiveThis post details the practical challenges and solutions encountered when enabling agentic RL training for the GPT-OSS model using the verl framework. Key contributions include: 1. Identifying and fixing a MoE log-probability mismatch that caused exploding KL divergence and non-increasing rewards in on-policy PPO training. 2. Implementing attention sink support in FlashAttentionV3 to handle the specific attention patterns required by GPT-OSS. 3. Introducing memory-efficient training techniques, specifically mitigating FSDP memory blow-ups caused by repeated MoE expert materialization and enabling sequence parallel with Flash Attention V3.
  2. Train your first Decision TransformerIntroduces Decision Transformers as a novel approach to Reinforcement Learning by framing it as a conditional-sequence modeling problem solvable with Transformers. Details the architecture, including modality-specific embeddings and GPT-2 processing for autoregressive action prediction. Provides a practical guide to training an offline Decision Transformer from scratch using the 🤗 Trainer and a custom data collator, demonstrating its application on the HalfCheetah environment. Includes code snippets for dataset loading, custom data collator implementation (including state normalization, return discounting, and trajectory length sampling), and model training setup.
  3. Introducing ⚔️ AI vs. AI ⚔️ a deep reinforcement learning multi-agents competition systemIntroduces a deep reinforcement learning multi-agent competition system called AI vs. AI, designed for training and evaluating agents in competitive environments like Soccer Twos. This system facilitates self-play and agent interaction, advancing research in multi-agent reinforcement learning.
  4. Proximal Policy Optimization (PPO)Introduced Proximal Policy Optimization (PPO) as an advancement over A2C, focusing on stabilizing reinforcement learning training by constraining policy updates. Explained the intuition behind PPO, the Clipped Surrogate Objective function, and the role of the probability ratio in measuring policy divergence. Provided a breakdown of the objective function's components and a practical coding example for a PPO agent.
  5. Advantage Actor Critic (A2C)This post introduces Advantage Actor Critic (A2C), a hybrid method combining policy-based and value-based approaches to reduce variance in policy gradient estimation. It details the Actor-Critic architecture, where an Actor controls the agent's behavior and a Critic evaluates actions. The post explains the training process, including how the Actor and Critic are optimized using Q-values and state-action pairs, and demonstrates A2C implementation with PyBullet for robotics simulations.
  7. Policy Gradient with PyTorchThis post introduces Policy Gradient methods as an alternative to value-based methods like Q-Learning. It explains the core concepts of policy gradients, their advantages (stochastic policies, continuous action spaces) and disadvantages (local optima, variance). The post details the REINFORCE algorithm, a Monte Carlo Policy Gradient method, and outlines its update mechanism using gradient ascent to maximize the expected return. It also provides links to a PyTorch implementation tutorial and relevant environments (CartPole-v1, PixelCopter, Pong).
  8. Deep Q-Learning with Space InvadersThis post introduces Deep Q-Learning as an extension of Q-Learning for handling large state spaces. It details the Deep Q-Network (DQN) architecture, input preprocessing techniques (resizing, grayscale, frame stacking), and the core Deep Q-Learning algorithm which uses a neural network to approximate Q-values and a loss function with gradient descent for training. Key stabilization techniques like Experience Replay, Fixed Q-Targets, and Double DQN are explained, along with their benefits in improving training efficiency, stability, and reducing Q-value overestimation. The post demonstrates the application of DQN in playing Atari games.
  9. An Introduction to Q-Learning Part 2/2This post details the Q-Learning algorithm, an off-policy value-based method using TD learning. It explains the Q-function and Q-table, the Q-Learning pseudocode (initialization, epsilon-greedy strategy, action execution, and Q-value update), and the distinction between off-policy and on-policy learning. A practical example with a maze scenario illustrates the algorithm's application.
  10. An Introduction to Q-Learning Part 1This post introduces the core concepts of value-based Reinforcement Learning methods. It defines state-value and action-value functions, explains their relationship to policies, and introduces the Bellman Equation for simplifying value estimation. It also contrasts Monte Carlo methods with Temporal Difference learning, setting the stage for Q-Learning.