Train Your First Reinforcement-Learning Trading Bot with FinRL

Reinforcement learning has moved from game-playing headlines to practical finance experiments. Training an agent to make buy, sell, and hold decisions from raw price data sounds futuristic, but the tools are already within reach of any Python programmer. FinRL, an open-source deep reinforcement learning framework for quantitative finance, lowers the barrier so you can train your first trading bot in an afternoon.

This tutorial walks through the entire pipeline: installing FinRL, configuring a trading environment, training two popular algorithms, PPO and SAC, and comparing the learned strategies against a simple buy-and-hold benchmark. We will use the well-known Stock_NeurIPS2018 example so you can reproduce every step without hunting for private data. The goal is not to hand you a profitable strategy. The goal is to teach you how reinforcement learning fits into algorithmic trading, where it shines, and where it can mislead you.

What Is FinRL?

FinRL is an open-source Python library built on top of stable, familiar machine-learning tools. It wraps OpenAI Gym-style environments, Yahoo Finance data downloaders, and Stable Baselines3 agents into a coherent framework for financial reinforcement learning. The project is maintained by the AI4Finance Foundation and has become a standard starting point for students and researchers who want to explore deep reinforcement learning in markets.

The core idea is simple. You define a market environment where the agent observes a state, such as prices, technical indicators, and current holdings. At each time step, the agent selects an action, such as buying shares, selling shares, or holding cash. The environment responds with a reward, typically the change in portfolio value, and transitions to the next state. Over thousands of episodes, the agent learns a policy that maximizes cumulative reward.

FinRL handles much of the boilerplate. It downloads historical data, computes technical indicators, splits data into training and testing sets, defines action and observation spaces, and integrates with Stable Baselines3 for training. It also provides prebuilt configuration dictionaries for common datasets, including the one we will use here.

FinRL is not a black-box money printer. It is a research tool. The value it provides is structure: a clean separation between data, environment, agent, and evaluation. Once you understand that structure, you can swap datasets, change reward functions, and experiment with more sophisticated agents without rewriting everything from scratch.

Why Reinforcement Learning for Trading?

Traditional trading strategies are usually rule-based. A moving-average crossover system might buy when a short-term average crosses above a long-term average and sell on the reverse. These rules are easy to understand and test, but they are rigid. They do not adapt to changing volatility, they do not learn from experience, and they cannot optimize position sizing in a continuous way.

Reinforcement learning reframes the problem. Instead of writing a fixed rule, you define a reward signal and let the agent discover a policy. The agent can learn to hold more cash during uncertain periods, increase exposure when trends align, and exit positions before large drawdowns. Because the policy is learned from data, it can in principle adapt to patterns that are hard to express as explicit rules.

That flexibility comes at a cost. RL agents are data-hungry, sensitive to hyperparameters, and prone to overfitting. A poorly specified reward function can lead to reward hacking, where the agent finds a loophole that maximizes reward during training but fails in live markets. For example, an agent rewarded only for returns might learn to take enormous leverage during a bull market, then collapse during the first correction. Controlling risk in the reward function, usually through a Sharpe ratio or drawdown penalty, is essential.

For this tutorial, we choose reinforcement learning because it naturally handles the sequential nature of trading. Every decision today affects the set of decisions available tomorrow. RL captures that dependency in a way that static prediction models do not.

Environment Setup

Before training any agent, you need a working Python environment. FinRL has several dependencies, so the cleanest approach is to create a dedicated virtual environment or conda environment. This avoids conflicts with other packages and makes reproduction easier.

Start by creating and activating a conda environment. If you prefer venv, the commands are similar.

conda create -n finrl python=3.10 -y
conda activate finrl

Install the core packages. FinRL depends on Stable Baselines3, Gymnasium, pandas, numpy, and several finance utilities. A minimal installation looks like this:

pip install finrl stable-baselines3 gymnasium yfinance

If you encounter dependency conflicts, install FinRL from source into a fresh environment. The maintainers test against specific versions of Gymnasium and Stable Baselines3, and the rapidly evolving RL ecosystem can cause version mismatches.

git clone https://github.com/AI4Finance-Foundation/FinRL.git
cd FinRL
pip install -e .

After installation, verify that the imports work in a Python session.

import pandas as pd
import numpy as np
import gymnasium as gym
from stable_baselines3 import PPO, SAC
from finrl import config
print("FinRL ready")

The Stock_NeurIPS2018 Example

The Stock_NeurIPS2018 dataset is a classic benchmark in financial reinforcement learning. It was introduced for the NeurIPS 2018 paper on deep reinforcement learning for portfolio management and has since become the "Hello World" of FinRL. The dataset covers daily prices for Dow Jones Industrial Average tickers from 2009 to 2022, split into training, validation, and testing periods.

Using a public benchmark is important. It lets you compare your results with other researchers, detect bugs by reproducing known numbers, and avoid the temptation to mine a private dataset until something looks good. The Dow-30 universe is also liquid and well-known, which reduces concerns about survivorship bias and data quality that plague smaller stock samples.

In FinRL, loading the dataset is straightforward. The library includes a helper that downloads the data, adds technical indicators, and returns a clean pandas DataFrame.

from finrl import config_tickers
from finrl.meta.preprocessor.yahoodownloader import YahooDownloader
from finrl.meta.preprocessor.preprocessors import FeatureEngineer, data_split
from finrl import config
 
# Dow-30 tickers from FinRL config
 tickers = config_tickers.DOW_30_TICKER
 
# Download data
df = YahooDownloader(
    start_date="2009-01-01",
    end_date="2022-12-31",
    ticker_list=tickers
).fetch_data()
 
# Add technical indicators
fe = FeatureEngineer(
    use_technical_indicator=True,
    technical_indicator_list=config.INDICATORS,
    use_vix=True,
    use_turbulence=True,
    user_defined_feature=False
)
processed = fe.preprocess_data(df)

The processed DataFrame contains open, high, low, close, volume, and a set of technical indicators for each ticker and date. FinRL's default indicator list includes MACD, RSI, CCI, DX, and simple moving averages. You can extend this list, but the defaults are a solid baseline.

Splitting the data this way is critical. You must never train and test on the same period. A model that memorizes the 2020 crash and recovery will look amazing on that data and fail when 2023 behaves differently. The validation set gives you a way to tune hyperparameters without peeking at the final test set.

Defining the Trading Environment

FinRL's StockTradingEnv is the bridge between market data and the RL agent. It tracks the agent's portfolio, computes rewards, and enforces constraints such as transaction costs and maximum position sizes.

The state at each step includes market features for every ticker plus account-level information. Account information usually contains current stock holdings, available cash, and technical indicators. The action space depends on whether you use a discrete or continuous formulation. In a common continuous setup, the agent outputs a vector of target portfolio weights, one per stock plus cash. The environment then rebalances toward those weights, paying transaction costs along the way.

from finrl.meta.env_stock_trading.env_stocktrading import StockTradingEnv
 
# Split processed data
train = data_split(processed, "2009-01-01", "2014-12-31")
trade = data_split(processed, "2016-01-01", "2022-12-31")
 
# Environment parameters
stock_dimension = len(train.tic.unique())
state_space = 1 + 2 * stock_dimension + len(config.INDICATORS) * stock_dimension
buy_cost_list = sell_cost_list = [0.001] * stock_dimension
 
env_kwargs = {
    "hmax": 100,
    "initial_amount": 1000000,
    "num_stock_shares": [0] * stock_dimension,
    "buy_cost_pct": buy_cost_list,
    "sell_cost_pct": sell_cost_list,
    "state_space": state_space,
    "stock_dim": stock_dimension,
    "tech_indicator_list": config.INDICATORS,
    "action_space": stock_dimension,
    "reward_scaling": 1e-4,
}
 
e_train_gym = StockTradingEnv(df=train, **env_kwargs)

Transaction costs matter more than beginners expect. A 0.1 percent round-trip cost sounds small, but an agent that trades daily can pay several percent per month in friction. Always include realistic costs in your environment. If your strategy only works with zero fees, it is not a real strategy.

Training PPO

Proximal Policy Optimization, or PPO, is one of the most popular deep reinforcement learning algorithms. It is an on-policy method, which means it learns from the data collected by the current policy. PPO stabilizes training by clipping the policy update so the new policy does not diverge too far from the old one. This makes it forgiving for newcomers and a reliable first choice for trading experiments.

Training a PPO agent in FinRL follows the Stable Baselines3 API. You create a vectorized environment, instantiate the model, and call learn.

from stable_baselines3 import PPO
from stable_baselines3.common.logger import configure
 
# Wrap the environment for Stable Baselines3
env_train, _ = e_train_gym.get_sb_env()
 
# PPO model
ppo_model = PPO(
    "MlpPolicy",
    env_train,
    learning_rate=0.0003,
    n_steps=2048,
    batch_size=64,
    n_epochs=10,
    gamma=0.99,
    gae_lambda=0.95,
    clip_range=0.2,
    verbose=1,
    tensorboard_log="./tensorboard/"
)
 
# Train
ppo_model.learn(total_timesteps=100000)

The hyperparameters above are sensible defaults. n_steps=2048 controls how many timesteps the agent collects before performing a policy update. clip_range=0.2 limits how aggressively the policy can change in one update. gamma=0.99 determines how much future rewards are discounted. These values are not magical; they are starting points. You should tune them on the validation set.

During training, monitor the episode reward. In early episodes, the agent often loses money or matches cash. After several thousand timesteps, you should see the cumulative reward trend upward on the training data. Do not celebrate yet. Training performance is not the goal; out-of-sample performance is.

Training SAC

Soft Actor-Critic, or SAC, is an off-policy algorithm designed for continuous action spaces. Unlike PPO, SAC can learn from data collected by older versions of the policy, which makes it more sample-efficient in many continuous control tasks. SAC also maximizes an entropy term, encouraging exploration and often producing more diverse policies.

For trading, SAC can be appealing because portfolio weights are naturally continuous. Instead of choosing among a few discrete actions, the agent outputs a smooth allocation vector. This can lead to more nuanced position sizing.

from stable_baselines3 import SAC
 
# SAC model
sac_model = SAC(
    "MlpPolicy",
    env_train,
    learning_rate=0.0003,
    buffer_size=100000,
    batch_size=64,
    ent_coef="auto",
    gamma=0.99,
    tau=0.005,
    train_freq=1,
    gradient_steps=1,
    verbose=1,
    tensorboard_log="./tensorboard/"
)
 
# Train
sac_model.learn(total_timesteps=100000)

SAC keeps a replay buffer of past experiences and samples from it during training. buffer_size=100000 controls how much history the agent remembers. ent_coef="auto" lets SAC automatically tune the exploration-exploitation trade-off. Because SAC is off-policy, it can continue learning from old experience while the current policy explores new behavior.

Comparing Results to Buy-and-Hold

Once both models are trained, evaluate them on the test period. The test period must be completely unseen during training and validation. This is where you find out whether the agents learned something general or merely memorized the training data.

FinRL provides a prediction helper that runs a trained model through an environment and records daily account values. You can then compute returns, volatility, Sharpe ratio, maximum drawdown, and other metrics.

from finrl.agents.stablebaselines3.models import DRLAgent
 
# Test environment
e_trade_gym = StockTradingEnv(df=trade, **env_kwargs)
 
# PPO prediction
ppo_df_account_value, ppo_df_actions = DRLAgent.DRL_prediction(
    model=ppo_model,
    environment=e_trade_gym
)
 
# SAC prediction
sac_df_account_value, sac_df_actions = DRLAgent.DRL_prediction(
    model=sac_model,
    environment=e_trade_gym
)

The most important comparison is against buy-and-hold. Buy-and-hold means investing the initial capital equally across all stocks in the portfolio at the start of the test period and doing nothing else. It is a tough benchmark because it captures the long-term upward drift of equities and pays almost no transaction costs.

When you compare results, focus on risk-adjusted returns, not raw returns. A strategy that doubles the market return but triples volatility is not necessarily better. The Sharpe ratio and maximum drawdown capture the trade-off between profit and pain.

In many FinRL experiments, PPO and SAC can match or slightly exceed buy-and-hold on the test set, especially during trending periods. However, the margin is often small, and results vary with random seeds, hyperparameters, and transaction costs. If your agent underperforms buy-and-hold, that is not a failure. It is valuable information. It means the RL agent is not yet capturing predictive signal beyond the market beta.

Risks of RL Trading

Reinforcement learning is powerful but dangerous when applied to trading without discipline. The biggest risk is overfitting. An agent with millions of parameters can memorize the quirks of the training period, including exact crash dates and recovery patterns. It will look brilliant in backtests and useless in live trading.

Distribution shift is another major concern. Markets change. Regime shifts, such as moving from a low-volatility bull market to a high-inflation bear market, can break a policy trained on the previous regime. RL agents do not naturally generalize across regimes unless you explicitly train them on diverse conditions or add robustness techniques.

Reward hacking is subtle but common. If your reward is simply daily return, the agent may learn to take huge positions when it detects upward momentum and sit in cash otherwise. That works until momentum reverses. If your reward includes transaction-cost penalties but ignores slippage, the agent may trade more than is realistic. If you reward Sharpe ratio over a short window, the agent may avoid all risk and produce a flat equity curve.

Another risk is the gap between simulation and reality. Backtests assume you can execute at close prices, that your orders do not move the market, and that you can rebalance instantly. In reality, slippage, partial fills, and latency erode returns. Start with paper trading, then small live size, and only scale after you have verified execution quality.

Extending the Experiment

After you have the basic pipeline working, there are many directions to explore. You can change the reward function to optimize the Sortino ratio instead of return, add a drawdown penalty, or use a multi-objective reward that balances profit and risk. You can include more features, such as sentiment scores, macroeconomic indicators, or alternative data. You can experiment with different algorithms, such as A2C, DDPG, or TD3.

You can also move beyond the single-period train-test split. Walk-forward analysis trains on a rolling window and tests on the next window, repeating across the dataset. This gives a more realistic picture of how the agent would have performed if it were retrained periodically. It is more computationally expensive, but it is much closer to real deployment.

The FinRL ecosystem also includes notebooks for portfolio allocation, cryptocurrency trading, and high-frequency-style environments. Each variant introduces new state and action spaces, but the core pattern remains the same: data, environment, agent, evaluation.

FAQ

What is FinRL and who is it for?

FinRL is an open-source library that applies reinforcement learning to quantitative finance. It is designed for researchers, students, and retail traders who want to experiment with algorithmic trading strategies using deep reinforcement learning agents like PPO, SAC, and A2C.

Do I need a powerful GPU to train a FinRL trading bot?

Not for small experiments. Training on the Stock_NeurIPS2018 dataset with a few Dow-30 tickers runs in minutes on a modern laptop CPU. A GPU speeds up large portfolio experiments or transformer-based agents, but it is not required to get started.

How does PPO compare to SAC for trading?

PPO is an on-policy algorithm that tends to produce stable, interpretable policies with discrete or continuous actions. SAC is an off-policy algorithm optimized for continuous action spaces and can learn more sample-efficient policies. In trading, PPO often yields steadier returns, while SAC can capture richer position-sizing behavior.

Why should I compare my RL bot to buy-and-hold?

Buy-and-hold is a simple, transaction-cost-free baseline. If your RL agent cannot beat it on a risk-adjusted basis after accounting for fees, slippage, and taxes, the agent is adding complexity without adding value.

What are the main risks of using reinforcement learning for trading?

Key risks include overfitting to historical data, reward hacking, distribution shift when market regimes change, underestimation of transaction costs, and overestimation of out-of-sample performance. RL agents can fail silently when market conditions differ from training.

Can I deploy this bot to trade real money?

The tutorial code is for education and research only. Before deploying any real capital, you need rigorous out-of-sample testing, paper trading, risk controls, legal review, and a robust execution infrastructure. This guide does not constitute financial advice.

Where can I learn more about FinRL?

Start with the official FinRL GitHub repository and documentation. The project includes tutorials, example notebooks, and papers on portfolio allocation, stock trading, and crypto trading. The paper "FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance" is a good academic entry point.

Bottom Line

Training your first reinforcement-learning trading bot with FinRL is an excellent way to understand how sequential decision-making maps onto financial markets. You will learn how to structure an RL problem, how to train PPO and SAC agents, and how to evaluate results against a simple buy-and-hold benchmark.

The honest takeaway is that RL is a research tool, not a ready-made strategy. Most beginner agents will struggle to beat buy-and-hold after costs. That is normal. The real value is in the process: building intuition for reward design, overfitting, and regime risk. Start small, keep rigorous test sets, and never move to live capital until you have paper-traded your agent through at least one full market cycle.