I’ll say something controversial that you won’t read elsewhere…

It’s easy to outperform the market in a backtest.

Pictures of 3 algorithmic trading strategies that outperformed the market

A backtest is a simulation of how a set of rules (aka, your trading strategies) performed when applied to historical data.

They are extremely useful because they give us insight as to how a strategy performed in the past.

However, the cardinal rule of algorithmic trading and trading strategy optimization cannot be forgotten:

This article will discuss how to create a profitable algorithmic trading strategy that outperforms the market. It’ll talk about overfitting, a process that all intelligent ML algorithms suffer from, including our most powerful neural networks. I’ll then discuss ways to mitigate overfitting, and create a set of rules that are profitable throughout time.

So if you’re interested in deploying a strategy that outperforms the market, then stay tuned!

Creating an algorithmic trading strategy

Creating an algorithmic trading strategy that outperforms the market in backtesting is actually fairly trivial. The key words here being during backtesting — creating a strategy that translate into real-time trading is extremely difficult, for reasons I will soon outline.

To create a profitable strategy during backtests, there are a number of ways you can go about doing this, with one possible option being trading strategy optimization. This is a process in which an algorithm consistently improves the parameters of the strategy against a historical timeframe.

Another, possibly simpler, option is to copy from an existing strategy that is known to be profitable. This is like copying the homework from the class valedictorian. You can take a pre-configured strategy, copy it, and make improvements to it as you see fit. There are a few examples of profitable algorithmic trading strategies in the NexusTrade library.

However, both of these approaches suffer from overfitting. Overfitting is when an algorithm fits well to one set of data, but fails to generalize out of sample. In the context of trading, this means the algorithm does really well on backtests, but the performance fails to translate into live-trading.

As you can imagine, overfitting is a huge aspect of why algorithmic trading is so difficult. Unlike other types of data (like language modeling), stock prices are non-stationary and stochastic. This means that the data is somewhat random and unpredictable, and moreover that the distribution of data changes over time. An example of this can be the dotcom bubble and its rapid crashing afterwards. Tech stocks had widely different performance leading up to the pop, and then were in a horrible situation for years to come afterwards.

When thinking about mitigating overfitting, you have to think about what data scientists in academia and the industry do to allow their models to generalize to unseen data. How do we take lessons from them to mitigate this risk?

There are many approaches to doing this, but this article will discuss 3. These options are

• Out of sample testing using a validation set
• Real-time paper-trading
• Maintaining a list of uncorrelated strategies

Out of Sample (Validation Set) Testing

Out of sample testing is when you see the performance of your strategy on a set of data that comes after your training set. Importantly, the validation set does not overlay with the training set. It’s just a set of data that you can use to see if a strategy has actual potential.

In contrast, the training set is the data used to improve the parameters of a strategy. Anybody can build a strategy that does AMAZING in the training set. It takes a curious mind, with a set of powerful tools, to create a strategy that outperforms the market in the validation set.

Thankfully, the trading strategy optimization engine within NexusTrade allows traders to separate their data into the training and validation set. So, a trader can evaluate their optimized strategies systematically.

Real-Time Paper Trading

Another approach to this is seeing how well a strategy performs when deployed live to the market. This is one of the best approaches; it eliminates any and all sources of bias and forces you to confront with the objective reality of your strategies.

Thankfully NexusTrade allows all users (even free users) to deploy their first algorithmic trading strategy to the cloud with the click of a button. Once deployed, the strategy runs for real-time paper-trading.

Now, it’s important for me to make this distinction: this is not real-trading! It allows you to trade with monopoly money, and learn the dynamics of the market in a risk-free way.

However, importantly, even real-time paper-trading is not foolproof. For example, if the paper-trading platform has unrealistic fills or doesn’t appropriately account for fees and slippage, then the performance will not translate into real-time trading.

Additionally, we have to be assured that our simulation of the market is accurate. Market impact, risk management, and even emotionality play a huge role in how a portfolio performs in real-time. Some people, particularly data scientists, might choose to do hypothesis testing to determine if a strategy indeed has an edge over buying and holding SPY and VOO (an important baseline for algorithmic trading).

Moreover, we still have the problem of non-stationary data. Which means, even if the strategy performs well in the training set and it performs well in the validation set and paper-trading, the dynamics of the market might change, and the strategy might start performing poorly because of this. Maybe Congress passes a law against a certain industry, or they raise taxes in a way that affects some companies more than others? Anything can happen that might affect the market, so a trader needs a way to spread their eggs into multiple baskets.

Maintaining a list of uncorrelated strategies

Combined with the above two approaches, this is the most important step in creating a set of strategies that outperform the market. Don’t put all of your eggs in one basket.

Recognize that markets can and do change. Create a strategy, evaluate it, and slowly scale it up. When its performing well, invest more in that strategy. If it starts to do poorly, dial back on it.

And don’t just create one. Create dozens of strategies each with their own rules. Some strategies might focus on niche industries like biotechnologies. Other strategies might look at the free cash flow of cryptocurrency stocks. And even other strategies might just buy and hold VOO and sell call options on them. Each unique set of strategies should have a purpose for what its trying to accomplish.

NexusTrade allows users to create dozens of unique, independent portfolios, each with their set of trading strategies. This allows users to try out multiple ideas and see which ones are more likely to be profitable in the wild.