When we create a trading system, we don’t know the parameter settings that will work best in the future. That is why we need to cover a wide range to see what we can really expect from the strategy. The parameter permutation test will help us with this.
1. What is the parameter permutation test
Before the what, the why.
The objective with this test is to generate a variable sample of combinations within the optimizations that give us a vision of the behavior of our strategy.
Each optimization creates a curve of results, adding them all we have a hypothetical scenario.
It is not a question of seeing which curve is most likely to be fulfilled in the future, but of what we can expect by applying our trading system.
To answer this, we will use the parameter permutation test, which is based on optimizing the parameters and then evaluating the results of all the optimizations.
It is like performing a Monte Carlo analysis but from the optimization of the parameters.
2. The goal
With this test we want to answer two questions:
- Will my strategy still work? When we optimize we can fall into the unique case solution trap. This means that we can find a combination with very good results that is not representative for the rest of the optimizations.
- Are we really exploiting an advantage or is my strategy lowered on some meaningless combination of parameters? If we create a large enough set of simulations we can have a realistic behavior of the system.
3. Optimization
What we look for in the optimizations we perform:
- May all optimizations yield positive results.
- That the average profit of all be positive.
- That the distribution of the returns is as uniform as possible (that it does not go from positive to negative in each optimization).
- The 3D graph should appear stable.
4. Configuration in the cross check
StrategyQuant gives the possibility within its platform to carry out this test in a simple way.
Within the Opt section. Profile / System Permutation parameter you can choose the type of parameters to optimize and the maximum number of optimizations.
In the filters, you can configure what are the conditions for this test.
5. Results of optimizations
In the first section (top left) you can see the percentages of profitable optimizations.
The second (bottom left) you find a histogram of the net gain of all optimizations. Each bar shows the net gain in that optimization and the red line the average net gain of all. Here it is very interesting to see the changes from positive to negative and vice versa.
On the right we see the third section, which is a panel with the 3D optimization graph.
6. Results in the parameter permutation test
On the left are the mean values of all the statistics calculated by applying the SPP method on the optimization results.
On the right you can see the graphs with the frequency, with the median values.
7. Limitations of the test
If we have a strategy with 2 or 3 parameters, this can be done. But if we have a strategy with 6 parameters or more, the combination is larger and testing all the combinations takes a long time.
For this reason, it is recommended that the strategy have few parameters, not only because of the test, but also because of the robustness of the strategy itself, as we often comment.
8. Conclusions and extras
The main idea of this was developed by Dave Walton is available at this paper.
The main idea of this test is that if we have created a strategy through data mining with software like StrategyQuant, it is possible that we have selected the variables that give the best results and not the most robust ones (in the future they will behave better).
Through the permutation of parameters of a system we can test all the possible combinations and evaluate their performance.
The best information from this test (SPP) are the average values of the statistics: net benefit, drawdown, % drawdown, Sharpe index, etc. Thus, we can see the average net benefit, the average drawdown, etc.
These are the values that we can have as more realistic for the performance of our strategy.
Any questions, I’ll read you in comments.
Remember that if you want to learn how to do all this in a practical way and become part of the algorithmic trading community, you can do it here.