Development Set Testing Set Combined Set
Net profit Trades Win%
Max. consec. winners Max. consec. losers Ave. bars, winners Ave. bars, losers Profit factor Annual
Net profit Trades W i n %
Max. consec. winners Max. losers Ave. bars, winners Ave. bars, Profit factor Annual ROA Net Profit Trades W i n % Max. winners Max. consec. losers Ave. bars, winners Ave. bars, losers Profit factor 5 4 4 4 4.84 4 8 8 1 1 5 3.87 157.25% D-Mark $50.400.00 6 2 4 5 2.86 4 8 6 4 1 5 2.35 63.91% 5 3 5 1 3.45 7 4 6 7 1 6 3.58 1 8 2 0 5 0 5 0 7.00 6.08 4 4 3 3 7 9 7 3 1 0 11 7.00 6.08 3 7 9 % 3 2 9 . 6 2 % 1 8 1 9 3 9 4 2 2.22 2.09 2 2 3 3 7 2 6 9 2 1 2 1 1.41 1.52 21.75% 23.59% 2 2 5 0 2.43 6 4 5 3 1 6 $31.700.00 2 5 4 8 2.63 6 4 5 8 1 5 2.43 8 6 . 2 7 % 2.43 Annual ROA 96.33% 86.25%
234 Trading System Development and Testing
contains a formula that subtracts the current net profit from the net profit 255 trading days ago. When we plot the date versus this column, we have created a one-year moving window of equity.
Let’s look at examples of some of these charts for our currency trading. Figure 16.2 shows our system’s profit distribution chart for the devel- opment period for the Yen.
As in the classic channel breakout system, Figure 16.2 has more large winning trades than a standard distribution. Another interesting feature of this chart is the very large number of small winning trades between $0.00 and These small trades are caused by the time-based exit method we added to our system. Without this exit, many of these small trades would have become losing trades.
Figure 16.3 shows a profit distribution chart for the combined testing and out-of-sample periods for the Yen. Notice that both Figure 16.2 and Figure 16.3 have large positive tails and the same high number of small winning trades. It is very important that this two charts are similar. Even if the system has similar profitability during both periods, it would be risky to trade if these charts were not similar, because the distribution of
Testing, Evaluating, and Trading a Mechanical Trading System 235
FIGURE 16.3 The distribution of trade profits for the adaptive channel breakout system, using the combined testing and out-of-sample sets in Chapter 15.
FIGURE 16.2 The distribution of trade profits the adaptive channel breakout system, using the development set Chapter 15.
trades would have changed. Most of my research has shown that the dis- tribution of trades changes prior to a system’s failure. This change will often occur in the distribution of trades of a profitable system before a system actually starts losing money.
Figure 16.4 shows a one-year moving window of equity for the devel- opment set for the Yen. The one-year moving window of equity is almost always above zero and, in general, has an upward slope.
Let’s now briefly analyze this system. First, we learned that it has a very stable performance when it is viewed as trading a basket of curren- cies. For example, the system has performed well on the D-Mark over the past few years but has done better than expected on the Yen. When we view the results as a basket, they have remained similar over the de- velopment, testing, and combined sets. (We showed this for the develop- ment and testing sets in Chapter 15.) We also see that the distribution of trades has remained similar for the Yen during both the development and the combined sets, as shown in Figures 16.2 and 16.3. The results for the other two currencies are also similar enough to give us confidence in this
236 Trading System Development and Testing
FIGURE 16.4 The one-year moving average of equity for the channel breakout system on the Yen, using the development set.
system. Many factors, such as the average length of winning and losing trades, have also been relatively constant on the basis of a basket of cur- rencies. The one-year moving window of equity for the Yen (Figure 16.4) is above zero most of the time and has a general upward bias on the de- velopment set.
On the basis of our analysis, we can conclude that this system has a good probability of continuing to work well for this basket of three cur- rencies for some time to come. Now that we think we have a reliable sys- tem, we need to discuss actually trading it.
System Trading
To trade our system on all three currencies, we would need a minimum of This amount would give good returns and limit the maxi- mum to about 33 percent, with returns on the account of about 31 percent per year. Because winning trades last about four months and losing trades last about three weeks, it will take some discipline to trade this model, The problem is that if we don’t follow system exactly, we could lose money even if the system continues to Most mechanical
Testing, Evaluating, and Trading a Mechanical Trading System 237
trading systems make money based on the results of a few large winning trades. This makes it very important to follow each and every trade that the system generates. If we know we are not disciplined enough to trade the model, we can still trade it by giving it to a broker who is equipped to trade the system. The broker can be given a limited power of attorney and would be legally obligated to follow all of the system’s signals. We will use the results of the system’s live performance to adjust our slippage estimates and collect live trading performance data.
Live System Performance Data Collection
The live data collection process is the same as the historical data collec- tion process, except that it is based on results recorded since the system has gone on line.
During the process of trading the system, we collect data on the sys- tem just as we did during the historical testing period. We use these data to develop the same characteristic benchmarks we had during the devel- opment periods. We compare these to make sure that the live trading pe- riod is still similar to previous results. This process is the same as comparing the development set to the testing set or the out-of-sample set. If they are similar within the standard error of the samples, then the sys- tem is still tradable. Any change in the profile of the system’s perfor- mance must be explained-even an increased percentage of winning trades.
Live System Evaluation
Let’s look at some danger signs for a system. One bad sign would be a 150 percent increase in maximum since the development pe- riod. Another would be a 150 percent increase in the number of maxi- mum consecutive losing trades.
If the system performance degrades, you should stop trading the sys- tem. If the system is performing well, you should continue collecting the live (using the data collection process) and analyzing the data at regular intervals. In this way, if a problem occurs, you can stop trad- ing the system while minimizing losses. If the system is performing well, your analysis will give you confidence to trade the system and maximize your profits.