Performance Metric: Pessimistic Gain/MDD

As I have gained more experience in backtesting strategies, I have come across various performance metrics. Some of them are provided in the standard AB backtest report (e.g. Calmar ratio: CAR/MaxDD), some of them I have picked up and added to by Custom Backtest logic (e.g. David Varadi’s DVR).

I have noticed that I have developed a preference for the Calmar Ratio, especially when I am doing my first analysis of a new strategy. I like the measurement taking the MaxDD as an important factor, this fits my thinking well. One of the downsides of the standard implementation is that it calculates the CAR and MaxDD on the basis of one equity curve and one test only. This doesn’t allow for any statistical validation.

Some months ago I found a very interesting variant of the Calmar Ratio: Pessimistic Gain/MaxDD. The concept applies bootstrap-testing to the Calmar Ratio. In essence, the algorithm takes a list of trades and builds a distribution curve of the Gain and the MaxDD of x new equity curves that are created by drawing from the original list of trades (resampling).  When x>number of trades, the mean and std of the distributions of the Gain and the MaxDD may be used to build confidence intervals of the Gain and the MaxDD of the original list of trades (If you want to learn more about bootstrap-testing I higly recommend this post by Jez Liberty). The Pessimistic Gain/MaxDD is then calculated by subtracting 1xSD from the Gain and adding 1xSD to the MaxDD :  (Gain-SD_Gain)/ MaxDD-SD_MaxDD.

I take no credit for this application as I happened to come across it in the Amibroker group at Yahoo. Here I found an amibroker plugin dll that contains the implementation of the-concept (resample.dll). The plug-in – with its usage instructions – was posted by a user named dloyer123. I tried to email this user, but the user name is no longer in use at Yahoo. As the information is accessible for everybody in the group, I see no harm to post it here on the website as well. I have added a link at the download page.

At the end, some words of caution. When using the resample.dl I found that it relies on the AB internal quotes array to store trades. When the # trades exceeds the # quotes not all trades will be passed on to the resample.dll. The measure then is not based on the full list of original trades. A second finding is that the resample.dll uses a generic % of equity positionsize. It cannot work with other  equity allocation schemes.

QD

 

6 thoughts on “Performance Metric: Pessimistic Gain/MDD

  1. Hey QD thanks for the post! I’ve been playing around with the Pessimistic Gain/MDD, any idea what some thresholds for poor/great numbers would be? (Also I can’t seem to get your OutputStatistics.afl to work even after getting the resampling dll. Still new to programming, so if this is a noobie move on my part I apologize in advance.)

    • Hi Rano, there is no straight forward answer on your treshold question. As the % of equity allocated to trades plays a critical role, one should only compare strategies using same %.
      My suggestion would be to simply try it out and a get a good feel.
      -QD-

  2. I just found the Sortino ratio. It’s sort of a modification of the sharpe. Have you looked at that?

    Regards,

    John

  3. Just discovered this site while searching for something akin to what I have been doing for over a year.

    A good strategy for MEOM universe of ETFs using the pair switching ideas (for 2003-2012 it comes down to RS with the twist that if any of the top ranked etfs has returned less than TLt during the performance period, buy TLt instead of the etf.)
    is to rank the performance for the prior eight weeks and invest in the top ranked ETFs, with the caveat just described for fifteen weeks.

    All trades for this strategy for the period 2003-2011 are posted here. CAGR is a not too shabby 23%.

    https://docs.google.com/spreadsheet/ccc?key=0AjKtHcwKc58WdENsZF9UanRqajFRYjNOdjU1bWRqYUE

Leave a Reply to The Quanting Dutchman Cancel reply