One of the things with biomedical research of any genre is that so much of any benefit can only be seen with complex and powerful statistical tools. And more often than not these tools are applied to studies that have taken place in a laboratory on populations of mice that have been specially bred for the purpose of experimentation. All of these means that when I hear a news report that drug X performed well in trials for condition Y my brain always adds the rejoinder – in mice perhaps.
In many ways, the same is true for people looking at trading systems, granted the part of the mouse is played by the individual trader but the overall issue remains the same. That is one of often marginal improvement in returns. How else would you explain the desperate quest for the perfect indicator if it were not for tinkering around the edges looking for the one magical thing that works? Often the alteration is so marginal it is only in the eye of the beholder.
When something works it is obvious that it works – the change from baseline is extreme. Consider the following chart which admittedly is a medico-social issue but it is one of the most obvious curves I have even seen demonstrating an effect. The chart is taken from the paper – The Power of the Pill: Oral Contraceptives and Women’s Career and Marriage Decisions – Goldin, Claudia and Lawrence F. Katz. 2002. Journal of Political Economy 110(4): 730-770. The graph tracks the number of women entering various university courses post the availability of the pill.
There is no need to dick around looking for the P-value – the effect is obvious. Any change in your trading system should show an effect that is just as obvious. For example, if you decide to move from trading a daily time frame to a five minute time frame and your equity curve doesn’t show the same sort of improvement then you are wasting your time. You are in effect tinkering around the edges looking for an improvement that might not be there.
