Ability to delete or sort strategies and paper trading
Nothing new, but ability to delete strategies would be greatly appreciated. At least, being able to sort on e.g. instrument or score will help navigate them all. Also would be nice to paper trade more than the current five.
Thanks
MG
Comments are currently closed for this discussion. You can start a new one.
Support Staff 2 Posted by Guillaume on 20 Jul, 2010 08:08 AM
Hi MG,
We have implemented the ability to delete solutions, it should be available soon.
We haven't looked at the possibility to sort solutions (by instruments, score...) yet, we'll keep this in mind.
Regarding the limitation about paper trading, the idea (beyond performance and storage limitation) is that people should be able to test many different strategies, but focus on the most interesting ones when it comes to paper trading, to validate that strategies are not overfitted.
We would like to avoid situations where people try to paper trade a lot of strategies in order to perform some sort of optimization at this stage.
So the number of 5 may change, but the limitation to a relatively small number of paper traded strategies will probably stay.
Thanks for the feedback
Regards,
Guillaume
Guillaume closed this discussion on 20 Jul, 2010 08:08 AM.
skaak re-opened this discussion on 24 Jul, 2010 08:12 AM
3 Posted by skaak on 24 Jul, 2010 08:12 AM
You say paper trading is for more *interesting* strategies. I
understand and agree. Say I have 10 strategies that are interesting or
good, and now I want to test them using paper trading. I can only test
5. Herewith a suggestion. Why not allow paper trading on all
interesting or good strategies? Say limit paper trading to strategies
with a score of e.g. 400 or more. If resources is a problem, then a
more generous hard limit e.g. score more than 400 and in total not
more than 30 strategies.
Regards
MG
Support Staff 4 Posted by Eric on 26 Jul, 2010 03:09 PM
Well, the thing is, we want to encourage users to select strategies in which they believe, possibly as candidates for investing actually money. It is easy to have a high score by over-fitting the strategy. We'd rather see strategies that do well a (reasonably) long time after their author has estimated them to be ready for production.
For the moment, we are still discovering what users (and ourselves) want to do with the paper trading strategies.
Increasing the limit to a value higher than 5 is a possibility. We could also create a two-tiered paper trading feature: one level is for the users to do whatever they want, a second level would be to let us know about strategies that have been proven under paper trading.
We are curious to hear your thoughts on this.
Thanks,
Eric
5 Posted by skaak on 26 Jul, 2010 03:52 PM
As you know, I am all for testing strategies in hold out/real life
rather than jump to conclusions based on in sample performance, but.....
The current paper trading limit is too restrictive - using score is my
suggestion to lift it. It can be abused as you say, as nearly
everything else can, but that is not the intention. If you pick a high
enough score from the available distribution of scores, you can
prevent such abuse to a very large degree.
I am working on 5 minute strategies at present and have locally some
that do pretty well in my own hold out. Online I am already paper
trading 5 good strategies, so I can not test any more. This makes it
pointless to submit new ones. I have to finish testing one of the
current 5 and only then can I submit and test a new one. It is going
to take a loooong time to test strategies this way, and by the time
you pick the best strategies, a few potential ones will either be
switched off or not yet online.
Also, after around three months out of sample they start to make
losses and need to be updated. Since you will trade strategies for
this period, the implied idea is to get strategies that will do well
out of sample for the next three months and this is not really a
problem. Note, however, that if you test such a strategy for three
months and then decide to use it, you need to first update the
parameters.
A strategy that does well and then gets used without being updated
will create problems. Of course I want to get a strategy that works
well forever after, but that holy grail does not exist in my
experience and so you need to regularly update parameters. Just
another pitfall to watch out for when testing and eventually using
strategies. I know there is a big difference between refitting and
updating and in this particular case the latter applies. Far worse
abuse in my mind, since it implies losing real money, is to not update
strategies.
On a different note, I am spending so much time on money management,
which have to be developed from scratch it seems. If the API had
something like getUnrealisedProfitOnOpenPosition and
getRealisedProfitOnRecentlyClosedPosition or something like that, it
would be wonderful. The getEntryAmount and getUnitaryEntryPrice
functions are useless for this - or can you shed some light as you did
with the DateTime stuff which works wonderful BTW.
Regards
MG
PS : Curiosity killed the cat....
6 Posted by hzang12345 on 27 Jul, 2010 01:19 AM
I guess I am unsure of what the difference would be between submitting them and "paper trading" them, MG?
I don't think it was stated anywhere that "paper traded" strats were the only strats that were to be considered? Since there is currently a limit of five this was something implemented as just another feature / enhancement to the whole. According to the T&C it does not specify to which location, but only 3 months after submission to the actual website are you free to disseminate formulas in another venue if not selected by "Algodeal".
I am confident "Algodeal" can easily track the results of all submitted strategies to the website whether they are located in the "paper strategy" section or not? I was more or less under the assumption the paper strategy section is for the "users" amusement and not the definitive location for any strategy to be under consideration.
Please correct me if I am wrong.
7 Posted by skaak on 27 Jul, 2010 04:49 AM
I'm under the impression that paper trading is another hurdle a
strategy needs to pass to be selected, but let us hear what Algodeal
has to say....
Even if Algodeal can paper trade behind the scenes, I'd like to see my
strategies' results. Since this is useful to test and produce good
ones the limit should be more generous.
The alternative is to submit a strategy on a daily basis. Since we can
not delete or sort strategies yet, this is not a practical way of
paper trading them. Proper paper trading makes it easy for Algodeal
(and us) to know when the strategy went live and how it performed.
The other point is that a strategy needs tweaking on an ongoing basis.
I'd like to at least point it out but also for this to be accommodated
somehow. The T&C mentions this, but it seems to be more applicable
when something goes wrong than as an integral part of the whole
exercise.
If in real life quants built strategies and then didn't touch them for
a while, e.g. three months, they'd play a lot more chess. Ongoing
tweaking is a big part of the real thing.
Regards
MG
Support Staff 8 Posted by Alexandre on 27 Jul, 2010 10:41 AM
Hi MG, Hi HZ,
Paper is one step forward to help us filter trading strategies. As Eric pointed out earlier, at some point strategies in paper reflect your own choice of « valid » strategies. In the future we will probably shift scoring and ranking etc (found in the dashboard page) to be based on paper results rather than backtest ones.
The idea of not letting too many out of sample (paper) strategies running is to reduce severely the opportunity of data mining bias. I agree that 5 paper strategies might not be enough to paper test strategies, especially if it involves different instruments and/or trading styles. We might work out a higher limit in the future.
As MG pointed out, nothing prevents you from getting the same results by re-submitting backtests regularly. In the next release you will be able to delete submissions, it might prove helpful to do this. We prefer to release early and adjust regularly according to our users' input and our own needs. Paper will surely evolve as we get more experience on it. If you have some ideas for improving the backtesting workflow, we'd be delighted to hear from you.
About evolving strategies over time (e.g. tweaking parameters), I think one answer would be to allow code updates of a paper strategy, while still building on the performance of the past results. We might implement this, even though this is currently difficult as paper is a full backtest run, and results are not incrementally updated.
Regarding money management, we received some complaints and we will probably work something out. I think a new sample from Mathieu might help you on this, sample 3 in the advanced section. Currently we are putting more resources to work on data both in scope and quality.
I hope this clears some of your interrogations.
Alexandre
Alexandre closed this discussion on 27 Jul, 2010 10:41 AM.
skaak re-opened this discussion on 27 Jul, 2010 11:39 AM
9 Posted by skaak on 27 Jul, 2010 11:39 AM
Thanks for the feedback,
Some inputs on the backtest workflow as requested - see attached graph
of a five minute model.
Step 1 : this strategy was built using data up to 2009/12/31, it
performs well in sample and proceeds to the next step
Step 2 : the strategy is tested using the hold out sample, starting
2010/01/01, it does well and proceeds to the next step
Step 3 : the strategy uses parameters such as stop loss, this is
calibrated using the hold out performance only - in sample performance
is not even looked at now
Step 4 : once these are calibrated the model (not parameters - that
would be overfitting) is updated using the full sample
Step 5 : this model is now fully up to date and submitted for paper
trading (although in this case, since I am already running 5 on paper
trading, it does not happen)
Step 6 : note that the model does well but starts making losses in Apr
2010 - it needs to be updated at least once in three months it seems
Say we did step 1-5 and the model was accepted for real money. Say it
is now 1 Apr 2010. Everything looks very good, but we did not update
the parameters and start trading. Then we are punished as the model
makes some losses from here onwards. So somehow you need to have an in
sample period, a hold out period, a paper trading period and also
regularly update the model.
Hope this helps
MG
10 Posted by skaak on 27 Jul, 2010 12:42 PM
PS : That example would make a lot of sense if TransactionDetails was
properly documented.
Regards
MG
11 Posted by hzang12345 on 27 Jul, 2010 05:33 PM
I guess since everyone is putting in their 10 cents I will throw mine in the pot as well.
IMO as far as sorting strategies etc..
It would most likely be the easiest way to sort first by
1) Contract
--Then--
2) Score (Highest ->Lowest)
--Then as a column on the side--
3) Days until Non Compete Agreement has ended.
For Example
1) 600 89 Days Remaining
2) 400 78 Days Remaining
.. ....... .. ........
_______________
A simple dropdown box with Arrow could have the name of all the contracts, then once clicked would be automatically sorted by score with the days remaining until the non compete agreement attached to each somehow.
These personally are the three pertinent things relating to the whole purpose of the website or at least from the "users" point of view.
As far as if the strats should be "touched up" etc. as MG has suggested, well that is more of a subjective point of view and is open certainly open to discussion. In my mind if volatility somehow is accounted for in the formula and it is indeed a legitimate strategy then it should always outperform the contract it is based on. I am speaking only as to out-performance on the specific contract if that makes sense. An example would be for the ES. If the ES was up 15% on the year our program would be up >15% and if the ES was down 15% on the year our program would be > -15% speaking purely in terms of yield.
Again, that last paragraph is entirely up to debate and is purely a subjective point of view. In my mind if you can continuously beat the historical 7.6% yield on the SP500 you indeed have something valid. But this is overstepping my role as a "user" of this website and is entirely based on my own opinion and views of the investing world. I certainly do not think my opinion is any more valid than anyone else's.
12 Posted by skaak on 28 Jul, 2010 04:17 AM
For step 3, a nice article over here - maybe you can add such graphs to MarketRunner or the score machine.
Regards
MG
Support Staff 13 Posted by Alexandre on 28 Jul, 2010 10:21 AM
Thanks for the pointers.
Regards
Alexandre
Alexandre closed this discussion on 28 Jul, 2010 10:21 AM.