I’m attempting to validate my models by running them on data that was not used to train them, but I’m getting very strange results. During training I saved some models that appeared to be giving decent results, but when loading them back again and running them ON THE ORIGINAL TRAINING DATA they give no results at all!! Not all of them though, only 10 out of 12. Something is happening here and you don’t know what it is…
Well, I could validate two of the models. They didn’t perform as well on the test data as on the training data, but the difference was not so great. Some overfitting, or perhaps just regime change (of the fiscal, not political, variety). I’ll have to consider how to proceed. More data would be great, but the only way to do that is by going for shorter time periods. I want to end up with something usable, and preferably not with a trading bot. Is it worth going for 4 hour data? Will increase the total periods by fifty percent, but I’m not sure that will make much difference. There’s obviously a good reason why the Quantra course on RL was producing worse results than a simple buy and hold strategy (on the S&P500) even with ten years of five minute data.
With the experience of working through an actual project I can go back to the books and have a better understanding of the issues being discussed. It’s a bit hard to do that ‘in a vacuum’ so to speak. There are still a lot of avenues to explore. I’ve read good things about LSTMs, and PPO, and stuff like that. I might even find out what those acronyms mean. Should keep me busy for a long time.