Month: July 2024

The Bees Knees

I’ve been looking at Ivan’s implementation of the A2C (Advantage Actor Critic) approach to deep reinforcement learning for trading, which he said (in 2022, when the book was published) was the bees knees (my term, not his). So I applied it to my recently downloaded data for ETHUSDT. Results shown above. Not great, but it’s a start.

His construction of state is fairly basic, just the last 10 closing prices as far as I can tell. I’m sure I can do something about that. Also he’s using raw price data. Most people who talk about training models for trading recommend using returns (percent change) rather than actual prices as the latter don’t have a constant mean or variance. I’m not sure if that’s relevant to these RL models, but I have a feeling that it is. Also the trades are simple, just buy (or short) at the start of the day and sell or cover at the end. No holding until a signal to close. The actual code is going to take some study. I get the general idea of what the actor critic approach is trying to achieve, compared with the temporal difference approach which is what I have been looking at up ’til now. The devil is in the details however.

So, I’ve looked at the rather elaborate approach used by Quantra in their Deep Reinforcement Learning in Trading course, with state composed of ohlc data over several bars at three levels of granularity. Plus technical indicators, and calendar related inputs. Approach is temporal difference (I think that’s what it’s called). I’ve looked at a similar approach from DeepLizard which was created to solve a GridWorld environment, which I’ve rejigged to work with trading data. Not sure how successfully. And now this A2C approach from Ivan Gridin’s book.

I’m not at the point where I could write code to implement one of these without consulting references. Too many details that I haven’t totally internalized yet, especially concerning getting tensors into the right shape. It’s a language problem really, internalizing the grammar and vocabulary so that you can speak/write without thinking about it. I guess it’s just practice, practice, practice.

Whats Up Ivan?

I have a book by Ivan Gridin on Reinforcement Learning, and he has a very interesting looking chapter on using it for stock trading, with code versions using Tensorflow and PyTorch. It’s a later chapter and I’m going to have to study what comes before to follow the details, but looks promising.

However I recently came across this article on Medium written by the same Ivan, discussing the pitfalls of Reinforcement Learning in stock trading, and explicitly warning against the very approach he used in his book (Actor-Critic). The wisdom of experience perhaps. Anyway, as far as I understand what he’s saying one could deal with the issues with proper risk management. I’ll be interested to see what Yves Hilpisch has to say on the issue in his forthcoming book on reinforcement learning in trading. I guess since I don’t intend to do any serious trading, but am just looking at this as a ‘hobby’, I won’t be risking anything significant.

More Data, Baby

I believe it was Enrico Fermi who once said that you won’t solve a problem in 1000 years if you don’t think about it for 5 minutes. So I finally decided to write a loop to download data from Binance, given the limit of 1000 bars for any one download, and now have 48,000 one hour bars (OHLC data). I’ve gone for ETHUSDT this time, not sure why, maybe just so it’s easier to distinguish the files.

The training loop I have been using for the ADA data has cycled through the data quite a few times, due to there not being all that much of it (about 7000 bars) but I think that might have caused some of my problems. This time I’ll just go through it once. Perhaps I should add a couple more features, but getting data at the 1 hour granularity is not that easy. I could add in the BTC price (easy), or a genera crypto index (but getting that at 1 hour for six years might prove challenging). Perhaps I could use some other data at the daily level (such as S&P500) and resample to hourly. Must give it some thought.

I’ve hit a bit of a snag with the online course I’ve mentioned. It uses Tensorflow, which I can’t run on the GPU on my machine because of some conflicts that I don’t understand, but probably because I’ve set up PyTorch to run on the GPU. This means it’s pretty slow. Transferring from TF to PT is easy as far as creating the model is concerned, but the training is giving me some issues. PyTorch requires a custom training loop, and TF just calls a fit method. Not so hard with a standard supervised learning problem, but an RL problem is a bit different. I just don’t know TF well enough to work out the equivalents. I’ll have to give that aspect of the project a bit more thought. I’m probably making mountains out of molehills (or storms in a teacup, or whatever).

The main issue is understanding exactly what it is that I’m trying to optimize. I guess when updating the Q table one is working towards the point where the new value is close to the old value, and the difference between them (the loss) is minimal. Sometimes I think the fog is lifting, and sometimes not.

ETA: Facepalm! The Quantra course has about 10 years of data for the S&P500 at 5 minute granularity, as I’ve mentioned quite a few times. They even resample it to hourly as part of the state building process. Surely I can use that as in input feature for my crypto trading. What an idiot I am.

Validation?

I’m attempting to validate my models by running them on data that was not used to train them, but I’m getting very strange results. During training I saved some models that appeared to be giving decent results, but when loading them back again and running them ON THE ORIGINAL TRAINING DATA they give no results at all!! Not all of them though, only 10 out of 12. Something is happening here and you don’t know what it is…

Well, I could validate two of the models. They didn’t perform as well on the test data as on the training data, but the difference was not so great. Some overfitting, or perhaps just regime change (of the fiscal, not political, variety). I’ll have to consider how to proceed. More data would be great, but the only way to do that is by going for shorter time periods. I want to end up with something usable, and preferably not with a trading bot. Is it worth going for 4 hour data? Will increase the total periods by fifty percent, but I’m not sure that will make much difference. There’s obviously a good reason why the Quantra course on RL was producing worse results than a simple buy and hold strategy (on the S&P500) even with ten years of five minute data.

With the experience of working through an actual project I can go back to the books and have a better understanding of the issues being discussed. It’s a bit hard to do that ‘in a vacuum’ so to speak. There are still a lot of avenues to explore. I’ve read good things about LSTMs, and PPO, and stuff like that. I might even find out what those acronyms mean. Should keep me busy for a long time.

Feeding the Beast

I’m expanding the input features to my ADA neural network. Have added day of the week (one hot encoded), a measure of the range of each period (low/high), and an RSI indicator courtesy of the TA-lib. Plus I updated the data from Binance and now have over 8000 six-hour periods. The spreadsheet with the data looks quite impressive. Below is a screenshot showing the first 80 periods, or 20 days, one percent of the total.

Some of the column headings are not quite accurate. 30dayret is actually 30periodret, where the period is 6 hours, not 1 day. In the past I’ve mostly worked with daily data so it’s a habit to refer to everything as 30day, 60day, etc.

Running my training script on this data gave me an average return per trade of 0.5% (before transaction costs), and maybe 100 such trades per year. I guess if I cleared 0.3% per trade on 100 trades that would be about 30% per year, which is not too shabby. Still, rosy test results have cost me quite a bit in the past. Those trading gods are fickle, if not downright malicious.

I guess I’ll have to redo my hyperparameter tuning now that I’m using an altered data set, and some validation of course. And maybe explore different network topologies, more nodes, more layers, potentially different kinds of layer such as RNN or CNN.

So far I’m only looking at a long-only strategy. I could expand this to a long-short strategy, but that’s harder to actually trade now that Binance doesn’t allow margin trading (in Australia). Perhaps I should check out 1inch or similar. Binance was so convenient. Not going to get too excited. If all goes well I might put $100 into trading the strategy, just to maintain some interest.

An interesting possibility is that the model trained on ADA could be used on other coins. Seems to be a common practice, using pre-trained models for similar problems. I doesn’t take that long to train a model though. Currently about an hour for 1,000,000 episodes (each episode is one period of data)

Hyperparameter Tuning

Hyperparameter tuning sounds such a fancy term, but in reality it’s just adjusting a couple of variables to get the best result possible. Like finding the perfect temperature to cook crepes (I’ve been seasoning a new carbon steel crepe pan lately, with mixed resuls).

I’ve been exploring various values of the learning rate. An ML algorithm starts of by ‘guessing’ how important any given input feature is in determining the final result, and adjusting the importance depending on how wrong the predictions are. The size of the adjustment is the learning rate, and the best rate has to be determined on a case by case basis. So, try a whole bunch of values, within a range that ‘seems reasonable’, and find the best by trial and error. A lot of that in machine learning.

Another ‘hyperparameter’ commonly experimented with is the optimizer algorithm used to go from first guess to best result. I’ve tried Adam, SGD (Stochastic Gradient Descent) and RMSprop. Also AdamW, which is supposed to be an improved Adam but in my case gave worse results. I don’t intend to modify the actual network much until I get some more consistent results. So far they’re very variable. I think I need a wider range of inputs.

I haven’t found that using a gpu is faster than using the cpu, however I’ve seen charts that show that for large, complicated problems it is faster, but not necessarily for smaller, simpler problems. However one big disadvantage of using the cpu for my machine learning problem is that it uses all the processing power and I can’t do anything else on the computer while waiting for the training to complete. And when you’re doing trial and error, that can take a long time. Using the gpu for training leaves me with enough processing power on the cpu to do most of the other things I use my computer for. Definitely the way to go.

Too Good to be True

My early tests on my RL trading app gave promising results, which at the time I thought were a little ‘too good to be true’. Well, that feeling was justified, as I later discovered that I had written the code in a way that it repeatedly learned from a small subset of the data, thus essentially ‘rote learning’ (called overfitting in ML lingo) rather than learning more general principles that could generalize to unseen data.

So after rewriting the code, and fixing many other errors besides (for which the logging I’ve incorporated has proved somewhat helpful), and also downloading 6 hour data from Binance for the entire 6 years that they’ve had ADAUSDT on their exchange, I’ve been running the app again with a variety of optimizers but always getting a similar result, which is that almost no learning takes place!! My average return over several thousand trades is approx 0.03%. Not enough to even cover the fees (which I haven’t included in the algorithm).

There’s not much point trying to do further optimization, or explore a range of different network topologies, when the baseline is so close to zero. I think I’m going to have to address the ‘what data to use’ issue first up, until I do actually get some learning, and then try to improve on it. That Quantra course used quite a lot of input features, including several technical indicators and what day of the week it was. I’m going to have to enlarge my ‘state space’ a bit. It’s interesting to see what other people (authors of books/courses) are using for their features. Ideally I should be using some measure of market sentiment. Perhaps I need to learn how to scrape X for tweets (?) relating to crypto. Another day, perhaps.

New Tool for the Toolbox

These machine learning apps are a bit bigger and a bit more complex than the code I’ve been working on for the past few years. Troubleshooting is proving more challenging than it has in the past. So I’ve decided to get up to speed with logging in Python, which Jason Brownlee of machinelearningmastery claims is an indispensable tool for debugging. I certainly hope it makes that job easier.

Most instruction on logging is a bit light on, or else overly complex, which is par for the course as far as learning almost anything goes. I guess my goldilocks zone is pretty narrow. Anyway, there’s a Udemy course just on that subject, and an ebook from Amazon Kindle. Should get me up to speed. And then back to the debugging!

Slide Rules to GPUs

A bit of history. In my last year of secondary education, in 1967, I was using a slide rule and four figure math tables for calculations in maths and physics. In my first year of university I acquired a four function calculator (add, subtract, multiply, divide). I bought a PC (MicroBee) in about 1983 as the IBM PC at that time was too expensive for a hobbyist.

A few years later I had a Commodore 128, the first computer to do ‘useful’ stuff on such as word processing. I did keep keep learning new stuff (languages, applications) on the MicroBee however.

And now I’m training a DDQN on the GPU of my current workstation. In the meantime I did actually do an academic program in IT and spent a few years teaching basic courses, essentially first two years of tertiary. Also lots of books, lots of courses. Anyway, quite a change in nearly 60 years.