Category: Machine Learning

[[[1]]]

Dimensions are becoming the bane of my life. Take the title to this post. The value is the number 1, but what all those brackets mean is this. Imagine you have an MS Excel workbook (or whatever spreadsheet app you use). This workbook has the possibility of several worksheets. Each worksheet has a number of rows, and each row a number of columns. The above notation means the value 1 in the first column of the first row of the first worksheet in the workbook. I’m pretty sure this makes it a 3 dimensional value.

Some of the issues I’m having converting code to work with different data are related to this issue of dimensions. The result of

torch.tensor([1, 2, 3])

is not the same as

torch.tensor([[1, 2, 3]])

That extra set of brackets creates an extra dimension, even though the values are the same. Problem is functions expect input data to be in a certain format and spit the dummy throw Exceptions if the format isn’t what they expect. Only by carefully stepping through the code, while it’s running, with the debugger allows me to see what types and dimensions data has. I’m still pretty new to PyTorch (and tensors generally) so it might take a while to get up to speed on this.

Found It

I’m very particular in how I like information to be presented to me. Not too fast, not too slow, right level of difficulty – you get the idea. I’ve finally found a reference that presents an explanation of the implementation of a Deep Q Network to solve a basic Reinforcement Learning problem (Frozen Lake 4×4) that fits the bill. Here it is, on YouTube.

Roadblocks

Whatever resource I’m studying I run into roadblocks. Following the development of a topic, A -> B ->C and all good, but suddenly E appears with no obvious transition from what went before. Perhaps the author/tutor plans to cover C -> D and D -> E later, but I have difficulty when I lose track of an argument/development of an idea. Perhaps I’m just not very flexible.

My usual response is to go to another resource/reference. At the moment I’ve gone back to a Quantra course that I purchased quite a few months ago on Deep Reinforcement Learning in Trading. I think I abandoned that when I first looked at it because it uses TensorFlow for the NN part and I want to stick with PyTorch. These days I’m a little more comfortable with rewriting code that is designed for TF into the PyTorch version. It’s not really that hard.

Anyway I’m making progress with that course and don’t anticipate any further serious roadblocks. Pity I don’t plan to actually do any (short term) trading going forward. At the moment it’s just an intellectual exercise.

Exploration, Exploitation

Finished another chapter, discussing one of the most basic problems in human life. Decisions. Specifically, how to transition from collecting information required to make a decision, to actually implementing that decision.

In ML terms the initial phase is called exploration. In the case of our one armed bandits (see previous post) it involves trying them all out to see which one (if any) gives the best payout. Exploitation is then using that best bandit to make money. Problem is if you spend too much time on exploration you’re spending a lot of money on losing machines. If you spend too little time/money on exploration then the ‘best’ one may be just a fluke, and not profitable in the long term.

A couple of the policies regarding the transition discussed in the book look at a soft transition from exploration to exploitation. During the exploitation phase occasionally recheck the ‘losing’ machines to gather a bit more data to confirm the decision. And if the ‘winning’ machine starts to produce bad results in the long term recheck the others more often. Rather like a bad marriage I guess.

Bandits

I’ve worked through the chapter on OpenAI Gym, even got the scripts working, adjusting for the changes to the API since the book was published. This is a problem with all IT related stuff, changes are so frequent that just about any tutorial/book material is out of date to some extent. Makes learning harder. Anyway, the approach to those games wasn’t very sophisticated. Just demonstrating that if one selects from available actions at random one doesn’t get very far.

So now we’ve moved on to One Armed Bandits, a nickname for slot machines. Looking at strategies for exploring a set of such machines to determine if any has a better probability of reward than the others. This could be relevant to trading. Given a situation where funds could be allocated to a range of strategies, how to determine the most profitable course of action. Especially if you don’t have past history to work with.

Maintaining Interest

I’m still searching for something that holds my interest for more than a few days. I’ve spent a lot of time on quantitative trading over the past few years, getting nowhere. or rather, going backwards (losing money). More recently I’ve spent time on Machine Learning, hoping it could improve my trading in some way. Not sure I will even need to do trading anymore. If I have enough to live on after buying an apartment I won’t actually need to make any income.

So, what to do? I’m coming around to revisiting reinforcement learning, which is a very interesting area of ML, and could be related to trading should I ever go back to that. It’s a complex area of ML, but I think I’ve got enough of a handle on the basics to take another shot at it. And once I’ve got it set up, finding the right data will be the biggest challenge. Garbage In, Garbage Out, they say, so identifying useful inputs will be the challenge. Or I could input everything and let the algorithm sort out what’s useful. That’s the beauty of reinforcement learning. The code does all the work.

So, I’m re-reading Practical Deep Reinforcement Learning with Python by Ivan Gridin. He gives examples in both PyTorch and TensorFlow, so I should be able to follow along fairly easily, at least as far as the ML library is concerned. However I have had problems with Reinforcement Learning in the past because of changes to other libraries commonly used. A favourite is openai.gym which gives me some headaches. Setting up a system that actually works with the code is a real headache in this area of ML. Let’s see how it goes this time.

So (my favourite word for starting paragraphs), I could develop an RL (reinforcement learning) model that has various trading strategies as actions, PnL as reward, and an environment that includes all the data that might be relevant. Perhaps the simplest starting point would be to concentrate on opening/closing positions on a single instrument. After that I could move on to managing a portfolio.

Today’s Plan

Will focus on regression problems, perhaps using several datasets from the UCI repository, plus a few of my own, including synthetic data and some crypto data. Plan is to set up data with Dataset/Dataloader, to explore a broad range of parameters using Optuna, and then to fine-tune the more important parameters, again with Optuna. The aim is to have a practised pipeline so that I can run any data through the process without having to think too much.

Thinking about this a bit more, I can leverage the Dataset class by implementing different subclasses to perform different data preparation protocols. A simple way to have different ways to prepare the data, and keep them all separate and easy to work on, and then to use in the training by simply specifying which subclass to use. I think my background with Java is showing here.

Aaargh!!!

Why is everything so difficult? I’m sure I’ve asked that question a few times already. So, hyperparameter tuning. Requires lots of iterations to find the best values. Need for speed. So run it all on the GPU naturally. Except that the approach my current book uses, which is skorch to leverage sklearns grid search, won’t run on the GPU. I tried it on Google Colab with a GPU runtime, but it took 3 times longer than using the CPU on my local machine. I’m not sure what Google Colab actually does when it says it’s using a GPU, but I suspect it’s not doing Torch tensor operations there.

So I went looking for alternatives. Optuna is on open source hyperparameter tuning library that works with PyTorch (among other platforms) and doesn’t seem too complicated. I don’t want to have to spend a month learning a new sophisticated app. Most online tutorials I found were overly complicated. Nobody seems to have heard of the KISS principle. Anway it seems that about six months ago I actually bought a course on Udemy on Hyperparameter tuning, and it does actually have a section on Optuna, so I’m looking at that. Maybe I’ll get it to work with PyTorch NNs running on a GPU. Who knows? And is it actually worth all the trouble? And will it run on Google Colab, with a GPU?

Well, I got a very simple example working, and with the GPU. Funny thing was the GPU took 3 times longer than the CPU did!! I think I read somewhere that GPUs work faster on large datasets, and this one was pretty small. I used the BTC MLP with a single input variable. Model was pretty simple, only a single layer in addition to the input and output layers.

import torch
from torch.autograd import Variable
import numpy as np
import pandas as pd
from sklearn.metrics import mean_squared_error
import optuna
from optuna.trial import TrialState

DEVICE = torch.device("cpu")

df = pd.read_csv('data/btc.csv', usecols=['date', 'close'], index_col='date', parse_dates=True)
df_returns = df['close'].to_frame().pct_change()
df_returns.rename(columns={'close': 't'}, inplace=True)
df_returns.insert(0, 't-1', df_returns['t'].shift(1))
df_returns.dropna(inplace=True)

X = df_returns['t-1'].to_numpy(dtype=np.float32).reshape(-1, 1)
y = df_returns['t'].to_numpy(dtype=np.float32).reshape(-1, 1)

train_limit = 700

X_train, X_test = X[:train_limit], X[train_limit:]
y_train, y_test = y[:train_limit], y[train_limit:]


class MLP_LinReg(torch.nn.Module):
    def __init__(self):
        super(MLP_LinReg, self).__init__()
        self.fc1 = torch.nn.Linear(1, 5)
        self.act = torch.nn.ReLU()
        self.fc2 = torch.nn.Linear(5, 1)

    def forward(self, x):
        out = self.act(self.fc1(x))
        out = self.fc2(out)
        return out

# Optuna function defining model and ranges of it's hyperparameters
# Returns the value to be optimized
def objective(trial):
    model = MLP_LinReg().to(DEVICE)
    criterion = torch.nn.MSELoss()
    lr = trial.suggest_float("lr", 0.001, 0.01)
    optimizer = torch.optim.SGD(model.parameters(), lr=lr)

    # train the model
    for epoch in range(100):
        inputs = Variable(torch.from_numpy(X_train)).to(DEVICE)
        targets = Variable(torch.from_numpy(y_train)).to(DEVICE)
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, targets)
        loss.backward()
        optimizer.step()

    # run model on test inputs to get predictions
    with torch.no_grad():
        input = Variable(torch.from_numpy(X_test)).to(DEVICE)
        y_pred = model(input).data.cpu().numpy()

    # compare predictions with actual returns (y_test)
    loss = mean_squared_error(y_test, y_pred)
    return np.sqrt(loss)


if __name__ == '__main__':
    study = optuna.create_study(direction="minimize")
    study.optimize(objective, n_trials=10)

    pruned_trials = study.get_trials(deepcopy=False, states=[TrialState.PRUNED])
    complete_trials = study.get_trials(deepcopy=False, states=[TrialState.COMPLETE])

    print("Study statistics: ")
    print("  Number of finished trials: ", len(study.trials))
    print("  Number of pruned trials: ", len(pruned_trials))
    print("  Number of complete trials: ", len(complete_trials))

    print("Best trial:")
    trial = study.best_trial

    print("  Value: ", trial.value)

    print("  Params: ")
    for key, value in trial.params.items():
        print("    {}: {}".format(key, value))

Grid Search

After trying out various versions of my models I’m becoming quite enamoured with the idea of automating that process. Luckily the book I’m currently working through, Deep Learning with PyTorch by Adrian Tam, has me covered. There’s a chapter on setting up a Torch model to work with Scikit-Learn’s GridSearch, an automated hyperpameter tuning functionality. I’ll definitely be giving that a go shortly. I guess while I’m working on my financial issues I can have the grid search running in the background! Maybe it’s time to get that better computer, or else just do it in the cloud on a GPU from Google Colab.

Actually this heralds a change in focus, from becoming familiar with neural networks, picking up the jargon and a general idea of what’s going on, to actually using neural networks to solve problems. Not for the solutions themselves, but to become competent in the use of neural networks. Not sure if I’ll ever be using them to solve anything significant, but I’m enjoying the process.

Random Forest in Python

Random Forest Regressor.

Random Forest Classifier.

Here is an example of using a Random Forest Classifier with a dataset consisting of one feature (lagged MACD histogram value) and nominal target (positive or negative return) encoded as 1/0.

import numpy as np
import pandas as pd
import talib.abstract as ta
from datetime import datetime
from sklearn.ensemble import RandomForestClassifier

data = pd.read_csv('data/BTCUSDT.csv', index_col = 0, parse_dates=True)
data['return'] = data['close'].pct_change()
data['good'] = np.where(data['return'] > 0, 1, 0)

data_values = pd.DataFrame(data['good'].values)

macd = ta.MACD(data)
macd_values = pd.DataFrame(macd.macdhist.values)

df = pd.concat([macd_values.shift(1), data_values], axis=1)
df.columns = ['macd', 'target']
df.dropna(inplace=True)
df.reset_index(drop=True, inplace=True)

# df.to_csv('data/macd.csv', index=False)

X = df.values

# train, test, etc. are numpy arrays
test_start = datetime(2023,1,1,0,0,0)
test_size = data.index.get_loc(test_start)

train, test = X[0:test_size], X[test_size:]

train_X, train_y = train[:,0], train[:,1]
test_X, test_y = test[:,0], test[:,1]

train_X = train_X.reshape(-1, 1)
test_X = test_X.reshape(-1, 1)

clf=RandomForestClassifier()
clf.fit(train_X, train_y)

predictions = clf.predict(test_X)

print (clf.score(train_X, train_y))
print(clf.score(test_X, test_y))

Baseline Models

While studying the WEKA application I encountered the concept of a baseline model, a very simple model that provides a prediction, to be used as a starting point to evaluate the effectiveness of other models. ZeroR seems to be the preferred baseline algorithm for supervised learning – it simply takes the average of all target values in the training data and predicts that for all test instances. For classification it finds the most populous class and predicts that for all test data.

Jason Brownlee argues that for time series forecasting it is better to use an algorithm that takes the sequential nature of time series into account. He proposes a persistence algorithm, which simply states that whatever happened yesterday will happen today. A bit like the weather. If you predict that today’s weather will be the same as yesterday’s weather you’ll be right more often than not.