Author: christinanorwood

Uni Layer Perceptron

RMSE = 0.02477

I set up a simple MLP in PyTorch, but I only used one node in one hidden layer, so I’m calling this a Uni Layer Perceptron. I figure this should give me the same result as the previous trial with the LinearRegression class from Scikit-Learn, as it’s only using one weight and one bias, equivalent to y = mx + c. Indeed, with sufficient iterations of the training loop, the results were almost identical.

I made the model a bit more complex, with a couple of layers (5 nodes each) and a ReLU activation layer, but the result was almost identical. I guess with a single input feature there’s not much scope for improvement. The code (simple version):

import torch
from torch.autograd import Variable
import numpy as np
import pandas as pd
from sklearn.metrics import mean_squared_error

df = pd.read_csv('data/btc.csv', usecols=['date', 'close'], index_col='date', parse_dates=True)
df_returns = df['close'].to_frame().pct_change()
df_returns.rename(columns={'close': 't'}, inplace=True)
df_returns.insert(0, 't-1', df_returns['t'].shift(1))
df_returns.dropna(inplace=True)

X = df_returns['t-1'].to_numpy(dtype=np.float32).reshape(-1, 1)
y = df_returns['t'].to_numpy(dtype=np.float32).reshape(-1, 1)

test_limit = 700

X_train, X_test = X[:test_limit], X[test_limit:]
y_train, y_test = y[:test_limit], y[test_limit:]

class MLP_LinReg(torch.nn.Module):
    def __init__(self):
        super(MLP_LinReg, self).__init__()
        self.fc1 = torch.nn.Linear(1, 1)

    def forward(self, x):
        out = self.fc1(x)
        return out

model = MLP_LinReg()

criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

for epoch in range(1, 10001):
    inputs = Variable(torch.from_numpy(X_train))
    targets = Variable(torch.from_numpy(y_train))
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = criterion(outputs, targets)
    loss.backward()
    optimizer.step()
    if epoch % 10 == 0:
        print('epoch {}, loss {}'.format(epoch, loss.item()))

with torch.no_grad():
    y_pred = model(Variable(torch.from_numpy(X_test))).data.numpy()

loss = mean_squared_error(y_test, y_pred)
print(loss)
print(np.sqrt(loss))

It’s easy to get caught up on simple details. Initially I was getting an error running the code because apparently one of the arrays was float and the other was double, and PyTorch didn’t like that. With so many ways to create Torch tensors from Numpy arrays it can be hard to find the right syntax to specify the data type. but eventually I found a way to do it that didn’t give me further errors. One reason I’m writing this blog is so that I can find the correct syntax fairly easily when I need it again in future. Not sure how well that will work though.

Simple Linear Regression

RMSE = 0.02478

Using the same data as the persistence model (i.e. daily returns with one lag as input variable) I trained a Linear Regression model (from scikit-learn) on the training set, made predictions on the test set, and calculated the mean squared error and then the root mean squared error (RMSE), result shown above. This is better than the persistence model, so making progress. Code is as follows:

import numpy as np
import pandas as pd
from sklearn import linear_model
from sklearn.metrics import mean_squared_error

df = pd.read_csv('data/btc.csv', usecols=['date', 'close'], index_col='date', parse_dates=True)

df_returns = df['close'].to_frame().pct_change()
df_returns.rename(columns={'close': 't'}, inplace=True)
df_returns.insert(0, 't-1', df_returns['t'].shift(1))

df_returns.dropna(inplace=True)

X = df_returns['t-1'].to_numpy().reshape(-1, 1) # matrix required
y = df_returns['t'].to_numpy()

test_limit = 700

X_train, X_test = X[:test_limit], X[test_limit:]
y_train, y_test = y[:test_limit], y[test_limit:]

model = linear_model.LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

loss = mean_squared_error(y_test, y_pred)
print(loss)
print(np.sqrt(loss))

Uh, Oh

That article on machinelearningmastery.com about LSTMs using PyTorch led me to a new book on the site, by Adrian Tam this time not Jason, called Deep Learning with PyTorch. I bought it. I like Jason’s style, and am hoping that his fellow authors adopt a similar style. I guess I’ll find out soon enough. Adrian has a couple of other books on the site, one specifically on LSTMs. Not sure if I’ll get that too, I guess it depends on what this current one gives me.

Baseline Loss – Persistence Model

RMSE = 0.03616

So, using daily returns as data (where 0.01 is equivalent to 1%), and using a persistence model as my baseline model (today’s predicted return is the same as yesterday’s return), we have the above value for the RMSE (root mean squared error) as calculated by scikit-learn. That is nearly 4% and actually seems pretty big to me given that daily returns are usually less than that. I guess there must be a few big values in there somewhere. Hopefully it won’t be too hard to improve on that result. The code is:

import pandas as pd
from sklearn.metrics import mean_squared_error

df = pd.read_csv('data/btc.csv', usecols=['date', 'close'], index_col='date', parse_dates=True)

df_returns = df['close'].to_frame().pct_change()
df_returns.rename(columns={'close': 't'}, inplace=True)
df_returns.insert(0, 't-1', df_returns['t'].shift(1))
df_returns.dropna(inplace=True)

X = df_returns['t-1'].to_numpy()
y = df_returns['t'].to_numpy()

test_limit = 700

X_train, X_test = X[:test_limit], X[test_limit:]
y_train, y_test = y[:test_limit], y[test_limit:]

loss = mean_squared_error(y_test, X_test)
print(loss)

Because the usual procedure for developing an ML model is to train the model on a portion of the data and then test it on the remaining data I have only used the test data here for calculating the MSE, even though the persistence model doesn’t require training. Just to have a result that is comparable to results produced by actual ML models.

Decisions, Decisions

I’m really bad at making decisions. I try to avoid having to. However with this ML project there are plenty of decisions to be made. I’m intending to compare several different models for predicting the price of BTC. However time series models usually use lags – values from previous days are moved to the same row as the value you are trying to predict. Problem is, for every day further back the lag is, you lose that row of data from the dataframe, because all previous days have NaNs in that column. For example the first row that can have a value from 7 days ago is day 8 (or row 8). So unless the decision is made up front as to how far back lags will go in any model, different data will be used for each model.

Perhaps in a learning exercise this isn’t very important. Perhaps the most important thing is to be aware that the problem exists. No doubt a ‘final’ model will be the result of tuning hyper parameters on the model that appears to be significantly better than all others, so such fine-grained issues in initial exploration is completely irrelevant. Probably. Still, I hate making decisions.

Step by Step

I keep going back to Jason Brownlee’s book (ebook) Deep Learning Time Series Forecasting that I mentioned finding in my collection some time back. He covers pretty much the same content as the Deep Learning for Time Series Cookbook, but I just prefer his style. He does use Tensorflow instead of Pytorch, but I’ve decided to handle that by simply trying to write his code using Pytorch. I don’t think the conversion should be too difficult. I’m just going to confuse myself (and my computer) if I try to use two ML frameworks. I’ve already run into problems with TF after installing the GPU version of PyTorch, something about something can’t be registered because something else already is.

Anyway, Jason has an eight step process before starting any ML project, so I’ll have a go at that here.

  1. Inputs vs Outputs
  2. Endogenous vs Exogenous
  3. Regression vs Classification
  4. Structured vs Unstructured
  5. Univariate vs Multivariate
  6. Single-step vs Multi-step
  7. Static vs Dynamic
  8. Contiguous vs Discontiguous

1. Inputs vs. Outputs

Output is some measure of price change for the following day, either price or something related to it such as return or log return or some other derivative of price, or rather, price change from the ‘current’ day. The inputs here will be some number of previous prices/returns/log returns from historical data. Some testing will probably be required to find an appropriate number of previous values.

2. Endogenous vs. Exogenous

Previous days prices are endogenous. An exogenous input could be something like the Fear and Greed Index, which is not directly related to BTC price data. At the outset I don”t intend to use exogenous data, although I may include that after preliminary investigations.

3. Regression vs. Classification

I’ve thought about making this a classification problem, such as whether the outcome is ‘good’ or ‘bad’ by some criterion, such as achieving a return of at least 2% over the next five days, but for starters I’ll just stick to a regression problem as it’s a bit less complicated and this is a learning exercise. Keep it Simple, Stupid (aka the KISS principle).

4. Unstructured vs. Structured

This refers to any obvious patterns, such as seasonality. With BTC there is a four year boom and bust cycle on top of a general uptrend, but these occur over a relatively long timeframe and you don’t really see it in, say, a year’s worth of data. Besides, if I work with returns instead of raw prices there is no such trend, although there may be seasonality (generally positive returns then generally negative returns over four years).

5. Univariate vs. Multivariate

For my initial experiment I’ll be using only the daily close prices (or returns calculated from those), so univariate. Further down the track I will probably include additional input variables. I wonder if using lags of the close price is considered a multivariate input?

6. Single-step vs. Multi-step

I’ll just be looking to predict the following day’s price, so single-step. I wonder if turning this into a classification problem that uses some aggregate of multiple following days to determine the class, such as I mentioned in point 3 above, would make this a multi-step problem.

7. Static vs. Dynamic

If I was actually using a model produced in this exercise in some useful way I would anticipate updating the data and the model on a regular basis, hence a dynamic problem. While I’m exploring, however, I’ll stick with the data I have so fair comparisons can be made between various models.

8. Contiguous vs. Discontiguous

Crypto data is very contiguous. Exchanges trade 24/7. More so than equities exchanges which trade only a few hours per day, and not on weekends or public holidays. No more annualizing statistics by multiplying by square root of 252, because that’s ‘the number of trading days in a year’. Root 365.25 is the new norm.

How helpful has this exercise been? For this initial project, not very, but I’m going with all the vanilla options. It will probably be more useful when things get more complicated.

ETA: I’ve just found an article on Jason’s website about using an LSTM with PyTorch for time series prediction. Mirabile Dictu!

The Best Laid Plans…

Why is everything so difficult? I wrote a short script to get 1000 days of data on BTC prices from Binance REST API.

from datetime import datetime, timezone
import requests        # for making http requests to binance
import json            # for parsing what binance sends back to us
import pandas as pd    # for representing/processing data

'''
LOOKS LIKE I CAN'T RUN THIS FROM GOOGLE COLAB!!

root_url = 'https://api.binance.com/api/v1/klines'

symbol = 'BTCUSDT'
interval = '1d'
limit = '1000'

url = root_url + '?symbol=' + symbol + '&interval=' + interval + '&limit=' + limit
data = json.loads(requests.get(url).text)

df = pd.DataFrame(data)
df.columns = ['date', 'open', 'high', 'low', 'close', 'v', 'close_time',
                'qav', 'num_trades', 'taker_base_vol', 'taker_quote_vol',
                'ignore']
df['date'] = [datetime.utcfromtimestamp(x/1000.0) for x in df.date]
df = df.set_index('date')

'''

But it won’t run. The error message suggests Binance blocked the request from Google Colab. No worries, I ran it on my local computer. But then I got warnings about using datetime.utcfromtimestamp, declaring this is deprecated and I need to use a timezone aware function. But that gives me dates with date, time and timezone offset, when all I need is the date. It just complicates my dataframe unnecessarily. Fortunately the ‘old’ code still works and gives me what I want, but for how much longer? I guess I’ll just have to use an old version of python when the next one forces me to take what it thinks is good for me. I couldn’t find any straight forward approach to removing the offset, and frankly I don’t even want the time for daily candles.

Anyway, I uploaded the data to Google Drive and I guess I can now start running some ML algorithms on it. No doubt that will bring me more hassles. Tomorrow, maybe.

Start Again

I’ve started ML projects before, but, well, I’m starting again. As this is a learning exercise (which actually applies to this whole ML ‘thing’ I’m doing) I’ll pick a context that has some topical interest for me with no expectation of actually discovering anything useful. So, it’s the predicting the price of BTC.

Which brings me to the first issue. The ideal time series for ML is stationary, i.e. it has a constant mean and variance. Time series for financial assets are far from stationary, and I have seen recommendations to use the return rather than the actual price, i.e. the fractional change from period to period. I think I’ll go with daily data, so the daily return is what I’m looking at. Why daily? Why not? I have to pick something and that’s it. Maybe as a variation I can use a different timeframe later and see if I get a better result.

So how much data? As this is an exercise it doesn’t really matter, although ML is said to work better with more data, although I acquired a book recently (but haven’t read it yet) that challenges this idea. Can’t imagine how that could be correct but I guess I’ll find out when I get around to reading it. Anyway, I can download 1000 data points from Binance in a single request from the REST API, so that’s easy.

What else do I need? To start with some baseline. Jason Brownlee suggests a persistence model is appropriate for time series data, so I might start with predicting that tomorrow’s return will be the same as today’s. And since I’m predicting a continuous value it will be a regression problem and I can use the Mean Square Error as an appropriate measure of loss. I’ll probably get around to actually doing this tomorrow. Just thinking about it for the moment.

NN Basics

I’ve just finished the chapter on the basics of neural networks, coding simple feed-forward, recurrent (both regular and LSTM) and convolutional networks. Each example used essentially the same synthetic data (in either 2D or 3D form depending on the requirements of the network) and the same loss function and optimizer, so it was easy to compare what was required to create each one. Pretty straight-forward really.

I think I’ve got my head around some ambiguities. In RNNs, for example, we have both a hidden layer and a hidden state, which appear to have nothing to do with each other, except that the size of the hidden state is partly dependent on the size of the hidden layer. Channels in CNNs are equivalent to input features in other kinds of network. I’m probably misunderstanding something, but it seems that becoming familiar with the jargon is a large part of learning neural networks. Of course actually applying them to real problems is a different story.

Familiarisation

So I’m starting to implement a neural network on data that isn’t just a book exercise. Dealing with categorical data, tried a couple of different approaches but ended up using one hot encoding. Getting consistent datatypes between pandas dataframes and torch tensors. Trying out different optimizers (SGD and Adam). Number of layers, number of nodes per layer. All just becoming more familiar with the requirements of this kind of model.

I was using the Abalone dataset referred to in a previous post, and was getting an MSE (mean squared error) on the age of approx 4. The mean age of those abalones was approx 10. Is this a good result? From my previous study of ML I’d have to run some kind of baseline model to answer that. I guess the simplest baseline model would be to assume that every abalone was the average age, and work out an MSE on that basis. The oldest abalone was 30, so I think that would give me a pretty big MSE!! Still, I’m not trying to produce the best model here, just to become more familiar with the process.

UCI ML Repository

The University of California at Irvine (UCI) has a very well known repository of ML datasets that can be used for learning and testing models. They even provide some very handy code to get the data for a given dataset into a python script (such as a Google Colab notebook). An example is:

%pip install ucimlrepo

This goes in one cell of the notebook, and then

from ucimlrepo import fetch_ucirepo 
  
# fetch dataset 
abalone = fetch_ucirepo(id=1) 
  
# data (as pandas dataframes) 
X = abalone.data.features 
y = abalone.data.targets 
  
# metadata 
print(abalone.metadata) 
  
# variable information 
print(abalone.variables)

This gets the data (in this case the Abalone dataset) and imports it as DataFrames. Very handy. I don’t even need to save any files anywhere, althought I guess I could write out dataframes if I wanted to.