Month: May 2024

Bike Experiment

UCI has a dataset on bike sharing. This is for a regression problem, to predict the number of bikes shared on any given day. I cleaned it up a bit (removed index, date) and one-hot encoded features such as day of week and others. Then separated out the count as the label (target).

I created a simple MLP consisting of a couple of layers, 35 inputs down to 1 output. I initially tried just a single layer for the simplest possible model, but got unexplained errors (all the predictions were null (nan), so I made it a little more complex (35->2, ReLU, 2->1) and that worked.

I set up a grid search in Optuna (which is not documented as well as it might be) with a couple of optimizers and a range of learning rates. SGD gave the best and worst results, usually with the same learning rate! Adam and RMSprop were much more consistent, although not quite as good as SGD’s best, but hard to trust predictions when the optimizer has such huge variability.

A couple of the features had much higher values than the rest. Bike use counts vary from about 1000 to 5000 per day, and two features tracked how many were casual users and how many registered users, whereas everything else was in the range approx 0 – 1. So I produced a feature casual/registered and used that instead, but it made almost no difference. Maybe with a more complex model it might make a bigger difference, more testing needed.

Best result was 5330 which actually was produced by Adam, not SGD, and with the modified dataset, but it was something of an outlier. Still, that’s a pretty high value when very few days had a count that high.

ETA: I realized that the above result was achieved with only 10 epochs in the training loop. Increased to 1000 and the best result was below 2000 (average of 10 trials). 1000 epochs gave a significantly better result than 500 even, so that’s something to consider in future.

I decided to switch to median rather than mean to get the following results (third column is the RMSE – Root Mean Squared Error), where each row represents 100 trials. RMSprop is consistently better than Adam, and the higher learning rates give better results. It appears that a learning rate of 0.1 is the largest value the algorithms will accept.

params_lr params_optimizer        
0.01      Adam              4397.0
          RMSprop           4695.0
0.02      Adam              2598.0
          RMSprop           2477.0
0.03      Adam              2226.0
          RMSprop           1951.0
0.04      Adam              1964.0
          RMSprop           1475.0
0.05      Adam              1739.0
          RMSprop           1226.0
0.06      Adam              1557.0
          RMSprop           1159.0
0.07      Adam              1427.0
          RMSprop           1135.0
0.08      Adam              1338.0
          RMSprop           1127.0
0.09      Adam              1281.0
          RMSprop           1125.0
0.10      Adam              1245.0
          RMSprop           1126.0

As a comparison I ran a Linear Regression from scikit-learn on the same data and got approx 1100 as a result, same as best results above.

Why Oh Why?

TL:DR Whinge, whinge whinge.

I set up an ML experiment with Optuna using some bike sharing data from UCI. Despite using almost the same code as in previous, successful, experiments, I was getting errors everywhere. Well, I’ve made so many mistakes coding over the years that I’m pretty experienced at finding where the problem is, but in this case something that worked yesterday (on different data) didn’t work today.

I was using a simple model, only one layer with inputs equal to the number of features, and one output (being a regression problem). I tried this on my BTC data a while back and it worked fine. Now, however, the array of predictions was full of null values. If I made the model a little more complex, an extra layer or an activation layer, it worked. But I didn’t need those with the BTC example. So what gives? Usually when I find a problem I understand why it occurred, but not this time. Grr!!

Today’s Plan

Will focus on regression problems, perhaps using several datasets from the UCI repository, plus a few of my own, including synthetic data and some crypto data. Plan is to set up data with Dataset/Dataloader, to explore a broad range of parameters using Optuna, and then to fine-tune the more important parameters, again with Optuna. The aim is to have a practised pipeline so that I can run any data through the process without having to think too much.

Thinking about this a bit more, I can leverage the Dataset class by implementing different subclasses to perform different data preparation protocols. A simple way to have different ways to prepare the data, and keep them all separate and easy to work on, and then to use in the training by simply specifying which subclass to use. I think my background with Java is showing here.

Human Learning

Having been a teacher most of my life I’m interested in the learning process. There’s a concept(?) called ‘the magic number 7, plus or minus 2’ which refers to how much new information a person can process at one time. Too many tutorial producers don’t understand this, and will try to teach 6 different ways to do one thing without realizing that they make it impossible to learn anything else at that time. The short term memory buffer is full. Better to learn one way to do six different things. But people do love their options.

Another issue is chunking. What is ‘one thing’? For a child learning to read, a letter is one thing. Then a word becomes one thing. Then a phrase. By the time the child has grown a book is ‘one thing’. Too many experts can no longer break down a topic into simple components because they no longer even see the components. Thinking at too high a level.

When I was teaching mathematics at high school, and a student was having a problem, the question I always asked was ‘what do they need to know already to understand this new idea?’ Quite often the prerequisite had not been consolidated, hence the difficulty. Learning is a building process, and every prior stage needs to be solid.

A while back I read this article, and found it very challenging. But after doing some study of the basics (wherever I could find someone who just dealt with the basics) I can now read through it and work out what it’s doing at each step. Can even incorporate some of the steps into my own problems. There are some interesting ways of doing things here. I guess I’ve made the foundation strong enough at last.