Hyperparameter tuning sounds such a fancy term, but in reality it’s just adjusting a couple of variables to get the best result possible. Like finding the perfect temperature to cook crepes (I’ve been seasoning a new carbon steel crepe pan lately, with mixed resuls).
I’ve been exploring various values of the learning rate. An ML algorithm starts of by ‘guessing’ how important any given input feature is in determining the final result, and adjusting the importance depending on how wrong the predictions are. The size of the adjustment is the learning rate, and the best rate has to be determined on a case by case basis. So, try a whole bunch of values, within a range that ‘seems reasonable’, and find the best by trial and error. A lot of that in machine learning.
Another ‘hyperparameter’ commonly experimented with is the optimizer algorithm used to go from first guess to best result. I’ve tried Adam, SGD (Stochastic Gradient Descent) and RMSprop. Also AdamW, which is supposed to be an improved Adam but in my case gave worse results. I don’t intend to modify the actual network much until I get some more consistent results. So far they’re very variable. I think I need a wider range of inputs.
I haven’t found that using a gpu is faster than using the cpu, however I’ve seen charts that show that for large, complicated problems it is faster, but not necessarily for smaller, simpler problems. However one big disadvantage of using the cpu for my machine learning problem is that it uses all the processing power and I can’t do anything else on the computer while waiting for the training to complete. And when you’re doing trial and error, that can take a long time. Using the gpu for training leaves me with enough processing power on the cpu to do most of the other things I use my computer for. Definitely the way to go.