ML seems to be a very ‘trial and error’ discipline. My first serious study was a course provided by the University of Waikato in New Zealand, the developers of the WEKA ML platform. I was surprised to see that in tackling any given problem, the general approach was to try a bunch of algorithms and see which one gave the best results. Then, for any given algorithm, try a bunch of different values of hyperparameters and see which ones gave the best results.
And of course for gradient descent, the approach is to start at some random values and then try to improve on that, although in this case there actually is a process (differentiaton) for working out in which direction to go to get a better result.
When it comes to neural networks there don’t seem to be too many guidelines for such things as number of layers, number of nodes, activation functions, etc. Just try out a bunch of stuff, see what works best.
As I’m considering using a greater variety of features for my next experiment I’m faced with the issue of scaling. With only one feature (input) in my last experiment it wasn’t an issue, but I believe the ranges of values for various features need to be pretty consistent for the optimization algorithms to work efficiently. So, do I use a Standard Scaler, or normalization, or MaxMin Scaling, or something else? Guess I’ll just have to try out a bunch of options and see what works best. Start at some random point, estimate how bad a result I get, and then try to improve on it. Way to go. I guess that’s why some authors devote some time to developing some kind of test harness so one can automate, to some extent, the process of trial and error.