It's All Numerical Optimization

Okay, now we’re here - I hope that you agree that our objective is no longer to make hard-rules. We reject “input” + “rules” → “answers” mental model and accept slight errors.

Enter Machine Learning

Or not. Before we see how we make a machine learn, let’s see how humans learn first, shall we?

Suppose you have a Japanese language vocabulary exam tomorrow and you decide to cram. What do you do?

• Open a dictionary and try to remember the words?
• Maybe you use flashcards?
• Then try some mock exam?

Wait, don’t these… look somewhat familiar?

Insanity is doing the same thing over and over again and expecting different results
- Albert Einstein Narcotics Anonymous

gasp

_{It’s not Einstein’s quote?}

It’s not insanity, it’s practice

Because something did change!

Whenever you attempt to learn, something changes. You get better. Microscopically, the connections between neural cells inside your body change. Your actions tell your neural cells to rearrange their networks to better adapt to your environment.

Now, how do these neural cells adapt? You did not give them an explicit rule to follow - you just performed an action and received feedback.

You got an answer wrong on the mock exam, recognized the mistake, and your brain rewired itself to avoid repeating it. You (unconsciously) performed error correction.

… And now we just need to apply that, to a computer!

How Wrong?

A computer does not have biological synapses, nor does it feel the frustration of failing a mock exam. It only understands numbers.

So we must first:

1. make it understand the “input” and “answers” (by turning them into numbers),
2. make it understand the concept of “correct” and “wrong”, and
3. how wrong their understanding are

… basically, we have to design a ruthless teacher, with the sole task of “grading the exam”: Look at the machine’s guess, compare it to the correct answer, and calculate a single score representing how spectacularly the machine failed.

We call this the “Loss Function”.

Once we have that, we design the machine to “practice” based on the function’s response. This is the “error-correction” part.

But how does the machine actually know what-and-how-much to tweak? It relies on an even bigger boss.

It’s All Numerical Optimization

… and math.

Now, before we progress any further, allow me to explain why I brought back math.

A machine learning model is actually just a mathematical function… and improving the “model” actually means tuning the coefficients—the so-called weights—of the function.

But first, we have to choose the underlying function. For example, do we want to try fitting $f(x)=ax+b$, or do we want to try fitting $f(x)=\frac{1}{1+e^{ax+b}}$ ?

Important: the selection of that function shapes what you can learn.

Let’s say we choose $f(x)=ax+b$ as our model and we want to tune $a$ and $b$. We initialize them with random values and our “ruthless teacher” comes back and hands us a very very bad score.

Let’s say we do it several times and we have several 3-d coordinates of $(a,b,Error)$. Now, based on these coordinates, can you guess the coordinates of $(a,b)$ that might yield the lowest $Error$?

However, while guessing all of the combinations of $a$ and $b$ is feasible, what happens if the number of coefficients increase? what if it has hundreds, or millions of coefficients? This brute-force approach simply does not scale well.

Now, with math, you don’t need to guess. By saying that we want the lowest $Error$, we already shape the problem into an optimization problem.

Instead of trying to mapping the whole space, we use calculus. Specifically, derivatives.

A derivative measures the slope of a curve. Imagine you are blindfolded on a mountain and want to reach the lowest valley. You don’t need a complete 3D map of the entire mountain range to get down. You just need to feel the ground immediately under your feet. If the ground slopes downward to your left, you take a step to the left.

“Gradient Descent” algorithm operates exactly like this. The math calculates the derivative (the slope) of the Loss Function with respect to our current weights, $a$ and $b$.

• If the math indicates that increasing $a$ makes the error go up, the system decreases $a$.
• If it indicates that decreasing $b$ makes the error go down, the system decreases $b$.

The machine evaluates the error, calculates this local slope, takes a small step downhill to adjust the weights, and repeats. It loops this calculation thousands of times until it hits the bottom of the error valley.

No exhaustive coordinate guessing. No biological rewiring. Just a highly efficient, iterative mathematical descent.

Sounds easy, right? But actually, gradient descent isn’t perfect.

Why? well, let’s say the reason(s) is left as an exercise to the reader

Previous: Pattern Recognition with Mathematics and Programming
Next: TBD