What a stubborn training log taught me about measuring before tuning
I learned this one the slow way: most “stuck” models were never moving in the first place.
I spent days staring at a training log that refused to budge. Validation loss bouncing in a narrow band, accuracy wobbling literally around the same values, epoch after epoch…I did the natural thing, I started fine tuning hyper-parameters: I changed the learning rate, then the dropout, then the scheduler…I kept a mental leaderboard of configs 🙃
What I was actually watching was noise and I was calling it progress. This is the kind of mistake that slips through the cracks quietly, because every step feels like reasonable engineering.
The Tournament of Luck
The first thing I missed: my validation set was really really tiny. A handful of samples. (I had no choice, very limited data available). It is easy to forget why that matters. When the validation set is that small, one sample flipping its prediction changes the score by a huge visible jump. So when I compared run A against run B and concluded a winner, I was not comparing models. I was comparing which random init got lucky on a couple of data points.
And the part that stings: every config, including the “best” one, sat barely below what a blind guess would score. I had never bothered to compute the chance level. If your classifier has N classes, a model predicting uniform noise gets a known loss and a known accuracy. Once I printed that number next to my metrics, the picture changed completely. My “improvements” were ripples around the I-don’t-know-anything floor.
The lesson that I had learned million times: you cannot interpret a number without a baseline. Simple to say and simple to forget.
Deep Is Not Big
The second mistake was even prettier, so it survived longer 😜
The model was deep. Dozens of transformer layers. It sounded impressive in the config file but it was absolutely a skyscraper built out of straws. Each layer was nearly empty, attention split into many heads, each with almost nothing to work with, and a feed-forward layer with no room to expand. Information went in narrow and came out narrow, fifty times in a row. Well, depth without width is not capacity, it is a long hallway with a low ceiling. You feel it?
The fix was almost embarrassing: fewer layers, much wider, room for the network to actually transform what passes through it. Same family of model, same attention heads, the only real change was giving information somewhere to go. I was basically chocking the network!
One more trap, specific to transfer learning, and it explained a mystery: My original tiny model kept scoring “best.” Why? It was grafted onto a pretrained backbone, and it was so small it barely disturbed the pretrained knowledge. It looked like learning. It was mostly just not destroying what it inherited. When I widened the model, the new shape could not reuse the old weights and on a dataset this small, starting from scratch meant uniform predictions. The “regression” was not the new architecture failing. It was the truth being revealed.
Measure, Then Act
This is the right order you should always follow, because skipping steps is exactly how the cracks open:
1- Learn the chance level for the task and print it beside every metric.
2- Run the overfit test, basically a tiny slice of data, no regularization, can the model memorize it? If not, there is a bug (yeah, a bug in the data!) or a mismatch, and no hyperparameter will save you.
3- Verify the transfer actually transferred, do not assume the weights loaded.
4- Now you can touch the architecture.
As you know already, my motto has been always: metrics, measuring, then action. I broke my own rule. I acted on numbers I had never put in context, and I spent days tuning a model that was not learning at all. The barrier was never in the loss landscape. it was in my ability to measure.
Pull the curtain back before you tune. That plateau you are fighting might be a floor you never rose above.