Before anyone builds the model
Machine learning tends to get pitched to companies as one big promise, and most of the companies on the receiving end have no real way of telling a good model from a bad one. That is completely understandable, and the good news is that they don’t need the maths to tell the difference. What they need is a handful of questions, asked before anyone starts building, and a clear idea of what a good answer sounds like. Over the last few weeks I have been building, and deliberately breaking, forecasting and churn models on two years of sales from a real UK online wholesaler (a public dataset of about a million transactions, from a business turning over roughly £9m a year), and each question below comes with what it showed.
What does it beat?
Every model should be measured against the simplest guess a sensible person would make without one, and the difference should be stated in pounds. For a demand forecast that guess is usually something like the average of the last four weeks. On thirteen weeks of sales held back for testing, across the top 500 products, that simple guess was £1.24m off. The best model was £1.14m off, which makes it 8% better. That is a normal, honest result on messy, real-world data, and it is worth having. It also sets the scale for what you should expect to hear: if a vendor tells you their model is ‘50% better’, the first question is better than what, as the answer is often a comparison chosen to make the model look good.
Which mistake costs more?
A forecast can be wrong in two directions, and they rarely cost the same. Forecasting too low means running out of stock and losing the sale, while forecasting too high means being left with stock on the shelves. Once running out was counted at three times the cost of leftover stock, the model which had been the most accurate came last, and a model trained with that cost in mind won instead, by 9%. Nothing in the data tells you what that ratio should be: it depends on margins, shelf life and how customers react to an empty shelf. Deciding which mistake costs more is the business’s call, not the data scientist’s, and if nobody asks the business, the model will be built around a choice that nobody made.
Has it run for real?
A single test gives you a single number, and that number can hide a lot. I replayed the forecast month by month from March to October as if it had been live, retraining it each month on only what would have been known at the time. It lost the first month by 3%, then won every month after that, by 4% growing to 10%. A copy which was never retrained after March still beat the simple guess, but fell behind the retrained one by about £75k over the eight months, without a single error showing anywhere. So it is worth asking whether the model has been run alongside the current process, on live data, before anyone relies on it, and what alerts someone when it starts to slip, because nothing will on its own.
Who keeps it alive?
The model itself was about 4 of the forecast’s 190 lines of code. The rest is the upkeep that sits around it: getting the data in every week, checking it before it is used, retraining on a schedule, keeping a record so any number can be reproduced, and a named person who owns all of it. One order in the data was worth £77k and turned out to be a typo, cancelled 16 minutes later; a check which catches that before it reaches the forecast is exactly the kind of unglamorous work that a model depends on. This upkeep is where most of the real cost of a model lives, and it doesn’t stop at launch. Platforms like Databricks or Azure ML supply the parts for most of it, but someone still has to decide when to retrain, what counts as a bad week, and who gets the call when it happens.
The answer nobody gives
The question that rarely comes up in a pitch is whether the model should be built at all. The churn model, which predicts which customers are about to stop ordering, earned about £6k per six months on assumed costs, for a business turning over £9m a year. Simply sorting customers by how long it had been since their last order came close to matching it. In a case like this the right advice is not to build it, and sometimes the most useful thing a company can be given is a saved filter in the CRM.