My first encounter with the importance of forecasting came during my first job more than 15 years ago. I was working in the back kitchen at Pizza Hut, and it was Mother’s Day. At first, I thought it was just another Sunday, the slowest day of the weekend. During usual Sundays, there would be one person (me) working in the back kitchen for most of the day, and the work entailed making preparations in the afternoon, making pizzas in the evening, and cleaning up at night. That Mother’s Day was very different. By around 5pm, orders started coming in, and unlike other Sundays it quickly ramped up to a pace where orders came in faster than I could complete them. By around 6pm, we’d already taken 100+ orders, and the pace wouldn’t slow down until 9pm. That night, we had over three times more orders than we were expecting.
The poor forecast caused a number of problems. Since we expected a normal Sunday, I was the only person in the back kitchen. This means that:
If we had correctly forecasted the customer demand that day, then we could have prepared more materials in advance. Had we scheduled two more staff in the back kitchen (plus one more staff in the dining area), we would have been able to make pizzas fast enough to keep up with the orders that came in, and the kitchen would have been mostly kept clean throughout the night, allowing us to go home on time. Instead, customers were left with worse-than-expected service, and I had a long, difficult night.
Business challenge
Ophelos at its core is a service company, part of which involves resolving customer enquiries with the best service possible. We employ amazing customer operations staff and strive to always do our best. However, just like the restaurant business, there are times when we have more demand, and times when we have less. And if we don’t have enough staff to service our customers, our team will become overwhelmed, leading to decreased well-being and poor service quality. In recent times, we have run into a few challenges resulting from our rapid growth. One of those challenges is that we had been getting more customer enquiries than we expected. Therefore, we decided to build a demand forecasting system. In hindsight, we should’ve done it much earlier, but as most startups go, we were busy with other fires.
How can demand forecasting solve the problem?
For Ophelos, demand forecasting means to predict the number of inbound customer enquiries in the future. Imagine a world where we knew exactly how many customers are going to contact us and what their needs are, then we can perfectly staff our operations team to ensure that all customers are provided the best service possible. Although real life is unpredictable, if we can build a reasonably accurate model to predict demand, it will help us avoid situations where actual demand is many times greater than our team’s capacity. In our case, if we had a demand forecasting system, we would have noticed earlier that our growth was so rapid that we’d soon outgrow our capacity to service customers to the highest standards. Of course, better late than never.
What are the technical challenges of forecasting demand for customer operations?
Demand forecasting is one of the most well-known problems in all of analytics, but it’s also one of the most challenging. The future is difficult to predict, especially when trying to forecast many months into the future. You could even argue that every crisis can be attributed to some form of mis-forecast.
In technical terms, forecasting is challenging for two reasons. First, there is a very limited amount of data available. Depending on the time frame of observations, we’re limited to only 365 observations per year if forecasting daily, 52 observations if forecasting weekly, 12 observations if forecasting monthly, and only 4 observations if forecasting quarterly. Even at the daily level, 365 observations per year is very limiting, and most models are likely to capture noise in the data rather than true signals.
Second, the future is often affected by events that have not existed in the past, making those effects nearly impossible to predict. For example, no one could have predicted that the world would be halted for many months in 2020 due to a pandemic. One industry that was heavily affected by the pandemic was the aviation industry. The dynamic pricing systems of airlines relied on forecasts of future demand to set prices. When the pandemic caused consumers to stop flying, the airlines’ demand forecasts were grossly overestimating and as a result prices were incredibly low. As a result, airlines had to shut down their dynamic pricing systems and rely on human pricing during the pandemic period in order to set prices that reduced their losses.
Our approach to forecasting demand
To account for the challenges of forecasting demand, our approach is to rely on a combination of intuition, first principles, and comprehensive testing to find models that are both accurate and consistent. Then, we will rely on the models to guide future staffing decisions.
In order to design a forecasting model, we have to first ask a set of questions. Each answer to a question results in a distinct model. We often do not know the correct answers to the questions, so we either have to use our judgement or explore various possibilities. The questions are:
What exactly are we forecasting?
We define a forecasting problem as a prediction problem, where each observation is characterised by a row of data with a set of attributes and a label. This question asks about what the label of a row of data should be. Customer demand is an abstract term, so we need to drill down into what it is exactly we’ll be forecasting. Here we consider three dimensions:
Imagine that we decide to forecast customer enquiries at the monthly level, for the whole company, and treat all inquiry types equally. This means that the label for a single observation is the total number of customer enquiries that we received that month. We expect the number of customer enquiries to be fairly stable, with variance likely to come from having more or less active clients. However, with this approach we would only have 12 data points over a whole year, which limits our models’ ability to find patterns, greatly hindering our ability to predict events that lead to significant spikes or troughs in demand.
Alternatively, let’s consider the other extreme, where we decide to forecast customer enquiries at the daily level, for each individual customer, and each inquiry type. This means the label for a single observation is the number of a specific inquiry type that we receive from a customer in a day. If we have 1 million customers and 15 inquiry types, then we would have 15 million data points over a year. That is a lot of data for our models to learn from, but we would have sparsity because most customers will not contact us on most days, especially for different inquiry types. Moreover, this dataset is likely to be highly correlated over certain day(s), client(s), and/or inquiry types. Our models must first capture these correlations and then identify other patterns in the data in order to be useful.
Ultimately we decided to test all combinations of the three dimensions. Our goal is to have a set of accurate and robust models, and testing all combinations provides us with stronger intuition about customer demand, and with comprehensive testing we are able to more reliably determine which models perform better.
For when are we forecasting?
An important factor to determine in forecasting problems is the forecast horizon, i.e., for when are we forecasting? Imagine that we’re forecasting at the month level, a natural question to ask is: do we want to forecast 1, 2, 6, or 12 months into the future? The further ahead we forecast the more likely we are to be wrong, but there is also more value when we’re correct. If we can accurately forecast that customer demand will double 6 months into the future, then we will have more time to hire the necessary staff to serve the demand. If we can only forecast this demand increase 1 month in advance, then it’s likely that we won’t have enough time to hire all the staff we need.
The tradeoff between accuracy and value cannot be made without knowing the accuracy of our models when forecasting further into the future. For some problems (such as airline demand), forecasting 6 months ahead is probably not much harder than 1 month ahead due to strong seasonality of the business, whereas for other problems (such as stock prices), forecasting even 2 days ahead is likely to be much harder than 1 day ahead.
Which models will we use to forecast?
As with any prediction problem, a number of models exist that can be used to forecast demand. In our case, due to the number of factors that we must consider when solving this problem, it is likely that different models will be suitable for different dimensions of consideration. For example, when forecasting at the client and month or week level, we might want to use simple linear models, whereas when forecasting at the individual customer and day level, it might be more suitable to use a tree-based machine learning model. Alternatively, deep learning models such as RNN or transformers are worth testing as well since they have been shown to produce state-of-the-art results in some problems. Since we ultimately seek a solution that is as reliable as possible, we actually test many models and choose ones that give us accurate and consistent forecasts. The list of models include: linear regression, logistic regression, XGBoost, RNN, LSTM, and transformers.
Deployed forecasting model and demand simulator
After identifying a set of suitable forecasting models we deploy them both as a dashboard and a simulator. Unlike most ML models, our demand forecasts are used to plan future staffing decisions, which means that it needs to generate forecasts under different scenarios. For example, we might expect to onboard three new clients over the next two months, and thus need to forecast the expected demand. However, what if our business does so well that instead of three new clients we actually onboard six? How many customer operations staff will we need then? Should we onboard all clients over the same month, or stagger their starting dates? Only with the ability to test different scenarios will the model be able to help generate business value.
Final words
As a team, we’ve put a lot of effort into making the very best demand forecasting model. However, as is the nature of forecasting, only time will tell how accurate our model is. While we strive for accuracy and reliability, the truth is we don’t know what we don’t know.
