Business forecasting 101
Residuals indicate the difference between your chosen forecasting method and actuals. You can look at residuals over time and their distribution to understand how well the chosen forecast method fits
A residual time graph shows the difference between forecasts (red line) and actuals (blue line).
Reading a Residual Time Graph
The residual time graph illustrates the difference between forecast values and your actual historical data over time.
Ideally, a residual graph will look like noise. It should contain no discernible patterns or repeated column groupings. Otherwise, the graph indicates that the forecasting method is not picking up on seasonality, recurring promotions, price changes, or other events. You may also be missing exponential growth or decay.
Reading a Residual Distribution Graph
The peak of this graph should always be centered at or around zero. If it is not peaked near zero, it indicates that your forecasting method is biased, meaning it is systematically over- or under-forecasting. For example, your data might contain a growing or declining trend but is being modeled with a level method. Alternatively, your data might contain exponential growth but is being modeled with a linear trend method.
If your graph does not have a bell shape (e.g. hill shape), it most likely means that you have outliers that are skewing your bell curve to have a longer tale on one side versus the other. In some cases, you might just have too little data to create a smooth distribution graph; in this
Determining Forecast Fit
To determine whether your forecast method fits well, check out the following:
– Forecast Fit – Residual Analysis
– Out of Sample Testing / Holdout Sample
– Forecast Error
Vanguard business forecasting applications display the forecast, actuals, residuals, errors, and the holdout sample for a complete view of your forecast fit.
Out of Sample Testing / Holdout Sample
To conduct this test, take out the most recent periods of demand history (the holdout sample) as if it did not exist. The number of time periods that you remove should correlate to your normal forecast horizon (e.g. if you forecast three months into the future, the holdout should be at least three months long). You can then apply different forecasting methods to see what the holdout sample error is or Mean Absolute Deviation/Error (MAD or MAE). This will tend to be higher than the mean error. The method with the lowest MAD will likely be more accurate.
The holdout sample is strictly anecdotal data since it covers only a limited number of periods. So while it’s helpful to test different methods, it does not, by itself, determine which method is best.
Straight Line Forecast Myth
Symmetric Mean Absolute Percent Error (SMAPE)
SMAPE is the forecast minus actuals divided by the sum of forecasts and actuals as expressed in this formula:
Mean Absolute Percent Error (MAPE)
With zeros or near-zeros, MAPE can give a distorted picture of error. The error on a near-zero item can be infinitely high, causing a distortion to the overall error rate when it is averaged in. For forecasts of items that are near or at zero volume, Symmetric Mean Absolute Percent Error (SMAPE) is a better measure.
MAPE is the average absolute percent error for each time period or forecast minus actuals divided by actuals:
Last Absolute Deviation Z-Score
Z-Score, unlike other measures, relies on the most recent data rather than an average of all historic forecasts and actuals. The Z-Score represents the standard deviation from the last period’s forecasts and its actuals. A Z-Score or standard deviation of two or more indicates a recent significant structural change. Was an item discontinued? Did a large new client recently join? Did a competitor exit, causing sales to ramp up?
The Z-score is obtained using this formula:
Mean Absolute Deviation (MAD), Mean Absolute Error (MAE)
MAD is most useful when linked to revenue, APS, COGS or some other independent measure of value. MAD can reveal which high-value forecasts are causing higher error rates.
MAD takes the absolute value of forecast errors and averages them over the entirety of the forecast time periods. Taking an absolute value of a number disregards whether the number is negative or positive and, in this case, avoids the positives and negatives canceling each other out.
MAD is obtained by using the following formula:
Subscribe to our blog
Never miss industry news, technical information, or thought leadership articles.