Forecast Tool Accuracy Comparisons
The table below shows the accuracy of various forecast
applications when applied to 3,003 sets of test data used in the
M3 competition. Lower values represent lower forecast error
(better performance). The Vanguard results are based on a 100%
automatic batch forecast run (about 2 minutes run time).
Vanguard produced the most accurate forecasts in all time
horizons, beating all other software and academic teams.
Average Symmetric MAPE (Mean Absolute
Percent Error)
Figures in bold represent first place finishes.
| |
Forecasting
Horizons |
|
Average
of Forecasting Horizons |
| SOFTWARE |
1 |
2 |
3 |
4 |
5 |
6 |
8 |
12 |
15 |
18 |
|
1-4 |
1-6 |
1-8 |
1-12 |
1-15 |
1-18 |
| Vanguard |
8.1 |
9.1 |
10.5 |
11.8 |
12.4 |
13.0 |
11.8 |
12.7 |
15.4 |
17.3 |
|
9.88 |
10.82 |
11.04 |
11.40 |
11.84 |
12.40 |
| Forecast Pro V3 |
8.6 |
9.6 |
11.4 |
12.9 |
13.3 |
14.3 |
12.6 |
13.2 |
16.4 |
18.3 |
|
10.64 |
11.69 |
11.86 |
12.14 |
12.60 |
13.19 |
| Autobox-1 |
9.8 |
11.1 |
13.1 |
15.1 |
16.0 |
16.8 |
14.2 |
15.4 |
19.1 |
20.4 |
|
12.30 |
13.67 |
13.78 |
14.00 |
14.56 |
15.23 |
| Autobox-2 |
9.5 |
10.4 |
12.2 |
13.8 |
13.8 |
14.9 |
13.2 |
15.2 |
18.2 |
19.9 |
|
11.48 |
12.44 |
12.63 |
13.10 |
13.70 |
14.41 |
| Autobox-3 |
9.7 |
11.2 |
12.9 |
14.6 |
15.8 |
16.5 |
14.4 |
16.1 |
19.2 |
21.2 |
|
12.08 |
13.43 |
13.64 |
14.01 |
14.57 |
15.33 |
| Autocast |
9.1 |
10.0 |
12.1 |
13.5 |
13.8 |
14.7 |
13.1 |
14.3 |
17.7 |
19.6 |
|
11.20 |
12.21 |
12.40 |
12.80 |
13.34 |
14.01 |
| ForecastX |
8.7 |
9.8 |
11.6 |
13.1 |
13.2 |
13.9 |
12.6 |
13.9 |
17.8 |
18.7 |
|
10.82 |
11.73 |
11.89 |
12.22 |
12.81 |
13.49 |
| SmartForecasts |
9.2 |
10.3 |
12.0 |
13.5 |
14.0 |
15.1 |
13.0 |
14.9 |
18.0 |
19.4 |
|
11.23 |
12.34 |
12.49 |
12.94 |
13.48 |
14.13 |
| ACADEMIC TEAMS |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| ADAPTA |
15.5 |
16.2 |
17.1 |
18.2 |
18.1 |
18.9 |
15.7 |
18.6 |
20.9 |
22.3 |
|
16.74 |
17.33 |
16.98 |
17.21 |
17.62 |
18.12 |
| AMM1 |
9.8 |
10.6 |
11.2 |
12.6 |
13.0 |
13.5 |
14.1 |
14.9 |
18.0 |
20.4 |
|
11.04 |
11.76 |
12.43 |
13.04 |
13.77 |
14.63 |
| AMM2 |
10.0 |
10.7 |
11.3 |
12.9 |
13.2 |
13.7 |
14.3 |
15.1 |
18.4 |
20.7 |
|
11.21 |
11.95 |
12.62 |
13.21 |
13.97 |
14.85 |
| ARARMA |
9.7 |
10.9 |
12.6 |
14.2 |
14.6 |
15.6 |
13.9 |
15.2 |
18.5 |
20.3 |
|
11.83 |
12.92 |
13.12 |
13.54 |
14.09 |
14.74 |
| AutomatANN |
9.0 |
10.4 |
11.8 |
13.8 |
13.8 |
15.5 |
13.4 |
14.6 |
17.3 |
19.6 |
|
11.23 |
12.38 |
12.58 |
12.96 |
13.48 |
14.11 |
| B-J automatic |
9.2 |
10.4 |
12.2 |
13.9 |
14.0 |
14.8 |
13.0 |
14.1 |
17.8 |
19.3 |
|
11.42 |
12.41 |
12.54 |
12.80 |
13.35 |
14.01 |
| COMB S-H-D |
8.9 |
10.0 |
12.0 |
13.5 |
13.7 |
14.2 |
12.4 |
13.6 |
17.3 |
18.3 |
|
11.10 |
12.04 |
12.13 |
12.40 |
12.91 |
13.52 |
| DAMPEN |
8.8 |
10.0 |
12.0 |
13.5 |
13.7 |
14.3 |
12.5 |
13.9 |
17.5 |
18.9 |
|
11.05 |
12.04 |
12.14 |
12.44 |
12.96 |
13.63 |
| FLORES-PEARCE-1 |
9.2 |
10.5 |
12.6 |
14.5 |
14.8 |
15.3 |
13.8 |
14.4 |
19.1 |
20.8 |
|
11.68 |
12.79 |
13.03 |
13.31 |
13.92 |
14.70 |
| FLORES-PEARCE-2 |
10.0 |
11.0 |
12.8 |
14.1 |
14.1 |
14.7 |
12.9 |
14.4 |
18.2 |
19.9 |
|
11.96 |
12.77 |
12.81 |
13.04 |
13.61 |
14.29 |
| HOLT |
9.0 |
10.4 |
12.8 |
14.5 |
15.1 |
15.8 |
13.9 |
14.8 |
18.8 |
20.2 |
|
11.67 |
12.93 |
13.11 |
13.42 |
13.95 |
14.60 |
| NAIVE1 |
11.6 |
12.5 |
14.6 |
16.1 |
16.5 |
16.9 |
15.4 |
16.0 |
20.5 |
22.1 |
|
13.69 |
14.70 |
14.92 |
15.24 |
15.84 |
16.59 |
| NAIVE2 |
10.5 |
11.3 |
13.6 |
15.1 |
15.1 |
15.9 |
14.5 |
16.0 |
19.3 |
20.7 |
|
12.62 |
13.57 |
13.76 |
14.24 |
14.81 |
15.47 |
| RBF |
9.9 |
10.5 |
12.4 |
13.4 |
13.2 |
14.2 |
12.8 |
14.1 |
17.3 |
17.8 |
|
11.56 |
12.28 |
12.42 |
12.77 |
13.25 |
13.75 |
| ROBUST-TREND |
10.5 |
11.2 |
13.2 |
14.7 |
15.0 |
15.9 |
15.1 |
17.5 |
22.2 |
24.3 |
|
12.38 |
13.40 |
13.73 |
14.57 |
15.42 |
16.30 |
| SINGLE |
9.5 |
10.6 |
12.7 |
14.1 |
14.3 |
15.0 |
13.3 |
14.5 |
18.3 |
19.4 |
|
11.73 |
12.71 |
12.84 |
13.13 |
13.67 |
14.32 |
| THETA |
8.4 |
9.6 |
11.3 |
12.5 |
13.2 |
14.0 |
12.0 |
13.2 |
16.2 |
18.2 |
|
10.44 |
11.49 |
11.62 |
11.95 |
12.42 |
13.01 |
| THETAsm |
9.8 |
11.3 |
12.6 |
13.6 |
14.3 |
15.0 |
12.7 |
14.0 |
16.2 |
18.3 |
|
11.81 |
12.76 |
12.77 |
13.04 |
13.40 |
13.88 |
| WINTER |
9.1 |
10.5 |
12.9 |
14.6 |
15.1 |
15.9 |
14.0 |
14.6 |
18.9 |
20.2 |
|
11.77 |
13.01 |
13.19 |
13.48 |
14.01 |
14.65 |
Data from Hibon M. and Makridakis S. [2000]. The M3
Competition: results, conclusions and implications.
International Journal of Forecasting 16, 451-476. MAPE for
Vanguard and some other software in this table was compiled
after the M3 Competition.
|
