How to Compare Forecast Models Before Selecting One in AWS Supply Chain

This tutorial shows how to compare forecasting candidates before selecting a model for operational use in AWS Supply Chain.

1. Hold the dataset constant

Use the same history, forecast horizon, product-location grain and exclusions for all candidates. Model comparison is meaningless if each candidate is evaluated on a different frame.

2. Generate comparable candidates

Run several forecasting approaches and capture their performance using the same evaluation periods and business segments.

3. Compare metrics by segment

Review error, bias and stability by category, demand pattern and business importance rather than choosing the model with the best single aggregate score.

4. Review structural errors

Look for systematic under-forecasting, over-forecasting, poor treatment of intermittency and unstable behavior around promotions or launches.

5. Validate on a holdout period

Do not select a model only from in-sample performance. Use a period the model did not see during fitting and compare it with the incumbent method.

6. Document the selection decision

Record why the model was selected, who approved it, the segments where it applies and the threshold that would trigger rollback.

Failure controls

• No automatic model switch without a predefined gain threshold.
• Stable segmentation across candidates.
• Explicit rollback criteria.
• Post-deployment drift monitoring.