Forecast Challenge

Forecast and observation are compared in the same units (°F as stored). The penalty is deliberately nonlinear: a small miss keeps nearly full credit, larger misses lose points progressively faster, and the score reaches zero at 12°F of error.

temperatureError    = abs(forecastTemperature − observedTemperature)
temperatureAccuracy = 100 × max(0, 1 − (temperatureError / 12)²)
                      clamped to 0 … 100

When a forecast contains both a daily high and a daily low, each is scored separately and the temperature score is their mean:

highAccuracy        = temperatureAccuracy(forecastHigh, observedHigh)
lowAccuracy         = temperatureAccuracy(forecastLow,  observedLow)
temperatureAccuracy = (highAccuracy + lowAccuracy) / 2

Observed high and low are the maximum and minimum hourly 2 m temperature inside the valid period. If only one target was forecast, that single score is used.

The scored quantity is probability of measurable precipitation, not amount. The forecast probability is stored 0–100 and used as a fraction from 0 to 1. Measurable precipitation means at least 0.01 in of liquid equivalent accumulated during the valid period at the verification point. The threshold is fixed in configuration and is never changed for forecasts already submitted.

observedPrecipitation  = 1 if accumulation ≥ 0.01 in, else 0
brierError             = (forecastProbability − observedPrecipitation)²
precipitationAccuracy  = clamp(100 − 200 × brierError, 0, 100)

A confident, correct probability scores near 100; a confident, wrong probability is penalised heavily. A 50% forecast always scores 50.

Exactly one wind target is scored — currently Peak sustained wind — so sustained wind and gusts are never mixed under one metric. The other wind value is archived with the forecast and shown as unscored. The 20 mph error scale lives in a central scoring configuration so it can be reviewed and revised without rewriting application logic.

windError    = abs(forecastWind − observedWind)
windAccuracy = 100 × max(0, 1 − (windError / 20)²)
               clamped to 0 … 100

A Core Forecast contains all scored variables:

• Forecast high temperatureF
• Forecast low temperatureF
• Probability of precipitation%
• Peak sustained windmph

overallAccuracy = (temperatureAccuracy + precipitationAccuracy + windAccuracy) / 3

example: (94.1 + 82.0 + 88.7) / 3 = 88.27
display: 88.3%      awarded: 88.27 leaderboard points

Rounding happens only for display. The higher-precision score is what is stored and ranked.

A forecaster's rating is the mean of their eligible verified forecast scores, never the cumulative total, so volume alone cannot raise a rank.

leaderboardRating = sum(verifiedForecastScores) / numberOfVerifiedForecasts

• Fewer than 10 verified forecasts: Unranked
• 10–29 verified forecasts: Provisional
• 30 or more verified forecasts: Ranked
• A rating appears on a public leaderboard only at 10 or more verified forecasts, and the verified count is always shown beside it.
• Ties are broken by the larger verified sample, so a short streak never outranks sustained accuracy.
• Verified forecasts cannot be deleted from a forecaster's statistical record, including inaccurate ones.

Category boards rank the mean of a single category score, so a forecaster who only forecasts one variable is never compared directly against forecasters completing full Core Forecasts:

• Overall — Mean Core Forecast accuracy — forecasts containing all three scored variables
• Temperature — Mean temperature accuracy
• Precipitation — Mean precipitation-probability accuracy
• Wind — Mean wind accuracy

Forecast difficulty varies geographically, and version 1 does not attempt a climatological adjustment. Instead the same raw accuracy is published across nested scopes — global, country, region and state or province where enough forecasters exist — and the verification region is stored on every forecast so further geographic boards can be generated later without redesigning anything.

Current scopes: Global, United States, Michigan, Florida, Midwest, Southeast, Northeast.

This first-generation number is raw verified accuracy, not difficulty-adjusted forecast skill. The architecture leaves room for an optional Skill-Adjusted Rating that compares a forecaster's error against a fixed reference model issued for the same location, valid time and lead time; it stays unimplemented until those reference forecasts are actually archived.

• A published forecast locks at the start of its valid period and cannot be edited.
• Verification runs 1.5 h after the valid period ends, once observations have settled.
• The valid period must be between 60 min and 7.0 d long, issued at most 10.0 d ahead and at least 6.0 h before it starts to be scored.
• Duplicate forecasts within 6.0 h for the same location and category are rejected.
• If observations are unavailable for a metric it is marked unverifiable rather than counted as a miss, and it is excluded from the forecaster's statistics.