01Can pilot reports teach a forecast where it's wrong?
Turbulence forecasts are good where they have been tuned against aircraft measurements, and much rougher elsewhere.
This post covers the data we have so far, how we frame the learning problem, and how we plan to train and test a calibrated model, starting where today's forecast is weakest.
02The weather models underneath
Every number in Bumpy starts as a forecast from a weather model. Turbulence has several causes, and each forecast catches some far better than others.
A model can recalibrate what a forecast sees. It can't recover what the forecast never saw.
We use two forecasts:
- Over North America: GTG. NOAA's Graphical Turbulence Guidance combines several turbulence measures from a detailed weather model, tuned against turbulence measured by airliners. It gives separate clear-air and mountain-wave forecasts, and we keep the larger.
- Everywhere else: a GFS estimate. We estimate turbulence from the winds of GFS, NOAA's global weather model, using a wind-shear index. It's a fair guide to jet-stream turbulence on a coarse grid, and not much more.
Pick a cause below to see which inputs capture it.
Storms
Thunderstorms are the biggest gap. Neither forecast we use has a storm component, so storms only reach Bumpy indirectly: through wind shear around large systems, and through reports from pilots who already flew through the bumps.
Four planned inputs would add a direct storm signal: storm warnings (SIGMETs), radar echo tops, lightning and instability (CAPE). They would feed both the model and a separate storm layer in the app, since flights usually route around storm cells rather than through them.
Pilots avoid storms, so reports near them are scarce. Storms will need their own evaluation.
03Data: pairing forecasts with what pilots felt
Pilot reports (PIREPs) give a position, an altitude and how rough the ride was. Pilots also report smooth air.
Each hour, every new report is matched against each forecast run we still hold. We record what that run predicted at the report's exact place and time, plus the strongest forecast nearby. Since several runs are compared with each report, one report yields several examples at different lead times.
Select a run to see the example it produces.
04The dataset so far
Logging began on 1 November 2026. These figures cover the first two weeks, up to 14 November.
Where and when reports come from
The reports follow the traffic, and mostly US traffic. The main source is US pilot reports. The rest of the world shows up mainly through automated airline position reports (AIREPs) on oceanic routes.
Why the data is sparse
- It follows the traffic. Reports cluster on busy US routes in daytime. Oceans, nights and most of Asia, Africa and South America barely appear.
- Bumps get reported, and bad ones are rare. Pilots report bumps more readily than smooth air, yet only 86 reports in two weeks were severe or extreme, often several from one patch of sky.
- Some pairs are incomplete. 4,164 have no forecast because the report is below the grid's lowest level. The 2,170 AIREPs carry a flight number where the aircraft type would be.
- Short leads, one global source. Each report is only compared with runs made within about two hours of it. The global WAFS product was unavailable throughout, so every pair outside the GTG area uses the GFS estimate.
How today's forecast scores
For each report we take the newest run made before it, which gives 28,501 fair pairs. The forecast ranks turbulence well but reads low: most moderate-or-worse reports get a forecast below our moderate line.
05Problem formulation
Our forecasts use eddy dissipation rate (EDR), a standard measure of how turbulent the air is, independent of the aircraft. Pilot reports use words instead: smooth, light, moderate, severe and the steps between.
We map each word to an approximate EDR for a medium-size airliner, so reports and forecasts share one scale. The first model is then a yes-or-no question: given the forecast and its context, how likely is a pilot to report turbulence at or above a threshold?
Switch the threshold to see which reports count as "yes".
06From the air to the cabin: aircraft size
EDR describes the air, not the ride. The same patch of turbulence feels very different in a four-seat Cessna and in an A380, because heavier aircraft respond less to the same gusts.
- Reports depend on the aircraft. A "moderate" from a regional jet and a "moderate" from a widebody aren't the same observation.
- Today we assume one size. Our forecasts and thresholds assume a medium-size airliner.
- The plan is five size classes. They follow the way air traffic control already groups aircraft by size and wake.
We considered a separate model for each aircraft class and decided against it. Pilot reports are already sparse. Split five ways, the smallest and largest classes would have almost no turbulence reports to learn from. Instead, aircraft class is an input to one shared model.
07Features
Each training example starts from the forecast at the report's position and adds the context that changes how far that forecast can be trusted. Some of these are recorded today; others need the forecast pipeline to log more of what it already computes.
| Group | Features | Why it matters |
|---|---|---|
| Forecast | Point EDR, strongest EDR nearby, forecast source | The signal itself, plus tolerance for position error and a flag for which product produced it |
| Where | Flight level, latitude, terrain height | Jet-stream turbulence peaks near cruise levels; mountain waves follow terrain |
| When | Lead time, month | Error grows with lead time; the jet stream strengthens in winter |
| Who | Aircraft size class | Converts air turbulence into what the cabin feels |
| Atmosphere (planned) | Wind speed, vertical shear, deformation, Ellrod index, Richardson number | Lets a model learn from the ingredients of turbulence, not only from our final number |
| Convection (planned) | Distance to the nearest active storm cell or convective SIGMET, echo top relative to flight level, lightning density, CAPE | Adds the storm signal that today's forecast inputs mostly lack |
08Modelling approach
Expected gain depends more on the region than on the method. Inside the GTG domain the forecast comes from GTG, which NOAA already calibrates against in-situ EDR, so improvement there is unlikely. Outside it the forecast is the uncalibrated GFS Ellrod-Knapp estimate, which is where we expect calibration to help most.
09Calibrating forecast probabilities
Our first model is a calibration layer. It learns a mapping from forecast EDR to the probability of moderate-or-worse turbulence, using isotonic or logistic regression. There's one mapping per forecast source and altitude band.
Calibration doesn't move where the forecast puts turbulence. It changes what the numbers mean. It runs when the forecast grid is built, so nothing downstream has to change. Drag the slider to apply it.
10Where the model runs
The model is applied when the forecast grid is built, once an hour. Its inputs are global weather grids, and its output is written into the same tiles the app already downloads.
Training is separate and slower: a weekly fit on the paired reports and forecasts collected so far.
Two rough numbers size the job (back-of-the-envelope estimates, not measurements):
- About a billion cells an hour. One global run is a 0.25° grid × 22 levels × about 36 hours, and the model scores every cell.
- Minutes of CPU. A calibration table is one lookup per cell and costs almost nothing. A small tree model over a full run, and each weekly retraining, are estimated at minutes on the same server.
11Evaluation and rollout
PIREPs are strongly correlated in space and time: a single weather system can generate dozens of reports. A random row split puts correlated rows in both train and test and overstates skill. We split by time instead, with whole weeks held out.
Metrics and acceptance criteria:
- Detection. Probability of detection (POD) and false-alarm ratio (FAR) for moderate-or-greater at the operating threshold.
- Probabilities. Brier score, ROC AUC and the reliability diagram above.
- By region. All metrics reported separately for the GTG domain and the GFS region, and by altitude band.
- Acceptance. Better Brier score than the baseline on held-out weeks in the GFS region, with no regression inside the GTG domain.
Slicing the results
A single average hides the cases that matter most. So every metric is broken down by forecast source, altitude band, lead time, aircraft class and season.
Moderate-or-worse reports are rare, so some slices will be small. We report how many turbulence cases each slice has. Confidence intervals come from a block bootstrap that resamples whole days, since reports from the same day are related.
Shadow mode before launch
A model that passes offline tests won't go straight to users. It first runs in shadow mode inside the hourly job. It scores every run alongside the current forecast, and both are logged against the pilot reports that come in afterwards.
Only when the new model beats the current one on later weeks it has never seen does the forecast switch over. If it later does worse, the previous model is switched back in for the next hourly run.
12Challenge: forecast lead time
A report can only be compared with forecast runs we still have when it arrives. We keep the three most recent hourly runs, so in practice every pair falls within about two hours either side of the report.
The app shows forecasts up to about 35 hours ahead, and errors grow with lead time. A model fitted only on short leads would be over-confident further out. Covering longer leads means storing more runs, at roughly 600 MB each.
13Challenge: pilot reports as labels
Label noise
Even a perfectly representative set of reports would be noisy:
- Position. Often approximate, sometimes given relative to a beacon tens of kilometres away.
- Time. Can lag the encounter by several minutes.
- Intensity. A judgment call in a busy cockpit, and the line between light and moderate is genuinely fuzzy.
We handle this three ways:
- Tolerance. The nearby-maximum feature absorbs small position and timing errors.
- Both readings. Borderline categories are tested both ways rather than forced to one side.
- Simple first. We prefer simple models until there's enough data for more.
Automated in-situ EDR reports from equipped airliners are a natural extension: they are denser, cover smooth air as well as turbulence, and don't depend on how a pilot describes the ride.
Further out, the phone itself could become a sensor. Its accelerometer measures what a passenger actually feels, in the exact aircraft we're forecasting for. It would have to be strictly opt-in and summarised on the device. We haven't built it, but it's the most direct label for the question Bumpy answers.
14Failure modes we're watching for
- Learning the US and applying it everywhere. Most training data comes from US airspace, where the forecast is GTG. A model can look good overall while learning little about the GFS regions it is meant to fix. Per-source metrics and acceptance criteria guard against this.
- Convective turbulence. Today's forecast inputs carry little storm signal, and the GFS estimate carries none. Calibration can make the numbers more honest, but it cannot add skill the inputs don't contain; that needs the observation-based storm inputs described under The weather models underneath.
- Seasonal drift. The jet stream moves and strengthens with the seasons. A model fitted in autumn may be miscalibrated by spring, which is one reason to retrain weekly and to evaluate by season.
- Upstream changes. When NOAA updates a forecast model, the relationship between its output and real turbulence can shift overnight. Shadow-mode logging makes these shifts visible quickly.
- Over-confidence at long lead times. Until the dataset covers longer leads, the model is evaluated only on short ones, and the app should treat long-range answers with more caution.