01Is two weeks of pilot reports enough to learn from?
The short version. There is enough data to check and correct today's forecast over North America, but not enough to train a model of our own.
- Few rough reports. 192 are moderate-to-severe or worse.
- Little outside North America. About 700 reports.
- Short lead times. Every forecast was under two hours old. The app looks days ahead.
02About one report a minute
The forecast grid is rebuilt every hour. Each build is a run, and the newest three are kept.
Each new report is logged once against each of those three runs, so one report becomes three rows.
Logging began on 1 November at about 06:00 UTC. Daily volume swings from about 1,500 to 3,100 reports with the weather and weekday traffic. 14 November is a partial day.
Reports logged per day
03Where the reports come from
97.6% of reports are over North America. That is the source, not a filter of ours. The data comes from aviationweather.gov, which is mostly US PIREPs: a pilot radios in how the ride feels.
The rest of the world only shows up through AIREPs. These are automated position reports from an airline's aircraft, sometimes with a turbulence note. 97% of them are from United over the Pacific and Atlantic.
The dense lines on the map are the busy US routes.
| Region (rough boxes) | Reports | Share |
|---|
04What pilots said
Pilot reports are words, not measurements. The pilot picks a category for how the ride felt in their own aircraft.
To compare words with a forecast, each category is turned into a rough EDR for a medium jet, from 0.15 for light to 0.60 for extreme. EDR is eddy dissipation rate, the standard measure of turbulence strength. Higher is bumpier.
In a forecast, we call 0.10 light, 0.22 moderate and 0.34 severe.
Smooth reports are logged too, and they are 27% of the total. A model needs them as much as the bumps. Without them it only sees places where something happened.
Reports by what the pilot said
05Altitude: about half are at cruise
A flight level (FL) is altitude in hundreds of feet, so FL350 is about 35,000 ft.
About half of all reports are between FL300 and FL400, where airliners cruise. The low levels are busy too, with climbs, descents and small aircraft.
Smooth reports are much more common down low. Switch to “Share smooth” to see it.
Reports by flight level
06Aircraft: mostly 737s and A320s
PIREPs record the aircraft's type code, such as B738 for a 737-800. AIREPs don't. They record the flight number instead (UAL58), so their aircraft type is unknown.
The type codes are sorted into five size classes with the same rules the forecast uses to score flights. Anything the rules don't recognise counts as medium.
- Super light: pistons and light turboprops, such as the Cessna 172, Pilatus PC-12 and King Air.
- Light: business jets, such as Citations, Challengers and Gulfstreams.
- Medium: narrowbodies, regional jets and airline turboprops, such as the 737, A320, E175 and CRJ-900.
- Heavy: widebodies, including the 747.
- Super: the A380.
Reports by aircraft size, split by what was reported
07Time of day
Reports follow US daytime traffic. “Local solar time” shifts each report's UTC time by its longitude, so it shows the time of day where the aircraft was.
The overnight hours are thin. A model will be least sure about overnight flights.
Reports per hour
08Lead time: two hours either way, no more
Lead time is how long before the report the forecast was made. Each report is compared with the three runs on disk when it's logged. Those runs range from about two hours older to two hours newer than the report.
A negative lead means the run came after the pilot reported, so it isn't a fair test.
The app shows forecasts for flights hours or days away. The log can't say yet how good a 6-hour or 24-hour forecast is, and that is the number a nervous flyer cares about.
Rows by lead time, in minutes
09How the forecast compares with what pilots felt
Over North America the forecast comes from GTG (Graphical Turbulence Guidance), NOAA's turbulence forecast. Everywhere else it is a rougher estimate worked out from GFS, NOAA's global weather model.
For each report we take the newest run made before it. The median lead is 29 minutes. That gives 28,501 fair pairs.
The chart shows how the forecast EDR is spread for each kind of report. Drag the slider to move the cut-off and see what it would have caught.
It ranks well
Pick a random moderate-or-worse report and a random smooth one. Inside the GTG area, the bumpy one has the higher forecast 79% of the time (an AUC of 0.79). Outside, it's 77%.
So the forecast does know where the air is rougher.
It reads low
At our 0.22 moderate cut-off, only 9% of moderate-or-worse reports inside the GTG area get a moderate forecast. The median forecast at those reports is 0.137.
This is the clearest finding so far, and a recalibration could fix it now. It holds for every size class: the median is 0.137 for mediums and 0.150 for heavies.
One possible cause, not yet tested: averaging fine GTG cells into the coarser grid flattens peaks.
| Reported | Area | Pairs | 10th pct | Median | 90th pct | Nearby max | Forecast moderate+ |
|---|
10What one record looks like
Each record pairs one report with one forecast run. This is a real record, a pilot report of moderate turbulence. The raw field is the text the pilot or airline system sent.
11What this sample can and can't support
What it can do now
Recalibrate how forecast EDR maps to smooth, light, moderate and severe over North America at short lead. There are about 6,600 moderate-or-worse and 7,400 smooth fair pairs. That is plenty for a simple correction, and it would be the first real test of what the app's words mean.
What it can't do yet
- Longer leads. It can't judge forecasts more than two hours ahead, which is most of what the app shows. The log has to record longer leads first.
- Severe turbulence. 86 severe-or-extreme reports, often several from one patch of sky, are too few to learn from.
- The rest of the world. There are about 700 reports outside North America, nearly all from one airline.
- The seasons. This is two weeks of November. The jet stream, summer storms and winter mountain waves each need their own months.
A trained model starts to make sense after about a year of logging that includes 6–24-hour leads. At today's rate that is about 790,000 reports, about 5,000 of them moderate-to-severe or worse.
12Gaps in the data
- No global turbulence forecast. WAFS (the World Area Forecast System), the global turbulence product, has never loaded: NOAA's public server refused or dropped the requests, and the product has since left it. So outside the GTG area, every forecast in this log is the GFS estimate.
- Lead time stops at two hours. Checking day-ahead forecasts means sampling each run against reports that arrive hours and days later. The plan is to keep a thinned copy of older runs, since a full run is about 570 MB.
- AIREPs have flight numbers, not aircraft types. That affects 2,170 reports. The plan is to look up the type from the flight number when the report is logged.