HadISD v3.4.1.202411p; Precipitation values; GHCNh

We have just released v3.4.1.202411p of HadISD.  The ISD is still being updated at NOAA, and we hope to release the v3.4.1.2024f update early in January 2025.

We have also been made aware that as a result of how we prioritise different messages in the ISD, not all precipitation reports are present in the HadISD.  The ISD has a number of sub-daily message types (e.g. FM-12, FM-15, FM-16 etc) stored to minute-precision.  When selecting observations for the HadISD, we do not take all possible entries in the ISD, but prioritise those which contain Temperature or Dew Point Temperature values.  To quote from Dunn et al, 2012:

As both temperature and dewpoint temperature are required to be measured simultaneously for any study on humidity to be reliably carried out, reports that have both temperature and dewpoint temperature observations are favoured (under the assumption that the readings were taken at close proximity in space and time) over those reports that have one or the other (but not both), even if the reports with both observations are further from the full hour. In cases where observations only have temperature or dewpoint temperature (and never both), then those with temperature are favoured, even if these are further from the full hour (00 min). All variables in a single HadISD hourly time step always derive from a single ISD time step, with no blending between the various within-hour reports. However the HadISD times are always converted to the nearest whole hour.

However, this can result in that the selected report may not include all metrics, and so there are gaps for that timestamp in the HadISD.  For precipitation variables, when summing to daily, monthly, or annual totals, this will result in an apparent undercatch, where totals are lower than derived from other data sources (e.g. GSOD, also based on ISD).  We've also been made aware that the logical QC check being applied to the precipitation data is not always working as intended - so it may be worth looking in the flagged_values field of the netCDF files if something looks awry.

Finally, in case you'd not seen, version 1.0.0 of GHCNh is available for download at NOAA.

HadISD v3.4.1.202409p

The update to HadISD v3.4.1.202409p has just been released, now that the data services at NOAA NCEI are back online following the flooding from Hurricane Helene.  Although many files are updated through to 30 September 2024, it is possible there are some data gaps resulting from missing ingested data.  NOAA NCEI are working through these, but it could take a while for all data gaps to be filled.  More may be apparent in the next release (v202410p).

It is also not yet clear if the date scheduled for the termination of updates to the ISD (31st October 2024 as last I heard) will now move later.  We intend to continue updating HadISD for as long as updates are appended to the ISD.

Delays to HadISD updates

Following Hurricane Helene's passage over North Carolina, the extensive flooding in the Asheville area has caused an outage of some of the NOAA-NCEI websites and datasets.  This means that the update to HadISD due early October 2024 (v3.4.1.202409p) will be delayed until these services are up and running again.

Looking towards GHCNH

Last week NCEI announced the release of the GHCNH (Global Historical Climate Network Hourly) dataset:

https://www.ncei.noaa.gov/news/next-generation-climate-dataset-built-seamless-integration

The GHCNH replaces the ISD, and as such I'm still expecting the ISD to be turned off in the next few months.  This will obviously result in the HadISD having no further updates.  

As many of the QC tests being applied in GHCNH are based on those in the HadISD, it does not make sense to pass the GHCNH data through the HadISD QC system. Therefore the HadISD in its current form will transform to a static dataset, and at some point in the future, will be retired and archived (though this is some time off!).

At the moment there is no plan to immediately work on a wrapper for the GHCNH data and release in a "HadISD" format. However this may change in the future as we move to using GHCNH in other systems as well.

We'll post updates on this blog over the next months during the transition from ISD to GHCNH.

HadISD v3.4.0.2023f & future look

We released updated versions of HadISD, and this time two versions have been released at the same time.  As described in this post, we noted that the buddy/neighbour checks had not been running since 2018.  We have released a version of HadISD which correctly implements these checks as intended (v3.4.0.2023f), but for those who may wish to do their own comparison or use a version where these checks are absent as per the last few years of updates, then v3.3.1.202312p is also made available.

As we noted in our earlier post, the missing buddy checks also affect some of the other QC checks - predominantly those where a comparison with neighbouring stations can lead to flags set being removed. The Odd Cluster (Fig 1) and Climatological (Fig 2) checks show clear increases in the fractions of observations flagged by these checks across most stations.

Fig 1: Odd cluster checks for Dewpoint. Top - v3.3.1.202312p, Bottom - v3.4.0.2023f

Fig 2: Climatological Outlier checks for Tempeature. Top - v3.3.1.202312p, Bottom - v3.4.0.2023f

Although there is a general increase in the amount of observations flagged, most of these are in the lowest categories of fractions of the total record (to be expected).  We also expected changes in the flagging rates for the Distributional Gap check, but saw only very slight differences.

The other test with a clear impact is that of Dewpoint Depression (Fig 3).

Fig 1: Dewpoint Depression checks. Top - v3.3.1.202312p, Bottom - v3.4.0.2023f

Future Look

As noted in another earlier post, the ISD will be pausing updates during 2024.  The timeline for this is now looking like end March 2024 rather than being December 2023, and we'll post on here when we get further details.  In the meantime, we will continue HadISD updates (under v3.4.1.2024XXp) until ISD updates cease.