HadISD v3.4.1.202409p

The update to HadISD v3.4.1.202409p has just been released, now that the data services at NOAA NCEI are back online following the flooding from Hurricane Helene.  Although many files are updated through to 30 September 2024, it is possible there are some data gaps resulting from missing ingested data.  NOAA NCEI are working through these, but it could take a while for all data gaps to be filled.  More may be apparent in the next release (v202410p).

It is also not yet clear if the date scheduled for the termination of updates to the ISD (31st October 2024 as last I heard) will now move later.  We intend to continue updating HadISD for as long as updates are appended to the ISD.

Delays to HadISD updates

Following Hurricane Helene's passage over North Carolina, the extensive flooding in the Asheville area has caused an outage of some of the NOAA-NCEI websites and datasets.  This means that the update to HadISD due early October 2024 (v3.4.1.202409p) will be delayed until these services are up and running again.

Looking towards GHCNH

Last week NCEI announced the release of the GHCNH (Global Historical Climate Network Hourly) dataset:

https://www.ncei.noaa.gov/news/next-generation-climate-dataset-built-seamless-integration

The GHCNH replaces the ISD, and as such I'm still expecting the ISD to be turned off in the next few months.  This will obviously result in the HadISD having no further updates.  

As many of the QC tests being applied in GHCNH are based on those in the HadISD, it does not make sense to pass the GHCNH data through the HadISD QC system. Therefore the HadISD in its current form will transform to a static dataset, and at some point in the future, will be retired and archived (though this is some time off!).

At the moment there is no plan to immediately work on a wrapper for the GHCNH data and release in a "HadISD" format. However this may change in the future as we move to using GHCNH in other systems as well.

We'll post updates on this blog over the next months during the transition from ISD to GHCNH.

HadISD v3.4.0.2023f & future look

We released updated versions of HadISD, and this time two versions have been released at the same time.  As described in this post, we noted that the buddy/neighbour checks had not been running since 2018.  We have released a version of HadISD which correctly implements these checks as intended (v3.4.0.2023f), but for those who may wish to do their own comparison or use a version where these checks are absent as per the last few years of updates, then v3.3.1.202312p is also made available.

As we noted in our earlier post, the missing buddy checks also affect some of the other QC checks - predominantly those where a comparison with neighbouring stations can lead to flags set being removed. The Odd Cluster (Fig 1) and Climatological (Fig 2) checks show clear increases in the fractions of observations flagged by these checks across most stations.

Fig 1: Odd cluster checks for Dewpoint. Top - v3.3.1.202312p, Bottom - v3.4.0.2023f

Fig 2: Climatological Outlier checks for Tempeature. Top - v3.3.1.202312p, Bottom - v3.4.0.2023f

Although there is a general increase in the amount of observations flagged, most of these are in the lowest categories of fractions of the total record (to be expected).  We also expected changes in the flagging rates for the Distributional Gap check, but saw only very slight differences.

The other test with a clear impact is that of Dewpoint Depression (Fig 3).

Fig 1: Dewpoint Depression checks. Top - v3.3.1.202312p, Bottom - v3.4.0.2023f

Future Look

As noted in another earlier post, the ISD will be pausing updates during 2024.  The timeline for this is now looking like end March 2024 rather than being December 2023, and we'll post on here when we get further details.  In the meantime, we will continue HadISD updates (under v3.4.1.2024XXp) until ISD updates cease.

Pausing HadISD updates in 2024

The HadISD dataset builds on NOAA NCEI's ISD dataset. There is work underway to replace the ISD with a new GHCNh (Global Historical Climate Network Hourly) product at NOAA, which will sit alongside the existing daily and monthly products under the GHCN brand.

As a result of this, when the ISD is no longer operationally updated, the HadISD will also cease to be updated.  Once this happens (likely at the end of this calendar year - the original notice from NOAA is already out of date) we will produce a final version of the HadISD and leave this available for some time on the home page.  A version will also be lodged at CEDA as usual.  This will allow any monitoring occurring on a calendar-year basis to happen on a complete dataset.

In due course we may look into the new GHCNh product to see whether we can build a "HadGHCNh" product from that.  Many of the quality control tests are similar in this new GHCNh and so we will need to do some careful investigation to ensure we are not erroneously keeping bad or removing good values if we apply the HadISD QC suite on top of these already QC'd data.

Next steps

Given the issues with the buddy check described in a previous post, we intend to release two versions in early 2024:  

• v331_202312p which follow on from other versions, with the buddy checks not being applied
• v340_2023f where we will reinstate the buddy checks.

Thereafter updates to HadISD will cease for the foreseeable future. 

 

We hope the approach of these two releases will give clarity and consistency to users of HadISD, and also enable us to perform some further investigations on the impacts of the inclusion of the buddy checks (and corrected unflagging steps) on the data at this point.  Users can also ensure they pick a dataset version which is consistent with any other approaches they have done. It also means that those who are using HadISD for climate monitoring can assess the calendar year 2023 and then have time to plan to use GHCNh.

 

As always, if you see anything untoward in the HadISD, do let us know!

Bug in the Buddy Checks

We have recently the noticed that the checks using the neighbouring stations in the HadISD are not running as intended, and are setting no flags at all (see Fig. 1 and also e.g. v331_202309p_Buddy_check).  It appears this has been the case since v202_2017p in 2018!  Although the initial releases of version 2 did include buddy checks, adaptations to run on a new job management system resulted in an bug where the data being read in for the buddy station was identical to the target station being assessed.  Unfortuntately we have only just picked this up.

This error affects the temperature, dew point and sea-level pressure variables which would use the buddy check to identify further spurious values.  We show differences between v201_2016f and v202_2017f in Fig. 1 (to keep changes to station counts to a minimum), which clearly demonstrates the effect of this error.  Although the majority of stations would only have had a few observations (<0.1% of the total in their record) flagged by this test, it is pervasive across all continents.

Fig. 1: Flagging rates for temperature neighbour check, Top - v201_2016f, Bottom - v202_2017f

Also, the neighbours are used to help unset some flags (tentatively) identified by earlier checks.  If there are insufficient neighbours, no unsetting occurs.  However, where there are enough neighbours, then as these contain identical data to the target station unflagging occurs as the observations from the neighbours appear to be a sufficiently good match.

This affects the climatological (temperature & dew point), distributional gap (temperature, dew point & SLP), odd cluster (temperature, dew point & SLP but not wind speed) and dew point depression checks.  The greatest reduction in numbers of observations flagged by any test are in the odd cluster and dew point depression checks (see Figs. 2 & 3) with lesser impacts in the climatological, and minor ones in the gap check.

Fig. 2: Flagging rates for temperature odd cluster check, Top - v201_2016f, Bottom - v202_2017f

Fig. 3: Flagging rates for dewpoint depression check, Top - v201_2016f, Bottom - v202_2017f

In terms of the impact on the dataset as a whole, the absence of the buddy checks along with the additional erroneous unflagging means that the data are not as clean and quality controlled as we had hoped (and have been stating).  We extend heartfelt apologies to all users.

However, there are no other impacts on the data other than some erroneous values are not being flagged that should be.  Although the set of automated QC tests applied to the HadISD would never have been a perfect system, we're sorry that it has not been running as effectively for the last few years.  The way the QC suite was designed is that individual observations can be flagged by many different tests.  Therefore, although some tests are not working as we had intended, in many cases, erroneous observations will be being flagged by other tests. The overall flagging rates across all tests are very similar (Fig. 4), but depending on the application, those values which are currently retained in error may be important. 

Fig. 4: Flagging rates for all dew point checks combined, Top - v201_2016f, Bottom - v202_2017f

As the dataset has been run with this error for a number of years (since 2018), we have decided to continue updates as they have been, i.e. without the buddy checks running, at this point in time for consistency with previous releases.  Given the pause to HadISD updates in early 2024 (see separate post), there are reasons for this approach.

Next steps

Given the issues with the buddy check described here and the forthcoming pause to HadISD updates, we intend to release two versions in early 2024:  

• v331_202312p which follow on from other versions, with the buddy checks not being applied
• v340_2023f where we will reinstate the buddy checks.

We hope this will give clarity and consistency to users of HadISD, and also enable us to perform some further investigations on the impacts of the inclusion of the buddy checks (and corrected unflagging steps) on the data at this point.  Users can also ensure they pick a dataset version which is consistent with any other approaches they have done.

Correction to T_wet calculation in the humidity files

Following the change to the formula used in HadISDH to calculate the wet-bulb temperature (see details on the HadISDH blog) we updated the forumla used for the humidity data files in HadISD for versions v3.3.0.2022f onwards.

It has recently come to our attention that in doing so we introduced a bug into how this updated formula was being called, and so the Twet values for versions v3.3.0.2022f to v3.3.1.202307p were incorrect (an ice bulb vapour pressure was being used in the call to the routine).  This has now been corrected in v3.3.1.202308p.