Delays to HadISD updates

Following Hurricane Helene's passage over North Carolina, the extensive flooding in the Asheville area has caused an outage of some of the NOAA-NCEI websites and datasets.  This means that the update to HadISD due early October 2024 (v3.4.1.202409p) will be delayed until these services are up and running again.

Looking towards GHCNH

Last week NCEI announced the release of the GHCNH (Global Historical Climate Network Hourly) dataset:

https://www.ncei.noaa.gov/news/next-generation-climate-dataset-built-seamless-integration

The GHCNH replaces the ISD, and as such I'm still expecting the ISD to be turned off in the next few months.  This will obviously result in the HadISD having no further updates.  

As many of the QC tests being applied in GHCNH are based on those in the HadISD, it does not make sense to pass the GHCNH data through the HadISD QC system. Therefore the HadISD in its current form will transform to a static dataset, and at some point in the future, will be retired and archived (though this is some time off!).

At the moment there is no plan to immediately work on a wrapper for the GHCNH data and release in a "HadISD" format. However this may change in the future as we move to using GHCNH in other systems as well.

We'll post updates on this blog over the next months during the transition from ISD to GHCNH.

HadISD v3.4.0.2023f & future look

We released updated versions of HadISD, and this time two versions have been released at the same time.  As described in this post, we noted that the buddy/neighbour checks had not been running since 2018.  We have released a version of HadISD which correctly implements these checks as intended (v3.4.0.2023f), but for those who may wish to do their own comparison or use a version where these checks are absent as per the last few years of updates, then v3.3.1.202312p is also made available.

As we noted in our earlier post, the missing buddy checks also affect some of the other QC checks - predominantly those where a comparison with neighbouring stations can lead to flags set being removed. The Odd Cluster (Fig 1) and Climatological (Fig 2) checks show clear increases in the fractions of observations flagged by these checks across most stations.

Fig 1: Odd cluster checks for Dewpoint. Top - v3.3.1.202312p, Bottom - v3.4.0.2023f

Fig 2: Climatological Outlier checks for Tempeature. Top - v3.3.1.202312p, Bottom - v3.4.0.2023f

Although there is a general increase in the amount of observations flagged, most of these are in the lowest categories of fractions of the total record (to be expected).  We also expected changes in the flagging rates for the Distributional Gap check, but saw only very slight differences.

The other test with a clear impact is that of Dewpoint Depression (Fig 3).

Fig 1: Dewpoint Depression checks. Top - v3.3.1.202312p, Bottom - v3.4.0.2023f

Future Look

As noted in another earlier post, the ISD will be pausing updates during 2024.  The timeline for this is now looking like end March 2024 rather than being December 2023, and we'll post on here when we get further details.  In the meantime, we will continue HadISD updates (under v3.4.1.2024XXp) until ISD updates cease.

Pausing HadISD updates in 2024

The HadISD dataset builds on NOAA NCEI's ISD dataset. There is work underway to replace the ISD with a new GHCNh (Global Historical Climate Network Hourly) product at NOAA, which will sit alongside the existing daily and monthly products under the GHCN brand.

As a result of this, when the ISD is no longer operationally updated, the HadISD will also cease to be updated.  Once this happens (likely at the end of this calendar year - the original notice from NOAA is already out of date) we will produce a final version of the HadISD and leave this available for some time on the home page.  A version will also be lodged at CEDA as usual.  This will allow any monitoring occurring on a calendar-year basis to happen on a complete dataset.

In due course we may look into the new GHCNh product to see whether we can build a "HadGHCNh" product from that.  Many of the quality control tests are similar in this new GHCNh and so we will need to do some careful investigation to ensure we are not erroneously keeping bad or removing good values if we apply the HadISD QC suite on top of these already QC'd data.

Next steps

Given the issues with the buddy check described in a previous post, we intend to release two versions in early 2024:  

• v331_202312p which follow on from other versions, with the buddy checks not being applied
• v340_2023f where we will reinstate the buddy checks.

Thereafter updates to HadISD will cease for the foreseeable future. 

 

We hope the approach of these two releases will give clarity and consistency to users of HadISD, and also enable us to perform some further investigations on the impacts of the inclusion of the buddy checks (and corrected unflagging steps) on the data at this point.  Users can also ensure they pick a dataset version which is consistent with any other approaches they have done. It also means that those who are using HadISD for climate monitoring can assess the calendar year 2023 and then have time to plan to use GHCNh.

 

As always, if you see anything untoward in the HadISD, do let us know!

Bug in the Buddy Checks

We have recently the noticed that the checks using the neighbouring stations in the HadISD are not running as intended, and are setting no flags at all (see Fig. 1 and also e.g. v331_202309p_Buddy_check).  It appears this has been the case since v202_2017p in 2018!  Although the initial releases of version 2 did include buddy checks, adaptations to run on a new job management system resulted in an bug where the data being read in for the buddy station was identical to the target station being assessed.  Unfortuntately we have only just picked this up.

This error affects the temperature, dew point and sea-level pressure variables which would use the buddy check to identify further spurious values.  We show differences between v201_2016f and v202_2017f in Fig. 1 (to keep changes to station counts to a minimum), which clearly demonstrates the effect of this error.  Although the majority of stations would only have had a few observations (<0.1% of the total in their record) flagged by this test, it is pervasive across all continents.

Fig. 1: Flagging rates for temperature neighbour check, Top - v201_2016f, Bottom - v202_2017f

Also, the neighbours are used to help unset some flags (tentatively) identified by earlier checks.  If there are insufficient neighbours, no unsetting occurs.  However, where there are enough neighbours, then as these contain identical data to the target station unflagging occurs as the observations from the neighbours appear to be a sufficiently good match.

This affects the climatological (temperature & dew point), distributional gap (temperature, dew point & SLP), odd cluster (temperature, dew point & SLP but not wind speed) and dew point depression checks.  The greatest reduction in numbers of observations flagged by any test are in the odd cluster and dew point depression checks (see Figs. 2 & 3) with lesser impacts in the climatological, and minor ones in the gap check.

Fig. 2: Flagging rates for temperature odd cluster check, Top - v201_2016f, Bottom - v202_2017f

Fig. 3: Flagging rates for dewpoint depression check, Top - v201_2016f, Bottom - v202_2017f

In terms of the impact on the dataset as a whole, the absence of the buddy checks along with the additional erroneous unflagging means that the data are not as clean and quality controlled as we had hoped (and have been stating).  We extend heartfelt apologies to all users.

However, there are no other impacts on the data other than some erroneous values are not being flagged that should be.  Although the set of automated QC tests applied to the HadISD would never have been a perfect system, we're sorry that it has not been running as effectively for the last few years.  The way the QC suite was designed is that individual observations can be flagged by many different tests.  Therefore, although some tests are not working as we had intended, in many cases, erroneous observations will be being flagged by other tests. The overall flagging rates across all tests are very similar (Fig. 4), but depending on the application, those values which are currently retained in error may be important. 

Fig. 4: Flagging rates for all dew point checks combined, Top - v201_2016f, Bottom - v202_2017f

As the dataset has been run with this error for a number of years (since 2018), we have decided to continue updates as they have been, i.e. without the buddy checks running, at this point in time for consistency with previous releases.  Given the pause to HadISD updates in early 2024 (see separate post), there are reasons for this approach.

Next steps

Given the issues with the buddy check described here and the forthcoming pause to HadISD updates, we intend to release two versions in early 2024:  

• v331_202312p which follow on from other versions, with the buddy checks not being applied
• v340_2023f where we will reinstate the buddy checks.

We hope this will give clarity and consistency to users of HadISD, and also enable us to perform some further investigations on the impacts of the inclusion of the buddy checks (and corrected unflagging steps) on the data at this point.  Users can also ensure they pick a dataset version which is consistent with any other approaches they have done.

Correction to T_wet calculation in the humidity files

Following the change to the formula used in HadISDH to calculate the wet-bulb temperature (see details on the HadISDH blog) we updated the forumla used for the humidity data files in HadISD for versions v3.3.0.2022f onwards.

It has recently come to our attention that in doing so we introduced a bug into how this updated formula was being called, and so the Twet values for versions v3.3.0.2022f to v3.3.1.202307p were incorrect (an ice bulb vapour pressure was being used in the call to the routine).  This has now been corrected in v3.3.1.202308p.

Calm winds in ISD, HadISD and GSOD

This post summarises an issue found earlier this year in the representation of calm periods (0 m/s) in the wind speed fields of ISD.  For full details, see our recently published paper on this at Environmental Research Communications.

We noted that in plots of the regional (and global) average wind speed, used in the BAMS State of the Climate report, that there was a significant inhomogeneity especially in Asian regions, see Figure 1, and wanted to understand the cause.

Fig 1: Time series of global and regional annual average wind speeds taken from stations in the HadISD.  For more details see Dunn et al, (2022) and the Surface Winds section in the BAMS State of the Climate.

The inhomgeneity is more prominent when looking at the calm fraction (percent of non-missing observations which measure 0m/s), see Figure 2. The drastic reduction for the two Asian regions, and also over Europe in 2013 is unlikely to be a natural feature of the climate at that date.

Fig 2: Time series of global and regional annual calm fraction taken from stations in the HadISD.

In looking more deeply at example stations, we noted that after 1 May 2013, there were no periods of 0m/s wind speed in the station time series in HadISD (Figure 3).  And this is a wide spread issue for stations across the globe (Figure 4).

Fig 3: Sub-daily wind measurements for the HadISD station 226760-99999 over its complete record.  [Sura, Russia, 63.58 N, 45.63E, 62.0m a.s.l.]

Fig 4a: Calm fraction for 2012

Fig 4b: Calm fraction for 2014

By tracing this back, we found this was also the case in the ISD, suggesting it has not been noticed, and affects all downstream products of the ISD (including HadISD and GSOD, the Global Summary of the Day). Investigations by our colleagues at NOAA/NCEI and their contacts in the USAF Weather Squadron found the issue as being an error in how calm winds were encoded in their outputs from the GTS.  This started on 1st May 2013, and has been corrected from 15th March 2022 going forwards.  Work is being done to correct the intevening years, and release that data into the databases, but this is still being done.

In HadISD, we can use the measurement flag which is in the ISD data files to recover calm periods assigned as missing.  This could also recover true missing data where the measurement flag has been erroneously given the value of calm, but we beleive this to be a small fraction of the observations.

By applying this simple correction to HadISD, we recover calm periods in stations between 2013 and 2022.  For those who use surface winds in their analyses, the addition of these calm periods (which used to be represented by missing data) will reduce the average wind speeds over this time range.  We show in Figure 5 how this impacts the global and regional time series, when compared to Figure 1.  There is a reduction in the magnitude of the reversal in global average surface wind speeds, which has been observed since around 2010.

 

Fig 5: Time series of global and regional annual average wind speeds taken from stations in the corrected HadISD.

There are other studies using independent data sources which reproduce both the long term decline of wind speeds since the beginning of the bulk of the HadISD records (1973) until around 2010, and also the slight reversal in global wind speeds since that date.  However, by including these previously missing calm periods means the magnitude of this reversal is reduced by around 30%.

For more details, please read the paper linked below (Open Access) or get in touch.

Reduction in reversal of global stilling arising from correction to encoding of calm periods, Dunn, Azorin-Molina, Menne, Zeng, Casey & Shen, ERC, 2021, https://iopscience.iop.org/article/10.1088/2515-7620/ac770a