tag:blogger.com,1999:blog-23939735365762538662024-02-22T10:18:20.727+00:00HadISD: interesting features of sub-daily climate dataNews, updates and interesting features of the HadISD datasetRobert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.comBlogger44125tag:blogger.com,1999:blog-2393973536576253866.post-73026386950823932382024-01-15T12:33:00.002+00:002024-01-15T12:33:30.154+00:00HadISD v3.4.0.2023f & future look<p><span style="font-family: arial;">We released updated versions of HadISD, and this time two versions have been released at the same time. As described in <a href="https://hadisd.blogspot.com/2023/10/bug-in-buddy-checks.html">this post</a>, we noted that the buddy/neighbour checks had not been running since 2018. We have released a version of HadISD which correctly implements these checks as intended (<a href="https://www.metoffice.gov.uk/hadobs/hadisd/v340_2023f/index.html">v3.4.0.2023f</a>), but for those who may wish to do their own comparison or use a version where these checks are absent as per the last few years of updates, then <a href="https://www.metoffice.gov.uk/hadobs/hadisd/v331_202312p/index.html">v3.3.1.202312p</a> is also made available.</span></p><p><span style="font-family: arial;">As we noted in our earlier post, the missing buddy checks also affect some of the other QC checks - predominantly those where a comparison with neighbouring stations can lead to flags set being removed. The Odd Cluster (Fig 1) and Climatological (Fig 2) checks show clear increases in the fractions of observations flagged by these checks across most stations.</span></p><div class="separator" style="clear: both; text-align: center;"><div style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI2DXav4_Mmx-x_l3rAhpXCCjpYT0dyoYpxAFPdqQxko9RH9KDQDzzOPOwzD37S0Uax98TZeaBquJ_4Glav-rdU7dB9JuHP4QKXgDR3Ws8dnbz9-IDdkxJdG3kA83ZThcd5XhH39Br75hQWlC1jbDf06VSWvXZ1w47qf-8oIU83qXua4rsp7rsq8gpSbE/s800/All_fails_OCD_20240105.png" imageanchor="1" style="font-family: arial;"><img border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI2DXav4_Mmx-x_l3rAhpXCCjpYT0dyoYpxAFPdqQxko9RH9KDQDzzOPOwzD37S0Uax98TZeaBquJ_4Glav-rdU7dB9JuHP4QKXgDR3Ws8dnbz9-IDdkxJdG3kA83ZThcd5XhH39Br75hQWlC1jbDf06VSWvXZ1w47qf-8oIU83qXua4rsp7rsq8gpSbE/w400-h300/All_fails_OCD_20240105.png" width="400" /></a></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAbffGKPzGMvQE4XeYw2fPJITS3VYkG7ViaRYLgcU4PGZOxNQMbYSiuh_1PaWSZaOyVxdmSjBL0lqn8l1J8m7x2cd_18NXyBluA52I9p4j-gW6KANPxiS5OzdWzRL7y0IAhbKiCKoqUvmBU9lPVayIEs-LoXzjCRPwSr26B_THboOBa5Dlo2xGXIuzWoU/s800/All_fails_OCD_20240109.png" imageanchor="1" style="font-family: arial; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAbffGKPzGMvQE4XeYw2fPJITS3VYkG7ViaRYLgcU4PGZOxNQMbYSiuh_1PaWSZaOyVxdmSjBL0lqn8l1J8m7x2cd_18NXyBluA52I9p4j-gW6KANPxiS5OzdWzRL7y0IAhbKiCKoqUvmBU9lPVayIEs-LoXzjCRPwSr26B_THboOBa5Dlo2xGXIuzWoU/w400-h300/All_fails_OCD_20240109.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig 1: Odd cluster checks for Dewpoint. Top - v3.3.1.202312p, Bottom - v3.4.0.2023f<br /></span></td></tr></tbody></table></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgi3oG9Fj9I-szxCDaLdbz-Rl0aY-ZLDoo4eF3-JPyLevkt1O7-YhA2YGGtGpRkulP2qpc-TJm5pwRP20bR8jluwKxytC7VfVtp2VmpCtjNMZjvD4jTztqUvTkuu2yd8J_cd5J_Q90P4S1_IduXc3zqVugqoht0UEJaGyJn6TKpNJCNCMqfqjZmoSaXSio/s800/All_fails_TCM_20240105.png" imageanchor="1" style="font-family: arial; margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgi3oG9Fj9I-szxCDaLdbz-Rl0aY-ZLDoo4eF3-JPyLevkt1O7-YhA2YGGtGpRkulP2qpc-TJm5pwRP20bR8jluwKxytC7VfVtp2VmpCtjNMZjvD4jTztqUvTkuu2yd8J_cd5J_Q90P4S1_IduXc3zqVugqoht0UEJaGyJn6TKpNJCNCMqfqjZmoSaXSio/w400-h300/All_fails_TCM_20240105.png" width="400" /></a></div><span style="font-family: arial;"><br /></span><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiikoR1ZlsIWBO8Y_xXe-1AMB00YKDeJxFsIwzoo_Dx_DVavBE5I-U8pvAXNIXTgkT6NkqGHdTLx0rne7vxCfiazM1v7LOd5hAQMGhfqPJPo2cTJ78MBquR-QNQ3ABMYk-PNFm0pN8PQWaIK_Mzk4wDO4KvFV-CZ2hTqUTv8tSr7ab4J2w993XyhtPx338/s800/All_fails_TCM_20240109.png" imageanchor="1" style="font-family: arial; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiikoR1ZlsIWBO8Y_xXe-1AMB00YKDeJxFsIwzoo_Dx_DVavBE5I-U8pvAXNIXTgkT6NkqGHdTLx0rne7vxCfiazM1v7LOd5hAQMGhfqPJPo2cTJ78MBquR-QNQ3ABMYk-PNFm0pN8PQWaIK_Mzk4wDO4KvFV-CZ2hTqUTv8tSr7ab4J2w993XyhtPx338/w400-h300/All_fails_TCM_20240109.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig 2: Climatological Outlier checks for Tempeature. Top - v3.3.1.202312p, Bottom - v3.4.0.2023f<br /></span></td></tr></tbody></table><span style="font-family: arial;"><br /></span><p><span style="font-family: arial;">Although there is a general increase in the amount of observations flagged, most of these are in the lowest categories of fractions of the total record (to be expected). We also expected changes in the flagging rates for the Distributional Gap check, but saw only very slight differences.</span></p><p><span style="font-family: arial;">The other test with a clear impact is that of Dewpoint Depression (Fig 3).</span></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXaJhcuG8uOqL7nIrPSnx_YCbron5pK6rvFus4LQwk_Ih96wCwulKb77xyspsdr0VpDcl2QZOh1KWX3FMAvDbdbcyh5Jdq3CWXmUjPTUYCGTeu74UCx8Rc1E0rXv5GuR107BipwicSAG0BgPsDGZAwVRuAiUYPh4oInEDLuwY8KqqikWf7jSw2fbCuagU/s800/All_fails_DPD_20240105.png" imageanchor="1" style="font-family: arial; margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXaJhcuG8uOqL7nIrPSnx_YCbron5pK6rvFus4LQwk_Ih96wCwulKb77xyspsdr0VpDcl2QZOh1KWX3FMAvDbdbcyh5Jdq3CWXmUjPTUYCGTeu74UCx8Rc1E0rXv5GuR107BipwicSAG0BgPsDGZAwVRuAiUYPh4oInEDLuwY8KqqikWf7jSw2fbCuagU/w400-h300/All_fails_DPD_20240105.png" width="400" /></a></div><span style="font-family: arial;"><br /></span><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiW2KcWDEQfYP3WZZouoDwElo9OO-RHaLcmAfed54IRI7lM_6smxRyy1fbJaVO8dLaizbnn6Y_MrXY5etc0lv-a5ofSI65w0s7lnoPr3IlO_xPBtlWhRU1axCv8Mx4dYy6xwMQ8daGdCbPTG30JhiGdrjLgHkzgTQOzgLVH45ZOy3SMhUuvGDdHKMaeMJc/s800/All_fails_DPD_20240109.png" imageanchor="1" style="font-family: arial; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiW2KcWDEQfYP3WZZouoDwElo9OO-RHaLcmAfed54IRI7lM_6smxRyy1fbJaVO8dLaizbnn6Y_MrXY5etc0lv-a5ofSI65w0s7lnoPr3IlO_xPBtlWhRU1axCv8Mx4dYy6xwMQ8daGdCbPTG30JhiGdrjLgHkzgTQOzgLVH45ZOy3SMhUuvGDdHKMaeMJc/w400-h300/All_fails_DPD_20240109.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig 1: Dewpoint Depression checks. Top - v3.3.1.202312p, Bottom - v3.4.0.2023f<br /></span></td></tr></tbody></table><span style="font-family: arial;"><br /></span><p></p><h3 style="text-align: left;"><span style="font-family: arial;">Future Look<br /></span></h3><p></p><p><span style="font-family: arial;">As noted in another <a href="https://hadisd.blogspot.com/2023/10/pausing-hadisd-updates-in-2024.html">earlier post</a>, the ISD will be pausing updates during 2024. The timeline for this is now looking like end March 2024 rather than being December 2023, and we'll post on here when we get further details. In the meantime, we will continue HadISD updates (under v3.4.1.2024XXp) until ISD updates cease.<br /><br /><br /></span></p><p></p>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-71395155199589899132023-10-11T15:46:00.003+01:002023-10-11T15:46:51.792+01:00Pausing HadISD updates in 2024<p><span style="font-family: arial;">T</span><span style="font-family: arial;">he HadISD dataset builds on <a href="https://www.ncei.noaa.gov/">NOAA NCEI</a>'s <a href="https://www.ncei.noaa.gov/products/land-based-station/integrated-surface-database">ISD</a> dataset. There is work underway to <a href="https://www.nesdis.noaa.gov/about/documents-reports/notice-of-changes">replace the ISD with a new GHCNh</a>
(Global Historical Climate Network Hourly) product at NOAA, which will
sit alongside the existing daily and monthly products under the GHCN
brand.</span></p><p><span style="font-family: arial;">As
a result of this, when the ISD is no longer operationally updated, the
HadISD will also cease to be updated. Once this happens (likely at the
end of this calendar year - the original notice from NOAA is already out
of date) we will produce a final version of the HadISD and leave this
available for some time on the home page. A version will also be lodged
at <a href="https://catalogue.ceda.ac.uk/uuid/f579035b3c954475922e4b13705a7669">CEDA</a> as usual. This will allow any monitoring occurring on a calendar-year basis to happen on a complete dataset.<br /></span></p><p><span style="font-family: arial;">In
due course we may look into the new GHCNh product to see whether we can
build a "HadGHCNh" product from that. Many of the quality control
tests are similar in this new GHCNh and so we will need to do some
careful investigation to ensure we are not erroneously keeping bad or
removing good values if we apply the HadISD QC suite on top of these
already QC'd data.<br /></span></p><p></p><p><span style="font-family: arial;"></span></p><h3 style="text-align: left;"><span style="font-family: arial;">Next steps<br /></span></h3><div style="text-align: left;"><p><span style="font-family: arial;"><span style="font-weight: normal;">Given the issues with the <a href="https://hadisd.blogspot.com/2023/10/bug-in-buddy-checks.html">buddy check described in a previous post</a>, we intend to release two versions in early 2024: </span></span></p><ul style="text-align: left;"><li><span style="font-family: arial;"><span style="font-weight: normal;">v331_202312p which follow on from other versions, with the buddy checks not being applied</span></span></li><li><span style="font-family: arial;"><span style="font-weight: normal;">v340_2023f where we will reinstate the buddy checks.</span></span></li></ul><span style="font-family: arial;"><span style="font-weight: normal;">Thereafter updates to HadISD will cease for the foreseeable future. </span></span></div><div style="text-align: left;"><span style="font-family: arial;"><span style="font-weight: normal;"> </span></span></div><div style="text-align: left;"><span style="font-family: arial;"><span style="font-weight: normal;">We
hope the approach of these two releases will give clarity and consistency to users of HadISD, and
also enable us to perform some further investigations on the impacts of
the inclusion of the buddy checks (and corrected unflagging steps) on
the data at this point. Users can also ensure they pick a dataset
version which is consistent with any other approaches they have done. It also means that those who are using HadISD for climate monitoring can assess the calendar year 2023 and then have time to plan to use GHCNh.<br /></span></span></div><div style="text-align: left;"><span style="font-family: arial;"><span style="font-weight: normal;"> </span></span></div><span style="font-family: arial;"><span style="font-weight: normal;">As always, if you see anything untoward in the HadISD, do let us know! </span></span>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-75731187465129942792023-10-11T15:44:00.001+01:002023-10-11T15:47:26.413+01:00Bug in the Buddy Checks<p><span style="font-family: arial;">We have recently the noticed that the checks using the neighbouring stations in the HadISD are not running as intended, and are setting no flags at all (see Fig. 1 and also e.g. <a href="https://www.metoffice.gov.uk/hadobs/hadisd/v331_202309p/images/All_fails_TOT_20231005.png">v331_202309p_Buddy_check</a>). </span><span style="font-family: arial;">It appears this has been the case since v202_2017p in 2018! </span><span style="font-family: arial;">Although
the initial releases of version 2 did include buddy checks, adaptations
to run on a new job management system resulted in an bug where the
data being read in for the buddy station was identical to the target
station being assessed. Unfortuntately we have only just picked this up.<br /></span></p><p><span style="font-family: arial;">This error affects the <u>temperature</u>, <u>dew point </u>and <u>sea-level pressure</u> variables which would use the buddy check to identify further spurious values. We show differences between v201_2016f and v202_2017f in Fig. 1 (to keep changes to station counts to a minimum), which clearly demonstrates the effect of this error. Although the majority of stations would only have had a few observations (<0.1% of the total in their record) flagged by this test, it is pervasive across all continents.<br /></span></p><p></p><div style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2tWCpQFipPjMEjUMnvSJAv8MnfOc3yiOF3zuLkSGk_Kn9HHQ1ALyzLRHm-zahXeEb2tzuGQkgzvI2bop9hCA2q_yfeTrqdgd6chMqRcJYb9p_R7iA8vFX2gwHB0_ttvDxcqKq9Sh0NphYCMezawfmayzBF_L4UOamWqrc9WitVAEoo9rIwk_B7WZqi1Q/s800/All_fails_TOT_20170330.png"><img alt="https://www.metoffice.gov.uk/hadobs/hadisd/v201_2016f/images/All_fails_TOT_20170330.png" border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2tWCpQFipPjMEjUMnvSJAv8MnfOc3yiOF3zuLkSGk_Kn9HHQ1ALyzLRHm-zahXeEb2tzuGQkgzvI2bop9hCA2q_yfeTrqdgd6chMqRcJYb9p_R7iA8vFX2gwHB0_ttvDxcqKq9Sh0NphYCMezawfmayzBF_L4UOamWqrc9WitVAEoo9rIwk_B7WZqi1Q/w400-h300/All_fails_TOT_20170330.png" width="400" /></a></div><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaOmwAGS0SlnrxgBBp57XjofxEq9pdLyqptievKY-qAWPVdMdl1vPVpk6pQzlo-3GZekXUwQdfixXS-lqFkq-1GZBT7tWMEWoOTMzFSoK4T5ndF8XFH8BWzOtwmiuhjx9P5mobmfHgWuqdffHxRwXbsYrS8l8eaBDaor_44LjAd9-tErGDsuq5Wtg344U/s800/All_fails_TOT_20180314.png" style="margin-left: auto; margin-right: auto;"><img alt="https://www.metoffice.gov.uk/hadobs/hadisd/v202_2017f/images/All_fails_TOT_20180314.png" border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaOmwAGS0SlnrxgBBp57XjofxEq9pdLyqptievKY-qAWPVdMdl1vPVpk6pQzlo-3GZekXUwQdfixXS-lqFkq-1GZBT7tWMEWoOTMzFSoK4T5ndF8XFH8BWzOtwmiuhjx9P5mobmfHgWuqdffHxRwXbsYrS8l8eaBDaor_44LjAd9-tErGDsuq5Wtg344U/w400-h300/All_fails_TOT_20180314.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig. 1: Flagging rates for temperature neighbour check, Top - v201_2016f, Bottom - v202_2017f<br /></span></td></tr></tbody></table><br /><span style="font-family: arial;"></span><p></p><p><span style="font-family: arial;">Also, the neighbours are used to help unset some flags (tentatively) identified by earlier checks. If there are insufficient neighbours, no unsetting occurs. However, where there are enough neighbours, then as these contain identical data to the target station unflagging occurs as the observations from the neighbours appear to be a sufficiently good match. <br /></span></p><p><span style="font-family: arial;">This </span><span style="font-family: arial;">affects the <u>climatological</u> (temperature & dew point), <u>distributional gap</u> (temperature, dew point & SLP), <u>odd cluster</u> (</span><span style="font-family: arial;">temperature, dew point & SLP but </span><span style="font-family: arial;">not wind speed) and <u>dew point depression</u> checks. The greatest reduction in numbers of observations flagged by any test are in the odd cluster and dew point depression checks (see Figs. 2 & 3) with lesser impacts in the climatological, and minor ones in the gap check.</span></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8lHTd-_wcbXzWQLs1tOb31IXQfCMJ6TAge25EMX2kG11OjI7cDbsFEMVLVGqsKcA5I8VP_b419bC-OSdo3bQwtbhH58X9SBrg_xHTqnw63Obd6h-KHkY1bkGsZkevG72M1KzkAunu8ZXEQqj0n0bnS_u8-OyA5uoSN2vE5C8sROtVAtNC6vrUtnC4PXk/s800/All_fails_OCT_20170330.png" style="margin-left: 1em; margin-right: 1em;"><img alt="https://www.metoffice.gov.uk/hadobs/hadisd/v201_2016f/images/All_fails_OCT_20170330.png" border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8lHTd-_wcbXzWQLs1tOb31IXQfCMJ6TAge25EMX2kG11OjI7cDbsFEMVLVGqsKcA5I8VP_b419bC-OSdo3bQwtbhH58X9SBrg_xHTqnw63Obd6h-KHkY1bkGsZkevG72M1KzkAunu8ZXEQqj0n0bnS_u8-OyA5uoSN2vE5C8sROtVAtNC6vrUtnC4PXk/w400-h300/All_fails_OCT_20170330.png" width="400" /></a></div><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivrq1ph9vODO5p0NNsKEIRceD_yo1baMFzFue1_oKsrmnuyotvGGRXRlToTY9l9GNYvemQzYEHwr_OvUB8UzknfUX3rchHZyEapdBugS_j8PrgGJgGt5nz3RCtjkMaQMHQHiqBIpe4puhSzBGS3xlFaQY6Gn3tKw6uvNx0ND0SQU-A9kLrqVY53_hPiq0/s800/All_fails_OCT_20180314.png" style="margin-left: auto; margin-right: auto;"><img alt="https://www.metoffice.gov.uk/hadobs/hadisd/v202_2017f/images/All_fails_OCT_20180314.png" border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivrq1ph9vODO5p0NNsKEIRceD_yo1baMFzFue1_oKsrmnuyotvGGRXRlToTY9l9GNYvemQzYEHwr_OvUB8UzknfUX3rchHZyEapdBugS_j8PrgGJgGt5nz3RCtjkMaQMHQHiqBIpe4puhSzBGS3xlFaQY6Gn3tKw6uvNx0ND0SQU-A9kLrqVY53_hPiq0/w400-h300/All_fails_OCT_20180314.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig. 2: Flagging rates for temperature odd cluster check, Top - v201_2016f, Bottom - v202_2017f</span></td></tr></tbody></table><br /><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhD9M4mtSqK4ZS488iq5In1bRKw0WVUyTodqfzBguX7uHrwpFgLVN1CEfudGpXl870b-u_FM2DCNbZ7roqUln88NOhysuYNBPnIO89Ucpyts5SJnI47NSdGCFskr8O4FYl5O_fuQ374MDJF0UzvfQqfUw3vZFZEKLopINqFyNEfrhCGwa5zN6JkqJRSzEg/s800/All_fails_DPD_20170330.png" style="margin-left: 1em; margin-right: 1em;"><img alt="https://www.metoffice.gov.uk/hadobs/hadisd/v201_2016f/images/All_fails_DPD_20170330.png" border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhD9M4mtSqK4ZS488iq5In1bRKw0WVUyTodqfzBguX7uHrwpFgLVN1CEfudGpXl870b-u_FM2DCNbZ7roqUln88NOhysuYNBPnIO89Ucpyts5SJnI47NSdGCFskr8O4FYl5O_fuQ374MDJF0UzvfQqfUw3vZFZEKLopINqFyNEfrhCGwa5zN6JkqJRSzEg/w400-h300/All_fails_DPD_20170330.png" width="400" /></a></div><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgg-xSd7GhuILRbHQHvsngzzo1VIXUv5BH4fZmqiEJniOq-hTROjLOE9Jxuo_yKemD6FJftfn34j8fQPi3c34gNk1bl7C0GyaNj3HMfG8nv8tJWnh2XDhAdM178mkXSTv011qVwv4heZ7fqoI-CL6xzMO1ZoO1M5lzTgxelnT40WpZS4Ablv6qYTDHM1fc/s800/All_fails_DPD_20180314.png" style="margin-left: auto; margin-right: auto;"><img alt="https://www.metoffice.gov.uk/hadobs/hadisd/v202_2017f/images/All_fails_DPD_20180314.png" border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgg-xSd7GhuILRbHQHvsngzzo1VIXUv5BH4fZmqiEJniOq-hTROjLOE9Jxuo_yKemD6FJftfn34j8fQPi3c34gNk1bl7C0GyaNj3HMfG8nv8tJWnh2XDhAdM178mkXSTv011qVwv4heZ7fqoI-CL6xzMO1ZoO1M5lzTgxelnT40WpZS4Ablv6qYTDHM1fc/w400-h300/All_fails_DPD_20180314.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig. 3: </span><span style="font-family: arial;">Flagging rates for dewpoint depression check, Top - v201_2016f, Bottom - v202_2017f</span></td></tr></tbody></table><span style="font-family: arial;"></span><p></p><p><span style="font-family: arial;">In terms of the impact on the dataset as a whole, the absence of the buddy checks along with the additional erroneous unflagging means that the data are not as clean and quality controlled as we had hoped (and have been stating). We extend heartfelt apologies to all users. <br /></span></p><p><span style="font-family: arial;">However, there are no other impacts on the data other than some erroneous values are not being flagged that should be. Although the set of automated QC tests applied to the HadISD would never have been a perfect system, we're sorry that it has not been running as effectively for the last few years. The way the QC suite was designed is that individual observations can be flagged by many different tests. Therefore, although some tests are not working as we had intended, in many cases, erroneous observations will be being flagged by other tests. The overall flagging rates across all tests are very similar (Fig. 4), but depending on the application, those values which are currently retained in error may be important. <br /></span></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmH7Opnaf0wKeDKqRB0HWFxFc7RwkkD-_eD5TKz_TSBdWq5DTyAyX4bdgM2bi2u2mF71vrjSFBbqnGM1dIFz1JWerELiHWspVs58eSDedxgdboBvEOyQYLDj96hMv4GXqzfYvltUmplz3k1xQaqzWrUTL-hV_QlJ-rVsWrDZeX96SsQjg2yH2Dz2c2CoE/s800/All_fails_ALL_Td_20170330.png" style="margin-left: 1em; margin-right: 1em;"><img alt="https://www.metoffice.gov.uk/hadobs/hadisd/v201_2016f/images/All_fails_ALL_Td_20170330.png" border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmH7Opnaf0wKeDKqRB0HWFxFc7RwkkD-_eD5TKz_TSBdWq5DTyAyX4bdgM2bi2u2mF71vrjSFBbqnGM1dIFz1JWerELiHWspVs58eSDedxgdboBvEOyQYLDj96hMv4GXqzfYvltUmplz3k1xQaqzWrUTL-hV_QlJ-rVsWrDZeX96SsQjg2yH2Dz2c2CoE/w400-h300/All_fails_ALL_Td_20170330.png" width="400" /></a></div><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilnbU7t73ooqb9wKHdlb7B4_FpUZIrT488_fOaoBRnDSw7c6-kcu11xGBi0fsM3T0l5RjxOwefko9__ltyKVNqi0HCilZCEsE2jo3cQqZoYkfe-E6wIZZEvYkK4TsMp57hhWdYMenj9CmfBkskpuxOmIeBVgxrXGOMYfV24cj50DPkW8z5DiWAKTLzxC4/s800/All_fails_ALL_Td_20180314.png" style="margin-left: auto; margin-right: auto;"><img alt="https://www.metoffice.gov.uk/hadobs/hadisd/v202_2017f/images/All_fails_ALL_Td_20180314.png" border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilnbU7t73ooqb9wKHdlb7B4_FpUZIrT488_fOaoBRnDSw7c6-kcu11xGBi0fsM3T0l5RjxOwefko9__ltyKVNqi0HCilZCEsE2jo3cQqZoYkfe-E6wIZZEvYkK4TsMp57hhWdYMenj9CmfBkskpuxOmIeBVgxrXGOMYfV24cj50DPkW8z5DiWAKTLzxC4/w400-h300/All_fails_ALL_Td_20180314.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig. 4: </span><span style="font-family: arial;">Flagging rates for all dew point checks combined, Top - v201_2016f, Bottom - v202_2017f</span></td></tr></tbody></table><span style="font-family: arial;"><br /></span><p></p><p><span style="font-family: arial;">As the dataset has been run with this error for a number of years (since 2018), we have decided to continue updates as they have been, i.e. without the buddy checks running, at this point in time for consistency with previous releases. Given the pause to HadISD updates in early 2024 (see separate post), there are reasons for this approach.</span></p><h3 style="text-align: left;"><span style="font-family: arial;">Next steps<br /></span></h3><div style="text-align: left;"><p><span style="font-family: arial;"><span style="font-weight: normal;">Given the issues with the buddy check described here and the <a href="https://hadisd.blogspot.com/2023/10/pausing-hadisd-updates-in-2024.html">forthcoming pause to HadISD updates</a>, we intend to release two versions in early 2024: </span></span></p><ul style="text-align: left;"><li><span style="font-family: arial;"><span style="font-weight: normal;">v331_202312p which follow on from other versions, with the buddy checks not being applied</span></span></li><li><span style="font-family: arial;"><span style="font-weight: normal;">v340_2023f where we will reinstate the buddy checks.</span></span></li></ul><span style="font-family: arial;"><span style="font-weight: normal;">We
hope this will give clarity and consistency to users of HadISD, and
also enable us to perform some further investigations on the impacts of
the inclusion of the buddy checks (and corrected unflagging steps) on
the data at this point. Users can also ensure they pick a dataset
version which is consistent with any other approaches they have done.</span></span></div><div style="text-align: left;"><br /> </div><br />Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-48952325242327194962023-09-06T16:20:00.002+01:002023-09-06T16:20:38.788+01:00Correction to T_wet calculation in the humidity files<p><span style="font-family: arial;">Following the change to the formula used in HadISDH to calculate the wet-bulb temperature (see details on the <a href=" https://hadisdh.blogspot.com/2023/01/new-wetbulb-temperature-algorithm-for.html. ">HadISDH blog</a>) we updated the forumla used for the humidity data files in HadISD for versions v3.3.0.2022f onwards.</span></p><p><span style="font-family: arial;">It has recently come to our attention that in doing so we introduced a bug into how this updated formula was being called, and so the Twet values for versions v3.3.0.2022f to v3.3.1.202307p were incorrect (an ice bulb vapour pressure was being used in the call to the routine). This has now been corrected in <a href="https://www.metoffice.gov.uk/hadobs/hadisd/v331_202308p/index.html">v3.3.1.202308p</a>.<br /></span></p>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-48985152444555993602022-06-15T14:39:00.000+01:002022-06-15T14:39:37.630+01:00Calm winds in ISD, HadISD and GSOD<p><span style="font-family: arial;">This post summarises an issue found earlier this year in the representation of calm periods (0 m/s) in the wind speed fields of ISD. For full details, see our recently published paper on this at <a href="https://iopscience.iop.org/article/10.1088/2515-7620/ac770a" target="_blank">Environmental Research Communications</a>.</span></p><p><span style="font-family: arial;">We noted that in plots of the regional (and global) average wind speed, used in the <a href="https://www.ametsoc.org/ams/index.cfm/publications/bulletin-of-the-american-meteorological-society-bams/state-of-the-climate/">BAMS State of the Climate </a>report, that there was a significant inhomogeneity especially in Asian regions, see Figure 1, and wanted to understand the cause.<br /></span></p><p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><span style="font-family: arial;"><img border="0" data-original-height="480" data-original-width="640" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjp3pUVAiELGJhnbiJ7QMk8wWoxYrrIX9SDln1PlL3pq66_bThS4m4GhypQyu9u_B_29DbniI2Kq66R4O9fDRtrN5I3UHyByS-g6nJIPHTTgAm4892SQ5NIy1D6uIpR7_lQoHlQIPq61o3L2jWFXc6GhqBLjlNujz55xfErqWuu8GCLPxRm5oL4Lg3G/s320/Figure6.png" style="margin-left: auto; margin-right: auto;" width="320" /></span></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: small;"><span style="font-family: arial;">Fig 1: Time series of global and regional annual average wind speeds taken from stations in the HadISD. For more details see Dunn et al, (2022) and the Surface Winds section in the BAMS State of the Climate.</span></span></td><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;"><br /></span></td><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;"><br /></span></td></tr></tbody></table></p><div class="separator" style="clear: both; text-align: center;"><div style="text-align: left;"><span style="font-family: arial;">The inhomgeneity is more prominent when looking at the calm fraction (percent of non-missing observations which measure 0m/s), see Figure 2. The drastic reduction for the two Asian regions, and also over Europe in 2013 is unlikely to be a natural feature of the climate at that date.<br /></span></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><span style="font-family: arial;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgk7CwVfg1c1Z2D1vLxgiYa4XBlR9JJprfQr6Jwe-rZ2suwVr0uHyad8tIm4gNDJU4Pma7ylE_EGTu56oZAhz3KmjoHUrq-cs9rIGAcFQwErKR7rn6savwUgdkd1PmjOoC_bwXXQwasrjGoZWcoLr-AjE_ZnYsX-H_u80Z4U-g1PFBAof2hwxojMJHC/s640/Figure3.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgk7CwVfg1c1Z2D1vLxgiYa4XBlR9JJprfQr6Jwe-rZ2suwVr0uHyad8tIm4gNDJU4Pma7ylE_EGTu56oZAhz3KmjoHUrq-cs9rIGAcFQwErKR7rn6savwUgdkd1PmjOoC_bwXXQwasrjGoZWcoLr-AjE_ZnYsX-H_u80Z4U-g1PFBAof2hwxojMJHC/s320/Figure3.png" width="320" /></a></span></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig 2: Time series of global and regional annual calm fraction taken from stations in the HadISD.<br /></span></td></tr></tbody></table></div><span style="font-family: arial;">In looking more deeply at example stations, we noted that after 1 May 2013, there were no periods of 0m/s wind speed in the station time series in <a href="https://www.metoffice.gov.uk/hadobs/hadisd/">HadISD</a> (Figure 3). And this is a wide spread issue for stations across the globe (Figure 4). <br /></span><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><span style="font-family: arial;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPe9KVchPmZnG0bAxdg_A7EhHBgjsvbiRLXbkeRUsxFabvIkZHNGjSuzk1G2TNRus9P5qPfD7zn6GvdIHncSA3AMEzAfPVW06P8oFtFy-WtUGJQw3Q_tq4341TWPLhIGzxkXSuaC4fKSlF7Eu_q-yElG4mLsSz-msvaBGoyfT0cd1MMQX4uWF3m1X-/s640/Figure2a.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPe9KVchPmZnG0bAxdg_A7EhHBgjsvbiRLXbkeRUsxFabvIkZHNGjSuzk1G2TNRus9P5qPfD7zn6GvdIHncSA3AMEzAfPVW06P8oFtFy-WtUGJQw3Q_tq4341TWPLhIGzxkXSuaC4fKSlF7Eu_q-yElG4mLsSz-msvaBGoyfT0cd1MMQX4uWF3m1X-/s320/Figure2a.png" width="320" /></a></span></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;"><span style="font-size: small;">Fig 3: Sub-daily wind measurements for the HadISD station 226760-99999 over its complete record. [<span style="line-height: 107%;">Sura, Russia, 63.58 N, 45.63E, 62.0m a.s.l.]</span></span></span><!--[if gte mso 9]><xml>
<o:OfficeDocumentSettings>
<o:RelyOnVML/>
<o:AllowPNG/>
</o:OfficeDocumentSettings>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:DoNotShowPropertyChanges/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-GB</w:LidThemeOther>
<w:LidThemeAsian>X-NONE</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
<w:UseFELayout/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="false"
DefSemiHidden="false" DefQFormat="false" DefPriority="99"
LatentStyleCount="376">
<w:LsdException Locked="false" Priority="0" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 9"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="header"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footer"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index heading"/>
<w:LsdException Locked="false" Priority="35" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of figures"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope return"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="line number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="page number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of authorities"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="macro"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="toa heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 5"/>
<w:LsdException Locked="false" Priority="10" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Closing"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Signature"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="true"
UnhideWhenUsed="true" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Message Header"/>
<w:LsdException Locked="false" Priority="11" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Salutation"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Date"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Block Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Hyperlink"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="FollowedHyperlink"/>
<w:LsdException Locked="false" Priority="22" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Document Map"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Plain Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="E-mail Signature"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Top of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Bottom of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal (Web)"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Acronym"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Cite"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Code"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Definition"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Keyboard"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Preformatted"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Sample"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Typewriter"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Variable"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Table"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation subject"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="No List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Contemporary"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Elegant"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Professional"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Balloon Text"/>
<w:LsdException Locked="false" Priority="39" Name="Table Grid"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Theme"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" QFormat="true"
Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" QFormat="true"
Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" QFormat="true"
Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" QFormat="true"
Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" QFormat="true"
Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" QFormat="true"
Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" SemiHidden="true"
UnhideWhenUsed="true" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="TOC Heading"/>
<w:LsdException Locked="false" Priority="41" Name="Plain Table 1"/>
<w:LsdException Locked="false" Priority="42" Name="Plain Table 2"/>
<w:LsdException Locked="false" Priority="43" Name="Plain Table 3"/>
<w:LsdException Locked="false" Priority="44" Name="Plain Table 4"/>
<w:LsdException Locked="false" Priority="45" Name="Plain Table 5"/>
<w:LsdException Locked="false" Priority="40" Name="Grid Table Light"/>
<w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="46" Name="List Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Mention"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Smart Hyperlink"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Hashtag"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Unresolved Mention"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Smart Link"/>
</w:LatentStyles>
</xml><![endif]--><!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin-top:0cm;
mso-para-margin-right:0cm;
mso-para-margin-bottom:8.0pt;
mso-para-margin-left:0cm;
line-height:107%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;
mso-fareast-language:EN-US;}
</style>
<![endif]--></td></tr></tbody></table><span style="font-family: arial;"><br /></span><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><span style="font-family: arial;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnxwDzkC39cqYn2wRnGR1TSHlwvuMu0z550mzC9DVpFLAMl-sjm9uKqP6R22t-NUDmirWVyeM5i7jhg50h6RV-0Xj_Z1eLTzSRIFhNePesBkb2fjNND2yi6ha-rC2bbYbPDGe7eWklxtjKBO-WzSLWOJc8_vuxhRtplXP3R5chltppy5qGVajlsKIV/s800/Figure1a.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="500" data-original-width="800" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnxwDzkC39cqYn2wRnGR1TSHlwvuMu0z550mzC9DVpFLAMl-sjm9uKqP6R22t-NUDmirWVyeM5i7jhg50h6RV-0Xj_Z1eLTzSRIFhNePesBkb2fjNND2yi6ha-rC2bbYbPDGe7eWklxtjKBO-WzSLWOJc8_vuxhRtplXP3R5chltppy5qGVajlsKIV/s320/Figure1a.png" width="320" /></a></span></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;"><span style="font-size: small;">Fig 4a: Calm fraction for 2012</span><br /></span></td></tr></tbody></table><span style="font-family: arial;"></span><p></p><p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><span style="font-family: arial;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5VpASQOPdv1PPKeDkL79T87UeV2BR4FBosqMIOBpuRTeHAI1EJSsAYsq8L6AT2EivHc5Lo8XjLWu1n9IC2sWN3PhEI7morexNjeOBXzSy3yit3u7cGxFn0h05rLvJfYShTLOCaCuaY54ZLkafzubkHQ5k_cs7ewETJgrD3h1H10D6ddWVzIYG-rzZ/s800/Figure1b.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="500" data-original-width="800" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5VpASQOPdv1PPKeDkL79T87UeV2BR4FBosqMIOBpuRTeHAI1EJSsAYsq8L6AT2EivHc5Lo8XjLWu1n9IC2sWN3PhEI7morexNjeOBXzSy3yit3u7cGxFn0h05rLvJfYShTLOCaCuaY54ZLkafzubkHQ5k_cs7ewETJgrD3h1H10D6ddWVzIYG-rzZ/s320/Figure1b.png" width="320" /></a></span></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;"><span style="font-size: small;">Fig 4b: Calm fraction for 2014</span><br /></span></td></tr></tbody></table><span style="font-family: arial;">By tracing this back, we found this was also the case in the <a href="https://www.ncdc.noaa.gov/isd">ISD</a>, suggesting it has not been noticed, and affects all downstream products of the ISD (including HadISD and <a href="https://www.ncei.noaa.gov/access/metadata/landing-page/bin/iso?id=gov.noaa.ncdc:C005" target="_blank">GSOD</a>, the Global Summary of the Day). Investigations by our colleagues at <a href="https://www.ncei.noaa.gov/" target="_blank">NOAA/</a><a href="https://www.ncei.noaa.gov/">NCEI </a>and their contacts in the USAF Weather Squadron found the issue as being an error in how calm winds were encoded in their outputs from the GTS. This started on 1st May 2013, and has been corrected from 15th March 2022 going forwards. Work is being done to correct the intevening years, and release that data into the databases, but this is still being done.</span></p><p><span style="font-family: arial;">In HadISD, we can use the measurement flag which is in the ISD data files to recover calm periods assigned as missing. This could also recover true missing data where the measurement flag has been erroneously given the value of calm, but we beleive this to be a small fraction of the observations.</span></p><p><span style="font-family: arial;">By applying this simple correction to HadISD, we recover calm periods in stations between 2013 and 2022. For those who use surface winds in their analyses, the addition of these calm periods (which used to be represented by missing data) will reduce the average wind speeds over this time range. We show in Figure 5 how this impacts the global and regional time series, when compared to Figure 1. There is a reduction in the magnitude of the reversal in global average surface wind speeds, which has been observed since around 2010.<br /></span></p><p><span style="font-family: arial;"> </span><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><span style="font-family: arial;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgG-iVzpQjG0RD7qnpkULpIYyFFx0cV7blGSG47w9Zj-fAKth0lHc3O0B-HwEpYga9pnaqzU8lH3edcuKGy9Arhyi45VGlNoMlsV-miB-dklmzXPEpsf9b6JS2K7swgG1FNwJN-C7Kxg4yWlNGsIoR8GAC-0AG58I75KYFOnVT0dCnmNX8J6vZICO8L/s640/Figure6b.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgG-iVzpQjG0RD7qnpkULpIYyFFx0cV7blGSG47w9Zj-fAKth0lHc3O0B-HwEpYga9pnaqzU8lH3edcuKGy9Arhyi45VGlNoMlsV-miB-dklmzXPEpsf9b6JS2K7swgG1FNwJN-C7Kxg4yWlNGsIoR8GAC-0AG58I75KYFOnVT0dCnmNX8J6vZICO8L/s320/Figure6b.png" width="320" /></a></span></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig 5: Time series of global and regional annual average wind speeds taken from stations in the corrected HadISD.<br /></span></td></tr></tbody></table><span style="font-family: arial;"><br />There are other studies using independent data sources which reproduce both the long term decline of wind speeds since the beginning of the bulk of the HadISD records (1973) until around 2010, and also the slight reversal in global wind speeds since that date. However, by including these previously missing calm periods means the magnitude of this reversal is reduced by around 30%.<br /></span></p><p><span style="font-family: arial;">For more details, please read the paper linked below (Open Access) or get in touch.<br /></span></p><h1 class="wd-jnl-art-title" itemprop="headline"><span style="font-family: arial;"><span style="font-size: x-small;"><span style="font-weight: normal;">Reduction in reversal of global stilling arising from correction to encoding of calm periods, Dunn, Azorin-Molina, Menne, Zeng, Casey & Shen, ERC, 2021, <a href="https://iopscience.iop.org/article/10.1088/2515-7620/ac770a">https://iopscience.iop.org/article/10.1088/2515-7620/ac770a</a></span></span><span style="font-size: small;"><span style="font-size: x-small;"> </span><br /></span></span></h1>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-11479451257032690702022-02-09T15:10:00.000+00:002022-02-09T15:10:08.466+00:00HadISD 3.3.0.202201p<p><span style="font-family: arial;">A new year and another larger version change. </span></p><p><span style="font-family: arial;">The final version for 2021 (<a href="https://www.metoffice.gov.uk/hadobs/hadisd/v320_2021f/index.html">v3.2.0.2021f</a>) was released in January, and is now also available at the CEDA Archive. We have released the first preliminary version for this calendar year, <a href="https://www.metoffice.gov.uk/hadobs/hadisd/v330_202201p/index.html">v3.3.0.202201p</a>.<br /></span></p><p><span style="font-family: arial;">A couple of bugs have come to light over the last month, which means this new version is has a larger version increment than normal.</span></p><h3 style="text-align: left;"><span style="font-family: arial;">Record Check</span></h3><p><span style="font-family: arial;">A user pointed out that the record check was failing known records for stations in the Middle East. This was due to the HadISD code using the values for the temperature records available at <a href="http://wmo.asu.edu">Arizona State University</a> for just continental Europe, rather than the area reflecting WMO Region VI. This has been fixed in the latest run. We've also updated the values used by this check to account for recent extreme events which have set new records.<br /></span></p><h3 style="text-align: left;"><span style="font-family: arial;">Calm Periods</span></h3><p><span style="font-family: arial;">We noticed that calm periods, especially for stations in Asia and parts of Europe, were not being correctly converted from the ISD because of a change in encoding for these measurements which started in mid-2013. We have been in contact with NOAA/NCEI who maintain the ISD to find out what occurred. However, as a interim we have adjusted how calm periods are read in from the ISD so that these are now correctly presented. We hope to be able to give further details on this change, how it arose and its implications in due course.<br /></span></p><p><span style="font-family: arial;"><br /></span></p>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-68634141246484940862021-09-20T15:02:00.000+01:002021-09-20T15:02:15.803+01:00Adapting the QC to account for the June 2021 North American Heatwave (part 3)<p><span style="font-family: arial;">We have seen in parts <a href="https://hadisd.blogspot.com/2021/07/adapting-qc-to-account-for-june-2021.html">1</a> and <a href="https://hadisd.blogspot.com/2021/09/adapting-qc-to-account-for-june-2021.html">2</a> that the Climatological and Distribution checks have been adjusted because some values at some stations were likely being erroneously flagged during the heatwave over North America in June 2021.</span></p><p><span style="font-family: arial;">In order to see how much of an effect these updated checks have had on the flagging rates, we look at a spatial distribution of the stations during the last days of June 2021 showing the temperatures at each station, with observations flagged with our unmodified QC appearing in green (Figure 1). As is clear, a large number of stations are flagged during this time. <br /></span></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPsilyVfgQCH1d8qko3W2nzu-1C8lXIdmlwAcZThYCQNOzT1djD-GoYtXE_1XdckpH6_j2wiOz3-x-UGOzLolLq9AudPgBac9n4FaMNtZGWSDZoKT_F5gWF7wbce0to8fQYRuJSFZWwBA/s640/0024.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPsilyVfgQCH1d8qko3W2nzu-1C8lXIdmlwAcZThYCQNOzT1djD-GoYtXE_1XdckpH6_j2wiOz3-x-UGOzLolLq9AudPgBac9n4FaMNtZGWSDZoKT_F5gWF7wbce0to8fQYRuJSFZWwBA/w400-h300/0024.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Figure 1. Temperatures at HadISD stations on 28-June-2021 at 00:00UTC (17:00PDT on 27th June). Flagged stations are shown in green, and non-reporting as transparent.</span><br /></td></tr></tbody></table><p><br /><span style="font-family: arial;">We produced the same map after the modifications in the two QC tests, and also the appropriate adjustment to the buddy check to allow any tentative flags to be unset (Figure 2). Many fewer stations are flagged with these updated tests. However, a number are still flagged. These are flagged throughout the period of interest rather than during the hottest part of the day, and so were likely the result of a test which flags an entire month. After some spot checks on these stations, these flags are </span><span style="font-family: arial;">from the excessive variance test.</span></p><p></p><p><span style="font-family: arial;"></span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8z4nYFW9zLNIkYrm7XEaR9vbXD5AvdiyiN-aA4H9pAETGAt3SOruS5GcyZt29kupFDjz2UFwXDYCdbwKyZV2iKKUH2omecjJ3JJ3sUkZZ0HUeF0XBLIiriA7uSbaRLng6vjH8phyV3fQ/s640/0024.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8z4nYFW9zLNIkYrm7XEaR9vbXD5AvdiyiN-aA4H9pAETGAt3SOruS5GcyZt29kupFDjz2UFwXDYCdbwKyZV2iKKUH2omecjJ3JJ3sUkZZ0HUeF0XBLIiriA7uSbaRLng6vjH8phyV3fQ/w400-h300/0024.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Figure 2. As for Figure 1 but after updated QC tests.</span><br /></td></tr></tbody></table><span style="font-family: arial;"><br />The excessive variance test looks at the distribution of the within-month variance, and identifies months with exceptionally low or exceptionally high variance. The scaling used for this test is the interquartile range, and months which have a variance more than 8 IQR from the average are flagged. As can be seen for the example of Osoyoos (712150-99999, latitude 49.033, longitude -119.433), the variance for June 2021 is much larger than any previously seen June (Figure 3). <br /></span><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbeYo9iJ59SE2hbba5aLUGoA0tlLIntrAGmF9qBKoptu1G-AEFWRMTQwiU57IETUZsMsnfGHEfqXrVKGatVyOTEZoXpOuIuXW9IwbC_Zr2rduUgh2e4KPsdiTv1vMR8Sad9Y7uzYS4mv4/s640/712150-99999_Osoyoos_variance_June21.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbeYo9iJ59SE2hbba5aLUGoA0tlLIntrAGmF9qBKoptu1G-AEFWRMTQwiU57IETUZsMsnfGHEfqXrVKGatVyOTEZoXpOuIuXW9IwbC_Zr2rduUgh2e4KPsdiTv1vMR8Sad9Y7uzYS4mv4/w400-h300/712150-99999_Osoyoos_variance_June21.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Figure 3. The Variance Check for Osoyoos (BC, 49.033, -119.433) showing the all of June 2021 flagged.</span><br /></td></tr></tbody></table><br /><p><span style="font-family: arial;">The variance check uses a fixed threshold of 8 IQR rather than thresholds generated from the properties of the distribution (as in the climatological and distribution checks). To update this check to determine thresholds from the distribution itself (as in the e.g. climatological and distributional checks) would be a larger change than the relatively small ones we have done so far. Also, in the example in Figure 3, a reasonable threshold determined from the distribution may still have excluded June 2021 (note, the y-axis is a log-scale) and we might struggle to be objective in this change if tailoring to this specific event, perhaps causing issues in other regions. </span><span style="font-family: arial;"><span style="font-family: arial;">In
contrast, the changes in the climatological and distributional checks
were easily motivated (and perhaps should have been spotted during
development of the monthly updates).</span></span></p><p><span style="font-family: arial;"><span style="font-family: arial;">As
we noted in the HadISD papers, the automated QC is a balance between
removing erroneous/dubious observations but retaining true extremes, and
what we do not want to do is make changes with inadvertently large
impacts elsewhere. </span>Our plan at this point in time is to note this as an issue for this test (and event) to look at in the future in any next major update to HadISD. Any flagged data in HadISD is removed from the netCDF data fields, but remains available within the netCDF files should users wish to access it. If you have thoughts on this, please do get in touch or comment below.<br /></span></p><p><span style="font-family: arial;">The next update to HadISD (in October 2021) will show a version number increment to reflect these changes in the QC tests (3.2.0.202109p).</span><br /></p><p><span style="font-family: arial;">[Animations of three days of the heatwave showing the flagged stations before and after the QC test updates are available on the <a href="https://hadleyserver.metoffice.gov.uk/hadisd/generalmaterial.html">HadISD homepage</a>].</span><br /></p>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-90796760737479213072021-09-14T16:26:00.001+01:002021-09-14T16:26:30.333+01:00Adapting the QC to account for the June 2021 North American heatwave (part 2)<p><span style="font-family: arial;">Continuing this took longer than planned, so there has been another version release of HadISD (v3.1.2.202107p) in the meantime.</span></p><p><span style="font-family: arial;">In the <a href="https://hadisd.blogspot.com/2021/07/adapting-qc-to-account-for-june-2021.html">last post</a>, we went through the changes that were made to the Climatological Outlier check as a result of the temperatures experienced in North America in June 2021. Since then, there have continued to be heatwave events across the world, with temperatures and impacts around the Mediterranean being the current focus (at time of writing). We will continue to use the North American heatwave for these changes for consistency, but note that of course changes to our QC will affect all stations and variables, and hence events.</span></p><h3 style="text-align: left;"><span style="font-family: arial;">Distributional Gap Check</span></h3><p><span style="font-family: arial;">In this check there are in fact two. The first uses monthly aggregated data, to look for asymmetries in the distribution, and we haven't changed that one. The second is what we delve into here, which uses all observations within a calendar month, and identifies gaps in this distribution to decide where to flag. We'll use the same station for our plots as in the previous post, 711130-99999 (Agazziz, BC, Canada).</span></p><p><span style="font-family: arial;">As we use a very similar approach in this test, we also had the same issue where our diagnostic plots initially were not showing data from the incomplete calendar year. But that was an easy fix, see Figure 1.<br /></span></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibJm1dpTpLEeYkT1zrNvJmR0p_tR9uVCN60ch8N_7XCyicmJ1vaY_hCTWyEqcx0WJx62JWn-HkV07dpWebKZ4J_ZGqbJtlZdNrk3P2C0r5_QMSp16yypNmW4HERzcTUT3Owo81wNhYPNk/s640/711130-99999_Agassiz_TGP_June.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibJm1dpTpLEeYkT1zrNvJmR0p_tR9uVCN60ch8N_7XCyicmJ1vaY_hCTWyEqcx0WJx62JWn-HkV07dpWebKZ4J_ZGqbJtlZdNrk3P2C0r5_QMSp16yypNmW4HERzcTUT3Owo81wNhYPNk/w400-h300/711130-99999_Agassiz_TGP_June.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig. 1 the distribution of scaled anomalies for
June from Agassiz (711130-99999), with the flagged ones highlighted in
red. Distribution from all years before 2021 is in black, and from all years including 2021 in grey. Blue is the fit to the data including skew and kurtosis using Gauss-Hermite polynomials. Note the logarithmic <i>y</i>-axis.</span></td><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;"> </span></td></tr></tbody></table><p></p><div class="separator" style="clear: both; text-align: center;"><span style="font-family: arial;"></span></div><span style="font-family: arial;">Here again, a handful of observations were being flagged because they fall beyond the bulk, but only when ignoring others from the incomplete calendar year. What we also noted was the single observation at the low end being flagged. This test should look for gaps at least two bin-widths wide (so two consecutive empty bins), and this doesn't seem to have been the case. We fixed that at the same time as the other changes.</span><p></p><p></p><p><span style="font-family: arial;">As for the climatological check, we treated the complete and incomplete years separately, which meant that these observations were now tentatively flagged, which can be unset by the neighbour check (Figure 2).</span></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj14g7scqdYo6SnU2z3a8XV_cfXAi6if0UX8yCqNzVAdHkoq4Kftp0lDDQoniaknEjg1id1-Lb0Emftvvm3PGoD6WgRvp7fs1iz2iF1yFvmVA5-unCIJj0fUCOKFPKCBzw_v00n-xi_91g/s640/711130-99999_Agassiz_TGP_June_tentative_gapfix.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj14g7scqdYo6SnU2z3a8XV_cfXAi6if0UX8yCqNzVAdHkoq4Kftp0lDDQoniaknEjg1id1-Lb0Emftvvm3PGoD6WgRvp7fs1iz2iF1yFvmVA5-unCIJj0fUCOKFPKCBzw_v00n-xi_91g/w400-h300/711130-99999_Agassiz_TGP_June_tentative_gapfix.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig 2. Same as Fig. 1, but with the data from
June 2021 now being correctly marked as tentatively flagged. Orange vertical lines are derived from the fit of the distribution (blue) to complete
years only (black), and red ones are where a gap has been found. The Purple and Pink are derived when
including the final month.</span></td></tr></tbody></table><span style="font-family: arial;"></span><p><span style="font-family: arial;">The final thing that we wanted to change was the nature of the curve being used to fit the distribution. When putting this code together, we wanted to include skew and kurtosis, as the distributions were clearly non-gaussian. At the time, we used Gauss-Hermite polynomials to obtain the fit with these higher moments of the distribution. However, we have since found that these sometimes have artefacts which result in some "wiggles" in the distributions (see Figure 1). Although this approach is still useful for gauging where to start looking for gaps in the distribution, but we thought that this was an opportunity to see what else could be done. We tried using the same skewed distribution (no kurtosis) as for the climatological outlier check.<br /></span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9fehyphenhyphenJQKpyRZ_skjecf-Pzjik1IJhKyDfGC3gex2_J-IU0o8bMBbplzK729YzR0cKutt9bD0jD9ygQt0ehsWUzJFvP6illCAFev6Gsm1G8a1yKuVeV-PSSbj7LYzCjWCQSkphgnD22tU/s640/711130-99999_Agassiz_TGP_June_tentative_gapfix_skew.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9fehyphenhyphenJQKpyRZ_skjecf-Pzjik1IJhKyDfGC3gex2_J-IU0o8bMBbplzK729YzR0cKutt9bD0jD9ygQt0ehsWUzJFvP6illCAFev6Gsm1G8a1yKuVeV-PSSbj7LYzCjWCQSkphgnD22tU/w400-h300/711130-99999_Agassiz_TGP_June_tentative_gapfix_skew.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig. 3: Same as Fig. 2, but now using a skewed-Gaussian ditribution for the fit rather than the Gauss-Hermite polynomials. </span></td></tr></tbody></table><div class="separator" style="clear: both; text-align: center;"><span style="font-family: arial;"></span></div><span style="font-family: arial;"><br /></span><p><span style="font-family: arial;">For this month, it is a more sensible fit, and also has a co-benefit of moving the value from which this test starts searching for a gap to the right, and so includes all of the hot temperatures in June.</span><br /></p>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-26389511917953781042021-07-26T11:20:00.004+01:002021-07-26T11:20:51.983+01:00Adapting the QC to account for the June 2021 North American heatwave (part 1)<p><span style="font-family: arial;">As we noted on the previous post, the extreme temperatures during June 2021 in western Canada and USA were being erroneously flagged by some of the HadISD QC tests. This is the first in likely a number of posts as we delve deeper into the causes and resolutions.</span></p><h2 style="text-align: left;"><span style="font-family: arial;">Climatological Outlier Check</span></h2><p><span style="font-family: arial;">We go back to the station which we showed in the previous post, 711130-99999 (Agassiz, BC). The plot we showed was one of the raw diagnostic outputs from HadISD which we ran to see what was going on.</span></p><p><span style="font-family: arial;">Firstly, while implementing changes for this test, we noticed that the plot was incomplete. It does correctly show the distribution from which the flagging thresholds were calculated as well as the highlighted flags in red. </span></p><p><span style="font-family: arial;">However, for this test, the thresholds are determined from the data themselves, using the distribution of the anomalised observations. So that the addition of each month during the year does not affect the thresholds set by any test, the thresholds are calculated from a distribution using the complete years only. In this case, that's all data from all Junes up to the end of 2020. What was shown on that plot was the distribution (black) of all the observations from complete years only, and the fitted Gaussian (blue). The red values were the observations that were flagged, which were from data in 2021 only, but missing were data from June 2021 which were not flagged. The updated plot is below (Fig. 1), where the grey histogram includes all data from June 2021. This shows that there are other observations in June 2021 which are warmer than the average of previous years. Some are even warmer than all previous years, but not sufficiently so to be flagged by this test.<br /></span></p><p><span style="font-family: arial;">[As a reminder, this test fits a Gaussian to the histogram, and then uses this to determine a threshold (from where the fit crosses <span style="font-family: courier;">y=0.1</span>). Observations which are further from the peak than the threshold are treated in two ways. If they are separated by an empty bin from the main distribution, then they are flagged. However, if they are "attached" (no empty bin) then they are tentatively flagged (and could be re-instated by the buddy check).]<br /></span></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_c4gxv4YOQ1bmoOyh_cXxBMq4WPuUEk535FtSMRL3WShfrF7Q6PtfeIADkfJHuIRHuZz0rRi77B35TWIKPpukjAyit2UOZ6Hry5qMz7zJOXJ8c7PnEVUjHmDm1-qHquwHDNU7kdkiORw/s640/711130-99999_Agassiz_TCM_June_v2.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_c4gxv4YOQ1bmoOyh_cXxBMq4WPuUEk535FtSMRL3WShfrF7Q6PtfeIADkfJHuIRHuZz0rRi77B35TWIKPpukjAyit2UOZ6Hry5qMz7zJOXJ8c7PnEVUjHmDm1-qHquwHDNU7kdkiORw/w400-h300/711130-99999_Agassiz_TCM_June_v2.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig. 1: the distribution of scaled anomalies for
June from Agassiz (711130-99999), with the flagged ones highlighted in
red. Distribution from all years before 2021 is in black, and from all years including 2021 in grey. Note the logarithmic <i>y</i>-axis.</span></td></tr></tbody></table><span style="font-family: arial;"></span><p></p><p><span style="font-family: arial;">This updated plot highlighted something we hadn't realised when adapting this test to work for the monthly updates. The thresholds are set on the complete-year data (up to 2020), and because these data all fall into a single distribution, this test identifies any observations further than this value as bad and flags them (highlighted in red). However, when including the 2021 data in the monthly update, there isn't an empty bin in the distribution. We note that had some observations in June in earlier (complete) years been very hot or very cold, they would have been correctly flagged by the test.<br /></span></p><p><span style="font-family: arial;">So, the first thing we have done is to rectifiy this, and ensure for cases like this, rather than flags being set, only tentative flags are set (Fig. 2), as these values are part of a contiguous distribution rather than being separate from it. In the case of monthly updates, the threshold for flagging (requiring that empty bin) is re-estimated, and shown by the purple line in Fig. 2. </span><span style="font-family: arial;"><span style="font-family: arial;">As in the case of complete years, any observation
in the final year further from the peak of the distribution is only
tentatively flagged as there is no empty bin (pink).</span></span></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfPPhmqumzs1ua04_dADbkfENmGhBUjiCgRVJQhN1K0kYQjB7XN-0IqnKAZZRLyynz6BeoO5QmBJXhyphenhyphenY4UTvgp1EBRMS4JHn9xK4bgxM0ipVnaUxO4385lZ187z1QqkzZqaSPHljXnFOM/s640/711130-99999_Agassiz_TCM_June_amended.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfPPhmqumzs1ua04_dADbkfENmGhBUjiCgRVJQhN1K0kYQjB7XN-0IqnKAZZRLyynz6BeoO5QmBJXhyphenhyphenY4UTvgp1EBRMS4JHn9xK4bgxM0ipVnaUxO4385lZ187z1QqkzZqaSPHljXnFOM/w400-h300/711130-99999_Agassiz_TCM_June_amended.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig. 2: Same as Fig. 1, but with the data from June 2021 now being correctly marked as tentatively flagged. Red and Orange vertical lines are derived from the distribution from complete years only (black) and fit (blue). The Purple and Pink are derived when including the final month.</span><br /></td></tr></tbody></table><span style="font-family: arial;"></span><p></p><p><span style="font-family: arial;">The other thing to address was to allow for a skew in the Gaussian fit as it is clear that a symmetric function is not the best fit to these data, which is shown in Fig. 3. This now reduces the number of observations on which a tentative flag is set to single figures. However, at the low temperature end, the threshold for the tentative flag has reduced, so should Agassiz get a very cold June, then it's possible some values may get tentative flags set instead<br /></span></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlk9AZGX-S7Vk_N6Y9l-j1LKwvN73lVcdFDPXvIJ5JS8w-YWggYyF-AgpDxVCMuuMmVSpUpdyEyprBKMcj8_ISFgdAVnt_-9hbNz6PEv6WcOPFQER3Ti93T6dbXKHkrF3w5OEU0SmqOeU/s640/711130-99999_Agassiz_TCM_June_skew_amended.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlk9AZGX-S7Vk_N6Y9l-j1LKwvN73lVcdFDPXvIJ5JS8w-YWggYyF-AgpDxVCMuuMmVSpUpdyEyprBKMcj8_ISFgdAVnt_-9hbNz6PEv6WcOPFQER3Ti93T6dbXKHkrF3w5OEU0SmqOeU/w400-h300/711130-99999_Agassiz_TCM_June_skew_amended.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig. 3: Same as Fig. 2, but now using a skewed-Gaussian ditribution for the fit. </span></td></tr></tbody></table><p><span style="font-family: arial;">The automated quality control works on data from meteorogical stations from around the world. For the case of Agassiz in Canada, we could be reasonably confident that all the data from June 2021 has been included in this update to HadISD, and therefore we could not bother with the "complete" versus "in progress" year distinction. However, for other locations, we do get data coming through in earlier months than the most recent one (e.g. data filling in during January through to May for the release that included June). In that case it is possible (though maybe unlikely) that thresholds for this test in earlier months could change from monthly release to monthly release, resulting in values being flagged or unflagged in different releases. Our approach is more stable during the monthly updates, and so we keep this distinction.</span></p><p><span style="font-family: arial;">At the end of the calendar year, we run the QC on the data for the final data release of that HadISD version. For this release, it is on a complete year, so for that release only (the ones processed in January each year), then this distinction isn't made. Therefore all the June data will go into the distribution from which the thresholds are determined. The original form of the test would have received a contiguous distribution for this station, and so only set tentative flags. However, but updating it to use the skew-Gaussian, in fact no observations are flagged (Fig 4.).</span></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVMjMGN03zxtUQ11q19XIq5NVcIbF5W9BnwvnK_3vkTS3ge84Ivqh_G8u-43jHXozYoySDTuqUctjufUKooZk_M9i6GBybya0m5IOBceNVYYx8o2HwUbw6WIS_iYEvvMsWR9di3mT_G5o/s640/711130-99999_Agassiz_TCM_June_amended_annual.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVMjMGN03zxtUQ11q19XIq5NVcIbF5W9BnwvnK_3vkTS3ge84Ivqh_G8u-43jHXozYoySDTuqUctjufUKooZk_M9i6GBybya0m5IOBceNVYYx8o2HwUbw6WIS_iYEvvMsWR9di3mT_G5o/w400-h300/711130-99999_Agassiz_TCM_June_amended_annual.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: arial;">Fig 4. Same as Fig 3. but for the version of the test as would be run for the update at the end of a calendar year.</span><br /></td></tr></tbody></table><span style="font-family: arial;"></span><p></p><p><span style="font-family: arial;">We will continue checking other QC tests as well as run further diagnostics before these changes are implemented in the HadISD QC suite, with a version number increment to reflect the changes. It is likely that these will not be available in time for the release in August 2021 (including data up to the end of July).<br /></span></p><p></p>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-24901826780230490442021-07-09T10:13:00.008+01:002021-07-09T10:48:36.764+01:00The June 2021 North American Heatwave and v3.1.2.202106p<div style="text-align: left;"><p><span style="font-family: arial;">We have just run the automated quality control (QC) for the latest monthly update, which includes data from June 2021. I'm sure you are all aware of the <a href="https://public.wmo.int/en/media/news/june-ends-exceptional-heat">severe heat wave</a> which affected the western part of North America in the last part of that month. British Columbia (Canada), Oregon and Washington (USA) experienced a number of days of exceptionally high temperatures, well over 40C in some cases. </span></p></div><div style="text-align: left;"><p><span style="font-family: arial;">Records measured by stations did not only fall, they were smashed, with new values set that were up to 5C higher than the previous records. A report by the World Weather Attribution project indicates that this event was <a href="https://www.worldweatherattribution.org/western-north-american-extreme-heat-virtually-impossible-without-human-caused-climate-change/">virtually impossible without human induced climate change</a>. Given that this event was so much warmer than anything experienced in
this region in the past, we thought to check how the automated QC
handled these exceptional values.</span></p><p><span style="font-family: arial;"><span style="font-family: arial;">Any
QC procedure will always result in retaining some bad values (false
negatives), and also erroneously removing some good ones (false
positives). It is impotant, however, to minimise these as best
possible, and do "least harm". To this effect, observations that have
been flagged by the QC are removed from the main data stream, but are
available in a separate data field in the netCDF files, should any user
wish to re-insert them into the time series.</span> <br /></span></p></div><div style="text-align: left;"><h4 style="text-align: left;"><span style="font-family: arial;">HadISD QC <br /></span></h4></div><p style="text-align: left;"><span style="font-family: arial;">The town of Lytton (BC) recorded the highest temperatures during this event, but that station does not form part of the <a href="https://www.metoffice.gov.uk/hadobs/hadisd/">HadISD</a>. We had a look to find nearby stations, and show these in Figure 1 for Agassiz, which is south of Lytton and further down the Fraser River, towards Vancouver.<br /></span></p><div style="text-align: center;"><p><span style="font-family: arial;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8uas7M8eFH4WLrSRp-BhOJa6aJwfG7Wf1Qb5nvn1_hjbWVWNQWGo9U5-p3RLAcK0Nzi-hIk-N9wuGBn6b_slEyQZ3GMqo-iIQTYz4s_U89hZSTAfYaTolYsClqzzLKGSAoSWsM5k9mrU/s640/Agassiz_HadISD312_202107p_2021_TCM.png"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8uas7M8eFH4WLrSRp-BhOJa6aJwfG7Wf1Qb5nvn1_hjbWVWNQWGo9U5-p3RLAcK0Nzi-hIk-N9wuGBn6b_slEyQZ3GMqo-iIQTYz4s_U89hZSTAfYaTolYsClqzzLKGSAoSWsM5k9mrU/w400-h300/Agassiz_HadISD312_202107p_2021_TCM.png" width="400" /></a></span></p></div><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: left;"><tbody><tr><td style="text-align: center;"><p><span style="font-family: arial;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8miH_Zl14oMMBx7r_jQqg2u-B98mHzECV40bGuV-1kl9JSdQ-CwujUWPcXE9SRf3xwVjJgYr1HeZVUNEYbcuvF9Y-ebuRR9SpEs6UBOSVJqrgTrGD_y-QT6-6AbltzSW9Yz8t-McIWgw/s640/Agassiz_HadISD312_202107p_June2021_TCM.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8miH_Zl14oMMBx7r_jQqg2u-B98mHzECV40bGuV-1kl9JSdQ-CwujUWPcXE9SRf3xwVjJgYr1HeZVUNEYbcuvF9Y-ebuRR9SpEs6UBOSVJqrgTrGD_y-QT6-6AbltzSW9Yz8t-McIWgw/w400-h300/Agassiz_HadISD312_202107p_June2021_TCM.png" width="400" /></a></span></p></td></tr><tr><td class="tr-caption" style="text-align: center;"><p><span style="font-family: arial;">Figure 1. Temperature timeseries for Agassiz (BC, 711130-99999) for (a) all of 2021 and (b) the latter half of June 2021. Observations are in black with any flagged by the climatological QC test in red.</span></p></td></tr></tbody></table><p style="text-align: left;"><span style="font-family: arial;">As you can clearly see, the highest values at the peak of the heatwave on the 27-29th June have all been removed, in this case by the Climatological Outlier check. At some level, this is unsurprising, given that the temperatures experienced surpassed anything in the previous record. Climatologically speaking they are exceptional values, and so without any other information to go on, could be dubious. </span></p><p style="text-align: left;"><span style="font-family: arial;"></span></p><h4 style="text-align: left;"><span style="font-family: arial;">The Current QC <br /></span></h4><p style="text-align: left;"><span style="font-family: arial;">Of course we know that these are likely to be valid observations, and as HadISD has been designed to retain true extremes, some adjustments to the QC algorithms are necessary. Firstly, let's have a look at how the current test is identifying and flagging these values. </span><span style="font-family: arial;"><span style="font-family: arial;">For full details see the <a href="https://cp.copernicus.org/articles/8/1649/2012/">HadISD paper</a> (Dunn et al, 2012).</span></span></p><p style="text-align: left;"><span style="font-family: arial;">The Climatological Outlier check works on a monthly basis, and calculates climatological values for each hour of the day for each month using the winsorized observations (Winsorizing is a process where all values exceeding a certain threshold [5% & 95% in this case] are replaced by these threshold values). Using these 24 climatological values, anomalies are calculated, and then scaled using their inter-quartile range.</span></p><p style="text-align: left;"><span style="font-family: arial;">We have included a way to account for some of the effects of a shifting climate using a low-pass filter. However, this is only applied to complete years of data, and so on our monthly updates has so far not been included. The resulting distribution is fitted with a Gaussian, and we use where this fitted Gaussian crosses the <i>y=0.1</i> line to set our threshold, rounded up to the next whole degree.</span></p><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: left;"><tbody><tr><td style="text-align: center;"><p><span style="font-family: arial;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXX-5YrdYfWschd5NWIfhCqbntZ20aT1RXNPiE0U8ri21IdkHPEczad7Vyt_E4XnlTgjFZ6aXSnzjnEpg8Qq3FW6-2riEUoBzlmzGnXYtunD2WPJCJYQ_puJ8qrTCEpxS2oCchsLZjbkc/s640/711130-99999_Agassiz_TCM_June.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXX-5YrdYfWschd5NWIfhCqbntZ20aT1RXNPiE0U8ri21IdkHPEczad7Vyt_E4XnlTgjFZ6aXSnzjnEpg8Qq3FW6-2riEUoBzlmzGnXYtunD2WPJCJYQ_puJ8qrTCEpxS2oCchsLZjbkc/w400-h300/711130-99999_Agassiz_TCM_June.png" width="400" /></a></span></p></td></tr><tr><td class="tr-caption" style="text-align: center;"><p><span style="font-family: arial;">Figure 2: the distribution of scaled anomalies for June from Agassiz (711130-99999), with the flagged ones highlighted in red. Note the logarithmic <i>y</i>-axis.<br /></span></p></td></tr></tbody></table><p style="text-align: left;"><span style="font-family: arial;">The test operates two levels of flagging, depending whether there is an empty bin between those further from the centre than the threshold values. If there is a gap, then these are flagged, as shown in Figure 2. If there isn't an empty bin and the are bins part of a contiguous distribution but are further from the mean than the threshold, then these are "tentatively" flagged (see Figure 10 in the </span><span style="font-family: arial;"><span style="font-family: arial;"><span style="font-family: arial;"> <a href="https://cp.copernicus.org/articles/8/1649/2012/">HadISD paper</a>)</span></span>. When running the neighbour checks, these tentatively flags can be removed if sufficient neighbours indate these are reasonable.</span></p><p style="text-align: left;"><span style="font-family: arial;">In the case of Agassiz, the observations were so extreme, that this test has flagged them without the option of the neighbour check undoing this (Figure 2).<br /></span></p><h4 style="text-align: left;"><span style="font-family: arial;">Amending the QC</span></h4><p style="text-align: left;"><span style="font-family: arial;">There are a number of options as to what we could do to improve the actions of this automated QC. However, the important thing is to make sure that whatever we implement, there are as few knock-on effects in other regions and flags as possible. The intention being that we improve this test in a robust, responsible way.</span></p><p style="text-align: left;"><span style="font-family: arial;"></span></p><p style="text-align: left;"><span style="font-family: arial;">A number of options have so far come to mind, including:</span></p><ul style="text-align: left;"><li><p><span style="font-family: arial;">Amend the low-pass filter to include data from the year in progress.</span></p></li><li><p><span style="font-family: arial;">Amend the fitting function from a pure Gaussian, to one which allows skew or even kurtosis. As seen in Figure 2, the distribution has a high tail above the fitted Gaussian, and accounting for this will affect the threshold used. This approach is already used in a different check in the HadISD QC.<br /></span></p></li></ul><ul style="text-align: left;"><li><p><span style="font-family: arial;">Use a rolling range to determine the years contributing to the climatologies used when creating anomalies, so values from 1931 are not contributing to 2021.</span></p></li><li><p><span style="font-family: arial;">Amend the neighbour check so that spatially coherent anomalies result in flags being unset from a greater subset of QC tests. </span></p></li></ul><div style="text-align: left;"><p><span style="font-weight: normal;"><span style="font-family: arial;">All of these approaches will need to be tested with care to ensure that any updates do not result in detrimental performance of the QC suite elsewhere in time or space.<br /></span></span></p><p><span style="font-weight: normal;"><span style="font-family: arial;">We will release this version of HadISD
(v3.1.2.202106p) with a note that observations from this event have
been erroneously flagged. As this is a preliminary version of HadISD,
this is reasonable, and gives us time to implement a solution in "slow
time". Watch this space for an update.</span></span></p><p></p><h4 style="text-align: left;"><span style="font-family: arial;">References:</span></h4><p><span style="font-weight: normal;"><span style="font-family: arial;">Dunn, R. J. H., Willett, K. M., Thorne, P. W., Woolley, E. V., Durre,
I., Dai, A., Parker, D. E., and Vose, R. S.: HadISD: a
quality-controlled global synoptic report database for selected
variables at long-term stations from 1973–2011, Clim. Past, 8,
1649–1679, <a href="https://doi.org/10.5194/cp-8-1649-2012">https://doi.org/10.5194/cp-8-1649-2012</a>, 2012 </span></span></p><p><span style="font-weight: normal;"><span style="font-family: arial;"><i><span style="font-size: x-small;">[Edited 9-Jul-2021 10.50BST to add option of amending the neighbour check] </span></i><br /></span></span></p></div>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-8568163417850550872021-03-05T14:20:00.000+00:002021-03-05T14:20:02.076+00:00v3.1.2.202102p<p><span style="font-family: arial;">It's been over a year since I last posted an update on this blog about HadISD. I suppose that's because everything has been ticking along nicely.<br /></span></p><p><span style="font-family: arial;">The dataset has been updated every month with version <a href="https://www.metoffice.gov.uk/hadobs/hadisd/v312_202102p/index.html">3.1.2.202102p </a>just being released. That contains data from 1-Jan-1931 to 28-Feb-2021.</span></p><p><span style="font-family: arial;">In the recent update of all data in the deep past to create v3.1.2.202101p (and the new station selection that goes with that), we have filled in a gap during April 2015 which was inadvertently left in the ISD. There has been a healthy increase in the number of stations in that update as well, with now 9278 stations in the dataset.</span></p><p><span style="font-family: arial;">At the end of last year, we released a <a href="https://www.metoffice.gov.uk/hadobs/hadisd/hadisd_v3_product_user_guide.pdf">Product User Guide</a>, to help users with the dataset. If you have any thoughts or suggestions about this, or anything else with HadISD, do <a href="https://www.metoffice.gov.uk/hadobs/hadisd/contact.html">get in touch</a>.<br /></span></p><p><span style="font-family: arial;"><br /></span></p>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-61024360900628776052020-02-18T10:16:00.003+00:002020-02-18T10:16:46.989+00:00v3.1.1.202001p<span style="font-family: Arial, Helvetica, sans-serif;">As I'm sure you've been noticing, I'm managing to update HadISD on a rougly monthly basis (the exact release date depends on other things in my schedule, but it tends to be around the second week of the month). </span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">In January version 3.1.0.2019f was released, the final update to 2019 data (hence the "f"). I also ran the Pairwise Homogenisation Algorithm on the data to produce the homogeneity assessment information for this version. </span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">Then, earlier this month (February), the new version 3.1.1.202001p was commenced. As this is a new set of monthly updates in 2020, the station selection code was re-run resulting in the addition of around 300 stations, making a total of just over 8400 in the HadISD. All other processing completed as normal, and this version is available at <a href="http://www.metoffice.gov.uk/hadobs/hadisd">www.metoffice.gov.uk/hadobs/hadisd</a> as usual. All data before 2020 is now frozen in this version, with only monthly appends, resulting in changes in the 2020 data.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">As always, please let us know if you spot anything which doesn't look right or if you are having issues in obtaining the data.</span>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-13875109472804150102019-12-11T15:07:00.000+00:002019-12-11T15:07:15.579+00:00Another minor version change<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">The HadISD code for versions 2 and 3 is written in Python (having migrated from the IDL code in version 1). However, it has been using Python2.7, support for which is ending on <a href="https://pythonclock.org/">January 1st 2020</a>. Therefore, I have updated the codebase to ensure that it is Python3 compatible.</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">During this process, it is possible that I will have missed some of the changes needed to ensure continuity across the versions (e.g. integer versus float division as default). However I have found one bug which seems to have been present since the creation of the Python version in 2014.</span></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">The world records check compares observation values to the world records for each continent (WMO region) as held by the WMO. Unfortunately in prior versions only the global values were being used, and so this check was not as powerful as it could have been. For many cases, the observation values that exceeded regional records but not the global ones will have been picked up by other checks (e.g. climatological or distribution).</span></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Below I show the images for the old and new versions of the test. These are on very slightly different runs (one from v3.0.1.201910p and one from a test run of the new code, which also updated the station counts slightly). As a fraction of the number of observations in each station, the increase in flagging is less than 0.1% (and in the stations I've checked numbers in the range of single observation to a few tens).</span></span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6fsjxgZbUH93UJpEc9qvlxV0vdM1AKnhJuNK4DA2LWPsZZkwZd9kjSlW2cl-Z8C2sdCrCctqSnYmqLNGgSUcq8b84FglKW2Ti8gr8WNpGyEkxspFWfezlM4Y7SZI0fyv6OSZvV6jKi3g/s1600/All_fails_TRC_20191105.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6fsjxgZbUH93UJpEc9qvlxV0vdM1AKnhJuNK4DA2LWPsZZkwZd9kjSlW2cl-Z8C2sdCrCctqSnYmqLNGgSUcq8b84FglKW2Ti8gr8WNpGyEkxspFWfezlM4Y7SZI0fyv6OSZvV6jKi3g/s400/All_fails_TRC_20191105.png" width="400" /></a></span></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Fig 1: World record checks for temperature in v3.0.1.201910p (Python 2.7 live version)</span></td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_q08u75OdwvGIPSGs-bvco-8h33ZaQBlMn2cDdddMDtwkyqnjvC-QWBlQNhwZ4lhRIZnXeN3wIyOXan-hLHqrwAhn_DkRvvlbCg1ANl14ClH_gkPnP8tlxg2eSaWGlbMayLlobt_fWPM/s1600/All_fails_TRC_20191102.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_q08u75OdwvGIPSGs-bvco-8h33ZaQBlMn2cDdddMDtwkyqnjvC-QWBlQNhwZ4lhRIZnXeN3wIyOXan-hLHqrwAhn_DkRvvlbCg1ANl14ClH_gkPnP8tlxg2eSaWGlbMayLlobt_fWPM/s400/All_fails_TRC_20191102.png" width="400" /></a></span></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Fig 2: World record checks for temperature in v3.1.0.201910p (test version of Python 3)</span></td></tr>
</tbody></table>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">As a result of (a) the update to Python3, and (b) the bug-fix of the world record check, the version released in December (which includes data up to the end of November 2019) is 3.1.0.201911p (as opposed to 3.0.1.201911p). No other changes have been made, and we expect to release the final version for 2019 updates in early January (3.1.0.2019f).</span></span><br />
<br />
Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-31877132377619293192019-02-05T13:48:00.000+00:002019-02-05T13:50:27.955+00:00Monthly updates, extra variables, new version<span style="font-family: "arial" , "helvetica" , sans-serif;">Since the launch of HadISD, we have run an annual update cycle in two stages. In January we update to allow a preliminary look at the most recent calendar year, and then in roughly April, we run a final version. This was adopted to balance the need of users to access data from the most recent complete calendar year in a timely fashion with the issue that data do not always arrive into the NOAA/NCEI archives within days of their observation.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">For the last few years I have been working on adapting the Python code that compiles HadISD2 to run in such a way to enable monthly updates. To do this, I've adopted the following conventions and outlines for the processing.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span>
<br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Quality Control Tests</span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In some of the quality control tests, the entire period of record is used to e.g. determine parameters of a distribution or set threshold values. With monthly appends to the data, these parameters would change from month to month, resulting in changing threshold values with each release. I decided that the flutter that this would cause in whether observations were flagged or not would be undesirable to users and so developed the tests as follows.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Where thresholds were set from the parameters of the observations themselves, these are only ever calculated from the data occurring up to the last complete year (31st December at 2300). Therefore, adding extra months has no impact until a run in January of a following year.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Some tests do not have thresholds set in this way, and so these are not impacted by the monthly append of new data.</span><br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Update cycle and versioning schema</span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">To retain a stable set of stations that make up HadISD, we have decided to only recreate the station list on an annual cycle. At the same time, all data in the "deep past" can be updated (i.e. prior to the most recent complete calendar year). On monthly updates, only the current year will have any data updated, but due to the way that the ISD files are stored at NCEI, an entire year is downloaded so the update in November could also have changes in February.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">To allow users to clearly identify which version of HadISD they are using in any output, we are going to stick with the versioning scheme as of HadISD and HadISD2 (x.y.z.datelabel). However, the date stamp will be more important for the monthly updating dataset. As there will be significant changes to the code, variables and processing we have decided to increment the overall version number by one - forming HadISD version 3. This is documented in a </span><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="https://www.metoffice.gov.uk/learning/library/publications/science/climate-science-technical-notes">Met Office Hadley Centre Technical Note</a>.</span> </span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">We have been running monthly updates for testing and internal purposes during the last few months of 2018. The first release using this new code will be that in January 2019. This includes the addition of 2018 data over previous release (v2.0.2.2017f), but no changes in the deep past, and no reselection of the stations. This update could be called v3.0.0.201812p - the preliminary update including data to the end of December. But as this will be the final monthly update for 2018's data, it will be released under <a href="http://www.metoffice.gov.uk/hadobs/hadisd/v300_2018f/">v3.0.0.2018f</a>.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">In February, the update which will include January's data will also check for any updates in previous years (1931-2018). With a new station selection in this update, it will be released as v3.0.1.201901p. In March v3.0.1.201902p etc all the way to January 2020 with v3.0.1.2019f there will be updates where all data from 2019 is overwritten. Then in February 2020, there is another update to the deep past, to the station list and also to include January 2020 - resulting in v3.0.2.202001p (note the change in the date label as well as in the "z" label).</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Other bits and bobs - precipitation and station level pressure</span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">While implementing the changes to the HadISD code base, I decided that this was an opportunity to address the issue with the precipitation fields, outlined in the <a href="http://hadisd.blogspot.com/2018/03/precipitation-in-hadisd.html">previous post</a>. There are 4 precipitation fields in the ISD, each with an accumulation period, an accumulation amount and quality code. I've split these out into new fields in the netCDF files, one for each accumulation period. This should make it easier for users who wish to use the precipitation amounts. However it is important to note that at this point, these data have NOT been quality controlled.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">A user requested to have station level pressure (different to sea-level pressure that is currently in HadISD) included. This we have done, and added another QC test to compare the station and sea-level pressures. If the difference between them is greater or less than 4.5 median-absolute deviations from the median difference, then the station level pressure is flagged.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">A <a href="https://www.metoffice.gov.uk/learning/library/publications/met-office-technical-documents">Met Office Hadley Centre Technical Note</a> has being drafted and will shortly be available (also on the <a href="https://www.metoffice.gov.uk/hadobs/hadisd/">HadISD website</a>). We encourage users to provide feedback on the monthly updates during 2019.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<h4>
<span style="font-family: "arial" , "helvetica" , sans-serif;">[US Government Shutdown]</span></h4>
<span style="font-family: "arial" , "helvetica" , sans-serif;">As a result of the prolonged US Government Shutdown, the update to HadISD is a bit later this year.</span>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-35634992053514986132018-03-22T10:44:00.002+00:002018-09-14T16:22:12.220+01:00Precipitation in HadISD<span style="font-family: "arial" , "helvetica" , sans-serif;">We have included precipitation accumulations in HadISD since its launch (netCDF field “<span style="font-family: "courier new" , "courier" , monospace;">precip1_depth</span>”), as this information is used as part of the quality control suite to check for high humidity periods in the dewpoint depression check. We wanted to make the HadISD fully traceable, so that users could check our quality control decisions for themselves, should they wish to. These precipitation accumulations are not quality controlled and so we have urged users to take care when using these data in their analyses.
<br /><br />In the ISD data format there are four possible entries for the precipitation data. These are indicated by character code “<span style="font-family: "courier new" , "courier" , monospace;">AA1</span>” to “<span style="font-family: "courier new" , "courier" , monospace;">AA4</span>”. Each of these has period, depth, condition and quality entries. To assist in the quality control of the dewpoint temperature fields, we extracted the first of these four precipitation fields. The netCDF names we assigned these variables are “<span style="font-family: "courier new" , "courier" , monospace;">precip1_depth</span>” and “<span style="font-family: "courier new" , "courier" , monospace;">precip1_period</span>”, as these were from the first ISD precipitation field.
<br /><br />Recently, Kimberly Channell of the <a href="http://glisa.umich.edu/">Great Lakes Integrated Sciences and Assessments at the University of Michigan</a> highlighted a confusion with the description of the <span style="font-family: "courier new" , "courier" , monospace;">precip1_depth</span> field in the netCDF files. The metadata for versions up to and including v2.0.2.2017p states “Depth of Precipitation Reported over time period”. This, combined with the hourly time stamps, could easily result in an assumption that the <span style="font-family: "courier new" , "courier" , monospace;">precip1_depth</span> field only contains hourly accumulations. Furthermore, our naming of the netCDF variable inadvertently supports this interpretation, that the “<span style="font-family: "courier new" , "courier" , monospace;">1</span>” in the “<span style="font-family: "courier new" , "courier" , monospace;">precip1_depth</span>” suggests hourly accumulation values. Unfortunately, neither of these are the case. </span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The accumulation period for the <span style="font-family: "courier new" , "courier" , monospace;">precip1_depth</span> is given by the <span style="font-family: "courier new" , "courier" , monospace;">precip1_period</span>. Even if there are timestamps every hour, the accumulation period may be a mix of time periods (from hourly to daily). We now appreciate that the metadata for these two variables could have been clearer, and that our chosen naming could be confusing, especially without knowledge of the ISD naming conventions. We apologise to users if these issues have caused problems with their analyses. To properly use the precipitation information, the depth information should be combined with the period.
<br /><br />Within one of the ISD precipitation fields, it is possible to have a number of accumulation periods, rather than just a single one across the entire length of the station record. The ISD is itself made up of a number of underlying databases, drawing observations from across a variety of observation networks (e.g. SYNOP, METAR, GTS). Each of these may have a different accumulation period, and also conventions as to the time of observation of e.g. 24 hour accumulations (and to which day these are assigned). These have been combined together to form threaded records for single station locations where possible during both the ISD and HadISD development.
<br /><br />It may be that a station report type (e.g. GTS) was the primary source in the early period (e.g. filling <span style="font-family: "courier new" , "courier" , monospace;">AA1</span>), but that for a later period, a different source with a different standard accumulation period has a higher priority in a merging process, and so supersedes this. Therefore, at observation times where both sources have data, this could move the hourly accumulations down into the later precipitation fields (<span style="font-family: "courier new" , "courier" , monospace;">AA2-4</span>), resulting in the interleaving of the different accumulation periods in the first entry. The example station in Figure 1 <span style="font-family: Arial, Helvetica, sans-serif;">exhibits behaviour consistent with this. It is possible that further hourly accumulation values are</span></span><span style="font-family: Arial, Helvetica, sans-serif;"> present in <span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: "Courier New",Courier,monospace;">AA2-4</span></span> of the ISD file for this station, but as we have not extracted those, they are not available to users of the HadISD at this time.
</span><br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpRXb4W8QbZmj3Hbe9sUf4jczh5S2rFAxhNq3CwWr4lUNUEfQPztnWmq7X5qLFczgB8sZh6fMBu72sY82O5uMMvhgMyQGVWYsdDe278o53DrIsXYrh1WN8WsUcnXcK46Bcb48CWgLwMXw/s1600/724380-93819_precip1_depth_blog.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpRXb4W8QbZmj3Hbe9sUf4jczh5S2rFAxhNq3CwWr4lUNUEfQPztnWmq7X5qLFczgB8sZh6fMBu72sY82O5uMMvhgMyQGVWYsdDe278o53DrIsXYrh1WN8WsUcnXcK46Bcb48CWgLwMXw/s400/724380-93819_precip1_depth_blog.png" width="400" /></a></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhznHEDbHgsHJX5ZJiG1H6fI4ZZDgOH2-Cs2130Emi6YbWmSoeP5fx_XuFtXAcL4gspnAVnvaS2rDqW1rf4gFDaTelOZFtrOrLuxvrNhuQQnlk90UG-ydcI7t6mgD0GxIktzf_8X1zQ_ho/s1600/724380-93819_precip1_period_blog.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="600" data-original-width="800" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhznHEDbHgsHJX5ZJiG1H6fI4ZZDgOH2-Cs2130Emi6YbWmSoeP5fx_XuFtXAcL4gspnAVnvaS2rDqW1rf4gFDaTelOZFtrOrLuxvrNhuQQnlk90UG-ydcI7t6mgD0GxIktzf_8X1zQ_ho/s400/724380-93819_precip1_period_blog.png" width="400" /></a></td></tr>
<tr align="left"><td class="tr-caption">Figure 1 (top) <span style="font-family: "courier new" , "courier" , monospace;">precip1_depth</span>, and (bottom) <span style="font-family: "courier new" , "courier" , monospace;">precip1_period</span> for 724380-93819 (Indianapolis Airport). This station has been merged from two ISD stations (99999-93819 and 724380-93819). As the precipitation information is not quality controlled, likely erroneous observations like the ~160mm in the late 1970s are still present in the data files (using HadISD v2.0.2.2017p). </td></tr>
</tbody></table>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">
<div class="separator" style="clear: both; text-align: center;">
</div>
</span><span style="font-family: "arial" , "helvetica" , sans-serif;">Therefore, for any given station in the HadISD, it is very likely that the period over which the precipitation depth has been accumulated is not constant over the entire record. However, there still may be valid <span style="font-family: "courier new" , "courier" , monospace;">precip1_depth</span> measurements present at each hourly timestamp, but these may be a combination of hourly1, but also 3, 6, 12, and 24 hourly measurements. We advise users wishing to take advantage of the precipitation information to make plots like those in Figure 1 to check for themselves what data have been included.
<br /><br />In light of this possible confusion from our netCDF variable names, we have/will take a number of actions:
<br /><br />1) Added notes to the HadISD webpages to clarify our naming scheme and inform users about the need to use both the “<span style="font-family: "courier new" , "courier" , monospace;">precip1_depth</span>” and “<span style="font-family: "courier new" , "courier" , monospace;">precip1_period</span>” fields. We’ve also improved the metadata for these two variables on the webpages too.
<br />2) Improve the metadata of the “<span style="font-family: "courier new" , "courier" , monospace;">precip1_depth</span>” and “<span style="font-family: "courier new" , "courier" , monospace;">precip1_period</span>” fields in the netCDF files in the next update (v2.0.2.2017f).
<br />3) In the longer term, extract all four ISD precipitation fields where available, and attempt to disaggregate into 1, 3 , 6, 12, and 24 hourly accumulation fields within the netCDF files. However, it is unlikely that we will be doing any quality control on these data and so we will still advise caution when using these. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br />We note again that the precipitation information in HadISD is not quality controlled at the moment. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br />Please do get in touch if you would like more information.
</span>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-51935343923309202512018-01-23T14:33:00.000+00:002018-01-23T14:33:06.704+00:00HadISD v2.0.2.2017p<span style="font-family: "arial" , "helvetica" , sans-serif;">We have just released version <a href="https://www.metoffice.gov.uk/hadobs/hadisd/v202_2017p/index.html">2.0.2.2017p</a> of HadISD on the <a href="https://www.metoffice.gov.uk/hadobs/hadisd/">Hadobs website</a>. The data now cover 1931/1/1 to 2017/12/31.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Downloading the data from the ISD finished on Monday 15th January and the quality control and other processes ran over the following days.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">There are 8103 stations in this version of HadISD, a full 2000 more than in HadISD version 1.0.x. However, there have been no changes to the quality control tests over v2.01.2016f.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">As always, if you notice anything untoward in the dataset please do get in touch. We intend to run a final version in a few months time if there have been changes to the ISD data in 2017 or earlier years in the intervening time.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We hope to move to monthly updates during 2018, which entail some minor changes to the QC code, but which should not impact the annual update methods. We will post on here in due course when this project is nearing completion.</span>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-81559550216519919662017-08-22T08:29:00.002+01:002017-08-22T08:29:53.018+01:00Digitisation and reporting resolution<span style="font-family: Arial,Helvetica,sans-serif;">A couple of years ago <a href="http://www.ccrc.unsw.edu.au/ccrc-team/students/james-goldie">James Goldie</a> (<a href="http://www.ccrc.unsw.edu.au/">UNSW</a>) contacted me about an issue he found in HadISD relating to the reporting resolution of temperature and humidity information for stations in Australia.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">In the HadISD, the data vary between single-degree, half-degree and 1/10th degree resolution. However, variations between these can cause some interesting striations in derived quantities.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">James has written up his work, with some cool animated plots at <a href="http://rensa.co/writing/viz-data-without-research-question/">his blog</a></span>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-26144826487101167142017-06-07T17:01:00.000+01:002017-06-07T17:01:22.029+01:00High windspeed values<span style="font-family: Arial,Helvetica,sans-serif;">Thanks to Phil Jones (UEA) and colleagues for pointing out this issue.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">There are a number of stations which have wind values of 88 m/s which also stands out as a repeating value (see Figure 1). </span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><span style="font-family: Arial,Helvetica,sans-serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicNynVLetK7ZqgykOsPAHJxwYxGa7B-sHC7_0tdo87y3vjIMhZ_yh9flGKjk9TQ5iTTHhCDVVfM8xEEBDKW-lhZ9D6WVAF_IjPaaBH2mZy67TbyPq8O4h48SxMVNMvFNxlkobwGGe_w0s/s1600/151080-99999_windspeeds.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="825" data-original-width="1358" height="241" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicNynVLetK7ZqgykOsPAHJxwYxGa7B-sHC7_0tdo87y3vjIMhZ_yh9flGKjk9TQ5iTTHhCDVVfM8xEEBDKW-lhZ9D6WVAF_IjPaaBH2mZy67TbyPq8O4h48SxMVNMvFNxlkobwGGe_w0s/s400/151080-99999_windspeeds.png" width="400" /></a></span></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial,Helvetica,sans-serif;">Fig 1. Station 151080-99999 (Ceahlau Toaca, 46.983N, 25.950E, 1898.0m) showing the wind speeds and inhomogeneities (vertical lines). The cluster of high values between 1991 and 2001 is clear (v2.0.1.2016f).</span></td></tr>
</tbody></table>
<span style="font-family: Arial,Helvetica,sans-serif;">These may be the result of a mistyped missing data code in the original data. It is also clear that this station may have rounding or conversion problems - we have not had the chance to investigate in detail so far.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">The maximum wind speed used for the record check is 113.3m/s (derived from a maximum gust speed - https://wmo.asu.edu/content/world-maximum-surface-wind-gust), so this would not exclude these values. The wind speeds are not passed through the distributional or frequent value checks as the shape of the distribution is not gaussian and to this point, these tests have been written assuming this shape. Nor is the spike check applied. Therefore, unfortunately, our QC suite is not (yet) clever enough at identifying these erroneous values.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">At the current time we do not have a solution to these issues - we would rather make folks aware than try and implement a "quick fix" which causes issues elsewhere. We will look into this during the course of this year and hope to roll out improvements to the wind QC in the next update.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">The stations which have been noted as affected by repeated high values are:</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">151080-99999
<br />156150-99999<br />156270-99999<br />228370-99999</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"></span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"></span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"></span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"></span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"></span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">Though others are noted to have one or a few high values.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"></span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">Please do not hesitate to get in touch if you do spot any issues or would like more information on these.</span>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-1175129746481783162017-02-09T09:53:00.002+00:002017-02-09T09:53:48.824+00:00HadISD v2.0.1.2016p<span style="font-family: "arial" , "helvetica" , sans-serif;">We have just released HadISD version 2.0.1.2016p<span style="font-family: "arial" , "helvetica" , sans-serif;"></span>. All plots and files should be on the <a href="http://www.metoffice.gov.uk/hadobs/hadisd/">website</a>. Between the release of v2.0.0.2015p in September, there have been no </span><span style="font-family: "arial" , "helvetica" , sans-serif;">updates to years in the past. The <a href="http://www.ncdc.noaa.gov/isd">ISD</a> raw data were downloaded on 19th January 2017 and processed over the following days. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The station selection was re-run, and so the station list has updated, with now 7877 stations present in this version. There have also been some minor changes to the quality control tests (affecting wind measurements) outlined below. A file indicating which stations are new to HadISD and which are no longer included compared to v2.0.0 is available.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">As a result of requests from users, in this version we have passed the wind speed observations through the spike check, and also the wind direction observations through the repeated values (streak) check.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The threshold values used to activate flagging in the spike check are calculated from the properties of the data themselves, using the distribution of differences between one observation and the next.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">For the streak check, although the parameters are calculated using the distribution of repeated values, these are only used to flag values if they are less than the defaults used in HadISD versions 1.0.x. We ensure that no calm periods are assessed when applying the streak check. The default values depend on the resolution of the wind direction and <span style="font-family: "arial" , "helvetica" , sans-serif;">are in the table below (see Table 4<span style="font-family: "arial" , "helvetica" , sans-serif;"> in <a href="http://www.clim-past.net/8/1649/2012/">Dunn et al, 201<span style="font-family: "arial" , "helvetica" , sans-serif;">2</span></a> for more inf<span style="font-family: "arial" , "helvetica" , sans-serif;">ormation</span>)</span>.</span></span><br />
<br />
<table cellpadding="10" cellspacing="0" style="width: 100%px;">
<caption><span style="font-family: "arial" , "helvetica" , sans-serif;">Wind Direction Streak Check</span></caption>
<tbody>
<tr padding-bottom:4px="" padding-top:5px="">
<th><span style="font-family: "arial" , "helvetica" , sans-serif;">Resolution</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">(degrees) </span></th>
<th><span style="font-family: "arial" , "helvetica" , sans-serif;">Repeated Streak (h)</span></th>
<th><span style="font-family: "arial" , "helvetica" , sans-serif;">Repeated Streak (d)</span></th>
<th><span style="font-family: "arial" , "helvetica" , sans-serif;">Repeated Hours</span></th>
<th><span style="font-family: "arial" , "helvetica" , sans-serif;">Repeated Days</span></th>
</tr>
<tr>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">90</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">120</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">28</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">28</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">10</span></td>
</tr>
<tr>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">45</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">96</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">28</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">28</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">10</span></td>
</tr>
<tr>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">22</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">72</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">21</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">21</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">7</span></td>
</tr>
<tr>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">10</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">48</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">14</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">14</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">7</span></td>
</tr>
<tr>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">1</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">24</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">7</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">14</span></td>
<td style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">5</span></td>
</tr>
</tbody>
</table>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
</span><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">However if you find something strange, do let us know
using the contact details on the <a href="http://www.metoffice.gov.uk/hadobs/hadisd/contact.html">HadISD website</a>. Please note the stations which are known to have issues are documented on <a href="http://hadisd.blogspot.co.uk/2013/12/spurious-stations-bad-mergers-while.html">this blog</a> and on the <a href="http://www.metoffice.gov.uk/hadobs/hadisd/v201_2016p/onlinematerial.html">website</a>. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">The quality control code used in this version will be uploaded to the github repository in the coming days.</span></span>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-68721291819281569332016-09-16T16:29:00.003+01:002016-09-29T14:45:51.878+01:00HadISD version 2.0.0.2015p<span style="font-family: "arial" , "helvetica" , sans-serif;">We submitted the <a href="http://www.clim-past-discuss.net/cp-2015-137/">HadISD version 2</a> paper to <a href="http://www.clim-past.net/">Climate of the Past</a> in <a href="http://hadisd.blogspot.co.uk/2015/09/hadisd-version-2.html">September 2015</a><span style="font-family: "arial" , "helvetica" , sans-serif;">, with the data<span style="font-family: "arial" , "helvetica" , sans-serif;">set tentatively version<span style="font-family: "arial" , "helvetica" , sans-serif;">ed as <span style="font-family: "arial" , "helvetica" , sans-serif;">v2.0.0.201<span style="font-family: "arial" , "helvetica" , sans-serif;">4p.</span></span></span></span></span> It went through the review process with a number of helpful comments being given by the two referees. </span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Some of these comments were addressed to the way the stations were selected and to add more stringent tests during this part of our code. We adjusted our code in light of these comments, but as we were then about to perform our standard update on HadISD version 1 (to 1.0.4.2015p), we waited until we were able to include data from 2015 into an updated version 2 as well before re-running our station selection code.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Another of the comments was that the paper would be best suited to a different journal, and a sister journal of Climate of the Past - <a href="http://www.geoscientific-instrumentation-methods-and-data-systems.net/">Geoscientific Instrumentation, Methods and Data Systems</a> (GI) - was suggested.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We re-r<span style="font-family: "arial" , "helvetica" , sans-serif;">a</span>n the updated station selection code, all the quality control procedures, homogeneity assessment and comparison to <a href="http://www.metoffice.gov.uk/hadobs/hadisd/v104_2015p/download.html">HadISD.1.0.4.2015p</a> to create <a href="http://www.metoffice.gov.uk/hadobs/hadisd/v200_2015p/">HadISD.2.0.0.2015p</a>. We submitted our updated manuscript to GI<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> <span style="font-family: "arial" , "helvetica" , sans-serif;">on 17th March<span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span><span style="font-family: "arial" , "helvetica" , sans-serif;">2016. </span></span></span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">After a long tim<span style="font-family: "arial" , "helvetica" , sans-serif;">e in discussions with the ed<span style="font-family: "arial" , "helvetica" , sans-serif;">itori<span style="font-family: "arial" , "helvetica" , sans-serif;">al team to explain why th<span style="font-family: "arial" , "helvetica" , sans-serif;">e manuscript was very similar to <span style="font-family: "arial" , "helvetica" , sans-serif;">our Climate of the Past D<span style="font-family: "arial" , "helvetica" , sans-serif;">iscussion pape<span style="font-family: "arial" , "helvetica" , sans-serif;">r, the paper was <span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="http://www.geosci-instrum-method-data-syst-discuss.net/gi-2016-9/">published</a> <span style="font-family: "arial" , "helvetica" , sans-serif;">in <span style="font-family: "arial" , "helvetica" , sans-serif;">GID <span style="font-family: "arial" , "helvetica" , sans-serif;">on 12th Sept<span style="font-family: "arial" , "helvetica" , sans-serif;">ember 2016. The ed<span style="font-family: "arial" , "helvetica" , sans-serif;">itoria<span style="font-family: "arial" , "helvetica" , sans-serif;">l team have d<span style="font-family: "arial" , "helvetica" , sans-serif;">ecided that as this<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> manuscript has already go<span style="font-family: "arial" , "helvetica" , sans-serif;">ne through one round of<span style="font-family: "arial" , "helvetica" , sans-serif;"> open<span style="font-family: "arial" , "helvetica" , sans-serif;"> peer review <span style="font-family: "arial" , "helvetica" , sans-serif;">in an <span style="font-family: "arial" , "helvetica" , sans-serif;">EGU journal, th<span style="font-family: "arial" , "helvetica" , sans-serif;">e discussion p<span style="font-family: "arial" , "helvetica" , sans-serif;">hase <span style="font-family: "arial" , "helvetica" , sans-serif;">could be close<span style="font-family: "arial" , "helvetica" , sans-serif;">d before the u<span style="font-family: "arial" , "helvetica" , sans-serif;">sual period had <span style="font-family: "arial" , "helvetica" , sans-serif;">completed, on 13th September. We are now in the process of submitting our final files to <span style="font-family: "arial" , "helvetica" , sans-serif;">GI.</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">We have re<span style="font-family: "arial" , "helvetica" , sans-serif;">leased<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> Had<span style="font-family: "arial" , "helvetica" , sans-serif;">ISD.2.0.0.201<span style="font-family: "arial" , "helvetica" , sans-serif;">5p on the</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> HadOBS server at <a href="http://www.metoffice.gov.uk/hadobs/hadisd/v200_2015p/">www.metoffice.gov.uk/hadobs/hadisd</a>. We expected further comments from review process at GI, and had intended to u<span style="font-family: "arial" , "helvetica" , sans-serif;">pdate the datas<span style="font-family: "arial" , "helvetica" , sans-serif;">et in light of these, creating v2.0.0.2015f. </span></span>However this will now not happen<span style="font-family: "arial" , "helvetica" , sans-serif;">, and the first <span style="font-family: "arial" , "helvetica" , sans-serif;">re<span style="font-family: "arial" , "helvetica" , sans-serif;">lease of HadISD2 </span></span>will <span style="font-family: "arial" , "helvetica" , sans-serif;">remain at v2.0.0.2015p. We intend to <span style="font-family: "arial" , "helvetica" , sans-serif;">perform annual updates on HadISD2 on similar timescales as for Ha<span style="font-family: "arial" , "helvetica" , sans-serif;">dISD, and so expect the next update to be in January 201<span style="font-family: "arial" , "helvetica" , sans-serif;">7 to create <span style="font-family: "arial" , "helvetica" , sans-serif;">v2.0.1.2016p.</span></span></span></span></span></span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Please read the papers on HadISD version 2 before using this dataset and also get in touch if you have any concerns with the data so we can check and address them if necessary.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The major<span style="font-family: "arial" , "helvetica" , sans-serif;"> changes when going to version 2 from version 1 is th<span style="font-family: "arial" , "helvetica" , sans-serif;">e increase in <span style="font-family: "arial" , "helvetica" , sans-serif;">the timespan cover<span style="font-family: "arial" , "helvetica" , sans-serif;">e<span style="font-family: "arial" , "helvetica" , sans-serif;">d by the datas<span style="font-family: "arial" , "helvetica" , sans-serif;">et, bac<span style="font-family: "arial" , "helvetica" , sans-serif;">k to 1931. We have also <span style="font-family: "arial" , "helvetica" , sans-serif;">refreshed the station selection cr<span style="font-family: "arial" , "helvetica" , sans-serif;">iteria, an<span style="font-family: "arial" , "helvetica" , sans-serif;">d there are 7677 un<span style="font-family: "arial" , "helvetica" , sans-serif;">ique station I<span style="font-family: "arial" , "helvetica" , sans-serif;">Ds p<span style="font-family: "arial" , "helvetica" , sans-serif;">resent in this version; but this w<span style="font-family: "arial" , "helvetica" , sans-serif;">ill change on each new release</span>. A number o<span style="font-family: "arial" , "helvetica" , sans-serif;">f the QC te<span style="font-family: "arial" , "helvetica" , sans-serif;">sts have had minor t<span style="font-family: "arial" , "helvetica" , sans-serif;">weaks, and we have also <span style="font-family: "arial" , "helvetica" , sans-serif;">improved the l<span style="font-family: "arial" , "helvetica" , sans-serif;">evel of QC <span style="font-family: "arial" , "helvetica" , sans-serif;">applied to wind speed and dire<span style="font-family: "arial" , "helvetica" , sans-serif;">ction observations.</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><br />
<br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Update - 29/09/2016</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">The paper descri<span style="font-family: "arial" , "helvetica" , sans-serif;">bing HadISD.2.0.0.2015p has been <span style="font-family: "arial" , "helvetica" , sans-serif;">publis<span style="font-family: "arial" , "helvetica" , sans-serif;">hed in <a href="http://www.geosci-instrum-method-data-syst.net/5/473/2016/"> Geoscientific Instrumentation, Methods and Data Systems</a>. </span></span></span></span> As <span style="font-family: "arial" , "helvetica" , sans-serif;">stated above, the data are availab<span style="font-family: "arial" , "helvetica" , sans-serif;">le on the HadOBS serv<span style="font-family: "arial" , "helvetica" , sans-serif;">er.</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-65405353086188587742016-08-01T09:44:00.000+01:002016-08-01T09:44:19.596+01:00HadISD available at CEDA/BADC with updated licence<span style="font-family: Arial,Helvetica,sans-serif;">We have been working with the <a href="https://badc.nerc.ac.uk/home/index.html">British Atmospheric Data Centre</a> (BADC) over the past year or so to make some of the datasets currently available on the <a href="http://www.metoffice.gov.uk/hadobs/hadisd/">HadOBS</a> website also available through the BADC<a href="http://www.ceda.ac.uk/"> Centre for Environmental Data Analysis</a> (CEDA) data archive as part of the <a href="http://www.clipc.eu/">CLIPC</a> project.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">HadISD <a href="http://catalogue.ceda.ac.uk/uuid/229b53d2e44741ecbe70ba6299875a30">v1.0.3.2014f</a> and <a href="http://catalogue.ceda.ac.uk/uuid/7b6993cbf7ec45f9ad01b86bed537e4c">v1.0.4.2015p</a> are now available through that portal. Not all the facets available in the netCDF files hosted by HadOBS are included in these data, but the climate variables and quality information are present.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">We intend that CEDA becomes an alternate route for users to access our data and also be able use some of their download tools (e.g. via OpenDAP).</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">As part of this process we have updated the licence under which HadISD is released. This also ensures that the data comply with the licence issued by <a href="https://www.ncei.noaa.gov/">NOAA/NCEI</a> on the <a href="http://www.ncdc.noaa.gov/isd">ISD</a> dataset. The new licence is as follows:</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
"<span style="font-family: Arial,Helvetica,sans-serif;"><span style="color: black;"><i>HadISD is distributed under
the <a href="http://www.nationalarchives.gov.uk/doc/non-commercial-government-licence/non-commercial-government-licence.htm">Non
Commercial Government Licence</a>. The data are available for non-commercial use with attribution to the data providers. Please
cite <a href="http://www.clim-past-discuss.net/8/1763/2012/cpd-8-1763-2012.html">Dunn et al (2012)</a>. This product may contain data which are
governed by WMO Policy
following <a href="http://www.wmo.int/pages/prog/www/ois/Operational_Information/Publications/Congress/Cg_XII/res40_en.html">WMO Resolution
40</a> <a href="http://www.wmo.int/pages/prog/www/ois/Operational_Information/Publications/Congress/Cg_XII/annex1_en.html">Annex
1</a> alongside additional data that may have restrictions
placed on their commercial use by the data owners. Any redistribution of this product should be accompanied by a similar
statement of usage policy.</i>"</span></span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;"><span style="color: black;">Any users who have concerns about what HadISD can be used for are encouraged to <a href="http://www.metoffice.gov.uk/hadobs/hadisd/contact.html">get in touch</a> to discuss their project. </span></span>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-41360826473808073952016-07-22T14:40:00.001+01:002016-07-22T14:40:15.634+01:00HadISD 1.0.4.2015f<span style="font-family: "arial" , "helvetica" , sans-serif;">We have just released HadISD version 1.0.4.2015<span style="font-family: "arial" , "helvetica" , sans-serif;">f</span>. All plots and files should be on the <a href="http://www.metoffice.gov.uk/hadobs/hadisd/">website</a>. T</span><span style="font-family: "arial" , "helvetica" , sans-serif;">his version <span style="font-family: "arial" , "helvetica" , sans-serif;">supersedes</span> the preliminary version from earlier this year (<a href="http://hadisd.blogspot.co.uk/2016/01/hadisd1042015p.html">v1.0.4.2015p</a>). There have been further updates to the <a href="http://www.ncdc.noaa.gov/isd">ISD</a>
source data for the years <span style="font-family: "arial" , "helvetica" , sans-serif;">2007-10, 2012-13 and 2015 </span>since the preliminary
dataset was created in January, but no changes in <span style="font-family: "arial" , "helvetica" , sans-serif;">other</span> years. </span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The
raw data were downloaded on <span style="font-family: "arial" , "helvetica" , sans-serif;">17th June</span> 201<span style="font-family: "arial" , "helvetica" , sans-serif;">6</span>, and processed over
the <span style="font-family: "arial" , "helvetica" , sans-serif;">last month</span> We have made no substantial changes to the codes which do the
conversion to NetCDF files or the Quality Control suit<span style="font-family: "arial" , "helvetica" , sans-serif;">e, and so there is only the update <span style="font-family: "arial" , "helvetica" , sans-serif;">in the version number is f<span style="font-family: "arial" , "helvetica" , sans-serif;"></span>rom prelimi<span style="font-family: "arial" , "helvetica" , sans-serif;">nary to final. </span></span></span> </span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">This version still contains 6103 stations, with 40<span style="font-family: "arial" , "helvetica" , sans-serif;">50</span> passing the final
filtering checks, down slightly from the 4060 in v1.0.3.2014f (see the <a href="http://www.clim-past-discuss.net/8/1763/2012/cpd-8-1763-2012.html">HadISD paper</a>
Section 6). The patterns of flagging are very similar to
v1.0.3.2014f. However if you find something strange, do let us know
using the contact details on the <a href="http://www.metoffice.gov.uk/hadobs/hadisd/contact.html">HadISD website</a>. Please note the stations which are known to have issues are documented on <a href="http://hadisd.blogspot.co.uk/2013/12/spurious-stations-bad-mergers-while.html">this blog</a> and on the <a href="http://www.metoffice.gov.uk/hadobs/hadisd/v103_2014p/files/user_identified_bad_stations.txt">website</a>.</span><br />
<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6OWHaNcLIBJ3q2stZspceamTrQ_CcL1usJvPRBzyM1mhdzYRYwbRlLbf4MfbyQvw67wmknCd6-Y7CKjCh4L2WQbLCmXrJujp9wFBeFLtabn_ZTNHAnTNOOyqh8RriiIKc7EbQFiUV_PA/s1600/ALL_T_mapfails_mask2_20160720.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="285" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6OWHaNcLIBJ3q2stZspceamTrQ_CcL1usJvPRBzyM1mhdzYRYwbRlLbf4MfbyQvw67wmknCd6-Y7CKjCh4L2WQbLCmXrJujp9wFBeFLtabn_ZTNHAnTNOOyqh8RriiIKc7EbQFiUV_PA/s400/ALL_T_mapfails_mask2_20160720.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial,Helvetica,sans-serif;">Fig.1 The fraction of all temperature records flagged for each station</span></td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhP2rK0E_loZE01Xkc2BergtSqyGhI-f891dUwQnPndy-_ylCcyLMeWgAqgMrRM_c2rCrwYwY6oq8yyskC3kcLFQ9CNTsMbuBYk6IH5l1EI9MSjjf6DPxRKW1asz7h9qg1Slvn7TOt9fzo/s1600/ALL_Td_mapfails_mask2_20160720.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="285" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhP2rK0E_loZE01Xkc2BergtSqyGhI-f891dUwQnPndy-_ylCcyLMeWgAqgMrRM_c2rCrwYwY6oq8yyskC3kcLFQ9CNTsMbuBYk6IH5l1EI9MSjjf6DPxRKW1asz7h9qg1Slvn7TOt9fzo/s400/ALL_Td_mapfails_mask2_20160720.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial,Helvetica,sans-serif;">Fig. 2 The fraction of all dewpoint temperature records flagged for each station</span></td></tr>
</tbody></table>
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwimB0cR0J8At3mMfQLHniag6RewXopERE0WFh4eAiUsnFdbNbceXNxBwwBHXns0lgnYw0R_iP4yjHuAejpZPbWjaXQYAHRTanlsMHY94T5q8Lf9QIaGAi6usOQCAjzsTHj7Og3FpesnY/s1600/ALL_SLP_mapfails_mask2_20160720.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="285" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwimB0cR0J8At3mMfQLHniag6RewXopERE0WFh4eAiUsnFdbNbceXNxBwwBHXns0lgnYw0R_iP4yjHuAejpZPbWjaXQYAHRTanlsMHY94T5q8Lf9QIaGAi6usOQCAjzsTHj7Og3FpesnY/s400/ALL_SLP_mapfails_mask2_20160720.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Arial,Helvetica,sans-serif;">Fig. 3 The fraction of all sea-level pressure records flagged for each station</span></td></tr>
</tbody></table>
<br />
<br /><br />
<br /><br /><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The Homogeneity information for this version is also available on the <a href="http://www.metoffice.gov.uk/hadobs/hadisd/v103_2014p/onlinematerial.html">website</a> using the same procedure (PHA) as outlined in <a href="http://www.clim-past.net/10/1501/2014/">Dunn et al, 2014</a>.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">As always, if you see anything untoward in the data or are having problems using it, please do not hesitate to <a href="http://www.metoffice.gov.uk/hadobs/hadisd/contact.html">get in touch</a>.</span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>Licencing</b></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The licence<b> </b><span style="font-family: "arial" , "helvetica" , sans-serif;">under which HadISD is released has been changed to a <a href="http://www.nationalarchives.gov.uk/doc/non-commercial-government-licence/non-commercial-government-licence.htm">Non Commercial Government Licence</a><span style="font-family: "arial" , "helvetica" , sans-serif;">. Full d<span style="font-family: "arial" , "helvetica" , sans-serif;">etails are </span></span></span>on the <a href="file:///home/h05/rdunn/LocalData/HadISD/QC/hadobs_copy/v104_2015f/download.html">HadISD website</a><span style="font-family: "arial" , "helvetica" , sans-serif;">. <span style="font-family: "arial" , "helvetica" , sans-serif;">Please <span style="font-family: "arial" , "helvetica" , sans-serif;">ensure you read and follow <span style="font-family: "arial" , "helvetica" , sans-serif;">the terms of this <span style="font-family: "arial" , "helvetica" , sans-serif;">licence before using the data.<span style="font-family: "arial" , "helvetica" , sans-serif;"></span></span></span></span></span></span></span>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-90750174776267389292016-01-25T11:53:00.000+00:002016-01-25T11:53:46.622+00:00HadISD.1.0.4.2015p<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We have just released HadISD version 1.0.4.2015p. All plots and files should be on the <a href="http://www.metoffice.gov.uk/hadobs/hadisd/">website</a> This update extends the coverage of the dataset to the
end of 2015 (31 December at 2300 inclusive). It remains a preliminary
dataset as there could still be further updates to the <a href="http://www.ncdc.noaa.gov/oa/climate/isd/index.php">ISD </a>dataset
in the next few months. We hope to do a processing run for the final
version some time around Easter (to create 1.0.4.2015f). </span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The raw data were downloaded on 7th January 2015, and processed over
the subsequent days. There have been changes to all of the raw files in 2012-2014 as part of the normal ISD update process
We have made no substantial changes to the codes which do the
conversion to NetCDF files or the Quality Control suite. Hence the
version number has only incremented by 0.0.1 and the year. Any updates to these systems will be included in the future HadISD.2.0.0. </span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">This version still contains 6103 stations, with 4049 passing the final
filtering checks, down slightly from the 4060 in v1.0.3.2014f (see the <a href="http://www.clim-past-discuss.net/8/1763/2012/cpd-8-1763-2012.html">HadISD paper</a>
Section 6). The patterns of flagging are very similar to
v1.0.3.2014f. However if you find something strange, do let us know
using the contact details on the <a href="http://www.metoffice.gov.uk/hadobs/hadisd/contact.html">HadISD website</a>. Please note the stations which are known to have issues are documented on <a href="http://hadisd.blogspot.co.uk/2013/12/spurious-stations-bad-mergers-while.html">this blog</a> and on the <a href="http://www.metoffice.gov.uk/hadobs/hadisd/v103_2014p/files/user_identified_bad_stations.txt">website</a>.</span><br />
<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhi07eCUVf3f9NQjqkpNyZl4cTfPoLB-WwqrZ-F1SCTURO57LLGAHVwyCEBk1GIACvOjsiFGCunC3GSJgHtaVxdGlHUBLnIG9F1OArvN5i9KfuSTb8zH0TtJZbGXuTbsj8Q_Y2afqoQf-8/s1600/ALL_T_mapfails_mask2_20160118.png" style="margin-left: auto; margin-right: auto;"><img border="0" height="285" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhi07eCUVf3f9NQjqkpNyZl4cTfPoLB-WwqrZ-F1SCTURO57LLGAHVwyCEBk1GIACvOjsiFGCunC3GSJgHtaVxdGlHUBLnIG9F1OArvN5i9KfuSTb8zH0TtJZbGXuTbsj8Q_Y2afqoQf-8/s400/ALL_T_mapfails_mask2_20160118.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Fig. 1 The fraction of all temperature records flagged for each station</span></td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl5D8dL6puP9-JrXe_btCa7TLp4m3qAjvdxmpYalHw93qT0QKVOXCuktHvHa6kdbKLrl9yKm2bn0Ko0L9vhUpmdnFdBsvsNOiBBmKZQcdWmBIsgSitfKTmyrs2n0B-F8-14MAuROgIlNU/s1600/ALL_Td_mapfails_mask2_20160118.png" style="margin-left: auto; margin-right: auto;"><img border="0" height="284" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl5D8dL6puP9-JrXe_btCa7TLp4m3qAjvdxmpYalHw93qT0QKVOXCuktHvHa6kdbKLrl9yKm2bn0Ko0L9vhUpmdnFdBsvsNOiBBmKZQcdWmBIsgSitfKTmyrs2n0B-F8-14MAuROgIlNU/s400/ALL_Td_mapfails_mask2_20160118.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Fig. 2 The fraction of all dewpoint temperature records flagged for each station</span></td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiU-VKghL1qHD__HnRLDfDfD6fuiwH1z_kMJ5POwxSWd-CORqtFqi4mK5cjsvniaF_rB5v0EZ9Wea2V5kzWRwSd2MOhBqa06oIwU5_V0QYuVFzKc74EdIpcdzc-AsBFEtZm9nawkyB5-es/s1600/ALL_SLP_mapfails_mask2_20160118.png" style="margin-left: auto; margin-right: auto;"><img border="0" height="285" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiU-VKghL1qHD__HnRLDfDfD6fuiwH1z_kMJ5POwxSWd-CORqtFqi4mK5cjsvniaF_rB5v0EZ9Wea2V5kzWRwSd2MOhBqa06oIwU5_V0QYuVFzKc74EdIpcdzc-AsBFEtZm9nawkyB5-es/s400/ALL_SLP_mapfails_mask2_20160118.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Fig. 3 The fraction of all sea-level pressure records flagged for each station</span> </td></tr>
</tbody></table>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">The Homogeneity information for this version is also available on the <a href="http://www.metoffice.gov.uk/hadobs/hadisd/v103_2014p/onlinematerial.html">website</a> using the same procedure (PHA) as outlined in <a href="http://www.clim-past.net/10/1501/2014/">Dunn et al, 2014</a>.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">As always, if you see anything untoward in the data or are having problems using it, please do not hesitate to <a href="http://www.metoffice.gov.uk/hadobs/hadisd/contact.html">get in touch</a>.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-56610388659004688802015-09-30T10:10:00.000+01:002016-01-21T10:29:23.321+00:00HadISD version 2<span style="font-family: "arial" , "helvetica" , sans-serif;">The paper describing HadISD version 2.0.0 has just appeared in the Discussions section of Climate of the Past:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">http://www.clim-past-discuss.net/11/4569/2015/cpd-11-4569-2015.html</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://www.clim-past-discuss.net/11/4569/2015/cpd-11-4569-2015.html" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="207" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzfZjp-uD0KLW-BjuF06vaLeSCjLyqnVc4ExlhS_UqQ-3s2T_EObWzSgVVbDqXIf3HzUIvPls-Pl_EvxELjCxxXtsovVvaKpLC8vJtz6aJ7iicyEEXCRg0BbL_Q6IXICziPrnak56s-9Y/s400/Screenshot.png" width="400" /></a></div>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">There now follows 8 weeks of reviewing process. Two anonymous referees will be asked to make comments, which will appear online, and also anyone can make attributed comments (i.e. under their name) which will also appear. After that we will have the chance to respond (these will also be published) and then the final paper will appear thereafter. Once all that is done, then we can release the dataset and also the quality control code. Hopefully this will all go through before the end of the year so that I can also run an update in January to v2.0.1.2015p.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">For those of you who follow this blog, then a number of the sections in the paper will be familiar, however, the jist of the paper is expanding the time coverage of HadISD from 1973 back to 1931. At the same time we've readdressed the way stations are selected and merged, and so v2.0.0.2014f has 8113 stations, with around 2000 of these being composite.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">As part of the creation of HadISD.2.0.0, we have also re-written all code into Python for ease of use - and as such we were able to check and in some cases alter some of the QC tests to work a bit better. We have also added new checks for wind speed and direction.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We believe that the result of these changes are that HadISD.2.0.0 is a more useful dataset for the study of extreme events, but also model validation, for ingestion into reanalyses and many other applications.</span><br />
<br />
<b><span style="font-family: "arial" , "helvetica" , sans-serif;">Update: January 20<span style="font-family: "arial" , "helvetica" , sans-serif;">16</span></span></b><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;">After some useful <span style="font-family: "arial" , "helvetica" , sans-serif;">review comments from the referees and a discussion with the ed<span style="font-family: "arial" , "helvetica" , sans-serif;">itorial team, <span style="font-family: "arial" , "helvetica" , sans-serif;">it was suggested that we re-submit th<span style="font-family: "arial" , "helvetica" , sans-serif;">is paper to <a href="http://www.geoscientific-instrumentation-methods-and-data-systems.net/"><span style="font-family: "arial" , "helvetica" , sans-serif;">Geoscientific Instrumentation, Methods and Data Systems</span></a></span></span></span></span> a partner journal of Climate of the <span style="font-family: "arial" , "helvetica" , sans-serif;">Past. As </span>we are currently updat<span style="font-family: "arial" , "helvetica" , sans-serif;">ing Had<span style="font-family: "arial" , "helvetica" , sans-serif;">ISD.1.0<span style="font-family: "arial" , "helvetica" , sans-serif;">.4, we w<span style="font-family: "arial" , "helvetica" , sans-serif;">ill do <span style="font-family: "arial" , "helvetica" , sans-serif;">t<span style="font-family: "arial" , "helvetica" , sans-serif;">his once <span style="font-family: "arial" , "helvetica" , sans-serif;">all the annual data<span style="font-family: "arial" , "helvetica" , sans-serif;">set updates are complete</span></span></span>, and at the same <span style="font-family: "arial" , "helvetica" , sans-serif;">time update HadISD.2.0.0 to in<span style="font-family: "arial" , "helvetica" , sans-serif;">clude data from 2015. We aim to resubmit this in early sp<span style="font-family: "arial" , "helvetica" , sans-serif;">ring.</span></span></span></span></span></span></span></span></span></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span>Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0tag:blogger.com,1999:blog-2393973536576253866.post-5690247914687004502015-04-29T09:48:00.000+01:002015-04-29T09:48:03.686+01:00Neighbour (buddy) Check for v2.0.0<span style="font-family: Arial,Helvetica,sans-serif;">In HadISD v1.0.x, the neighbour check selects stations within 500m height and 300km distance of the target station. The bearing is also used to assign stations to quadrants (90-degree bins), and the closest 10 are chosen, ensuring that each quadrant contains at least two stations. When fewer neighbours are available the distribution of stations across the quadrants can be lop-sided.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">For HadISD v2.0.0, I wanted to improve the station selection as, just because a neighbour is close, it may not be very useful when running the buddy checks. So the new neighbour selection uses the correlation coefficient of the target and neighbour time series as well as the data overlap (very important in early years). The details of both at this point are as follows.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">Initially stations are selected on the basis of distance to ensure that the neighbours experience similar weather as the target. Then, the correlation of the two timeseries is obtained. However, so that the correlation is not dominated by the annual or diurnal cycle, the timeseries are processed to removed these. Firstly daily means are calculated for all days which have more than 6 observations, which are used to create the climate anomalies for each observation. To further remove the diurnal cycle, hourly means are calculated and used to create "anomalised climate anomalies" for each observation. These time series are used to calculate the correlation coefficients.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">The reason for using the data overlap as another criteria results from the lengthened data coverage of HadISD v2.0.0. Few stations will have coverage over the entire 1931-2014 period, and so it would be highly likely that neighbours selected in terms of distance alone have no concurrent data. I use the fraction of observations that are also present in the neighbour as the overlap value. </span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">The neighbours are then sorted by the linear combination of the correlation coefficient and the overlap fraction, and the top 10 are selected, again ensuring that there are at least two in each quadrant if possible.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;">In a perfect world (or at least one with infinite computing resources), I would select all stations within the 500m height--300km distance criteria and calculate the correlations and overlaps for all. However, this takes a while (it probably could be faster, but at some level, lots of file-read operations have to occur) and it is important that this dataset can be quality controlled within a reasonable time frame. Therefore, at the moment, only the nearest 20 stations are assessed for their correlation and overlap with the target. </span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">The process appears to take around the 5 minute mark per station on a ~2GHz processor - so 28 days of processing for 8000 stations if done just one. I'm hoping to use many more than just one to do my bidding!</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: Arial,Helvetica,sans-serif;"><br /></span>
<br />Robert Dunnhttp://www.blogger.com/profile/17316439468151853162noreply@blogger.com0