Years versus file size

During the review of the HadISD paper (see documents on Climate of the Past Discussions) we were asked to quantify how many stations report for how long in the ISD (Integrated Surface Dataset).  Our comment that "many stations report only rarely" could have been misleading.  We therefore did a quick analysis of the stations in the ISD in July 2012.  I've just re-run the code to update to the current status of the ISD, and thought the results might be of wide enough interest to not remain buried on the discussion paper.

Of the 29,678 unique station IDs present in the database (on 20 June 2013), 14,159 report in fewer than 10 years (almost half), almost 18,921 for less than 20 and almost 21,962 for less than 30 years.  One station reports in 81 years.  The mean length is 18.2 years, but the median is only 11.  The distribution is shown in Fig. 1.

Fig. 1 Number of stations against the number of years they report for.  The spike at 40 years is the result of a sudden increase in the number of stations in 1973.

We used the file size in bytes as an indicator of the number of records, as a station that reports only once a year for many years is not much use for climatological studies.  Most stations had sizes between 10^5 and 10^7 bytes.  The figure below shows the distribution of the number of years a station reports for against the file size.  The colour scale is logarithmic.

Fig. 2 Years with records in the ISD against total file size over all years for each station ID.  Created on 20/7/2013

This shows that there are some stations which have lots of data in them but only for a small number of years (bottom right of the figure).  The apparent diagonal cut-off, from bottom left to top right, shows the link between file size (in bytes) and years which have data, assuming a fairly constant set of reported variables.

The file size is not a perfect proxy to use when trying to assess the completeness of a record, but if combined with the number of years in which a station reports, many stations which only report for a few years or containing very little data can easily be excluded from any station selection made.

Station Reporting Interval

While working on a related project, the issue of station reporting interval came up.  The HadISD stations were selected to try and chose those in the ISD which reported very hour or every 3 hours.  To investigate how many of which type occurred at any year, I made the following plots.

Number of stations at each reporting interval

Proportion of stations at each reporting interval

The first plot shows the number of stations at each reporting interval.  I've included the option of 6 hourly as well.  Firstly, there is a drop in the number of active station after around 1990.  This plot shows the reporting intervals for the ~4200 "accepted" stations, i.e. those which are thought to be suitable for climatological studies.  The second plot shows how the proportion of the different reporting intervals changes.  The number of three hourly stations falls off with time, and the number of hourly stations increases.

If your application could depend on the reporting interval (or changes in it) then this may be of use.

Quality Control Code Released

We have released the IDL code which performs the detailed quality control (QC) on the HadISD data.  This is the code as it was used for the v1.0.1.2012p release.  We are not supporting the code, but there is a README file included in the zip archive, which can be found here.  Most of the necessary files should be available in the online material.  However if there are any input files or IDL program files that you feel are missing please do let us know so that we can include them or explain why they have not been included.

The code that has been released is scientific code.  During its development we have tried to ensure its readability.  However it is a complex piece of code, and we welcome all suggestions on how to improve it for future versions of HadISD.  The dataset is still under development, and so we envisage there being changes in the code (both language and logic) in the future.

Message from NCDC ISD team

There is a message from the ISD team at NCDC on the ISD FTP server about a problem with their merging process:

NCDC has identified a problem with merged ISD data. Under certain circumstances station merging is not taking place when it should, resulting in unsorted, non-merged output. This problem may be found in files on this FTP server from 2005-present. 

We are working to correct this issue and we apologize for any inconvenience this may cause.

This may have affected v1.0.1.2012p and we are investigating.  We will wait until this issue is resolved before finalising this update and creating v1.0.1.2012f.

Please let us know if you find anything untoward in the data.

2 Stations with the Wrong Latitudes

Thanks to Tim McVicar (CSIRO) we've found two stations that have been listed with the wrong latitude in our station listing files:

917650 61705 PAGO PAGO       +14332 -170711 +00030  (should be -14.3, -170.7)
    (American Samoa)
619670 70701 DIEGO GARCIA +07300 +072400 +00027  (should be -7.3, 72.4)

Both of these stations have the wrong latitudes in both v1.0.0.2011f and currently v1.0.1.2012p.  The listing files will be updated and synchronised with the ISD listing files before running the final v1.0.1.2012f quality control run.

Please do let us know if you find any other quirks in the data.

See also comment here regarding affect on HadISDH.

Version 1.0.1.2012p

#!/usr/bin/python
print "hello world"

This blog will hopefully act as a repository for ongoing developments with the HadISD dataset.  As this is a new dataset, all kinds of quirks or short interesting results are likely to come up, which might be of interest to users, and we can try and highlight these here. 

The current stable version of HadISD is 1.0.0.2011f, however, a preliminary version has just been released (1.0.1.2012p).  Although these versions are just a mass of numbers, hopefully the following will make them clear.

• The final letter indicates whether the dataset is f-final or p-preliminary.  A final dataset is stable, and will not change, whereas this preliminary dataset will be updated and overwritten at some point in the future.  This is because the source data for HadISD, the Integrated Surface Database (ISD), is still being updated for the 2012 data.  Once ISD is stable, we'll re-run our scripts and update the data
• The year stamp indicates the final complete year in the dataset (the first year is 1973, and for the moment is unlikely to move to earlier years)
• The remaining three numbers indicate major-moderate-minor changes. Major changes (e.g. a complete re-write of our scripts) would require an accompanying publication in a peer-reviewed journal.  Moderate changes (e.g. an update to one of the QC tests) would have a technical note to explain the change and the resulting differences available on the website.  Minor changes include update to past years (not just the most recent) and will be noted on the download page.

Now it should be nice and clear - the dataset is a preliminary version of the one which includes data up to the end of 2012.  There has been a minor change in one of the QC tests (the global and African record high temperatures have changed, see paper and WMO press release) and the ISD maintainers updated data in the years 2004-9 and 2011.

We have checked through some of the summary and diagnostic plots (all available on the website too) and found no major unexpected changes in the flagging rates, however it is still a preliminary dataset.  If you find anything untoward or have problems please contact the maintainers through the HadISD website.