What are the available sources for historical weather data?

weather-file

asked 2014-09-01 10:26:57 -0600

Peter Ellis

1059 ●5 ●15 https://bigladdersoftware.com/

updated 2020-01-20 12:09:28 -0600

__AmirRoth__

4446 ●5 ●17 http://bleedinggreenna...

For some modeling projects you need to obtain a weather file for a historical period (e.g., the previous year) in a given location in order to do a calibration or other kinds of analysis. What are the available sources for historical, measured weather data? Either ready-made weather files or raw data of any interval (hourly, monthly) that could be used to create a custom weather file would be great.

edit retag flag offensive close merge delete

add a comment

26

answered 2014-09-05 03:31:37 -0600

Julien Marrec

29716 ●94 ●76 http://www.effibem.com/

updated 2016-11-09 04:42:04 -0600

What you are looking for is called an AMY - Actual Meteorological Year - file. The companies that offer this product, at a cost, are White Box Technologies, Weather Analytics (Update: no longer providing weather files since June 2015) and Weather Source. I'm not advertising, I'm paraphrasing DOE's EREE website where you'll find more information.

As far as finding the raw data:

NCDC/NOAA: Integrated Surface Database (ISD), or
Weather Underground: see Joe Huang's post on E+ support group mentioning some cautions, and Jamie Bull's python snippet to download data from Wunderground to CSV

If you're ever doing a project in Northern Europe, check out the Real-Time Weather Converter and the associated thread on bldg-sim.

edit flag offensive delete link

Comments

1

We've also used WeatherSource.com to obtain data. Not sure if it's better or worse than the sources Julien mentioned, but it may be worth looking at.

Anonymous ( 2014-09-15 11:02:00 -0600 )edit

1

@Peter Ellis Let me know if you need me to add something else in order for you to accept the answer and mark the subject as resolved.

Julien Marrec ( 2014-09-15 11:26:57 -0600 )edit

@JO: I added Weather Source per your suggestion. It was a bias I had, sorry. I read the name on EREE but because I wasn't familiar with it I chose not to include it. My bad.

Julien Marrec ( 2014-09-17 11:41:36 -0600 )edit

I marked this one as the best answer because it is the most comprehensive and covers multiple available sources. @Julien Marrec, the one small improvement would be if you could add links for each of the companies (White Box Technologies, Weather Analytics and Weather Source) that offer the product.

Peter Ellis ( 2014-09-24 03:29:52 -0600 )edit

@Peter Ellis: thanks for marking the answer. Regarding the link, if you think so, I'll do it. I had voluntarily not included the links in the first place and rather one to the EREE website in order to limit advertising.

Julien Marrec ( 2014-09-24 04:20:12 -0600 )edit

see more comments

13

answered 2014-09-16 23:17:07 -0600

Joe Huang
2772 ●5 ●14

updated 2015-01-16 22:39:31 -0600

White Box Technologies has just completed processing 2014 weather data for 8,862 locations around the world, including 2,202 in the US, 290 in Canada, and 2,763 in Europe. Since this number of locations is almost 40% more than that in the ASHRAE Handbook, WBT has also calculated DDY files for all locations based on the long-term historical record. WBT has also added a feature to its Web site allowing users to create composite weather files combining data from two different years. (added Jan 16 2015)

White Box Technologies provides historical weather data for over 8,000 locations around the world (2,200 in the US) from the current year going back to 2001, processed to various formats (*.BINM, *.epw, *.CSV, *.TM2, etc.) for use in simulations. The raw data are official weather station broadcasts archived by the National Climatic Data Center as the Integrated Surface Hourly Database (ISD).

These files are available for purchase at http://weather.whiteboxtechnologies.com .

(Joe Huang is the president of White Box Technologies, Inc.)

edit flag offensive delete link

Comments

INTERPRETATION AND DATA QUALITY ISSUES : AN EXAMPLE by statistical analysis of downloaded real-time weather data I found that sky condition and visibility for full hours (hh:00) were inconsistent with values between hours (hh:50, hh:20) for a European weather station. E.g. at full hours sky was never "Clear" , visibility ranging from 0-80km, whereas in between full hours visibility ranged from 0-10km; verbal classification of sky condition appeared to use different sets of words for (hh:00) and for (hh:20) & (hh:50) resp. with only a common subset - as if from two distinct data sets.

OS-user-AT ( 2015-02-24 05:08:45 -0600 )edit

@osuserAT Your post is ambiguous about the "real-time weather data", but since it follows my earlier answer I will assume that you're speaking of WBT historical weather data. In any case, I think you're being misled by the different conventions being used to denote unlimited visibility, since I don't know how anyone could even measure visibility beyond 10 km, especially at an airport where most of the weather stations are located. Therefore, your concern that visibility ranges from 0-80 km in the "real-time data" and 0-10 km in some other data is not an indication of poor data quality.

Joe Huang ( 2015-02-24 12:07:31 -0600 )edit

(sorry, my comment should have followed the original question, therefore nothing re. WBT) My data source states unambiguously that all visibility values are in km, hence there is no need to worry about coding conventions. But if values of 11,12,..,30, 50,60,75km occur at full hours, and never in between full hours, this needs explanation, and a decision on how to treat the data for conversion into epw file.

OS-user-AT ( 2015-02-24 16:57:10 -0600 )edit

I frankly think you're making a mountain out of a molehill. Whether visibility max's out at 80 km or 10 km, both basically indicate unlimited visibility. Visibility is also not used in EnergyPlus or any other building simulation program of which I'm aware. It's basically recorded by weather stations due to aviation needs, and happened to be included in the newer weather file formats like TMY2 and TMY3s and then carried over as an element to the EPWs in the spirit of completeness. How are you using Visibility anyway?

Joe Huang ( 2015-02-24 17:41:53 -0600 )edit

it turns out the meteorological data are derived from METAR code at times hh:20, hh:50, and from AAXX code at hh:00, i.e. the data were apparently merged from two different devices or measurement methods. As you point out, visibility is not used for EnergyPlus simulations, but another problem was how to estimate sky cover from verbal sky condition data (with open questions re. interpretation and data quality). Sky Cover and Visibility were included in the DEF file for generating epw converter estimates of solar radiation because the radiation data were also missing.

OS-user-AT ( 2015-02-25 06:54:17 -0600 )edit

see more comments

2

answered 2015-02-26 15:52:49 -0600

OS-user-AT

2284 ●3 ●14 https://mak-analytics.com

updated 2015-03-04 13:22:20 -0600

In addition to the above mentioned sources you can also use the following data source:

Iowa Environmental Mesonet (IEM): The IEM maintains an ever growing archive of automated airport weather observations from around the world. (ASOS Network / ASOS-AWOS-METAR Data)

Here is Iowa State University of Science and Technology's website, which offers free download of historical weather data (decoded and METAR):

http://mesonet.agron.iastate.edu/requ...

With data from this source I was able to produce epw weather files for EnergyPlus.

edit flag offensive delete link

Comments

@osuserAT: Could you please tell us the steps and/or assumptions that you made in creating epw file from this source. epw weather file needs more weather parameters than available at this website e.g. solar radiation (different types), snow depth, days since last snow fall etc. I think this will have influence on the results. thanks

Waseem ( 2015-03-05 05:02:13 -0600 )edit

@Waseem, the thread on this question is becoming a bit long. I think that there is certainly enough content here to start a new question.

Neal Kruis ( 2015-03-05 09:23:05 -0600 )edit

I agree to Neal Kruis (more questions to come !). re. Waseem's question: the steps of how to do it have been described here: https://unmethours.com/question/3083/... (GENERAL HOW-TO for PREPARING A CUSTOM INPUT FILE FOR THE EPW WEATHER CONVERTER). Solar radiation data etc. are not required. I intend to augment my answer there with more details depending on incoming feedback.

OS-user-AT ( 2015-03-05 15:10:20 -0600 )edit

add a comment

1

answered 2018-11-08 16:52:22 -0600

rafael.alonso
933 ●1 ●6 ●12 http://atmoslab.io

Another option for converting the DATA from NOAA is the opensource plugin, Dragonfly for Grasshopper/Rhinoceros, from Ladybug tools. I haven't tried it myself, but one of the example files of their webpage is regarding AMY processing to EPW.

edit flag offensive delete link

Comments

How do you generate the solar radiation data?

Joe Huang ( 2018-11-15 20:32:00 -0600 )edit

1

As far as I know, the files from NOAA include several fields for Solar radiation. However, I don´t think all the weather stations record it. When they are not included (filled with 999 or similar), I believe Dragonfly keeps the values from the reference EPW

rafael.alonso ( 2018-11-16 12:34:41 -0600 )edit

1

You might be interested in this discussion - the radiation data is drawn from ERA5 reanalysis data so should be much more accurate than other methods:

https://discourse.ladybug.tools/t/loo...

Joseph Yang ( 2020-08-11 07:49:39 -0600 )edit

add a comment

1

answered 2018-11-08 15:45:54 -0600

carlobianchi89
159 ●1 ●7

I would like to add to this a (hopefully) useful tool I created during my PhD research period: https://energysystems.mech.utah.edu/laf/

LAF is FREE and OPENSOURCE.

It allows the user to download weather data from more than 9000 stations in the US. The data are pulled from multiple networks, including airport data provided by the National Weather Service. The user can select multiple variables, multiple years, from multiple locations and multiple networks. The data are downloaded, polished and the gaps (if any) are filled to provide a CSV file with 8760 data points for each selected variable.

After that, the user can create AMY files, ready to be used in EnergyPlus and OpenStudio. The AMYs are created splicing the dowloaded weather data over the TMY3 file corresponding to the considered location.

The tool has been packed for Mac and Windows, but it is more reliable to launch it from Python directly.