How to Extract weather data of past 2 years using OpenWeatherMap Plugin

BhaskarDevGoel

How to Extract weather data of past 2 years using OpenWeatherMap Plugin? I want to find the temp and humidity of a location for past 2 years. Can i use the OpenWeatherMap API for the same? if YES, then how? and if NO, then what other options do i have in datiaku?

Sarina

Hi @BhaskarDevGoel
,

The plugin only provides data from OpenWeatherMap within the past 5 days or next 7 days: https://www.dataiku.com/product/plugins/open-weather-map/

Note as well that the plugin uses an API, so must adhere to OpenWeatherMap's API rate limit, which is by default 2000 calls per day: https://openweathermap.org/faq

Going through an API is probably not the most efficient way to get a large amount of historical data. If you really want a lot of historical data, finding a publicly available large historical dataset and them importing the dataset into DSS (or Cloud Storage and then connecting to it from DSS) is probably a good solution.

Thanks,
Sarina

tgb417

@SarinaS

Are you aware of any "... publicly available large historical dataset..." about weather?

Is there a python library or R package that make accessing this kind of thing easy to get from public data source?

Back in 2019 the Dataiku Dark Sky, Forcast.io, and Wunderground Plugins by Jérémy Grèze were very helpful in such use cases, were I was successfully able to pull 20 years of historical data for a particular location. Since those plugins were pulled, I've not found it as easy to answer the kind of question that @BhaskarDevGoel
has.

I suspect that pulling the plugins had more to do about changes at the data providers than at Dataiku. But it would be great to have a "semi-standard" way to do this kind of enrichment. Particularly if the current OpenWeatherMap Plug-In data goes back only 5-7 days. It is hard to build a model on this little data.

Should we put in a Product Idea for this one.