How to Extract weather data of past 2 years using OpenWeatherMap Plugin
How to Extract weather data of past 2 years using OpenWeatherMap Plugin? I want to find the temp and humidity of a location for past 2 years. Can i use the OpenWeatherMap API for the same? if YES, then how? and if NO, then what other options do i have in datiaku?
Answers
-
Hi @BhaskarDevGoel
,
The plugin only provides data from OpenWeatherMap within the past 5 days or next 7 days: https://www.dataiku.com/product/plugins/open-weather-map/
Note as well that the plugin uses an API, so must adhere to OpenWeatherMap's API rate limit, which is by default 2000 calls per day: https://openweathermap.org/faq
Going through an API is probably not the most efficient way to get a large amount of historical data. If you really want a lot of historical data, finding a publicly available large historical dataset and them importing the dataset into DSS (or Cloud Storage and then connecting to it from DSS) is probably a good solution.
Thanks,
Sarina -
tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 NeuronOptionsAre you aware of any "... publicly available large historical dataset..." about weather?
Is there a python library or R package that make accessing this kind of thing easy to get from public data source?
Back in 2019 the Dataiku Dark Sky, Forcast.io, and Wunderground Plugins by Jérémy Grèze were very helpful in such use cases, were I was successfully able to pull 20 years of historical data for a particular location. Since those plugins were pulled, I've not found it as easy to answer the kind of question that @BhaskarDevGoel
has.I suspect that pulling the plugins had more to do about changes at the data providers than at Dataiku. But it would be great to have a "semi-standard" way to do this kind of enrichment. Particularly if the current OpenWeatherMap Plug-In data goes back only 5-7 days. It is hard to build a model on this little data.
Should we put in a Product Idea for this one.
