Join us on July 16th as we explore real-world Reinforcement Learning Learn more

Creating a data set from scratch in Python

Level 1
Creating a data set from scratch in Python


I would like to create a data set in a notebook out of financial data I am pulling out of a web service : Quandl.

They provide an API which allows me to download data in a dataframe

import Quandl as qd

import dataiku

import pandas as pd, numpy as np

from dataiku import pandasutils as pdu

df = qd.get("GOOG/NASDAQ_GOOGL")



DatetimeIndex(['2004-08-19', '2004-08-20', '2004-08-23', '2004-08-24', '2004-08-25', '2004-08-26', '2004-08-27', '2004-08-30', '2004-08-31', '2004-09-01',
'2015-09-24', '2015-09-25', '2015-09-28', '2015-09-29', '2015-09-30', '2015-10-01', '2015-10-02', '2015-10-12', '2015-10-13', '2015-10-14'], dtype='datetime64[ns]', name=u'Date', length=2803, freq=None, tz=None)

In [21]:



Index([u'Open', u'High', u'Low', u'Close', u'Volume'], dtype='object')

df is a dataframe.

I guess it messes up as the schema isn't initiated properly

I try to run:

 fdc = dataiku.Dataset("qdl")



and it fails with:

Unable to fetch schema for %s : %s'%(,err_msg)

Hence my question:

What is is the recipe/optimal way to create a dataset in python from a pandas dataframe




0 Kudos
1 Reply
Level 5

This works for me:

from dataiku import Dataset
import Quandl as qd
g = qd.get("GOOG/NASDAQ_GOOGL")
g["date"] = g.index

This is an easy way to initialize the schema (instead of defining it by hand). Once the schema is initialized, I can replace write_with_schema by write_from_dataframe.
0 Kudos
Labels (3)