I have my final dataset and but I want to change the column names based on a naming convention at the end of the code. Right now I'm using
There are no errors in the code, but when I go to explore the data, 'the relation does not exist'.
After Settings > 'Test Connection' and 'Create Table Now' works fine... the table has 0 records with the correct column names.
Is there a better function to do this?
The query has the potential of returning hundreds of millions of rows. Knowing this ,I was avoiding loading into Dataframes. (requires loading data through DSS memory right?)
If you do not want to go through pandas.DataFrame() then you can use another part of the Dataiku API:
This is assuming your input and output datasets are in SQL or HDFS connections. Otherwise, you will be required to go through the "regular" dataiku API using pandas.DataFrame. Note that you can do chunked reading and writing: https://doc.dataiku.com/dss/latest/python-api/datasets.html#chunked-reading-and-writing-with-pandas
Hope it helps,