Write a dataset with more than 10k rows

Solved!
dharman
Level 2
Write a dataset with more than 10k rows

From a Python recipe, I am trying to write a dataframe from a SQL query that has a little over 13k rows. Using the following code, DSS is only writing 10k rows. How do I get the remaining rows?

 

DataWithSample_df = SampleActiveMem_df # SQL output that I've reviewed

# Write recipe outputs
DataWithSample = dataiku.Dataset("DataWithSample")
DataWithSample.write_schema_from_dataframe(DataWithSample_df)
for df in DataWithSample.iter_dataframes(chunksize=15000):
    DataWithSample.write_from_dataframe(DataWithSample_df)

 

0 Kudos
1 Solution
dharman
Level 2
Author

Solved: user error by misnaming a dataframe. My mistake

View solution in original post

2 Replies
dharman
Level 2
Author

Solved: user error by misnaming a dataframe. My mistake

CoreyS
Dataiker Alumni

Thank you for sharing your solution with the Community @dharman!

Looking for more resources to help you use Dataiku effectively and upskill your knowledge? Check out these great resources: Dataiku Academy | Documentation | Knowledge Base

A reply answered your question? Mark as โ€˜Accepted Solutionโ€™ to help others like you!
0 Kudos

Labels

?
Labels (2)

Setup info

?
A banner prompting to get Dataiku