Faster option for writing small data to Snowflake from code studio
Jonny2650
Registered Posts: 1 ✭
I have a little python streamlit webapp I've made in code studio that outputs data generated to a snowflake table that's in the same project. The way I'm attempting to do it (code below) is taking up to 2 minutes to upload the data but it's only ever a couple of rows of data (<10), 20 columns and no large fields. I would have thought for data this small that it would take under 30s. Is there a faster way of doing this?
with st.spinner('Uploading to Snowflake...'):
output_ds = dataiku.Dataset("output_dataset")
output_ds.write_with_schema(df_to_write)
st.success('Successful Upload into Snowflake!', icon=" ")
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,088 Neuron
How long does it take to run in a normal Python recipe?