Submit your use case or success story to the 2023 edition of the Dataiku Frontrunner Awards ENTER YOUR SUBMISSION

Syncing S3 data to Snowflake and skipping malformed rows

yashpuranik
Syncing S3 data to Snowflake and skipping malformed rows

Hi All,

Suppose I have a malformed CSV in S3.

 

1.png

 

If I try to sync to Snowflake using S3 to Snowflake engine, I get the following error.

2.png

Of course I can get around it if I select DSS as the engine, but I don't want to unnecessarily use up a lot of disk space (even if temporarily) because I am looking to move massive datasets.

How do I modify the behaviour of the S3 to Snowflake stream as suggested in the error logs?

yashpuranik
0 Kudos
1 Reply
SarinaS
Dataiker

Hi @yashpuranik,

We have an existing request in our backlog to allow for use of additional Snowflake ON_ERROR options. At the moment, your only option would be to clean the data before the sync.  

I hope that information is helpful. 

Thanks,
Sarina

0 Kudos

Labels

?
Labels (1)
A banner prompting to get Dataiku