Survey banner
The Dataiku Community is moving to a new home! Some short term disruption starting next week: LEARN MORE

Syncing S3 data to Snowflake and skipping malformed rows

yashpuranik
Syncing S3 data to Snowflake and skipping malformed rows

Hi All,

Suppose I have a malformed CSV in S3.

 

1.png

 

If I try to sync to Snowflake using S3 to Snowflake engine, I get the following error.

2.png

Of course I can get around it if I select DSS as the engine, but I don't want to unnecessarily use up a lot of disk space (even if temporarily) because I am looking to move massive datasets.

How do I modify the behaviour of the S3 to Snowflake stream as suggested in the error logs?

yashpuranik
0 Kudos
1 Reply
SarinaS
Dataiker

Hi @yashpuranik,

We have an existing request in our backlog to allow for use of additional Snowflake ON_ERROR options. At the moment, your only option would be to clean the data before the sync.  

I hope that information is helpful. 

Thanks,
Sarina

0 Kudos

Labels

?
Labels (1)
A banner prompting to get Dataiku