Syncing S3 data to Snowflake and skipping malformed rows

[yashpuranik]

Hi All,

Suppose I have a malformed CSV in S3.

If I try to sync to Snowflake using S3 to Snowflake engine, I get the following error.

Of course I can get around it if I select DSS as the engine, but I don't want to unnecessarily use up a lot of disk space (even if temporarily) because I am looking to move massive datasets.

How do I modify the behaviour of the S3 to Snowflake stream as suggested in the error logs?

[SarinaS]

Hi @yashpuranik,

We have an existing request in our backlog to allow for use of additional Snowflake ON_ERROR options. At the moment, your only option would be to clean the data before the sync.  

I hope that information is helpful. 

Thanks,
Sarina