I have several Redshift datasets that have already been built in a project, and was trying to deploy some very basic recipes. For instance, on one dataset - let's call it dataset X - I was simply trying to create a new dataset with only the first two columns of X using a Redshift connection.
When I go to deploy the script (there are roughly 1.5 million rows and 2 columns that I want to write out), we see some errors. Upon inspecting the log files, I see the following issue:
It appears that writing out 10,000 lines is taking on average 5 minutes, which seems like a very long amount of time. I have tried implementing this process both using a visual lab and python recipe, but the same issue occurs in both scenarios.
I ended moving the DBs from Redshift and saving them as new DBs using a Postgres connection, and then everything went very fast again. So, I feel as if this must be some issue with SQL and Redshift not playing well together, but I'm not sure exactly why. Any insight into this would be very helpful; thanks!