Survey banner
Switching to Dataiku - a new area to help users who are transitioning from other tools and diving into Dataiku! CHECK IT OUT

Optimizing data upload to Teradata

Level 2
Optimizing data upload to Teradata
Can someone provide a high level overview of how DSS uploads data to Teradata? Is it just submitting a batch of inserts or using any Teradata utilities such as FastLoad or MultiLoad?

I need to move ~70M rows (about 8 CHAR columns of ~2-20 width and a couple INT columns) from SQL Server to Teradata where the bulk of my data resides to continue processing in-database but the upload to Teradata is rather slow. For my dataset it took approximately 7 hours.

I'm looking for any suggestions on optimizing the process. Thanks!
0 Kudos
1 Reply
Dataiker Alumni

When syncing a dataset from SQL server to Terada, DSS submits a series of batch inserts. As of the latest release today, it is not using the specific Teradata utilities you mention.

In the specific case of syncing between Teradata and HDFS, we do use the fast TDCH method:

To check if there can be any optimization, can you please add a screenshot to your question with the output dataset "Settings > Advanced" tab ?

Best regards,

0 Kudos


Labels (2)
A banner prompting to get Dataiku