Testing SQL recipes

rafael_rosado97
rafael_rosado97 Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 61 Partner

Hello everyone

I'm doing a speed test.

Test 1: Make a query using an SQL recipe to a table found in my database 1; I send the result to the filesystem.

Test 2: Make a query using an SQL recipe to a table found in my database 1; I send the result to another database 2 (but it is located within the same server).

Test 1 runs faster than test 2. What can influence this behavior? Does this have to do with the resources where database 2 is located?

In case 1, is the SQL engine of my database 1 used?

Thank you!!

Tagged:

Answers

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,090 Neuron

    It seems to me you are merely measuring write IO performance. Databases are never going to be as fast as writing plain files as they have overheads they add like how the data is structured in the database, indexes, constraints, redo logs, etc. What you should keep in mind is that a RDBMS database will give massive performance when trying to query the data, join the data, group the data, etc. try to join two large file system datasets and see what happens. So your test results are expected.

  • rafael_rosado97
    rafael_rosado97 Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 61 Partner

    Thank you, @Turribeach
    !!

    I'm going to test it!

    Rafael.

Setup Info
    Tags
      Help me…