Recipe output setting: write to temp table first

natejgardner
natejgardner Neuron, Registered, Neuron 2022, Neuron 2023 Posts: 151 Neuron

Some recipes take a very long time to execute. During execution, target datasets are typically unavailable since they've been dropped prior to data insertion. A current workaround is to add a sync recipe to these datasets, allowing them to be copied. However, this leaves duplicate tables sitting in the database and requires a manual disable for projects with pipelining enabled. This problem could be solved with a temporary table mode. If the output of a recipe could optionally be written to a temporary table to completion, then copied to the permanent target table, that would solve this problem without creating intermediate datasets or requiring special configuration for pipelining. The temporary table could be deleted after data is copied to the permanent table so space is conserved and so the flow remains uncluttered.

0
0 votes

New · Last Updated

Setup Info
    Tags
      Help me…