SQL Change Trigger in a Scenario is running the steps twice

rsingamsetty
rsingamsetty Dataiku DSS Core Designer, Dataiku DSS & SQL, Registered Posts: 18 ✭✭✭✭✭

Hi All

I have a FLOW which is automated thru a SCENARIO1 using SQL Change Trigger to run a python code STEP to RECURSIVE FORCE BUILD TARGET1

FLOW1 = SOURCE 1 --> PREPARE--> TARGET1

SCENARIO1: def

SQL Change Trigger = select max(rec_created_dt) from SOURCE1

STEP= scenario.build_dataset(TARGET1, RECURSIVE_FORCE_BUILD)

in above STEP im extract job details(job_name,execution_time) from list_jobs and Target metrics(ex record_count) and inserting them into a REDSHIFT table .

Here i am seeing whenever SOURCE1 is loaded with new data SCENARIO1 is triggring twice and job is executing twice.What could be the reason ? any help would be much appreciated.

In first attempt i see 0 records being calcualted on TARGET1 but in second run actual record count is being calculated.

Answers

  • Liev
    Liev Dataiker Alumni Posts: 176 ✭✭✭✭✭✭✭✭

    Hi @rsingamsetty

    It's a little hard to figure out what might be causing this behaviour without specifics of your tables, scenarios and other processes settings. When you say the scenario runs twice, what's the time difference in between the two runs? Could it be that your data is written into the database in two batches?

  • rsingamsetty
    rsingamsetty Dataiku DSS Core Designer, Dataiku DSS & SQL, Registered Posts: 18 ✭✭✭✭✭

    Hi @Liev
    ,

    I have figured out the issue. Actually from source side the tables are getting truncate and load

    well truncate table is also activating trigger so i have to modify the sql statement to avoid activation with 0 counts

Setup Info
    Tags
      Help me…