Join us on July 16th as we explore real-world Reinforcement Learning Learn more

SQL Change Trigger in a Scenario is running the steps twice

Level 3
SQL Change Trigger in a Scenario is running the steps twice

Hi All

I have a FLOW which is automated thru a SCENARIO1 using SQL Change Trigger to run a python code STEP to RECURSIVE FORCE BUILD     TARGET1

FLOW1 = SOURCE 1 --> PREPARE--> TARGET1

SCENARIO1: def

SQL Change Trigger = select max(rec_created_dt) from SOURCE1

STEP= scenario.build_dataset(TARGET1, RECURSIVE_FORCE_BUILD)

in above STEP im extract job details(job_name,execution_time) from list_jobs and Target metrics(ex record_count) and inserting them into a REDSHIFT table .

Here i am seeing whenever SOURCE1 is loaded with new data SCENARIO1 is triggring twice and job is executing twice.What could be the reason ? any help would be much appreciated.

 

In first attempt i see 0 records being calcualted on TARGET1 but in second run actual record count is being calculated.

 

0 Kudos
2 Replies
Dataiker
Dataiker

Hi @rsingamsetty 

It's a little hard to figure out what might be causing this behaviour without specifics of your tables, scenarios and other processes settings. When you say the scenario runs twice, what's the time difference in between the two runs? Could it be that your data is written into the database in two batches? 

 

0 Kudos
Level 3
Author

Hi @Liev ,

I have figured out the issue. Actually from source side the tables are getting truncate and load

well truncate table is also activating trigger so i have to modify the sql statement to avoid activation with 0 counts 

 

Labels (3)