SQL Change Trigger in a Scenario is running the steps twice
Hi All
I have a FLOW which is automated thru a SCENARIO1 using SQL Change Trigger to run a python code STEP to RECURSIVE FORCE BUILD TARGET1
FLOW1 = SOURCE 1 --> PREPARE--> TARGET1
SCENARIO1: def
SQL Change Trigger = select max(rec_created_dt) from SOURCE1
STEP= scenario.build_dataset(TARGET1, RECURSIVE_FORCE_BUILD)
in above STEP im extract job details(job_name,execution_time) from list_jobs and Target metrics(ex record_count) and inserting them into a REDSHIFT table .
Here i am seeing whenever SOURCE1 is loaded with new data SCENARIO1 is triggring twice and job is executing twice.What could be the reason ? any help would be much appreciated.
In first attempt i see 0 records being calcualted on TARGET1 but in second run actual record count is being calculated.
Answers
-
It's a little hard to figure out what might be causing this behaviour without specifics of your tables, scenarios and other processes settings. When you say the scenario runs twice, what's the time difference in between the two runs? Could it be that your data is written into the database in two batches?
-
Hi @Liev
,I have figured out the issue. Actually from source side the tables are getting truncate and load
well truncate table is also activating trigger so i have to modify the sql statement to avoid activation with 0 counts
