PySpark exit recipe with Warning status

Options
jacksonisaac
jacksonisaac Registered Posts: 2 ✭✭✭
edited July 16 in General Discussion

I have a PySpark recipe which reads a dataset, and extracts a column based on first index (first row).

In a scenario when the input dataset partition is empty, it throws a normal error: 'index out of range'.

To handle this I created a try except block, and want to end the recipe at that except block. I tried sys.exit(1) which returns error, and sys.exit(0) which returns successful.

I wanted to know if there is any way I can signal the PySpark recipe to flag this partition job as warning (so I see 'Yellow' on that partition instead of Green or Red button).

Here's a code snippet I have now:

try:
    var_x = input_df.select('var_x').drop_duplicates().take(1)[0]['var_x']
except:
    logger.warn(f"Empty Input partition for {var_a}|{var_b}")
    sys.exit(0)

Answers

Setup Info
    Tags
      Help me…