Dataiku Named a Gartner Magic Quadrant Leader 2 Years Running! Read More

Getting DSS to recognize early Termination Python Code recipe. Do I use sys.exit() ?

Getting DSS to recognize early Termination Python Code recipe. Do I use sys.exit() ?

I'm writing some Python 3 code that processes each of my records from a DSS dataset. This code may run into problems with off-site REST APIs or unexpected data configurations issues.

My basic library for REST communications has try / catch so I do get error messages.  However, in the case of this script, I'd like to just terminate my Python script without trying any more records.  Disregarding all of the other records.

When I use something like:

rsp_df = pd.DataFrame.from_dict(json_normalize(rsp), orient="columns")

if rsp_df.columns[0] == 'error':
logging.error('We have a problem')
sys.exit() ## <-------- This successfully terminates


So the sys.exit() works.

However, DSS thinks that everything worked successfully.


How can my Python 3 code terminatte abruptly but tell DSS that there was a problem.


I'm clearly missing something. 

Thanks for any help you can provide.



2 Replies


Running sys.exit() means actually sys.exit(0), aka "terminate the current process and return the code 0"

Return code 0 means "normal completion". From the point of view of the caller of the Python process (i.e. the DSS job), a process exiting with code 0 means that everything happened normally. It would be exactly the same behavior as if the Python script had reached its end (there is an implicit sys.exit(0) at the end of any script)

To indicate an abnormal completion, you must call sys.exit(X) with X a non-zero integer. By convention, most programs just sys.exit(1).

DSS will consider that as an abnormal termination and will mark the recipe (and the job) as failed.

Another possibility is to raise an exception. The advantage of raising an exception is that you can give more information about the issue in your exception message, which DSS will display more prominently than just the logs.

For example you could: raise Exception("Something bad happened, df columns were %s" % (rsp_df.columns))

Thanks. I will take a look at this the next time I'm with my code. On the wrong laptop right now.
0 Kudos
A banner prompting to get Dataiku DSS