Getting DSS to recognize early Termination Python Code recipe. Do I use sys.exit() ?
I'm writing some Python 3 code that processes each of my records from a DSS dataset. This code may run into problems with off-site REST APIs or unexpected data configurations issues.
My basic library for REST communications has try / catch so I do get error messages. However, in the case of this script, I'd like to just terminate my Python script without trying any more records. Disregarding all of the other records.
When I use something like:
rsp_df = pd.DataFrame.from_dict(json_normalize(rsp), orient="columns")
if rsp_df.columns[0] == 'error':
logging.error('We have a problem')
logging.error(rsp)
sys.exit() ## <-------- This successfully terminates
else:
logging.debug(rsp)
return(rsp)
So the sys.exit() works.
However, DSS thinks that everything worked successfully.
How can my Python 3 code terminatte abruptly but tell DSS that there was a problem.
I'm clearly missing something.
Thanks for any help you can provide.
--Tom
Answers
-
Hi,
Running sys.exit() means actually sys.exit(0), aka "terminate the current process and return the code 0"
Return code 0 means "normal completion". From the point of view of the caller of the Python process (i.e. the DSS job), a process exiting with code 0 means that everything happened normally. It would be exactly the same behavior as if the Python script had reached its end (there is an implicit sys.exit(0) at the end of any script)
To indicate an abnormal completion, you must call sys.exit(X) with X a non-zero integer. By convention, most programs just sys.exit(1).
DSS will consider that as an abnormal termination and will mark the recipe (and the job) as failed.
Another possibility is to raise an exception. The advantage of raising an exception is that you can give more information about the issue in your exception message, which DSS will display more prominently than just the logs.
For example you could: raise Exception("Something bad happened, df columns were %s" % (rsp_df.columns))
-
tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 NeuronOptionsThanks. I will take a look at this the next time I'm with my code. On the wrong laptop right now.
