API score Python Code optimization

[Kenza98]

Hello everyone, 

Hope you're doing well !

In order to optimize the following Python code that takes more than 6h40min for 4.5M rows,

# Compute recipe outputs from inputs
input_json = json.loads(input_df.fillna("").to_json(orient='index'))
row = 0
data = []    
while row < input_df.shape[0]:
    prediction = client.predict_record("random_forest", input_json[str(row)])
    prediction["result"]['Id_client'] = input_json[str(row)]['Id_client']
    prediction["result"]['proba_1']=prediction["result"]['probas']['1']
    row += 1
    data.append(prediction["result"])

application_scored_df = pd.DataFrame(data)


# Write recipe outputs
application_score = dataiku.Dataset("APPLICATION_SCORE")
application_score.write_with_schema(application_scored_df)

I wrote this code that uses the function "predict_records" instead of "predict_record",

#predict_records prend en param une liste de dictionnaires commençant par 'features' : [{'features':{...}}, {'features':{}}...]
l=[]
s=['features']
for i in range(0, input_df.shape[0]) :
    l0=input_df.iloc[[i,]].set_index([s])
    dicti=l0.to_dict('index')
    l.append(dicti)

prediction_test = client.predict_records("random_forest", l)
prediction_df=pd.DataFrame(prediction_test['results'])
prediction_df.insert(0, 'Id_client', input_df['Id_client'])
prediction_df['proba_1']=prediction_df['probas'].apply(pd.Series)[['1']]

application_score = dataiku.Dataset("APPLICATION_SCORE")
application_score.write_with_schema(prediction_df)

the code works very well on 5 rows but once applied on the full dataset, I have this error :

Do you know what this error is about and how can I solve this to reduce execution time ?

Thank's in advance,

Kenza 

[Clément_Stenac]

Hi,

For your actual error, could you please attach the full log (Actions > View full job log) ?

Since your code is running within DSS, do you really need to call an API node ? You could also use a scoring recipe locally, which would be much faster.

[Kenza98]

To use the scoring recipe, you have to be in the same flow as the one where the model was implemented, am I wrong ? 

For my case, I have the model in the development flow and want to use it in an other flow (production one), this is why I am using the API node.