Need help with Formula language in Custom Python functions in the Prepare Recipe

Hi @SohilRaceQuant
,

Formula language is not suitable for your use case.

One way to achieve what you want is to create a custom metric that calculates the average age for each 'Pclass' group excluding the empty age rows:

import numpy as np

def process(dataset, partition_id):
    df = dataset.get_dataframe()
    df.Age.fillna(np.nan, inplace=True)
    df_query = (df
                        .groupby('Pclass')
                        .agg({'Age':np.nansum})
                    )

    d={}
    
    for i in range(len(df)):
        d["Pclass_"+str(df.iloc[i, 1])]=df.iloc[i, 0]

    return d

After that you can access this metric in a custom Python function using Python API. In order to use dataikuapi package in a custom Python function of a prepare recipe you need to enable the option Use a real Python process (instead of Jython):

This is an example code that shows how to retrieve the custom metric value:

import dataikuapi

host="http://localhost:11200"
apiKey = "***********************"

def process(row):
    client = dataikuapi.DSSClient(host, apiKey) 
    project = client.get_project("COM31578")
    dataset = project.get_dataset("input")
    source_metrics = dataset.get_last_metric_values()
    if row['Age'] is None:
        return source_metrics.get_metric_by_id("python:Pclass_"+row['Pclass']+":avg")['lastValues'][0]['value']
    else:
        return row['Age']

I hope this helps.