"schema aware" custom recipes? - make propagate schema changes work

Highlighted
UserBird Dataiker
Dataiker
"schema aware" custom recipes? - make propagate schema changes work
Jump to solution
Is there a way to make the "propagate schema changes" work with a custom recipe?

This would be very handy especially if there are large chains of custom recipes and something at the very beginning changes..
0 Kudos
1 Solution

Accepted Solutions
UserBird Dataiker
Dataiker
Re: "schema aware" custom recipes? - make propagate schema changes work
Jump to solution

Not possible: https://doc.dataiku.com/dss/latest/schemas/recipes.html#python-r-pyspark-sparkr



It would be marvelous for productivity throughout flow development to be able to pass a schema method to the input/output dataframes which constructs the schema when doing a schema propagation.




def get_ouput_schemas():
result = {}
schema1 = ds_input1.get_schema()
schema2 = ds_input2.get_schema()
schema_1, schema_2 = do_something(schema1, schema2)
result['output_name_1'] = schema_1
result['output_name_2'] = schema_2

dataiku.get_schemas(get_output_schemas)


 



 

View solution in original post

0 Kudos
1 Reply
UserBird Dataiker
Dataiker
Re: "schema aware" custom recipes? - make propagate schema changes work
Jump to solution

Not possible: https://doc.dataiku.com/dss/latest/schemas/recipes.html#python-r-pyspark-sparkr



It would be marvelous for productivity throughout flow development to be able to pass a schema method to the input/output dataframes which constructs the schema when doing a schema propagation.




def get_ouput_schemas():
result = {}
schema1 = ds_input1.get_schema()
schema2 = ds_input2.get_schema()
schema_1, schema_2 = do_something(schema1, schema2)
result['output_name_1'] = schema_1
result['output_name_2'] = schema_2

dataiku.get_schemas(get_output_schemas)


 



 

View solution in original post

0 Kudos
Labels (2)