Using Dataiku

What type of connection is your dataset using?

I will suggest you move away from partitioning data in Dataiku. Partitioning data in Dataiku does not improve performance like in other technologies, it merely avoids having to compute the whole dataset. This may reduce compute time but it introduces a whole set of limitations and issues when using partitions in Dataiku. If you only aim for using partitioning in Dataiku is performance you should move away from that and look at data technologies that can handle your datasets matching your performance requirements. A few samples are Databricks, BigQuery or Snowflake.

This method validates both datasets and recipes: https://community.dataiku.com/t5/Using-Dataiku/API-to-check-managed-SQL-dataset-schema-consistency/m-p/38407/highlight/true#M14238