Incorrect column mapping in S3 dataset on Automation Node after redeployment

yasmine
yasmine Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 16 ✭✭✭✭

Hello,

We are facing an unexpected issue with an S3 dataset on the Automation Node.

The project uses S3 datasets:

  • Design Node: S3 connections pointing to pre-production data.
  • Automation Node: S3 connections pointing to production data.

The project had already been deployed several times successfully. However, after the 7th deployment, some columns of an input S3 dataset are no longer displayed correctly on the Automation Node.

More specifically, values belonging to one column appear under another column in Dataiku, as if the columns were shifted or incorrectly mapped.

The same production S3 data is displayed correctly in QuickSight, which suggests that the underlying S3 data itself is correct. The issue is only visible in Dataiku on the Automation Node.

These columns are used in a downstream join recipe, and because of this incorrect mapping, the join no longer finds matching records and the output dataset is empty.

Could you please help us understand the cause of this behavior and identify a solution to correct the column mapping on the Automation Node?

Thank you in advance for your support.

Best regards,
Yasmine

Dataiku version used: 14.7.2

Setup Info
    Tags
      Help me…