Request for Guidance on Adding a Column Name to Dataset Path for Versioning System Implementation

Options
Suhail
Suhail Registered Posts: 18 ✭✭✭✭

Hello Everyone,

I am currently working on implementing a data versioning system for my dataset. One approach I am considering involves appending a timestamp to each version and organizing the data into separate directories.

To accomplish this, I am planning to incorporate a version column into the dataset and use it to modify the s3 file path based on the associated timestamp. My inquiry pertains to the method for adding a column name to the dataset path.


Operating system used: macos

Answers

Setup Info
    Tags
      Help me…