Handling Concurrent Access (read & write) Standalone Dataset for Parallel Scenarios

Hello everyone,

I have a situation where I've created a standalone dataset to track the status of multiple scenarios. Each scenario reads the dataset, updates its corresponding row with the status 'Completed' and the 'Run date.' The challenge arises when these scenarios run in parallel and attempt to access the dataset simultaneously, resulting in an error stating that the root path of the dataset doesn't exist. This error is triggered because one scenario's custom Python recipe is trying to read the dataset while another scenario's recipe is attempting to write to it (building the dataset).

Running the scenarios in sequence is not an option since each one takes a considerable amount of time to complete. To address this issue, I'm considering partitioning the dataset to allow multiple recipes to read and write in parallel. However, I'm not sure if partitioning would be a viable solution, or if there are any settings I can modify within the dataset to enable concurrent access.

I'd greatly appreciate your insights and suggestions on how to achieve my requirement of maintaining a dataset that tracks the completion status of multiple scenarios while allowing them to run in parallel without conflicting access.

Thank you in advance for your help!