folder in s3 bucket in format /${project_key}/yyyy-mm-dd-filename.

One way to do this is by creating a variable ${current-yyyy-mm-dd-var} which you can spend append to the existing ${project_key}/${current-yyyy-mm-dd-var} the path of your S3 managed folder. But it's unclear if this is enough for your use case. If you need to create this dataset once or you need to create these every day at subfolders within your existing folder.

Here is a code sample on how to set the yyyy-mm-dd project variable.

import dataiku 
from dataiku.scenario import Scenario 
from datetime import datetime, timedelta, date 

client = dataiku.api_client() 
project = client.get_project(dataiku.default_project_key()) 
scenario = Scenario()

variables = project.get_variables()
current_date = str(date.today()) 
variables["standard"]["current-yyyy-mm-dd-var"] = current_date
project.set_variables(variables) 

Hi @Ankur30
,

Here is an example Python recipe that would do this.

import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu
from datetime import date

managed_folder_id = "xZUM5Sn9"
current_date = str(date.today()) 


# Read input dataset
my_dataset = dataiku.Dataset("my_dataset_name")
df = my_dataset.get_dataframe()

# Write recipe outputs
output_folder = dataiku.Folder(managed_folder_id)
#upload_data or upload_stream
output_folder.upload_stream("/" + current_date + "/" + "filename.csv", df.to_csv(index=False).encode("utf-8"))

The str(pd.to_datetime('now')) has a space in it.

e.g 2021-10-21 11:15:10.842966

That could be your issue when trying to check or retrieve the files, you can try looking directly in S3 to see if the files were created.

Can you share the code snippet and exact error from the job logs?

What is your dataset configuration? Blob path? This suggests the datasets can't connect to the specified path configured.

Perhaps continuing this over a support ticket and sending us the job diagnostics is best course of action here.

Unaware exactly what dataset type is failing to build if this is an S3 dataset or Files from Folder etc.

If it's an S3 Datasets you can find the details :

For Files in Folder Dataset :