Store Json configuration in S3 bucket and loading it into Dataiku global variables

sonal_18
Level 1
Store Json configuration in S3 bucket and loading it into Dataiku global variables

Hello Team,

I'm have storeed my JSON configuration on a S3 bucket and i wanted load it into dataiku global variables. But i'm facing challenges with the code to do it.

 

Could you please help me with it?

0 Kudos
6 Replies
HarizoR
Developer Advocate

Hi,

First you will need to load the JSON data from your file stored in S3 into a Python dictionary. It is possible by using the get_object() method from the boto3 package (assuming that the code is executed with the relevant AWS IAM permissions).

From there, you can update the global variables using the Python client of Dataiku's API.

 

Hope this helps!

Best,

Harizo

0 Kudos
sonal_18
Level 1
Author

Hello @HarizoR ,

i'm trying to do this with the below code and im getting error at the last line while setting the project variable.

import dataiku
import json
import pandas as pd
configuration = dataiku.Folder("U44wokeq")
configuration.get_info()
with configuration.get_download_stream("NGE_ODH_R3_DEV_CONFIG.json") as f:
data = f.read()
my_json = data.decode('utf8').replace("'", '"')
# Load the JSON to a Python list & dump it back out as formatted JSON
data = json.loads(my_json)
s = json.dumps(data, indent=4, sort_keys=True)
client = dataiku.api_client()
project = client.get_project(dataiku.get_custom_variables()['projectKey'])
project_variables = project.get_variables()
project_variables['standard'] = s
project.set_variables(project_variables)

 

Error :

def _perform_empty(self, method, path, paraDataikuException: com.dataiku.common.server.DKUControllerBase$MalformedRequestException: Could not parse a ProjectVariables from request body

 

0 Kudos
HarizoR
Developer Advocate

Hi,

What you describe here differs quite a bit from your original question:

- You are connecting to your S3 data through a managed folder instead of direct access.

- You are setting up a project variable and not a global variable.

That being said, the fix for your problem should be simple: you should directly pass the data dict object to project_variables["standard"] and not the s string:

project_variables["standard"] = data

 

Best,

Harizo

0 Kudos
sonal_18
Level 1
Author

in the managed folder can i read "json" already stored in s3?

My use case is i want to read json configuration stored from s3 and load it to dataiku variables.

0 Kudos
sonal_18
Level 1
Author

@HarizoR  : It worked with "data". Thanks ๐Ÿ™‚

0 Kudos
CoreyS
Dataiker Alumni

Hey @sonal_18 we're happy you were able to find a solution! When you have a chance feel free to accept a solution to ensure others on the community can make use of it in the future!

Looking for more resources to help you use Dataiku effectively and upskill your knowledge? Check out these great resources: Dataiku Academy | Documentation | Knowledge Base

A reply answered your question? Mark as โ€˜Accepted Solutionโ€™ to help others like you!
0 Kudos