BigQuery connection

rafael_rosado97
rafael_rosado97 Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 61 Partner

Hello, Dataiku Team.

If I want to deploy a project on Automation node, but my project has a connection with BigQuery, I need to create the same connection for Automation node, right?

The thing is that I do not have the same options in the automation node as design node.

In design node I have set a section called Path restrictions ----> Bucket and Managed datasets & folders ---> Managed data subpath. But there are not these options in automation node as you can see on images.

Do you know how I can set this options?

Thank you!!

bq_1.png

bq_2.png


Operating system used: Linux

Tagged:

Answers

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,090 Neuron

    I think you are getting confused. Below is my Designer New Google BigQuery connection screen, I am on v12.6.3. There is no concept of "path" in BigQuery connections and you can't create folders in BigQuery either. I think you are confusing the connection with Google Cloud Storage (GCS) which is a bucket connection not a BigQuery connection. And I think you are getting confused because BigQuery needs / uses a Google Cloud Storage for fast writes to allow new data to be loaded in BigQuery. So a connection to BigQuery usally involves two Dataiku connections, one for Google Cloud Storage and one for BigQuery which uses the Google Cloud Storage for staging data.

    Screenshot 2024-06-05 at 23.03.43.png

    Screenshot 2024-06-05 at 23.06.29.png

Setup Info
    Tags
      Help me…