How do I preserve global variables in prod after the import of a new bundle?

[AnnaProba]

Hello

I have a development project with project Global Variables for example

x = 10

This variable is updated by some of the steps in my scenario. Each time a scenario runs it will change the value of x.

I want to put my code in production.

For this, I will create my 1st bundle with the global variable "x" set to 0. In prod, the scenario will run for some time and the value of x will evolve. 

Now, I have made some modifications to my code and want to update the bundle in prod. Let's say, the current value of x in prod is 127 but I don't know this. I have no way of knowing it.

Question: How should I now set up my variables in the project development bundle if I don't know what the current value of x is in prod and I don't want to modify it?

I want the new bundle in prod to use the x that was established in the prod by the first bundle and not start from 0.

• Should I leave global_variables undefined in develop and set x only as a local variable?
• If the global variables are undefined, would it delete the global variables in the prod?
• How do I preserve global variables in prod after the import of a new bundle?

[NN]

Hi ,
I think you should Define Local Variables instead of global for specific instance.
Local variables are not included in the bundle so will not get overwritten in prod.

[Turribeach]

Luckily for you Dataiku already has this subject very well covered. 

• Should I leave global_variables undefined in develop and set x only as a local variable?
• No. Always define global. But only override local in Development. Since local overrides don't get added to the bundle.

• If the global variables are undefined, would it delete the global variables in prod?
• Yes. Whatever it's on your global variables will be taken in the bundle and applied to production. So if Global Variables are empty in Dev it will end up empty in Prod.

• How do I preserve global variables in prod after the import of a new bundle?
• Simple: Always define your variables in Global Variables and always override on Local Variables. That way all your variable definitions are carried to all environments and all your local overrides are preserved in each environment. Do not update Global variables unless you want that variable value to apply to all environments.

[AnnaProba]

Thank you @neuralmarket and @Turribeach for the answers but I am still not sure how to proceed.

Let's say the bundle1 has the following variables:

Global: {x=0}
Local: {}

Then bundle1 is sent to prod. At each trigger of a scenario x changes.
At some point in time in prod, x=127.

I am preparing a bundle2 to update the code in production, but I don't want to change the value of x (being 127 in prod but I don't know what the actual value is. It could be anything. Only prod knows.).

How do I set up my global and local variables in bundle2 to not reset the value of x in prod?:

Global:?
Local:?

When bundle2 goes into prod, the x should continue to evolve from the value it had before bundle2 went into prod (127).

Should I leave Global: {} or just empty?

[Turribeach]

Like I said do not override global variables if you need to keep them different in different environments. 

Dev:

Global: {x=0}
Local: {x=1}

Prod:

Global: {x=0}
Local: {x=127}

You can deploy any bundles you want from Dev to Prod and x will always be 127 or whatever value the last run set. Change your code to only override local variables for those variables that need to be kept different in each environment. Then it doesn't really matter what's on the bundle as it will never touch the local variable. 

See this post on how to update local variables:

https://community.dataiku.com/t5/Plugins-Extending-Dataiku/Update-Local-Variable-with-Python/m-p/348...

[AnnaProba]

Thanks  @Turribeach, I think I am getting closer to the solution. I beg you for a bit more patience with me 😉 I need to clarify one thing, currently, we don't use local variables neither in dev ni in prod! Everything is done with global variables. Global variables get updated by scenario in prod. Not local ones!

And I see now that this is problematic!!

I could envision the following scheme with two variables

bundle1
Global: x=0, update_local_x=True

bundle1 goes into prod where we copy value x into local variable x which will be updated by the bundle (because update_local_x=True).

At time=t0
Global x=0
Local x=0 (updated from global x)

At time t1
Global: x=0
Local: x=127 (updated by scenario)

Now we want to update the code, we create bundle2

bundle2
Global: x=0, update_local_x=False

We are now in prod with bundle2

At time t1
Global: x=0
Local: x=127

Then in some future event where I need to reset the x into some specific value I could do it in bundle3 with

Global: x=53, update_local_x=True

At time t3
Global: x=53
Local: x=53

Does this make sense?

So in summary, the prod, scenarios must only update local variable x, it should not touch global one.

[Turribeach · Accepted solution]

You are making this more complicated that it needs to be. This is how you should work:

1. All variables are always defined in Global Variables
2. If the variable is environment specific you will only override it on Local Variables in your scenario code. Otherwise you can override the variable on Global Variables. Crucially when you set a Local Variable with the code I suggested it the local variable doesn't have to exist or be defined already so all your logic around update_local_x is pointless and superfluous. 

The above two rules will work in all situations other than on the first ever deployment in Prod where x=0 and the local override will not exist. If there is a specific first time value for x that you need to use in your first ever run which is different than x=0 then your scenario code should handle that. Otherwise again I see no point in all your update_local_x since after the first run the scenario will override Local x with whatever value it wants and that value will not be changed by new bundles. And I fail to understand why you think you need to have Local x=0 for the first run. 

[AnnaProba]

Ok, I think I get it. Thanks