Multivariate LSTM in Dataiku

Options
Darius679
Darius679 Registered Posts: 6 ✭✭✭✭

I've completed this tutorial on how to implement an LSTM in dataiku.

https://knowledge.dataiku.com/latest/kb/analytics-ml/time-series/ts-forecast/time-series-code/deep-learning-ts.html

Basically I want to understand how to adapt this for multiple variables.

I have a data set that has many 'runs' of a chemical process over a 10 day period, and different observations of the process over those 10 days like pH, temperature, etc. I want to use an LSTM that will predict the observations of the next day given a a few of the previous days.

I can't find anything like this or any examples of any multivariate LSTMs implemented in dataiku. Do you know of any tutorials that could help?

Best Answer

  • Darius679
    Darius679 Registered Posts: 6 ✭✭✭✭
    Answer ✓
    Options

    I have figured it out after much effort. The gist of the changes are

    • Making the windows -- they should have the format [observations for time 1, observations for time 2 ...] for however long you want your windows to be.
    • the windowprocesser.py must change to accommodate a longer array of length window_size * num_features. You should also remove the scaling because it doesn't respect the ranges of different variables, only taking the max/min
    • When building the model, you should change the 1 to num_features in the reshape layer. This makes the array into the right shape for training the LSTM

Answers

  • CoreyS
    CoreyS Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Registered Posts: 1,150 ✭✭✭✭✭✭✭✭✭
    Options

    Thank you for sharing your solution with the rest of the community @Darius679
    !

  • NIDHS
    NIDHS Registered Posts: 1
    Options

    Thanks for sharing this. Could you maybe share the snippets of the changes that needs to be made to windowprocessor.py and Architecture to run multivariate LSTM?

Setup Info
    Tags
      Help me…