topic How can I compute an exponential moving average in a preparation script? in Using Dataiku DSS
https://community.dataiku.com/t5/Using-Dataiku-DSS/How-can-I-compute-an-exponential-moving-average-in-a-preparation/m-p/275#M16
Thu, 07 May 2015 03:06:35 GMTUserBird2015-05-07T03:06:35ZHow can I compute an exponential moving average in a preparation script?
https://community.dataiku.com/t5/Using-Dataiku-DSS/How-can-I-compute-an-exponential-moving-average-in-a-preparation/m-p/275#M16
Thu, 07 May 2015 03:06:35 GMThttps://community.dataiku.com/t5/Using-Dataiku-DSS/How-can-I-compute-an-exponential-moving-average-in-a-preparation/m-p/275#M16UserBird2015-05-07T03:06:35ZRe: How can I compute an exponential moving average in a preparation script?
https://community.dataiku.com/t5/Using-Dataiku-DSS/How-can-I-compute-an-exponential-moving-average-in-a-preparation/m-p/276#M17
Currently the visual preparation recipes are working line by line. This allows to work on very large datasets by streaming, potentially in parallel on Hadoop. The downside is that each line is processed independently, so, an exponential moving average cannot reliably be done in this type of recipe.<BR /><BR />If the dataset fits in memory, I would go with a Python recipe and use the functions from Pandas.Thu, 07 May 2015 03:09:13 GMThttps://community.dataiku.com/t5/Using-Dataiku-DSS/How-can-I-compute-an-exponential-moving-average-in-a-preparation/m-p/276#M17jrouquie2015-05-07T03:09:13ZRe: How can I compute an exponential moving average in a preparation script?
https://community.dataiku.com/t5/Using-Dataiku-DSS/How-can-I-compute-an-exponential-moving-average-in-a-preparation/m-p/9317#M4582
<P>Thank you Jrouquie.</P><P>Struggled with this as a new user so wanted to share the Python/Pandas recipe. I am strugling to post all of the code in - even if in a code block.</P><P>import dataiku<BR />import pandas as pd, numpy as np<BR />from dataiku import pandasutils as pdu</P><P># Read recipe inputs<BR />autoSleep_20190303_20200606_prepared = dataiku.Dataset("AutoSleep_20190303_20200606_prepared")<BR />autoSleep_20190303_20200606_prepared_df = autoSleep_20190303_20200606_prepared.get_dataframe()</P><P>#Apply rolling average here<BR />autoSleep_20190303_20200606_prepared_df['bedtime_fortmattedAvg7']=autoSleep_20190303_20200606_prepared_df['bedtime_formatted'].rolling(window=7).mean()</P><P>#Output to Dataiku<BR />autoSleep_RollingAverage_df = autoSleep_20190303_20200606_prepared_df</P><P>#Update column with MA<BR />autoSleep_RollingAverage = dataiku.Dataset("AutoSleep_RollingAverage")<BR />#autoSleep_RollingAverage.write_with_schema(autoSleep_RollingAverage_df)</P>Sat, 22 Aug 2020 20:34:08 GMThttps://community.dataiku.com/t5/Using-Dataiku-DSS/How-can-I-compute-an-exponential-moving-average-in-a-preparation/m-p/9317#M45827TonRobot2020-08-22T20:34:08Z