Interrow calculations in Analysis

[UserBird]

Does the Analyse feature support interrow calculations similar to what we have with window functions in SQL. As a simple example get the column value of the previous row in an ordered dataset.

[jereze]

Hi Uli,

No, in an Analysis, each row or line is calculated separately (as it can be multi-threaded). So you cannot use window functions in an Analysis.

You should use a SQL, Python or R recipe to do that.

(Similarly to the sorting.)

We have on the roadmap a Window recipe (similar to the Grouping recipe) that will work on datasets managed by SQL databases supporting such functions (PostgreSQL, etc.).

I hope that helps,

Jeremy

 

[ubethke]

Hi Jeremy.

Thanks for that.
Let's assume I am using Spark as my execution engine and let's assume that I apply a Grouping recipe and then an Analysis where I replace a value in a column for example. When this is pushed down into Spark does it require two passes over the data or is there some optimization in the background that generates code with one pass only?