Interrow calculations in Analysis

Options
UserBird
UserBird Dataiker, Alpha Tester Posts: 535 Dataiker
Does the Analyse feature support interrow calculations similar to what we have with window functions in SQL. As a simple example get the column value of the previous row in an ordered dataset.

Answers

  • jereze
    jereze Alpha Tester, Dataiker Alumni Posts: 190 ✭✭✭✭✭✭✭✭
    Options

    Hi Uli,

    No, in an Analysis, each row or line is calculated separately (as it can be multi-threaded). So you cannot use window functions in an Analysis.

    You should use a SQL, Python or R recipe to do that.

    (Similarly to the sorting.)

    We have on the roadmap a Window recipe (similar to the Grouping recipe) that will work on datasets managed by SQL databases supporting such functions (PostgreSQL, etc.).

    I hope that helps,

    Jeremy

  • ubethke
    ubethke Registered Posts: 21 ✭✭✭✭
    Options
    Hi Jeremy.

    Thanks for that.
    Let's assume I am using Spark as my execution engine and let's assume that I apply a Grouping recipe and then an Analysis where I replace a value in a column for example. When this is pushed down into Spark does it require two passes over the data or is there some optimization in the background that generates code with one pass only?
Setup Info
    Tags
      Help me…