How to append or insert record in final hive table

Options
SUSHIL
SUSHIL Registered Posts: 22 ✭✭✭
in Using Dataiku

Hi,

I am using the excel file to input the data.

Based on the visual recipe, I am transforming the input excel file as per standard format and storing the output in hive table.

The question is I need to regularly need to change my input based on the business date and then upload. These needs to flow and do transformation as per steps provided in the recipe and then it need to insert value into hive final table without overwriting the existing output or drop the existing value in table.

Can you provide me the steps to do the configuration to achieve the above results.

I don't want to add multiple files in the input as I need to upload on regular basis with removing existing file ans upload the new file as per my business case

· Share on FacebookShare on Twitter

Answers

  • Manuel
    Manuel Alpha Tester, Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Dataiku DSS Adv Designer, Registered Posts: 193 ✭✭✭✭✭✭✭
    Options

    Hi,

    In the input/output settings of a recipe, you can tick the append box for the output dataset, see attached.

    I hope this helps.

    · Share on FacebookShare on Twitter
  • SUSHIL
    SUSHIL Registered Posts: 22 ✭✭✭
    Options

    Thanks for your help.

    If we use recipe with csv (hive compatiable) mode in output, the above option get enabled. So that i can able enable and run the flow successfully.

    The data gets append instead of overwrite

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…