Data not writing to dataset in Python recipe

petertsui
petertsui Registered Posts: 2

I created a Python recipe that I get data from an API. I save the response to a panda dataframe, but when I use the DataSet.write_from_dataframe to write the dataframe result my output dataset. The rows is written, but the row is empty when I check the dnb_duns_detail_json dataset.

Operating system used: MacOS

Best Answer

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,248 Neuron
    Answer ✓

    As you screen shot shows Dataiku does see the row but doesn't show it so I suspect you have applied a filter in the sample pane or you may need to update the sample. Click on whole data and make sure the sample is updated.

    Incidentally why are you using write_from_dataframe() instead of write_with_schema()? In Python recipes the recommended method is write_with_schema() so that the schema is updated as the data changes. You can still pass to it a data frame.

Answers

  • petertsui
    petertsui Registered Posts: 2

    Thank you for the feedback. I did as suggested and updated the sample, but still no data show up.

    You suggestion to use the write_with_schema work when writing the dataframe to a filesystem_managed dataset.

    The reason I used write_from_dataframe is because I also wanted to write the data into an Oracle database where I predefined the table structure. I needed to put the "response" column as a CLOB. I tried changing the "Table creation mode" and set it to Manually define without success.

Setup Info
    Tags
      Help me…