Using Dataiku

Hi All,

I'm trying to export my final df of a python recipe to a managed folder & I get the below mentioned error:

Job failed: Error in Python process: At line 35: <class 'Exception'>: Managed folder EyABfYFjV cannot be used : declare it as input or output of your recipe

Any code snippet via which I can push my df data into a csv file & then when I go back to the recipes it should show the managed folder icon wherein my file should be present.

Thanks, I will try doing it thatway

In terms of only loading new files there are no built-in ways of doing this so you will have to develop custom Python code. Ideally you would want to have subfolders within your folder and move files as you process them. For instance files arrive in ./landing then they get moved to ./processing for loading and then after successful load you move them to ./loaded. That way you always know what files you processed and if it fails loading a file you have right there in the ./processing folder to fix the code or the file and try again the same file. This post will give you some guidance but you will need Python skills to achieve this requirement.

The simplest would be to use the Classify Text recipe showcased in this tutorial.

It does require a LLM connection, which can be either

• an API-based LLM (most are paid, some provide a free tier or trial)
• a Huggingface connection for local inference, which requires access to a GPU-enabled Kubernetes cluster

Hi @DogaS
,

This error usually occurs if there are null values in the input data (in this case, it would be the date column). Can you please check? If there are nulls, I'd recommend filling or removing them.

Note, you can fill empty values with a Prepare recipe using processor "fill empty cells...".

Thanks!

Hi @Devian31_M
,

The simplest way to upload an XML to DSS is by creating a dataset from the UI. In this case, the DSS autodetects the format (XML) and the schema (XPath).

For example, you could upload XML files to a managed folder (or connect your folder to an external database with XML files) > select the menu icon on the XML file > "create a dataset". In the next window, select "test & get schema" then "load preview"

And, the output should look something like this:

This is the simplest way to load an XML file. To do this in Python recipe instead, you will need to use pandas read_xml, in which case you would need to specify the XPath. You would also need a code env with pandas 1.3 and the library "lxml" installed in the env.

import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu

# Read recipe inputs
XML_folder = dataiku.Folder("3trhpQ3P")

with XML_folder.get_download_stream("books.xml") as f:
    data = pd.read_xml(f, xpath='/catalog/book')

books_df = data

# Write recipe outputs
books = dataiku.Dataset("books")
books.write_with_schema(books_df)

Let us know if you have questions!

Thanks