Using Dataiku

I think you should go back to your deployer node and go to your project's bundle deployment interface and go to the update logs and look at the traceback in more detail because you'll have more information. I don't see how I can help you without more information. :(

sometimes you have this kind of error because the bundle version doesnt match with the current project deployed.

I think you should go back to your deployer node and go to your project's bundle deployment interface and go to the update logs and look at the traceback in more detail because you'll have more information. I don't see how I can help you without more information. :(

sometimes you have this kind of error because the bundle version doesnt match with the current project deployed.

Sorry but this is a horrible solution. I am sure there are much better alternatives. Why can't you use a standard Sync recipe to copy the files between folders? Why are you not allowed to use Python? Dataiku without access to Python is pretty much useless. This doesn't sound right, there are aspects that you are not really clarifying. Can you explain the requirement in detail?

We don’t have all the facts to compare. You say you are running the queries in Dataiku, what specific Oracle driver version are you using in Dataiku? What Oracle database version do you connect to? Where is your DSD node located? Where are your docker containers located? What exactly do you mean by “I execute the same query from a Docker container on AWS”? You need to describe the whole technology stack there including all the driver and software stack versions used.

Finally how do you do go from a query that takes 1 second to run to taking 2 days? Are you running SQL queries in a for loop? That’s a really bad pattern. Explain exactly what is your requirement and how you are trying to achieve it.

@esmvy ,

I don't have a good answer for you at this time. And I don't have a bunch of time to followup. Several thoughts.

1. I've seen problems like this when using visual models. I'd add a column to datasets prior to the model inference. Then I'd have to regenerate the model to include the new column and re-publish the model. Then I could run the model on the new dataset.
2. Based on the Python you are showing, it appears that you are not using the visual model building approach. I've not build models in Dataiku that way, so, I'm not clear what you might be doing. Others may be able to understand exactly what you are doing from this bit of code. However, I am not.
3. If I were you, I would likely open up a support ticket. The Technical support team at Dataiku are often very good to excellent. They can get some diagnostics and more accurately identify the root cause.

—Tom

Hello all,

i am still looking for a solution to my problem. I have the following Jupyter Notebook:

import dataiku

import pickle

import pandas as pd

from sklearn.feature_extraction.text import TfidfVectorizer

from sklearn.metrics.pairwise import cosine_similarity

import dataikuapi



def load_data(folder_name="Recommender"):

    managed_folder = dataiku.Folder(folder_name)



    with managed_folder.get_download_stream("cosine_similarity.pkl") as stream:

        cosine_sim = pickle.load(stream)



    with managed_folder.get_download_stream("tfidf_vectorizer.pkl") as stream:

        vectorizer = pickle.load(stream)


    with managed_folder.get_download_stream("tfidf_matrix.pkl") as stream:

        X_tfidf = pickle.load(stream)  



    sachnummer = dataiku.Dataset("LTM_prep")

    df = sachnummer.get_dataframe()

    df.drop(['lieferant_name', 'lieferant_ort', 'LIEFERANT_NAME_ORT', 'LT_GEBINDE_NUMMER', 'MDI'], axis=1, inplace=True)


    return cosine_sim, vectorizer, X_tfidf, df



def recommend_filtered1(input_bennenung, vectorizer, X_tfidf, df, top_n=10):

    try:

        if not input_bennenung:

            return {"error": "Die Eingabe-Benennung darf nicht leer sein."}


        input_bennenung = input_bennenung.upper()

        input_vector = vectorizer.transform([input_bennenung])  

        similarities = cosine_similarity(input_vector, X_tfidf).flatten()  


        top_indices = similarities.argsort()[-top_n:][::-1]

        recommendations = [

            {"test": df.iloc[idx]['test'],

             "test2": df.iloc[idx]['test2'],

             "SIMILARITY_SCORE": round(similarities[idx], 2)}

            for idx in top_indices if similarities[idx] > 0

        ]


        return recommendations if recommendations else {"message": "Keine ähnlichen Benennungen gefunden."}

    except Exception as e:

        return {"error": f"Fehler: {str(e)}"}



def recommend_from_input(input_bennenung):


    folder_name = "Recommender"  


    if not input_bennenung:

        return {"error": "Fehlender Parameter 'input_bennenung'"}


    try:

        # Lade alle benötigten Objekte

        cosine_sim, vectorizer, X_tfidf, df = load_data(folder_name)  


        # Empfehlung berechnen

        return recommend_filtered1(input_bennenung, vectorizer, X_tfidf, df)

    except Exception as e:

        return {"error": f"Fehler beim Laden der Daten oder der Empfehlung: {str(e)}"}

and want to call the method recommend_from_input from it. I am in the API Designer. I have a managed folder called "Recommender," which I can also see in the Flow. The structure im Folder is

Unter Folder Settings i see type is Amazon S3 and i have a setted connection and see also the path in bucket. So when i call def recommend_from_input(input_bennenung): return input_bennenung in the api designer code section with the test query

{

"input_bennenung": "Stern"

}

there are no errors and i get "Stern back". So now i just pasted my notebook code in the api designer code Section and when i run it there is a error:

Result: 

{"error":"Fehler beim Laden der Daten oder der Empfehlung: Default project key is not specified (no DKU_CURRENT_PROJECT_KEY in env)"}

In the logs are no errors only info and debug.

I would appreciate any help. I have already read the documentation on Exposing Python Functions, but I still don't know where my mistake is.