Count Columns on file based dataset (from file, not from schema)

Hello,

You can get the columns count with a Python probe this way:

import dataiku

def process(dataset, partition_id):
    df = dataset.get_dataframe()
    columns_count = len(df.columns)
    return {'columns_count' : columns_count}

Though I am not sure it will help you. For that, I would need more context about what you want/need to do with this metric that you can't with the one already provided

Have a nice day,

Estelle

From what I understand, your flow takes as input a file whose number of columns will often change. But you only need 2 columns in it, which remain the same over time.

In DSS, some recipes select by default all columns in a dataset, like the recipes "Sync" or "Sample/Filter". It can require that you update the schema when it changes.

To work with your flow input, you can rather choose a recipe where you can specify the columns you want to work with, like a "Prepare" recipe (with the first step "Keep only these columns"), a "Group by" or a "Join" recipe - and SQL recipe if relevant. Then your number of columns in the output dataset will always stay the same and you won't need to update the schema.

Does that answer your needs?

Hello,

Thank you for this detail, it helps me understand better.

If you need to detect that a column was added to this file, you can go in Settings > Format/Preview.

Then at the bottom, you'll find this option: "Add. columns behavior (read)".

I personally don't know how to get easily a file "Warnings" message, but here is what you can do waiting for a better solution:

1/ Select "Error". It means that your dataset won't be built when a new column is added, but you maybe could find some turnaround.

2/ Then in a scenario, thanks to a scenario variable, you can add a reporter which sends you the 'message' computed by this Python step:

import dataikuapi

projectKey = dataiku.default_project_key()
client = dataiku.api_client()
project = client.get_project(projectKey)
dataset = project.get_dataset("test_csv")
metrics = dataset.compute_metrics()
runs = metrics['result']['runs']
def test_error_key(run):
    try:
        error = run['error']['message']
        result = 'Unexpected column found' in error
    except:
        result = False
    return result     

r = filter(test_error_key, runs)
if len(r) > 0:
    message = 'error: new columns detected in data'
else: 
    message = 'ok'

It worked for me in a Jupyter notebook. You'll need to adjust it depending on your needs.

I hope it helps you even if there is probably a better solution

Best