Ready for Dataiku 10? Try out the Crash Course on new features!GET STARTED

Reading Source files with different Schema

sj0071992
Level 3
Level 3
Reading Source files with different Schema

Hi,

 

I have one File System where all my log files are Stored (Every Day)

That log files contain Details about (SQL_CONNECTION, SQL_QUERY and LOCAL_PROCESS) Usage type

These each Usage type have different set of columns and in single Log file we have details about all these 3 Usage type

I want to build the separate process based Usage type so i used split recipe. But now when i ran for LOCAL_PROCESS the columns which i am expecting is not coming but in source log file i am able to see those columns

 

Could you please help here.

 

Thanks in Advance

0 Kudos
1 Reply
AlexT
Dataiker
Dataiker

Hi @sj0071992 ,

When creating the schema it may not find the columns in the entries it analyzes to build the schema. 

One way you can approach this is by setting "one record per line" and later using a prepare recipe to split to the columns you need.

Screenshot 2021-12-11 at 12.18.20.png

Let me know if that approach works for you if not can you please share a small sample( obfuscated if needed) of the first few lines from your log file/s. 

Thanks,

0 Kudos
A banner prompting to get Dataiku DSS