Submit your inspiring success story or innovative use case to the 2022 Dataiku Frontrunner Awards! ENTER YOUR SUBMISSION

Truncating a HDFS folder that Im accessing through Download Recipe

sarah2753
Level 1
Truncating a HDFS folder that Im accessing through Download Recipe

Hi, 

Is it possible to delete everything inside a filesystem HDFS folder that I'm downloading? 

I'm, copying them and deleting them like this:

for x in paths:
    with source_folder.get_download_stream(x) as f:
        data = f.read()
        with destination_folder.get_writer(x) as w:
    w.write(data)

But the files only get deleted temporarily . If I build the source folder again the files reappear? 

How can I delete them from the original place? Do I need the host, port... information? It doesn't seem to be working

 

Thank you so much in advance 🙂 

0 Kudos
0 Replies