How can I save a fine-tuned HuggingFace model to a Dataiku Managed Folder?

Bart · August 19

Hello,

I am having an issue with saving a HuggingFace model that I am fine-tuning.

The default implementation for saving HF models is to use:

model.save_pretrained("/path/to/directory")

However, as far as I can tell, this does not work in dataiku. Even after I use the following code to get the path:

folder = dataiku.Folder("xxxxxxx")

d = folder.get_info()

path = d['accessInfo']['root']

I get an error saying that my permission to the path is denied.

In the past I have used pickle objects and saved those, but the specific model I am using this time is not pickleable.

Any help would be much appreciated!

Operating system used: Windows

Turribeach · August 19

Where is the managed folder stored on? (What kind of technology)

Bart · August 19

Under the storage settings we have:

Type: Azure Blob Storage

Azure connection: fastpath_conn

Turribeach · August 19

Some folder methods only work on local storage. For cloud storage folders you need to use the corresponding read and write API methods.

Answers