Create a folder from existing items in object store or shared folder

jax79sg
Level 2
Create a folder from existing items in object store or shared folder

Hi, 

We have some pretrained models in object store (bucket in MINIO) and shared folder (E.g. \\computer1\sharedpath). I want to create a folder that directly references these models in the object store and shared folder so i can feed them to a python recipe. 

I can't seem to achieve this, please advise.

Side note: I am only allowed to create a folder and upload something into it, which i means i  have to 'Download', 'and upload' a duplicate copy somewhere. This is a huge waste of space given the sizes of these pretrained models.

3 Replies
jax79sg
Level 2
Author
Hi, just wondering if there's a potential answer to this question. The impetus is to avoid having duplicate data.
0 Kudos
Triveni
Dataiker

Hi!

Thanks for your question. While I am not personally familiar with MINIO if you can access your data via an API call, you can use a Python recipe to stream the data from your object store into a DSS Folder. The documentation here  outlines how to use our API to write into a managed folder. Essentially, you will create a new Python recipe with a Folder as the output, and leave the input field blank. Then you can use the MINIO API to load your objects into memory and write them out to the folder. The same process is possible for your file shared on the computer locally - just point to the path of where the file is stored to bring the data into memory and write out to the folder. Hope that helps!

0 Kudos
ATsao
Dataiker

Hi there,

Unfortunately, it is not possible to create a managed folder that points to two separate locations. You would need to create two folders, one containing the files from MinIO and the other pointing to your shared folder (assuming it is accessible from the DSS server itself), but you could have them both as inputs for your python recipe and access the underlying files accordingly using code. 

From looking at the MinIO documentation, MinIO seems to be an object storage server released under Apache License v2.0 that is compatible with Amazon S3 cloud storage service. So you may also be able to create an Amazon S3 connection to MinIO, something like:

image.png

You could then create a managed folder using this S3 connection and then update the folder path (via the Settings tab) to point to the appropriate location instead. 

Similarly, for your shared folder, is this path also directly accessible from the DSS server itself? If so, then you should be able to create a new filesystem connection (with "allow managed folders" param enabled so that you can create a managed folder using it) pointing to the appropriate directory. Otherwise, if this location is not directly accessible, then you may need to follow up with your sysadmins to see if this is something that can be mounted onto your server, so that it can then be accessed by DSS directly through the local filesystem.  

Best,

Andrew

0 Kudos