Submit your inspiring success story or innovative use case to the 2022 Dataiku Frontrunner Awards! ENTER YOUR SUBMISSION

Possible to run python script to create CSVs by scraping website?

info-rchitect
Level 3
Possible to run python script to create CSVs by scraping website?

Hi,

 

I have a Python script that hits internal website and scrapes it, creating CSV files (2 per product).  The creation of CSV files is because I was running it manually on my desktop.  In the end, I just want to load the data contained in these CSV files into two distinct Snowflake tables.  What is the best approach?

 

thx


Operating system used: Windows 10

0 Kudos
2 Replies
Manuel
Dataiker
Dataiker

Hi,

If you search Selenium in this community, you will find a few different posts, including this one, https://community.dataiku.com/t5/Using-Dataiku/Web-scrapping-any-tuto-to-explain-how-to-sart-with-da...

I hope this helps.

info-rchitect
Level 3
Author

Hi Manuel,

 

I used a Python recipe and got the dataframes I needed but having issues with the write_with_schema function.  Started a specific ticket related to that so I think this ticket can be closed.

 

thx

0 Kudos