Possible to run python script to create CSVs by scraping website?
Hi,
I have a Python script that hits internal website and scrapes it, creating CSV files (2 per product). The creation of CSV files is because I was running it manually on my desktop. In the end, I just want to load the data contained in these CSV files into two distinct Snowflake tables. What is the best approach?
thx
Operating system used: Windows 10
Answers
-
Manuel Alpha Tester, Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Dataiku DSS Adv Designer, Registered Posts: 193 ✭✭✭✭✭✭✭Hi,
If you search Selenium in this community, you will find a few different posts, including this one, https://community.dataiku.com/t5/Using-Dataiku/Web-scrapping-any-tuto-to-explain-how-to-sart-with-dataiku/m-p/19859#M7775
I hope this helps.
-
Hi Manuel,
I used a Python recipe and got the dataframes I needed but having issues with the write_with_schema function. Started a specific ticket related to that so I think this ticket can be closed.
thx