Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Possible to run python script to create CSVs by scraping website?
Added on February 14, 2022 2:24PM
Likes: 0
Replies: 2
info-rchitect
Registered Posts: 196 ✭✭✭✭✭✭
Hi,
I have a Python script that hits internal website and scrapes it, creating CSV files (2 per product). The creation of CSV files is because I was running it manually on my desktop. In the end, I just want to load the data contained in these CSV files into two distinct Snowflake tables. What is the best approach?
thx
Operating system used: Windows 10
Answers
-
Manuel Alpha Tester, Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Dataiku DSS Adv Designer, Registered Posts: 193 ✭✭✭✭✭✭✭Hi,
If you search Selenium in this community, you will find a few different posts, including this one, https://community.dataiku.com/t5/Using-Dataiku/Web-scrapping-any-tuto-to-explain-how-to-sart-with-dataiku/m-p/19859#M7775
I hope this helps.
-
Hi Manuel,
I used a Python recipe and got the dataframes I needed but having issues with the write_with_schema function. Started a specific ticket related to that so I think this ticket can be closed.
thx