Using Selenium Chrome Driver in python recipe

Options
sarahtan
sarahtan Registered Posts: 2

Hi, I have been trying to create a python recipe that web scrapes with Selenium, but have been facing this error:

unknown error: Chrome failed to start: exited abnormally. (unknown error: DevToolsActivePort file doesn't exist) (The process started from chrome location .../managed_folders/TEST_3/a4dI41U2/chromedriver is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

I have added the chromedriver as an input to the code and these are the libraries in my environment:

selenium
chromedriver-py
webdriver-manager
chromedriver-binary-auto
chromedriver-binary

This is the code I have been using:

import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

## Will throw a need permissions error
#from webdriver_manager.chrome import ChromeDriverManager
#driver = webdriver.Chrome(ChromeDriverManager().install())

# Read recipe inputs
chrome_driver = dataiku.Folder("a4dI41U2")
chrome_driver_info = chrome_driver.get_info()
driver_path = chrome_driver.get_path() + '/chromedriver'


# Compute recipe outputs
chrome_options = Options()
chrome_options.binary_location = driver_path
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--disable-extensions")
driver = webdriver.Chrome(executable_path=driver_path, chrome_options=chrome_options)
driver.quit()

What should I do to solve the error?

Thanks in advance!

Answers

Setup Info
    Tags
      Help me…