Scraping with Authentication

KELAS DATA SCRAPING

Kampus Remote Worker Indonesia | 2024

Introduction

When scraping data from websites, you will often encounter authentication mechanisms that protect content behind login forms or API credentials.

if user.logged_in: print("Success!")

Automate Login vs Payload

Automate Login using Selenium

driver = webdriver.Chrome(options=options)
driver.get(url)

username = driver.find_element(By.ID, "email")
username.send_keys("admin@example.com")

password = driver.find_element(By.ID, "password")
password.send_keys("password")

submit_button = driver.find_element(By.ID,"submit-button")
submit_button.click()

Login using Requests Payload

payload = {
            "email": "admin@example.com",
            "password": "password",
        }

response = requests.post(url, data=payload)

Class Activity

Create an account on https://webapi.bps.go.id/developer/ then automate your login process using both of selenium and requests payload.

Learn More

Join us on this self-study journey! Click the link below to get started.

• Scraping Websites With CSRF Token Authentication or WAF-Protected Websites

Discover Advanced Topic

Master this advanced topic and supercharge your skills.

• Can you scraping a website that use OAuth 2.0 like Google?

Homework Assignment

Find any website that using login and try login using the best approach!

Thank you

Any Question?

Kampus Remote Worker Indonesia | 2024