Chapter 12: Working with APIs & Web Scraping in Python

Previous Chapter Table of Contents Next Chapter

1. API Fundamentals

APIs (Application Programming Interfaces) allow different software systems to communicate.

REST APIs

Standard web APIs using HTTP methods:

  • GET - Retrieve data
  • POST - Create data
  • PUT/PATCH - Update data
  • DELETE - Remove data

API Responses

Common formats:

  • JSON (most common)
  • XML
  • CSV

Status Codes

Key HTTP responses:

  • 200 - Success
  • 400 - Bad request
  • 401 - Unauthorized
  • 404 - Not found

2. Using APIs in Python

Making API Requests

import requests

# GET request
response = requests.get('https://api.github.com/users/octocat')
if response.status_code == 200:
    data = response.json()
    print(data['name'], data['public_repos'])
Method Description Example
requests.get() Retrieve data requests.get(url, params={...})
requests.post() Send data requests.post(url, json={...})
response.json() Parse JSON response data = response.json()
response.status_code Check request status if response.status_code == 200:

Exercise 1: Weather API

Using OpenWeatherMap API:

  1. Fetch current weather for your city
  2. Display temperature and conditions
  3. Handle API errors gracefully

3. Web Scraping Basics

Web Scraping Process

Web scraping extracts data from websites

BeautifulSoup Example

from bs4 import BeautifulSoup
import requests

url = "https://books.toscrape.com/"
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

for book in soup.select('article.product_pod'):
    title = book.h3.a['title']
    price = book.select_one('p.price_color').text
    print(f"{title} - {price}")

BeautifulSoup

HTML parsing library:

  • find() - First matching element
  • find_all() - All matching elements
  • select() - CSS selector

Scraping Ethics

Best practices:

  • Check robots.txt
  • Limit request rate
  • Respect copyright

Selenium

For dynamic content:

  • Browser automation
  • JavaScript rendering
  • Form interaction

Exercise 2: News Headlines

Scrape a news website:

  1. Extract headlines and links
  2. Filter by date or category
  3. Save results to CSV

4. API Authentication

API Keys

import requests

API_KEY = "your_api_key_here"
headers = {"Authorization": f"Bearer {API_KEY}"}
response = requests.get("https://api.example.com/data", headers=headers)

API Keys

Simple authentication:

  • Passed in URL or headers
  • Rate limits often apply
  • Keep keys secret!

OAuth

Secure authorization:

  • Token-based
  • User permission levels
  • Refresh tokens

Session Cookies

For logged-in scraping:

  • Maintain session state
  • Handle login forms
  • Use with caution

Exercise 3: GitHub API

Using GitHub's API:

  1. Fetch your repositories
  2. Display name, language, and stars
  3. Sort by most recent update
Previous Chapter Next Chapter