← Previous Contents Next →

Chapter 14: Introduction to Web Scraping, Web Dev, AI & Data Science

14.1 Web Scraping with Python

Web scraping is the process of extracting data from websites automatically.

14.1.1 Basic Web Scraping with BeautifulSoup

import requests
from bs4 import BeautifulSoup

url = 'https://example.com'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

# Extract all paragraph texts
paragraphs = [p.get_text() for p in soup.find_all('p')]
print(paragraphs)

# Extract all links
links = [a['href'] for a in soup.find_all('a', href=True)]
print(links)

14.1.2 Scraping Best Practices

• Always check a website's robots.txt file (e.g., https://example.com/robots.txt)
• Respect crawl-delay instructions
• Use time.sleep() between requests to avoid overwhelming servers
• Set a proper User-Agent header

14.2 Web Development with Python

Python offers several frameworks for web development:

14.2.1 Flask - Micro Web Framework

from flask import Flask

app = Flask(__name__)

@app.route('/')
def home():
    return "Hello, World!"

if __name__ == '__main__':
    app.run(debug=True)

14.2.2 Django - Full-featured Framework

# Typically involves multiple files
# Command to start a new project:
# django-admin startproject mysite

# Key components:
# - Models (database)
# - Views (business logic)
# - Templates (presentation)
# - URLs (routing)

14.3 Introduction to AI & Machine Learning

Python is the dominant language for AI/ML with these key libraries:

14.3.1 Scikit-learn - Machine Learning

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

# Load data
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2)

# Train model
model = RandomForestClassifier()
model.fit(X_train, y_train)

# Evaluate
print("Accuracy:", model.score(X_test, y_test))

14.3.2 TensorFlow/PyTorch - Deep Learning

import tensorflow as tf

# Simple neural network
model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

14.4 Data Science Workflow

Typical data science process with Python:

1. Data collection (Pandas, Scrapy)
2. Data cleaning (Pandas, NumPy)
3. Exploratory analysis (Matplotlib, Seaborn)
4. Feature engineering (Scikit-learn)
5. Model building (Scikit-learn, TensorFlow)
6. Model deployment (Flask, FastAPI)