← Previous Contents Next →

Chapter 13: Data Science Basics with NumPy & Pandas

13.1 Introduction to NumPy

NumPy (Numerical Python) is the fundamental package for numerical computing in Python. It provides:

• Powerful N-dimensional array objects
• Sophisticated broadcasting functions
• Tools for integrating C/C++ and Fortran code
• Useful linear algebra, Fourier transform, and random number capabilities

13.1.1 Creating NumPy Arrays

import numpy as np

# Create a 1D array
arr1 = np.array([1, 2, 3, 4, 5])

# Create a 2D array
arr2 = np.array([[1, 2, 3], [4, 5, 6]])

# Special arrays
zeros = np.zeros((3, 4))  # 3x4 array of zeros
ones = np.ones((2, 2))     # 2x2 array of ones
range_arr = np.arange(0, 10, 2)  # array from 0 to 10 (exclusive) step 2

13.1.2 Array Operations

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

# Element-wise operations
print(a + b)  # [5 7 9]
print(a * b)  # [4 10 18]

# Dot product
print(np.dot(a, b))  # 32

# Universal functions
print(np.sqrt(a))    # [1.         1.41421356 1.73205081]
print(np.exp(a))     # [ 2.71828183  7.3890561  20.08553692]

13.2 Introduction to Pandas

Pandas is a powerful data manipulation library built on top of NumPy. Its key data structures are:

• Series: 1D labeled array
• DataFrame: 2D labeled data structure with columns of potentially different types

13.2.1 Creating DataFrames

import pandas as pd

# From dictionary
data = {'Name': ['Alice', 'Bob', 'Charlie'],
        'Age': [25, 30, 35],
        'City': ['New York', 'Paris', 'London']}
df = pd.DataFrame(data)

# From CSV file
df = pd.read_csv('data.csv')

13.2.2 Basic DataFrame Operations

# View first/last rows
print(df.head())  # first 5 rows
print(df.tail(3)) # last 3 rows

# Access columns
print(df['Name'])      # Single column
print(df[['Name', 'Age']])  # Multiple columns

# Filtering
print(df[df['Age'] > 30])  # Rows where Age > 30

# Basic statistics
print(df.describe())  # Summary statistics for numerical columns

13.3 Data Cleaning and Preprocessing

13.3.1 Handling Missing Data

# Check for missing values
print(df.isnull().sum())

# Drop rows with missing values
df_clean = df.dropna()

# Fill missing values
df_filled = df.fillna(0)               # Fill with 0
df_filled = df.fillna(df.mean())       # Fill with mean

13.3.2 Data Transformation

# Rename columns
df = df.rename(columns={'OldName': 'NewName'})

# Add new column
df['Salary'] = [50000, 60000, 70000]

# Apply functions
df['Age'] = df['Age'].apply(lambda x: x + 1)  # Increment age by 1

# Group by operations
grouped = df.groupby('City')['Age'].mean()  # Average age by city

13.4 Basic Data Analysis

# Correlation matrix
print(df.corr())

# Value counts
print(df['City'].value_counts())

# Pivot tables
pivot = pd.pivot_table(df, values='Age', index='City', aggfunc='mean')

13.5 Simple Data Visualization

import matplotlib.pyplot as plt

# Line plot
df.plot(x='Name', y='Age', kind='line')
plt.show()

# Bar plot
df.plot(x='Name', y='Age', kind='bar')
plt.show()

# Histogram
df['Age'].plot(kind='hist', bins=10)
plt.show()