Decision Boundaries

First: Preparing Your Data

◈

The 80/20 Rule of ML

In practice, 80% of ML work is data preparation, 20% is modeling. Raw data is messy: missing values, different scales, inconsistent formats. Before any model sees your data, it needs preprocessing.

1. Data Cleaning

Handle the messiness in real-world data:

Missing values: Drop rows, impute with mean/median, or use indicators
Outliers: Detect (z-score, IQR), decide whether to remove or cap
Duplicates: Identify and remove exact or near-duplicates
Inconsistent formatting: Standardize dates, units, categories

2. Feature Scaling

Many algorithms are sensitive to feature magnitudes:

Standardization (Z-score): μ=0, σ=1. Good for most cases
Min-Max Scaling: Range [0,1]. Good for bounded features
Robust Scaling: Uses median/IQR. Handles outliers better

Critical: Fit scaler on training data only, then transform val/test

3. Categorical Encoding

Convert categories to numbers:

One-Hot Encoding: Binary column per category. No ordinal assumption
Label Encoding: Integer per category. Only for ordinal data
Target Encoding: Replace with mean target. Powerful but leak risk

4. Class Imbalance

When one class dominates (fraud: 0.1%, normal: 99.9%):

Oversampling: SMOTE creates synthetic minority examples
Undersampling: Reduce majority class (loses information)
Class Weights: Penalize majority class errors more in loss
Stratified Splits: Maintain class ratios in train/val/test

Why Preprocessing Matters for Decision Boundaries

Unscaled features distort the geometry that classifiers learn. If feature A ranges 0-1 and feature B ranges 0-1000, the decision boundary will be dominated by B simply because its scale is larger. After standardization, both features contribute equally, and the boundary reflects their true importance.

The decision boundary you learn is only as good as the data you feed in. Garbage in, garbage boundary out.

What is a Decision Boundary?

Definition

A decision boundary is the hypersurface in feature space that separates regions assigned to different classes by a classifier. Points on one side are classified as class A; points on the other side as class B.

Intuition

Imagine a map where you're trying to draw a border between two countries. The border is your decision boundary—everything on the left belongs to Country A, everything on the right to Country B.

In classification, the "countries" are classes (spam/not-spam, cat/dog, tumor/healthy) and the "map" is the space of all possible inputs. The classifier's job is to learn where to draw the border based on training examples.

Technical

For a binary classifier outputting a score f(x), the decision boundary is defined by the set of points where:

Decision Boundary = {x : f(x) = threshold}

For probabilistic classifiers (like neural networks with softmax), the boundary typically occurs where P(class A | x) = P(class B | x) = 0.5.

In higher dimensions, this "boundary" is actually a hypersurface—a surface of dimension (d-1) in a d-dimensional space. In 2D it's a curve; in 3D it's a surface; in 100D it's a 99-dimensional manifold.

In Practice

When you use a spam filter, it implicitly computes features of your email (word frequencies, sender reputation, link counts) and checks which side of a learned boundary your email falls on.

The boundary isn't drawn by a human—it emerges from optimization. The training process adjusts model parameters until the boundary separates training examples as well as possible (while hopefully generalizing to new data).

Linear vs. Non-Linear Boundaries

The shape of the decision boundary depends on the model. Simple models produce simple boundaries; complex models can produce intricate boundaries that weave through the feature space.

Linear Boundaries

Straight lines, planes, and hyperplanes

Models: Logistic regression, linear SVM, perceptron

The boundary is defined by a linear equation:

w₁x₁ + w₂x₂ + ... + wₙxₙ + b = 0

Pros: Fast, interpretable, low variance, works well when classes are actually linearly separable.

Cons: Cannot capture complex relationships. If the true boundary is curved, a linear model will have high bias (systematic errors).

Non-Linear Boundaries

Curves, surfaces, and complex manifolds

Models: Neural networks, kernel SVM, decision trees, random forests, k-nearest neighbors

The boundary can take any shape the model architecture allows—circles, spirals, disconnected regions, or arbitrarily complex surfaces.

Pros: Can fit any pattern given enough capacity. Necessary for most real-world problems.

Cons: Risk of overfitting—the boundary might contort to fit noise in training data rather than true patterns.

◈

The XOR Problem

A famous example: the XOR (exclusive-or) function outputs 1 when exactly one input is 1. The four points (0,0)→0, (0,1)→1, (1,0)→1, (1,1)→0 cannot be separated by any straight line. This simple problem requires a non-linear boundary—and historically, solving XOR helped drive the development of multi-layer neural networks.

How Different Models Create Boundaries

━

Logistic Regression

Learns a single linear boundary. The weights define a hyperplane; the sigmoid function converts distance-from-boundary into probability. Fast and interpretable, but fundamentally limited to linearly separable problems.

Boundary shape: Straight line / hyperplane

◇

Decision Trees

Create axis-aligned rectangular regions by making sequential threshold decisions (if x₁ > 5, go left; else go right). Each leaf is a rectangular region. Can approximate any boundary but tend to be "blocky."

Boundary shape: Axis-aligned rectangles

◯

K-Nearest Neighbors

Classifies based on majority vote of k closest training points. The boundary is implicit—defined by the Voronoi regions around training points. Very flexible but computationally expensive and sensitive to noise.

Boundary shape: Piecewise linear (Voronoi)

∿

Neural Networks

Compose multiple linear transformations with non-linear activations. Each layer can be thought of as folding and warping space. Deep networks can create arbitrarily complex boundaries through this composition.

Boundary shape: Arbitrary smooth manifolds

Neural Networks: Composing Boundaries

Neural networks achieve complex boundaries by composing simple operations. Each neuron computes a linear boundary; layers of neurons, connected through non-linear activations, can combine these into any shape.

The Composition Principle

A single neuron with a step activation (or sigmoid) creates a linear boundary—it divides space into two half-spaces. But when you:

Combine multiple neurons in one layer → multiple linear boundaries, creating convex regions
Stack multiple layers → compositions of regions, creating arbitrary shapes

This is the fundamental insight: depth gives neural networks their representational power. A two-layer network with enough neurons can approximate any continuous function (Universal Approximation Theorem), but deeper networks can often do so more efficiently.

Building XOR from Linear Pieces

Layer 1, Neuron 1

Creates boundary: x + y = 0.5

Separates (0,0) from others

+

Layer 1, Neuron 2

Creates boundary: x + y = 1.5

Separates (1,1) from others

→

Layer 2

Combines: "between both boundaries"

XOR region isolated!

Each ReLU activation in a modern network creates a "fold" in space—where the function transitions from linear to flat (or vice versa). A network with many ReLU neurons has many folds, allowing it to carve out intricate decision regions.

Boundary Complexity and Overfitting

◈

The Bias-Variance Trade-off

A model that can only produce simple boundaries has high bias—it may systematically misclassify if the true boundary is complex. A model that can produce very complex boundaries has high variance—it may overfit to noise, producing a boundary that perfectly fits training data but fails on new data.

Boundary Complexity Effects

Boundary Type	Bias	Variance	Risk
Too simple (underfitting)	High	Low	Systematic errors on all data
Appropriate complexity	Balanced	Balanced	Good generalization
Too complex (overfitting)	Low	High	Fits noise, poor on new data

Visual Signs of Overfitting

An overfit boundary often looks "wiggly"—it contorts to wrap around individual training points rather than finding a smooth separation. If you see a boundary that makes sharp turns to include specific points, that's likely overfitting.

A well-generalized boundary is usually simpler than it could be—it captures the broad pattern without chasing outliers.

Common Misconceptions

✗

"More complex boundaries are always better"

A perfectly complex boundary can achieve 100% training accuracy by wrapping around every single training point—but this almost never generalizes well.

The accurate framing: The best boundary is the simplest one that captures the true pattern. Complexity should match the problem's actual structure, not the noise in your training data.

✗

"The boundary is just a line—easy to understand"

In 2D, boundaries are curves we can visualize. Real problems have hundreds or thousands of features—the boundary is a high-dimensional surface that cannot be directly visualized or easily intuited.

The accurate framing: 2D visualizations are pedagogical tools, not representations of real ML. In high dimensions, the boundary is a complex manifold whose properties we study through metrics (accuracy, margin, smoothness) rather than direct visualization.

✗

"Neural networks always find the optimal boundary"

Neural networks find a boundary that reduces training loss, not necessarily the best possible boundary. Different initializations, architectures, and hyperparameters produce different boundaries.

The accurate framing: Training finds a good local solution in parameter space, which corresponds to a particular boundary. This boundary depends on many factors including random initialization, and there's often no way to verify it's globally optimal.

⚗ Interactive Lab: Exploring Decision Boundaries

Experiment with different classifier types and see how they create different decision boundaries. Add points, switch models, and observe how complexity affects the boundary shape.

Dataset Pattern

Noise Level 10%

Logistic Regression

Accuracy: —

Decision Tree

Accuracy: —

K-Nearest Neighbors

Accuracy: —

Neural Network

Accuracy: —

See how a neural network builds complex boundaries by composing simple linear operations. Watch each layer transform the space.

Hidden Neurons 4

Hidden Layers 2

Input Space

→

After Layer 1

→

Final Boundary

What you're seeing: The network "unfolds" the data by applying linear transformations followed by ReLU activations. In the transformed space, a complex problem can become linearly separable.

See how model complexity affects the decision boundary. Too simple underfits; too complex overfits. Find the sweet spot.

Model Complexity (Polynomial Degree) 1

Training Accuracy —

Test Accuracy —

Complexity —

Here's how to implement the classifiers you're visualizing above using scikit-learn and PyTorch. These are production-ready patterns.

                                    Logistic Regression (scikit-learn)
                                    from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

# Always scale features for linear models
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_train)

# Train logistic regression
model = LogisticRegression()
model.fit(X_scaled, y_train)

# Predict
y_pred = model.predict(scaler.transform(X_test))
accuracy = (y_pred == y_test).mean()

# The decision boundary is: w·x + b = 0
# w = model.coef_, b = model.intercept_
                                

                                    Neural Network Classifier (PyTorch)
                                    import torch
import torch.nn as nn

class BinaryClassifier(nn.Module):
    def __init__(self, input_dim, hidden_dim=32):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),  # Non-linearity enables curved boundaries
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 1),
            nn.Sigmoid()  # Output probability in [0, 1]
        )
    
    def forward(self, x):
        return self.net(x)

# Training
model = BinaryClassifier(input_dim=2)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
criterion = nn.BCELoss()  # Binary Cross-Entropy

for epoch in range(100):
    y_pred = model(X_train)
    loss = criterion(y_pred, y_train)
    
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
                                

                                    Visualizing Decision Boundaries
                                    import numpy as np
import matplotlib.pyplot as plt

def plot_decision_boundary(model, X, y):
    # Create mesh grid
    h = 0.02  # Step size
    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
    xx, yy = np.meshgrid(
        np.arange(x_min, x_max, h),
        np.arange(y_min, y_max, h)
    )
    
    # Predict on every point in the grid
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    
    # Plot
    plt.contourf(xx, yy, Z, alpha=0.3, cmap='coolwarm')
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap='coolwarm', edgecolors='black')
    plt.title('Decision Boundary')
    plt.show()
                                

Key Pattern: The decision boundary's shape is determined by the model's inductive bias. Logistic regression = linear, trees = axis-aligned rectangles, neural nets = smooth curves (controlled by architecture).

Key Observations

Logistic regression always produces a straight line—it can't adapt to curved patterns.
Decision trees create rectangular regions—good for some patterns, awkward for circular ones.
K-NN creates highly local boundaries that conform closely to training points—flexible but noisy.
Neural networks can create smooth, complex boundaries—but need proper regularization to avoid overfitting.

Learning Objectives

First: Preparing Your Data

The 80/20 Rule of ML

1. Data Cleaning

2. Feature Scaling

3. Categorical Encoding

4. Class Imbalance

Why Preprocessing Matters for Decision Boundaries

What is a Decision Boundary?

Linear vs. Non-Linear Boundaries

Linear Boundaries

Non-Linear Boundaries

The XOR Problem

How Different Models Create Boundaries

Logistic Regression

Decision Trees

K-Nearest Neighbors

Neural Networks

Neural Networks: Composing Boundaries

The Composition Principle

Building XOR from Linear Pieces

Boundary Complexity and Overfitting

The Bias-Variance Trade-off

Boundary Complexity Effects

Visual Signs of Overfitting

Common Misconceptions

"More complex boundaries are always better"

"The boundary is just a line—easy to understand"

"Neural networks always find the optimal boundary"

⚗ Interactive Lab: Exploring Decision Boundaries

Logistic Regression

Decision Tree

K-Nearest Neighbors

Neural Network

Input Space

After Layer 1

Final Boundary

Logistic Regression (scikit-learn)

Neural Network Classifier (PyTorch)

Visualizing Decision Boundaries

Key Observations

✓ Check Your Understanding