znazz75.de

Building a Simple Neural Network from Scratch in Python

Claude
,

Building a neural network from scratch, without a framework doing the heavy lifting invisibly, is genuinely the best way to actually understand what’s happening beneath the abstraction layer of PyTorch or TensorFlow.

import numpy as np

class SimpleNeuralNetwork:
    def __init__(self, input_size, hidden_size, output_size):
        self.W1 = np.random.randn(input_size, hidden_size) * 0.01
        self.W2 = np.random.randn(hidden_size, output_size) * 0.01

    def sigmoid(self, x):
        return 1 / (1 + np.exp(-x))

    def forward(self, X):
        self.z1 = X.dot(self.W1)
        self.a1 = self.sigmoid(self.z1)
        self.z2 = self.a1.dot(self.W2)
        return self.sigmoid(self.z2)

    def backward(self, X, y, output, learning_rate=0.1):
        # Backpropagation: compute how much each weight contributed to the error,
        # then nudge each weight slightly to reduce that error
        error = y - output
        d_output = error * output * (1 - output)
        error_hidden = d_output.dot(self.W2.T)
        d_hidden = error_hidden * self.a1 * (1 - self.a1)

        self.W2 += self.a1.T.dot(d_output) * learning_rate
        self.W1 += X.T.dot(d_hidden) * learning_rate

Why This Exercise Is Genuinely Worth Doing

Frameworks like PyTorch handle backpropagation and gradient computation entirely automatically (autograd) — genuinely convenient, but it means the actual mechanism (how error genuinely propagates backward through the network, nudging each weight proportionally to its real contribution to the final error) stays completely invisible unless you implement it by hand at least once yourself.

What This Simple Example Deliberately Leaves Out

Production neural networks add batch normalization, dropout regularization, more sophisticated optimizers (Adam, not plain gradient descent), and vastly more layers and parameters — but the fundamental forward pass and backpropagation mechanism shown above is genuinely the same core idea underlying even today’s largest, most sophisticated language models.

, ,

Recommended

Categories