Strip away the hype and a neural network is just a stack of weighted sums and simple nonlinear functions, trained by repeatedly nudging its numbers in the direction that reduces error. Building the tiniest possible version by hand makes the whole concept far less mystical.
A Single Neuron
A neuron takes inputs, multiplies each by a learned weight, sums them plus a bias, and passes the result through an activation function:
import numpy as np
def sigmoid(x):
return 1 / (1 + np.exp(-x))
def neuron(inputs, weights, bias):
weighted_sum = np.dot(inputs, weights) + bias
return sigmoid(weighted_sum)
inputs = np.array([0.5, 0.8])
weights = np.array([0.4, 0.6])
bias = -0.1
print(neuron(inputs, weights, bias)) # a single number between 0 and 1
Why the Nonlinearity Matters
Without an activation function like sigmoid or ReLU, stacking layers would collapse mathematically into a single linear transformation, no matter how many layers you add — nonlinearity is precisely what lets a network learn curved decision boundaries and complex patterns rather than just straight lines.
Learning: Gradient Descent in One Sentence
# Simplified single-weight example
weight = 0.5
learning_rate = 0.1
for step in range(100):
prediction = weight * 2.0 # a toy "model"
error = prediction - 4.0 # target output is 4.0
gradient = error * 2.0 # how much the error changes per unit of weight
weight -= learning_rate * gradient # nudge weight toward less error
print(weight) # converges toward 2.0, the value that makes prediction == 4.0
Real networks do this across millions or billions of weights simultaneously, using backpropagation to efficiently compute how much each individual weight contributed to the overall error — but the core idea is exactly this loop, just at enormous scale.
From This to a Transformer
Everything from a simple image classifier to GPT-scale language models is built from these same fundamentals — layers of weighted sums, nonlinearities, and gradient-based learning — with architecture-specific innovations (convolutions, attention mechanisms) layered on top to handle specific kinds of data more effectively.