Activation Functions

Activation functions give neural networks their non-linearity. Compare the common ones side by side, read off their derivatives, and see why some of them make deep networks hard to train.

Beginner

Solid = f(x), dashed = f′(x)Saturated: |f′(x)| < 0.1 for SigmoidDrag or hover over the plot to move the x marker.

Controls

Marker position x = 2.50Highlight function (shades its saturation regions)Network depth for the gradient estimate: 10 layersShow derivatives

σ(x) = 1 / (1 + e^(−x)), σ′(x) = σ(x)(1 − σ(x))

Values at x = 2.50

The last column multiplies f′(x) by itself 10 times: roughly how much of the gradient survives 10 layers that all sit at this pre-activation (ignoring the weights).

Function	f(x)	f′(x)	f′(x)¹⁰
Sigmoid	0.924	0.070	2.87e-12
Tanh	0.987	0.027	1.77e-16
ReLU	2.500	1.000	1.00e+0

Red values have shrunk below 0.001: the gradient has effectively vanished. Sigmoid's derivative never exceeds 0.25, so even at x = 0 it shrinks by 4× per layer.

Softmax: logits to probabilities

p_i = e^z_i/T / Σ_j e^z_j/T. Edit the logits and the temperature T.

Logit z₁ = 2.0

65.9%

Logit z₂ = 1.0

24.2%

Logit z₃ = 0.1

9.9%

Temperature T = 1.00

Low T sharpens the distribution towards the largest logit (approaching argmax); high T flattens it towards uniform (1/3 = 33.3% each). Adding the same constant to every logit changes nothing.

How to use this visualizer

1
Toggle functions on and off with the coloured chips above the plot. Solid lines are f(x), dashed lines are the derivatives f′(x).
2
Hover or drag across the plot (or use the marker slider) to read f(x) and f′(x) for every visible function in the table.
3
Pick a function to highlight. The amber bands show where its derivative is below 0.1: the saturated regions.
4
Raise the network depth and watch f′(x) raised to that power collapse towards zero for sigmoid and tanh.
5
In the softmax panel, drag the logits and the temperature, and add or remove classes, to see how scores become probabilities.

Why networks need activation functions

A neuron computes a weighted sum z = w·x + b and then applies an activation a = f(z). Without f, stacking layers gains nothing: a composition of linear maps is still a single linear map, so a 100-layer network could only draw straight decision boundaries. A non-linear f is what lets the network bend its boundary to fit curved data.

During training, backpropagation multiplies the error signal by f′(z) at every layer it passes through. That is why the derivative matters as much as the function itself: if f′(z) is small, the signal shrinks; if it is small at many layers, it vanishes and the early layers stop learning.

Key formulas

Sigmoid and tanh

σ(x) = 1 / (1 + e^−x), σ′(x) = σ(x)(1 − σ(x)) ≤ 0.25

tanh(x) = 2σ(2x) − 1, tanh′(x) = 1 − tanh²(x) ≤ 1

ReLU family

ReLU(x) = max(0, x), LeakyReLU(x) = max(αx, x)

GELU(x) = x·Φ(x), softplus(x) = ln(1 + e^x)

Φ(x): standard normal CDF
φ(x): standard normal density

Softmax with temperature

p_i = e^z_i/T / Σ_j e^z_j/T

z_i: logit (raw score) of class i
T: temperature; T = 1 is the standard softmax

Vanishing gradients

The gradient reaching layer 1 of an n-layer network contains a product of n derivatives f′(z). With sigmoid each factor is at most 0.25, so after 10 layers the signal is at most 0.25¹⁰ ≈ 10⁻⁶ of its original size. ReLU passes a gradient of exactly 1 for positive inputs, which is a large part of why it made deep networks practical. Its own weakness is the opposite side: a unit whose input is always negative gets zero gradient forever (a "dead ReLU"), which Leaky ReLU and GELU avoid.

Comparison

Function	Output range	Max f′	Zero-centred	Saturates
Sigmoid	(0, 1)	0.25	No	Both sides
Tanh	(−1, 1)	1	Yes	Both sides
ReLU	[0, ∞)	1	No	x < 0 (gradient is exactly 0)
Leaky ReLU	(−∞, ∞)	1	Roughly	Never
GELU	≈ [−0.17, ∞)	≈ 1.13	Roughly	Far negative x
Softplus	(0, ∞)	1 (limit)	No	Far negative x

Hidden layers (default)

ReLU or one of its relatives. Cheap to compute and the gradient is exactly 1 for positive inputs.

Transformers

GELU is the standard choice in BERT and GPT-style models: a smooth ReLU that lets small negative values through.

Binary output

Sigmoid on the final layer turns a score into a probability, paired with binary cross-entropy.

Multi-class output

Softmax turns a vector of scores into a probability distribution, paired with cross-entropy.

In code

Python

import numpy as np
from scipy.special import erf

def sigmoid(x):  return 1 / (1 + np.exp(-x))
def d_sigmoid(x): s = sigmoid(x); return s * (1 - s)

def relu(x):     return np.maximum(0, x)
def d_relu(x):   return (x > 0).astype(float)

def gelu(x):     return x * 0.5 * (1 + erf(x / np.sqrt(2)))

def softmax(z, T=1.0):
    z = np.asarray(z) / T
    e = np.exp(z - z.max())        # subtract the max for numerical stability
    return e / e.sum()

print(softmax([2, 1, 0.1]))        # [0.659 0.242 0.099]

Practice problems

Problem 1: Gradient through a sigmoid stack

A 10-layer network uses sigmoid everywhere. Ignoring the weights, what is the largest possible factor by which the gradient can shrink between the output and the first layer? How does this compare with ReLU units that are all active?

Show solution

σ′(x) = σ(x)(1 − σ(x)) is largest at x = 0, where σ(0) = 0.5.

σ′(0) = 0.5 × 0.5 = 0.25

Best case after 10 layers: 0.25¹⁰ = 1 / 1,048,576 ≈ 9.5 × 10⁻⁷

Active ReLU units have f′ = 1, so 1¹⁰ = 1: no shrinkage from the activations.

Sigmoid: at least a million-fold shrinkage; ReLU: none.

Problem 2: Softmax and temperature

Compute the softmax of the logits [2, 1, 0] at T = 1 and at T = 0.5.

Show solution

T = 1: e² = 7.389, e¹ = 2.718, e⁰ = 1; sum = 11.107

p = [7.389, 2.718, 1] / 11.107 = [0.665, 0.245, 0.090]

T = 0.5: divide the logits by 0.5 to get [4, 2, 0]

e⁴ = 54.598, e² = 7.389, e⁰ = 1; sum = 62.987

p = [0.867, 0.117, 0.016]

Halving T makes the top class go from 66.5% to 86.7%.

Discussion

Questions, ideas or a different approach? Join the conversation.