PyTorch torch.nn.LeakyReLU Function

PyTorch torch.nn Reference Manual

torch.nn.LeakyReLU is the leaky ReLU activation function in PyTorch.

It allows negative values to have a small positive gradient, avoiding the "dying ReLU" problem.

Function Definition

torch.nn.LeakyReLU(negative_slope=0.01, inplace=False)

Parameters

• negative_slope: The slope for negative values, default is 0.01

Formula

f(x) = x, x > 0
f(x) = negative_slope * x, x <= 0

Usage Examples

Example 1: Basic Usage

Example

import torch
import torch.nn as nn

lrelu = nn.LeakyReLU(negative_slope=0.1)

x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
output = lrelu(x)

print("Input:", x.tolist())
print("Output:", output.tolist())
print("Negative values have small positive gradient")

Example 2: Usage in GAN

Example

import torch
import torch.nn as nn

# GAN generators commonly use LeakyReLU
generator = nn.Sequential(
    nn.Linear(100, 256),
    nn.LeakyReLU(0.2),
    nn.Linear(256, 512),
    nn.LeakyReLU(0.2),
    nn.Linear(512, 784),
    nn.Tanh()
)

z = torch.randn(4, 100)
output = generator(z)

print("Input:", z.shape, "-> Output:", output.shape)

Example 3: Preventing Dying Neurons

Example

import torch
import torch.nn as nn

# Compare ReLU and LeakyReLU
relu = nn.ReLU()
lrelu = nn.LeakyReLU()

# Negative value input
x = torch.randn(10, 100) - 5

# ReLU outputs all zeros
print("Non-zero ratio after ReLU:", (relu(x) != 0).float().mean().item())

# LeakyReLU has gradient
print("Non-zero ratio after LeakyReLU:", (lrelu(x) != 0).float().mean().item())

Use Cases

• GAN: Generator and discriminator
• Preventing dying neurons
• Sparse networks

PyTorch torch.nn Reference Manual