优化算法详解¶
概述¶
优化算法是机器学习的核心，决定了模型如何从数据中学习。本教程详解常用的优化算法及其原理。
梯度下降基础¶
基本原理¶
三种梯度下降¶
import numpy as np
# 1. 批量梯度下降（BGD）
def batch_gradient_descent(X, y, lr=0.01, epochs=1000):
w = np.zeros(X.shape[1])
for _ in range(epochs):
gradient = X.T @ (X @ w - y) / len(y)
w -= lr * gradient
return w
# 2. 随机梯度下降（SGD）
def stochastic_gradient_descent(X, y, lr=0.01, epochs=1000):
w = np.zeros(X.shape[1])
for _ in range(epochs):
for i in np.random.permutation(len(y)):
gradient = X[i] * (X[i] @ w - y[i])
w -= lr * gradient
return w
# 3. 小批量梯度下降（Mini-batch）
def mini_batch_gradient_descent(X, y, lr=0.01, epochs=1000, batch_size=32):
w = np.zeros(X.shape[1])
for _ in range(epochs):
indices = np.random.permutation(len(y))
for start in range(0, len(y), batch_size):
batch_idx = indices[start:start+batch_size]
gradient = X[batch_idx].T @ (X[batch_idx] @ w - y[batch_idx]) / len(batch_idx)
w -= lr * gradient
return w
动量优化¶
Momentum¶
def momentum_optimizer(X, y, lr=0.01, momentum=0.9, epochs=1000):
w = np.zeros(X.shape[1])
v = np.zeros_like(w) # 速度
for _ in range(epochs):
gradient = X.T @ (X @ w - y) / len(y)
v = momentum * v + lr * gradient
w -= v
return w
Nesterov Accelerated Gradient (NAG)¶
def nag_optimizer(X, y, lr=0.01, momentum=0.9, epochs=1000):
w = np.zeros(X.shape[1])
v = np.zeros_like(w)
for _ in range(epochs):
w_lookahead = w - momentum * v
gradient = X.T @ (X @ w_lookahead - y) / len(y)
v = momentum * v + lr * gradient
w -= v
return w
自适应学习率¶
1. AdaGrad¶
def adagrad(X, y, lr=0.01, eps=1e-8, epochs=1000):
w = np.zeros(X.shape[1])
G = np.zeros_like(w) # 累积梯度平方
for _ in range(epochs):
gradient = X.T @ (X @ w - y) / len(y)
G += gradient ** 2
w -= lr * gradient / (np.sqrt(G) + eps)
return w
# 特点：学习率逐渐减小，适合稀疏数据
2. RMSProp¶
def rmsprop(X, y, lr=0.001, decay=0.9, eps=1e-8, epochs=1000):
w = np.zeros(X.shape[1])
G = np.zeros_like(w)
for _ in range(epochs):
gradient = X.T @ (X @ w - y) / len(y)
G = decay * G + (1 - decay) * gradient ** 2
w -= lr * gradient / (np.sqrt(G) + eps)
return w
# 特点：解决了AdaGrad学习率过快衰减的问题
3. Adam¶
def adam(X, y, lr=0.001, beta1=0.9, beta2=0.999, eps=1e-8, epochs=1000):
w = np.zeros(X.shape[1])
m = np.zeros_like(w) # 一阶矩估计
v = np.zeros_like(w) # 二阶矩估计
t = 0
for _ in range(epochs):
t += 1
gradient = X.T @ (X @ w - y) / len(y)
# 更新矩估计
m = beta1 * m + (1 - beta1) * gradient
v = beta2 * v + (1 - beta2) * gradient ** 2
# 偏差修正
m_hat = m / (1 - beta1 ** t)
v_hat = v / (1 - beta2 ** t)
# 更新参数
w -= lr * m_hat / (np.sqrt(v_hat) + eps)
return w
# Adam是最常用的优化器，结合了Momentum和RMSProp的优点
4. AdamW¶
def adamw(X, y, lr=0.001, beta1=0.9, beta2=0.999,
eps=1e-8, weight_decay=0.01, epochs=1000):
w = np.zeros(X.shape[1])
m = np.zeros_like(w)
v = np.zeros_like(w)
t = 0
for _ in range(epochs):
t += 1
gradient = X.T @ (X @ w - y) / len(y)
m = beta1 * m + (1 - beta1) * gradient
v = beta2 * v + (1 - beta2) * gradient ** 2
m_hat = m / (1 - beta1 ** t)
v_hat = v / (1 - beta2 ** t)
# AdamW解耦了权重衰减
w -= lr * (m_hat / (np.sqrt(v_hat) + eps) + weight_decay * w)
return w
# AdamW是当前大模型训练的主流优化器
学习率调度¶
常用调度策略¶
import torch.optim.lr_scheduler as lr_scheduler
# 1. StepLR：每隔一定步数衰减
scheduler = lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)
# 2. CosineAnnealingLR：余弦退火
scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max=50)
# 3. OneCycleLR：一周期策略
scheduler = lr_scheduler.OneCycleLR(
optimizer, max_lr=0.01, steps_per_epoch=100, epochs=10
)
# 4. ReduceOnPlateau：指标停止改善时衰减
scheduler = lr_scheduler.ReduceLROnPlateau(
optimizer, mode='min', factor=0.1, patience=10
)
Warmup策略¶
def warmup_scheduler(optimizer, warmup_steps, total_steps):
def lr_lambda(step):
if step < warmup_steps:
return float(step) / float(max(1, warmup_steps))
return max(0.0, float(total_steps - step) / float(max(1, total_steps - warmup_steps)))
return lr_scheduler.LambdaLR(optimizer, lr_lambda)
# 大模型训练常用：warmup + cosine decay
二阶优化方法¶
牛顿法¶
def newton_method(f, grad_f, hess_f, x0, iterations=100):
x = x0
for _ in range(iterations):
gradient = grad_f(x)
hessian = hess_f(x)
x = x - np.linalg.solve(hessian, gradient)
return x
# 优点：收敛快
# 缺点：计算Hessian矩阵开销大
L-BFGS¶
# 拟牛顿法，不需要计算完整的Hessian矩阵
from scipy.optimize import minimize
def lbfgs_optimize(f, grad_f, x0):
result = minimize(f, x0, jac=grad_f, method='L-BFGS-B')
return result.x
优化器对比¶
# 优化器选择指南
SGD + Momentum
- 优点：泛化性好
- 缺点：需要仔细调参
- 适用：计算机视觉
Adam
- 优点：收敛快，调参简单
- 缺点：泛化性可能较差
- 适用：NLP、推荐系统
AdamW
- 优点：解耦权重衰减
- 缺点：内存占用大
- 适用：大模型训练
LAMB
- 优点：支持大batch训练
- 缺点：实现复杂
- 适用：分布式训练
学习率选择¶
# 学习率经验值
SGD: 0.01 - 0.1
Adam: 0.001 (默认)
AdamW: 0.001 - 0.01
# 学习率查找
1. 从小到大逐渐增加学习率
2. 记录每个学习率的loss
3. 选择loss下降最快的区间
# 经验法则
loss突然爆炸 → 学习率太大
loss下降缓慢 → 学习率太小
loss震荡 → 考虑使用动量
实战技巧¶
1. 梯度裁剪¶
2. 混合精度训练¶
# 使用FP16加速训练
from torch.cuda.amp import autocast, GradScaler
scaler = GradScaler()
for batch in dataloader:
with autocast():
output = model(batch)
loss = criterion(output, target)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
3. 梯度累积¶
# 模拟大batch
accumulation_steps = 4
optimizer.zero_grad()
for i, batch in enumerate(dataloader):
loss = model(batch) / accumulation_steps
loss.backward()
if (i + 1) % accumulation_steps == 0:
optimizer.step()
optimizer.zero_grad()