注意:这不是什么入门教程。
架构 一个全连接神经网络(Fully Connected Neural Network)是由全连接层(Fully Connected Layer)和损失函数 \(L\) 组成的。
数学上来说,一个层其实就是一个函数。如果输入是 \(n\) 维向量,输出是 \(m\) 维向量,层就是 \(\mathbb{R}^n\to\mathbb{R}^m\) 的一个映射。使用反向传播法训练的模型中,一个层至少要有两种方法:
前向(Forward Propagation)。就是作为一个函数使用,对给定的输入 \(X \in \mathbb{R}^n\) ,给出一个 \(Y \in \mathbb{R}^m\) 。 后向(Backward Propagation)。应该在前向之后被调用。给定输出关于损失函数的导数 \(\frac{\partial L}{\partial Y}\) ,计算刚才的输入 \(X\) 关于损失函数的导数 \(\frac{\partial L}{\partial X}\) 。同时,如果这个层有可优化的参数,后向函数应该完成对参数的优化。 而整个模型要做的事就很简单:将输入发给第一层的前向,然后把每一层的输出传给下一层的前向,得到预测输出 \(\hat{Y}\) ;计算损失 \(L(\hat{Y},Y)\) 以及 \(\frac{\partial L}{\partial \hat{Y}}\) ,再反过来把后一层后向的输出发给前一层的前向。
全连接层的数学原理 一个全连接层含有两组参数 \(W \in \mathbb{R}^{n\times m}\) 和 \(b \in \mathbb{R}^m\) 。同时有一个激活函数 \(f:\mathbb{R}^m \to \mathbb{R}^m\) 。前向的公式是:
\[ Y=f(W^TX+b) \]
后向的公式是:
\[ \begin{aligned} \frac{\partial L}{\partial X}&= \frac{\partial L}{\partial Y}\frac{\partial Y}{\partial (W^TX+b)}\frac{\partial (W^TX+b)}{\partial X} \\ &= \frac{\partial L}{\partial Y}J_f(W^TX+b)W^T \end{aligned} \]
由于 \(f(X)^{(i)}\) 只与 \(X^{(i)}\) 有关,所以 \(J_f\) 是一个对角阵的形式。满足 \(J_f(i;i) = \frac{\partial f}{\partial X^{(i)}}\) 。
数对向量求导(\(\frac{\partial L}{\partial Y}\) )是行向量,函数(\(\mathbb{R}^n \to \mathbb{R}^m\) )的雅克布矩阵是 \(\mathbb{R}^{n\times m}\) 。数对矩阵求导(\(\frac{\partial L}{\partial W}\) )是同形的矩阵。
实现时应该与数学形式统一。(即导函数要返回雅克布矩阵)
如果我们使用梯度下降法优化参数,我们还要求出 \(\frac{\partial L}{\partial W}\) 与 \(\frac{\partial L}{\partial b}\) 。其实很像啊:
\[ \begin{aligned} \frac{\partial L}{\partial W} &= X\frac{\partial L}{\partial Y}J_f(W^TX+b)\\ \frac{\partial L}{\partial b} &= \frac{\partial L}{\partial Y}J_f(W^TX+b) \end{aligned} \]
下面给出一些常用的激活函数:
Softmax \[ \begin{aligned} \operatorname{Softmax}(X)_{i}&=\frac{e^{X_{i}}}{\sum_j{e^{X_{j}}}} \\ J_{i,j} &= \left\{ \begin{aligned} &\frac{e^{X_i}\sum - (e^{X_i})^2}{(\sum)^2} &, i=j \\ &\frac{-e^{X_i}e^{X_j}}{(\sum)^2} &, i \ne j \end{aligned} \right. \end{aligned} \]
Sigmoid \[ \begin{aligned} \operatorname{Sigmoid}(X)_{i}&=\frac{1}{1+e^{-X_i}} \\ J_{i,j} &= \left\{ \begin{aligned} &\frac{e^{-X_i}}{(1 + e^{-X_i})^2} &, i=j \\ &0 &, i \ne j \end{aligned} \right. \end{aligned} \]
tanh \[ \begin{aligned} \tanh(X)_{i}&=\tanh(X_i) \\ J_{i,j} &= \left\{ \begin{aligned} &1-\tanh^2(X_i) &, i=j \\ &0 &, i \ne j \end{aligned} \right. \end{aligned} \]
ReLU \[ \begin{aligned} \operatorname{ReLU}(X)_{i}&=\max(X_i,0)\\ J_{i,j} &= \left\{ \begin{aligned} &1 ,& i=j,X_i > 0 \\ &0 ,& \text{otherwise.} \\ \end{aligned} \right. \end{aligned} \]
Leaky ReLU \[ \begin{aligned} \operatorname{LeakyReLU}(X)_{i}&=\max(X_i,\alpha X_i)\\ J_{i,j} &= \left\{ \begin{aligned} &1 ,& i=j,X_i > 0 \\ &\alpha ,& \text{otherwise.} \\ \end{aligned} \right. \end{aligned} \]
ELU \[ \begin{aligned} \operatorname{ELU}(X)_{i}&=\left\{ \begin{aligned} &X_i, &X_i>0\\ &e^{X_i}-1, &X_i \le 0 \end{aligned} \right.\\ J_{i,j} &= \left\{ \begin{aligned} &1 ,& i=j,X_i > 0 \\ &0 ,& \text{otherwise.} \\ \end{aligned} \right. \end{aligned} \]
损失函数的数学原理 给出一些常用的损失函数:
交叉熵(CrossEntropy) 常用于多分类问题。应该满足 \(\sum Y=\sum \hat{Y}=1\) 。
\[ \begin{aligned} L(\hat Y, Y) &= -\sum_i Y_i \log \hat Y_i \\ \frac{\partial L}{\partial \hat Y_i} &= -\frac{Y_i}{\hat Y_i} \end{aligned} \]
考虑到这玩意经常与 Softmax 一起使用,我们可以两个连在一起算,得到一个简化结果:
\[ \frac{\partial L}{\partial \hat Y} J_{\operatorname{Softmax}}=(Y - \hat Y)^T \]
均方误差 常用于回归任务。太简单了,就不写了。
正则化 原理是在损失函数上加一个惩罚项,参数越复杂,惩罚项越大。即按照奥卡姆剃刀原则以避免过拟合。
常见的有 \(L_0,L_1,L_2\) 等。\(L_0(x)=[x \ne 0],L_1(x) = |x|, L_2(x)=x^2\) 。以 \(L_2\) 正则项为例:
\[ L'(\hat Y, Y)=L(\hat Y, Y) + \lambda \sum L_2(W)+L_2(b) \]
求导的时候也把正则项加上就好。
代码实现 NeuralNetwork.py
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 import numpy as npfrom NNFunctions import *from NNLayers import *class FCNN : def __init__ (self, layers, lossFunc ) : assert (type (layers) == type ([])) assert (len (layers) > 0 ) self.lossFunc = lossFunc self.layers = layers def _forward (self, X ) : for layer in self.layers : X = layer._forward(X) return X def _backward (self, dY, skipLastCalc = False ) : for layer in self.layers[::-1 ] : dY = layer._backward(dY, skipLastCalc) skipLastCalc = False def _update (self, learningRate, lam ) : for layer in self.layers : layer._update(learningRate, lam) def trainBatch (self, X, Y, learningRate, lam ) : X, Y = np.array(X).astype(np.float64), np.array(Y).astype(np.float64) if X.ndim == 1 : X, Y = X.reshape(1 , -1 ), Y.reshape(1 , -1 ) loss = 0 for i in range (X.shape[0 ]) : yHat = self._forward(X[i].reshape(-1 , 1 )) if type (self.lossFunc) == type (CrossEntropy()) and \ type (self.layers[-1 ].func) == type (Softmax()) : self._backward(yHat.reshape(1 , -1 ) - Y[i].reshape(1 , -1 ), True ) else : self._backward(self.lossFunc.gradient(yHat, Y[i].reshape(-1 , 1 ))) loss += self.lossFunc(yHat, Y[i].reshape(-1 , 1 )) self._update(learningRate, lam) return loss / X.shape[0 ] def trainOne (self, X, Y, learningRate, lam ) : X, Y = np.array(X).astype(np.float64).reshape(-1 , 1 ), np.array(Y).astype(np.float64).reshape(-1 , 1 ) yHat = self._forward(X) if type (self.lossFunc) == type (CrossEntropy()) and \ type (self.layers[-1 ].func) == type (Softmax()) : self._backward(yHat.reshape(1 , -1 ) - Y.reshape(1 , -1 ), True ) else : self._backward(self.lossFunc.gradient(yHat, Y)) self._update(learningRate, lam) return self.lossFunc(yHat, Y) def testBatch (self, X, Y ) : X, Y = np.array(X).astype(np.float64), np.array(Y).astype(np.float64) if X.ndim == 1 : X, Y = X.reshape(1 , -1 ), Y.reshape(1 , -1 ) cntAcc = 0 for i in range (X.shape[0 ]) : yHat = self._forward(X[i].reshape(-1 , 1 )) cntAcc += 1 if np.argmax(yHat) == np.argmax(Y[i].reshape(-1 , 1 )) else 0 return cntAcc / X.shape[0 ] def testOne (self, X, Y ) : X, Y = np.array(X).astype(np.float64).reshape(-1 , 1 ), np.array(Y).astype(np.float64).reshape(-1 , 1 ) yHat = self._forward(X) return np.argmax(yHat) == np.argmax(Y) def predictBatch (self, X ) : X = np.array(X).astype(np.float64) if X.ndim == 1 : X = X.reshape(1 , -1 ) yHat = [] for i in range (X.shape[0 ]) : yHat.append(self._forward(X[i].reshape(-1 , 1 ))) return yHat def predictOne (self, X ) : X = np.array(X).reshape(-1 , 1 ).astype(np.float64) return self._forward(X)
NNFunctions.py
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 import numpy as np_maxV = np.log(np.finfo(np.float32).max ) def _exp (z ) : return np.exp(np.clip(z, -_maxV, _maxV)) def _log (z ) : return np.log(z, out=np.zeros_like(z), where=(z!=0 )) class ReLU : def __call__ (self, z ) : return np.where(z >= 0 , z, 0 ) def gradient (self, z ) : z = z.reshape(-1 ) return np.diag(np.where(z >= 0 , 1 , 0 )) class LeakyReLU : def __init__ (self, alpha = 0.01 ) : self.alpha = alpha def __call__ (self, z ) : return np.where(z >= 0 , z, self.alpha * z) def gradient (self, z ) : z = z.reshape(-1 ) return np.diag(np.where(z >= 0 , 1 , self.alpha)) class ELU : def __call__ (self, z ) : return np.where(z >= 0 , z, _exp(z) - 1 ) def gradient (self, z ) : z = z.reshape(-1 ) return np.diag(np.where(z >= 0 , 1 , _exp(z))) class Softmax : def __call__ (self, z ) : return _exp(z) / np.sum (_exp(z)) def gradient (self, z ) : res = np.zeros((z.shape[0 ], z.shape[0 ])) callz = self(z) for i in range (z.shape[0 ]) : for j in range (z.shape[0 ]) : if i == j : res[i, j] = callz[i, 0 ] * (1 - callz[i, 0 ]) else : res[i, j] = - callz[i, 0 ] * callz[j, 0 ] return res class tanh : def __call__ (self, z ) : return np.tanh(z) def gradient (self, z ) : z = z.reshape(-1 ) return np.diag(1 - np.tanh(z) * np.tanh(z)) class CrossEntropy : def __call__ (self, yHat, Y ) : assert (yHat.shape == Y.shape) return -np.sum (Y * _log(yHat)) def gradient (self, yHat, Y ) : assert (yHat.shape == Y.shape) return (-Y / yHat).reshape(1 , -1 )
NNLayers.py
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 import numpy as npclass FullyConnectedLayer : def __init__ (self, nx, ny, func ) : self.nx, self.ny = nx, ny self.w = np.random.randn(nx, ny).astype(np.float64) self.b = np.zeros(ny).reshape(ny, 1 ).astype(np.float64) self.dw = np.zeros((nx, ny)).astype(np.float64) self.db = np.zeros(ny).reshape(ny, 1 ).astype(np.float64) self.dcnt = 0 self.func = func def _forward (self, X ) : assert (X.shape == (self.nx, 1 )) self.X = X self.z = np.dot(self.w.T, X) + self.b return self.func(self.z) def _backward (self, dY, skipCalc = False ) : assert (dY.shape == (1 , self.ny)) dz = dY if skipCalc else np.dot(dY, self.func.gradient(self.z)) self.dw += np.dot(self.X, dz) self.db += dz.T self.dcnt += 1 return np.dot(dz, self.w.T) def _update (self, learningRate, lam ) : self.w -= (self.dw + self.w * lam * 0.5 ) / self.dcnt * learningRate self.b -= (self.db + self.b * lam * 0.5 ) / self.dcnt * learningRate self.dw = np.zeros((self.nx, self.ny)) self.db = np.zeros(self.ny).reshape(self.ny, 1 ) self.dcnt = 0