常见的机器学习算法（七） Softmax回归算法

╰+哭是因爲堅強的太久メ 2023-02-12 07:29 30阅读 0赞

Softmax 回归算法，又称为多项式或多类别的 Logistic 回归算法。

Softmax 回归模型有以下几个特点：

*  对于每个类别，都存在一个独立的、实值加权向量![format_png][]
 *  这个权重向量通常作为权重矩阵中的行。
 *  对于每个类别，都存在一个独立的、实值偏置量b
 *  它使用 softmax 函数作为其激活函数
 *  它使用交叉熵( cross-entropy )作为损失函数

**用 softmax 函数作为激活函数，将得分值转化为概率值形式。**![format_png 1][]属于类别 k 的输入向量的概率值为：

![format_png 2][]

同样地，我们可以通过矢量化来对所有类别同时处理，得到其概率输出。模型预测出的表示的是该类别的最高概率。

**计算整个训练集的损失值。**我们希望模型预测出的高概率值是目标类别，而低概率值表示其他类别。这可以通过以下的交叉熵损失函数来实现：

![format_png 3][]

在上面公式中，目标类别标签表示成独热编码形式( one-hot )。因此![format_png 4][]为1时表示![format_png 1][]的目标类别是 k，反之则为 0。

**对权重向量和偏置量，计算其对损失函数的梯度：**

![format_png 5][]

**对每个类别k，更新其权重和偏置值:**

![format_png 6][]

import numpy as np
    import matplotlib.pyplot as plt
    from sklearn.datasets import load_iris,make_blobs
    from sklearn.model_selection import train_test_split
    
    np.random.seed(123)
    x, y_true = make_blobs(n_samples=5000, centers=4)
    fig = plt.figure(figsize=(8,6))
    plt.scatter(x[:,0], x[:,1], c=y_true)
    plt.title('Dataset')
    plt.xlabel('First feature')
    plt.ylabel('Second feature')
    plt.show()
    
    y_true = y_true[:, np.newaxis]
    x_train, x_test, y_train, y_test = train_test_split(x, y_true)
    print('Shape of x_train: ', x_train.shape)
    print('Shape of y_train: ', y_train.shape)
    print('Shape of x_test: ', x_test.shape)
    print('Shape of y_test: ', y_test.shape)
    
    class SoftmaxRegressor:
        def __init__(self):
            pass
    
        def train(self,x,y_true,n_classes,n_iters=10,l_r=0.1):
            self.n_samples, self.n_features = x.shape
            self.n_classes = n_classes
            self.weights = np.random.rand(self.n_classes, self.n_features)
            self.bias = np.zeros((1, self.n_classes))
            all_losses = []
            for i in range(n_iters):
                scores = self.compute_scores(x)
                probs = self.softmax(scores)
                y_predict = np.argmax(probs, axis=1)[:, np.newaxis]
                y_one_hot = self.one_hot(y_true)
                loss = self.cross_entrpy(y_one_hot, probs)
                all_losses.append(loss)
                dw = (1 / self.n_samples) * np.dot(x.T, probs - y_one_hot)
                db = (1 / self.n_samples) * np.sum(probs - y_one_hot, axis=0)
                self.weights -= l_r * dw.T
                self.bias -= l_r * db
                if i % 100 == 0:
                    print('Iteration number: ', i, ' loss: ', np.round(loss, 4))
            return self.weights, self.bias, all_losses
    
        def predict(self,x):
            scores = self.compute_scores(x)
            probs = self.softmax(scores)
            return np.argmax(probs, axis=1)[:, np.newaxis]
    
        def softmax(self,scores):
            return np.exp(scores) / np.sum(np.exp(scores), axis=1, keepdims=True)
    
        def compute_scores(self,x):
            return np.dot(x, self.weights.T) + self.bias
    
        def cross_entrpy(self, y_true, scores):
            return - (1 / self.n_samples) * np.sum(y_true * np.log(scores))
    
        def one_hot(self,y):
            'Tranforms vector y of labels to one-hot encoded matrix'
            one_hot = np.zeros((self.n_samples, self.n_classes))
            one_hot[np.arange(self.n_samples), y.T] = 1
            return one_hot
    
    '初始化并训练'
    regressor = SoftmaxRegressor()
    w_trained, b_trained, loss = regressor.train(x_train, y_train, l_r=0.1, n_iters=800, n_classes=4)
    fig = plt.figure(figsize=(8,6))
    plt.plot(np.arange(800), loss)# 因为n_iters=800
    plt.title('Development of loss during training')
    plt.xlabel('Number of iterations')
    plt.ylabel('Loss')
    plt.show()
    
    '测试'
    n_samples_test, _ = x_test.shape
    y_predict = regressor.predict(x_test)
    print('Classification accuracy on test set: ', (np.sum(y_predict == y_test) / n_samples_test)*100, '%')

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5OTM4NjY2_size_16_color_FFFFFF_t_70][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5OTM4NjY2_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5OTM4NjY2_size_16_color_FFFFFF_t_70 2][]

[format_png]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9CblNORWFmaWNGQVl6QjdDeXhTU0picVVEbkRnZTFqRE1CRTB6bm1pYUZqUFEyMklRQmF1TUxaT2t0UTExZ3NzSGVEcUdiNDZGUDU1T0NXQlVHb2cwckt3LzY0MD93eF9mbXQ9cG5n?x-oss-process=image/format,png
[format_png 1]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9CblNORWFmaWNGQVl6QjdDeXhTU0picVVEbkRnZTFqRE13UlBpY1RqT1N6M3dZNjNoU0lLSnJtMERUdVRJQktLWllNbVBYOWVMd3NMdk9raWNhUktCaWJoN1EvNjQwP3d4X2ZtdD1wbmc?x-oss-process=image/format,png
[format_png 2]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9CblNORWFmaWNGQVl6QjdDeXhTU0picVVEbkRnZTFqRE12Tk1IcnZWQmJzM2RsclZrZ0pHMWtram41bEpncURXbDRVQ1BXQTZLZEw5Zm9aZW5nNk12RHcvNjQwP3d4X2ZtdD1wbmc?x-oss-process=image/format,png
[format_png 3]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9CblNORWFmaWNGQVl6QjdDeXhTU0picVVEbkRnZTFqRE1nYWN6c25jR0hkODJzUHJ6RzJSYWtMdVlEWGM4ajUxekdKUkJqdnlhemRKNnhkSHlzbVh0TFEvNjQwP3d4X2ZtdD1wbmc?x-oss-process=image/format,png
[format_png 4]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9CblNORWFmaWNGQVl6QjdDeXhTU0picVVEbkRnZTFqRE04NG1IVGJ1bXZJc3NqTzFOeVY3VXFzRmljdmhEc3RWNk1lR2t5Rjh3d1ozWW5Sc0RRWDRjc09nLzY0MD93eF9mbXQ9cG5n?x-oss-process=image/format,png
[format_png 5]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9CblNORWFmaWNGQVl6QjdDeXhTU0picVVEbkRnZTFqRE1ZV2U5Y0tZaWN0YXh6UlB6U3ppY3k2U3BUNGVQOGRGaFNNTG9kanlvZ3NmdXpXcUd3d1ZpY2ZYVlEvNjQwP3d4X2ZtdD1wbmc?x-oss-process=image/format,png
[format_png 6]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9CblNORWFmaWNGQVl6QjdDeXhTU0picVVEbkRnZTFqRE01RVI5S2dzZGliekg2OEl6UmR3ZktVVUdUaWFvVGVXM2c4c2JDVHBSdTBVczRsRkF6ZjgyM3VNUS82NDA_d3hfZm10PXBuZw?x-oss-process=image/format,png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5OTM4NjY2_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200525134615675.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5OTM4NjY2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5OTM4NjY2_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200525134641603.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5OTM4NjY2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5OTM4NjY2_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200525134707621.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5OTM4NjY2,size_16,color_FFFFFF,t_70