[吴恩达机器学习] C++实现梯度下降 & 牛顿迭代法

阳光穿透心脏的1/2处 2023-01-17 09:30 179阅读 0赞

URL：  
[https://www.bilibili.com/video/BV164411S78V?p=10][https_www.bilibili.com_video_BV164411S78V_p_10]

以 f ( x ) = x 3 − x f(x) = x^\{3\}-x f(x)=x3−x为例

d d x f ( x ) = 3 x 2 − 1 \\frac\{\\mathrm\{d\} \}\{\\mathrm\{d\} x\}f(x) = 3x^\{2\}-1 dxdf(x)=3x2−1  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0wxNTU4MTk4NzI3_size_16_color_FFFFFF_t_70]

**梯度下降公式：求极值点**  
 x = x − α d d x f ( x ) x = x - \\alpha \\frac\{\\mathrm\{d\} \}\{\\mathrm\{d\} x\}f(x) x=x−αdxdf(x)  
 α \\alpha α**为学习率**

**需要注意的是迭代的方向是想着函数值减小的方向，  
如果改方向没有极值点，那么就无法停止**

**牛顿迭代法：求零点**

x = x − f ( x ) d f ( x ) x = x - \\frac\{f(x) \}\{\\mathrm\{d\} f(x)\} x=x−df(x)f(x)  
 α \\alpha α**为学习率**

#include <iostream>
    #include <math.h>
    
    using namespace std;
    #define debug(x) cout<< #x <<": "<< (x) <<endl;
    
    bool lfeq(double a, double b) { 
        double delt = a-b;
        if (delt < 0) { 
            delt = -delt;
        }
        return delt < 1e-5;
    }
    
    double f(double x) { 
        return x* x* x - x;
    }
    double df(double x) { 
        return 3* x * x - 1;
    }
    
    bool gradientDescent() { 
    
        double alpha = 0.1;
        double x = 2;
    
        while (!lfeq(df(x), 0)) { 
            x = x - alpha * df(x);
            debug(x);
        }
    
        debug(sqrt(3) / 3)
    
        return true;
    }
    
    bool newtonIter() { 
    
        double x = 2;
    
        while (!lfeq(f(x), 0)) { 
            x = x -  f(x)/df(x);
            debug(x);
        }
        debug(1)
        return true;
    }
    
    int main()
    { 
        //梯度下降
        gradientDescent();
        //牛顿迭代法
        newtonIter();
        return 0;
    }

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0wxNTU4MTk4NzI3_size_16_color_FFFFFF_t_70 1]

--------------------

**关于学习率**

[视频][Link 1]

学习率过大：可能导致目标函数发散或者震荡  
学习率过小：目标函数收敛需要很长的时间

数学证明了当学习率足够小的时候，每次迭代目标函数一定会下降

[https_www.bilibili.com_video_BV164411S78V_p_10]: https://www.bilibili.com/video/BV164411S78V?p=10
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0wxNTU4MTk4NzI3_size_16_color_FFFFFF_t_70]: /images/20221021/3b1cc43e42f54027b20ac3a2d0047f3e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0wxNTU4MTk4NzI3_size_16_color_FFFFFF_t_70 1]: /images/20221021/61ccdf6a36b842acbc7c976b5cb1131b.png
[Link 1]: https://www.bilibili.com/video/BV164411S78V?p=21&spm_id_from=pageDriver