python3深度学习TensorFlow__模型的保存与恢复/tensorboard可视化

╰半夏微凉° 2022-02-16 20:49 267阅读 0赞

# 0.概要 #

> 通常情况下，我们需要使用训练好的模型来预测或者分类现有数据；但存在一个问题，即每次应用模型的时候都需要重新训练模型，这种方式将会极大提高模型训练成本，因此，需要通过tensorflow深度学习框架对训练完毕的模型进行存储。

# 1.粗粒度的模型保存与恢复 #

## 1.1 单次模型的保存与恢复 ##

### 1.1.1 存储 ###

save_path = "..."
    saver = tf.train.Saver()
    with tf.Session() as sess:
        ...
        # 模型相关定义
        ...
        
        saver.save(sess, save_path)

## 1.1.2 恢复 ##

save_path = "存储时的文件夹对应路径"
    saver = tf.train.Saver()
    with tf.Session() as sess:
        ...
        # 加载相关的代测试数据
        ...
        # 加载训练完毕模型
        saver.restore(sess, save_path)

## 1.2 多次模型的保存与恢复 ##

## 1.2.1 多次存储 ##

save_path = "..."
    saver = tf.train.Saver()
    with tf.Session() as sess:
        ...
        ...
        if epoch % 20 == 0:
            saver.save(sess, save_path, global_step=epoch)
    
    # global_step:参数设定每隔epoch次存储一次模型训练结果
    # (仅保存最近的5个模型文件，不会因为保存的模型文件过多导致存储不足)

### 1.2.2 恢复最新模型 ###

save_path = "上边模型存储的路径"
    model = tf.train.latest_checkpoint(save_path)  # 获取文件夹中最新的模型存储路径
    saver = tf.train.Saver()
    
    with tf.Session() as sess:
        ...
        ...
        # 加载最新模型
        saver.restore(sess, model)
        ...

# 2.细化的保存与恢复方法 #

## 2.1 存储文件说明 ##

> 在模型保存路径对应文件夹下共包含4个文件：
> 
> **checkpoint**：检查点文件，记录存储文件名称
> 
> **save\_model.ckpt.data-00000-of-00001**：等价于save\_model.ckpt，权重存储文件
> 
> **save\_model.ckpt.index**：存储权重目录
> 
> **save\_model.ckpt.meta**：模型的全部图文件

## 2.2 模型的存储与恢复 ##

### 2.2.1 模型存储 ###

> 在具体的模型存储过程中，模型权重在存储到save\_model.ckpt.data-00000-of-00001文件中是依据相应的名称来进行存储的，因此，**需要对模型中不同的权重进行重新注入的话，就需要在模型定义的过程中赋予不同的权重以名称**。

with tf.variable_scope("var"):
        a_val = tf.Variable(tf.random_normal([1]), name="a_val")
        b_val = tf.Variable(tf.random_normal([1]), name="b_val")
    
    x_input = tf.placeholder(tf.float32, name="input_placeholder")
    y_label = tf.placeholder(tf.float32, name="result_placeholder")

### 2.2.2 模型恢复 ###

> **图恢复**
> 
> 第一，恢复模型的整个图文件；
> 
> 第二，从图中获取整个图的信息；
> 
> 第三，根据节点名称将不同的变量或者占位符重新按名称赋值
> 
> **具体权重恢复**
> 
> 第四，具体权重的恢复，在对话中完成

saver = tf.train.import_meta_graph("meta文件对应的相对路径")
    
    graph = tf.get_default_graph()
    
    a_val = graph.get_tensor_by_name("var/a_val: 0")
    
    input_placeholder = graph.get_tensor_by_name("input_placeholder")
    labels_placeholder = graph.get_tensor_by_name("result_placeholder")
    y_output = graph.get_tensor_by_name("output: 0")  # 最终输出结果的张量
    
    
    with tf.Session() as sess:
        saver.restore(".ckpt对应文件的相对路径")
        ...
        ...
        # 程序运行
        ...

### 2.2.3 模型特定值的恢复 ###

> 仍然按照上述顺序，仅仅是针对单一变量。

# 3.tensorboard可视化运行结果\[LeNet为例\] #

## 3.1 图结构(命名空间与tensorboard上图节点) ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FkbWluX21heGlu_size_16_color_FFFFFF_t_70][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FkbWluX21heGlu_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FkbWluX21heGlu_size_16_color_FFFFFF_t_70 2][]

> ①**实线**边表示数据传输，**箭头**表达了数据传输的方向,如：output->accuracy
> 
> ②**双向箭头**(tf=1.0)或**黄色箭头**（tf=1.4）表示修改关联（值）
> 
> ③灰色实线上的**数字**表示张量tensor传递的维度尺寸(output->accuracy中“？x10”：？表示batch图像批量)
> 
> ④当**tensor个数大于1时**，可视化效果图上将只显示张量的个数（loss->train\_step线条上显示3 tensors）
> 
> ⑤边上的**粗细**表示两个节点传输张量纬度的总大小，而不是传输的张量个数
> 
> ⑥带有箭头的**虚线**表示依赖关系（fc1->init：全连接层进行操作之前，需要先初始化依赖张量）
> 
> ⑦节点结构相同，则显示的**节点颜色**也会相同
> 
> ⑧**形状**：圆角矩形代表“命名空间（tf.name\_scope("name")）”，椭圆代表“节点”

## 3.2 节点信息（时间、空间和设备） ##

init = tf.global_variables_initializer()
    with tf.Session() as sess:
    
        sess.run(init)
        mnist_data_set = input_data.read_data_sets("mnist", one_hot=True)
        start_time = time.time()
    
        saver = tf.train.Saver()
        writer = tf.summary.FileWriter("./log", sess.graph)
    
        iters = 10
        for i in range(iters):
            batch_xs, batch_ys = mnist_data_set.train.next_batch(200)
    
            # ==========tensorboard查看节点信息
            # 配置记录信息
            run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
            # 运行时记录信息的proto
            run_metadata = tf.RunMetadata()
    
            # ==========模型精确度变化
            train_accuracy = sess.run(accuracy, feed_dict={x: batch_xs, y_: batch_ys}, options=run_options, run_metadata=run_metadata)
            acc.append(train_accuracy)
            print("step %d, training accuracy %f" % (i, train_accuracy))
            c.append(train_accuracy)
            index.append(i)
    
            # ==========模型损失值变化
            loss = sess.run(cross_entropy, feed_dict={x: batch_xs, y_: batch_ys}, options=run_options, run_metadata=run_metadata)/900
            train_loss.append(loss)
    
            # ==========运行消耗的时间
            cost_time = time.time()-start_time
            avg_time.append(cost_time)
            start_time = time.time()
    
            # ===========借助训练数据训练模型
            sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
    
            # ===========记录节点信息
            writer.add_run_metadata(run_metadata=run_metadata, tag=("tag%d" % i), global_step=i)
            print("After %d training step(s), loss on training batch is %g." % (i, loss))
    
        saver.save(sess, save_model)

### 3.2.1 查看节点时间开销 ###

> 第一：“**Session runs**”中选择index较大的元数据标记，作为计算节点时间消耗标准，此做法能有效降低初始化init对性能的影响；
> 
> 第二：选择“**Compute time**”
> 
> 第三：命名空间或节点颜色越深，代表时间开销越大

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FkbWluX21heGlu_size_16_color_FFFFFF_t_70 3][]

### 3.2.2 查看节点空间开销 ###

> 第一：同3.2.1
> 
> 第二：选择“**Memory**”
> 
> 第三：命名空间或者节点颜色越深表明空间开销越大

### 3.2.3 查看节点设备开销 ###

> 第一：同3.2.1
> 
> 第二：选择“**TPU Compatibility**”（TPU为AlphaGo使用的模型训练芯片）

## 3.3 可视化监控指标（事件、图片、音频、张量分布监控数据） ##

> tensorborad除可以可视化上述图graph外，还可以可视化EVENTS\\IMAGES\\AUDIO\\HISTOGRAMS。以手写数字识别中，卷积层与全连接层“权重”和“偏执项”变化为例，进行程序编写，程序与结果如下所示。
> 
> 说明：
> 
> tf.**merged\_all**(): 合并默认图表中收集的所有摘要
> 
> 可能会出现问题“Nonetype问题”， 这种现象可能由**merge\_all函数包括了不必要的tf.summary.scalar对象**，该对象未被赋值导致。

![è¿éåå¾çæè¿°][70]

# # ========================================================32通道
    # import tensorflow as tf
    # import numpy as np
    # import pandas as pd
    # from tensorflow.examples.tutorials.mnist import input_data
    # import time
    # import matplotlib.pyplot as plt
    # import cv2
    # from PIL import Image
    #
    #
    # def weight_variable(shape):
    #     initial = tf.truncated_normal(shape, stddev=0.1)
    #     return tf.Variable(initial)
    #
    #
    # def bias_variable(shape):
    #     initial = tf.truncated_normal(shape, stddev=0.1)
    #     return tf.Variable(initial)
    #
    #
    # def conv2d(x, W):
    #     return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding="SAME")
    #
    #
    # def max_pool_2x2(x):
    #     return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding="SAME")
    #
    #
    # x = tf.placeholder("float32", [None, 784])
    # y_ = tf.placeholder("float32", [None, 10])
    #
    # x_image = tf.reshape(x, [-1, 28, 28, 1])
    #
    # # 卷积层1
    # W_conv1 = weight_variable([5, 5, 1, 32])
    # b_conv1 = bias_variable([32])
    # h_conv1 = conv2d(x_image, W_conv1) + b_conv1
    #
    # # 池化层1
    # h_pool1 = max_pool_2x2(h_conv1)
    #
    # # 卷积层2
    # W_conv2 = weight_variable([5, 5, 32, 64])
    # b_conv2 = bias_variable([64])
    # h_conv2 = conv2d(h_pool1, W_conv2) + b_conv2
    #
    # # 池化层2
    # h_pool2 = max_pool_2x2(h_conv2)
    #
    # # 全连接层
    # W_fc1 = weight_variable([7*7*64, 1024])
    # b_fc1 = bias_variable([1024])
    # h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
    # h_fc1 = tf.matmul(h_pool2_flat, W_fc1) + b_fc1
    #
    # # 输出层
    # W_fc2 = weight_variable([1024, 10])
    # b_fc2 = bias_variable([10])
    # y_conv = tf.nn.softmax(tf.matmul(h_fc1, W_fc2) + b_fc2)
    #
    # cross_entropy = -tf.reduce_sum(y_*tf.log(y_conv))
    # train_step = tf.train.GradientDescentOptimizer(0.0001).minimize(cross_entropy)
    #
    # # 模型评价
    # correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
    # accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
    #
    # # 定义存储变量
    # c = []
    # index = []
    # average_time = []
    # max_accuracy = []
    #
    # save_model = ".//model//LeNet_5_32.ckpt"
    #
    # init = tf.global_variables_initializer()
    # with tf.Session() as sess:
    #     sess.run(init)
    #     mnist_data_set = input_data.read_data_sets("mnist", one_hot=True)
    #     start_time = time.time()
    #
    #     train_writer = tf.summary.FileWriter(".//log", sess.graph)
    #     saver = tf.train.Saver()
    #
    #     # 迭代次数
    #     for i in range(2000):
    #         batch_xs, batch_ys = mnist_data_set.train.next_batch(50)
    #
    #         # 训练模型精度
    #         train_accuracy = sess.run(accuracy, feed_dict={x: batch_xs, y_: batch_ys})
    #         max_accuracy.append(train_accuracy)
    #         print("step %d, training accuracy %g" % (i, train_accuracy))
    #         c.append(train_accuracy)
    #         index.append(i)
    #
    #         # 建模时间
    #         cost_time = time.time()-start_time
    #         average_time.append(cost_time)
    #         start_time = time.time()
    #
    #         # 训练数据
    #         train_step.run(feed_dict={x: batch_xs, y_: batch_ys})
    #
    #     saver.save(sess, save_model)
    #
    # print("average_time:", np.average(average_time))
    # print("total_time:", np.sum(average_time))
    # print("max accuracy: %.4f" % np.average(max_accuracy[1500:]))
    #
    # # 测试数据
    # saver = tf.train.Saver()
    # with tf.Session() as sess:
    #     save_model = tf.train.latest_checkpoint(".//model")
    #     saver.restore(sess, save_model)
    #
    #     img = mnist_data_set.test.images
    #     labels = mnist_data_set.test.labels
    #     test_accuracy = sess.run(accuracy, feed_dict={x: img, y_: labels})
    #     print("test_accuracy:%.4f" % test_accuracy)
    #
    #
    # # 训练结果绘图
    # plt.figure(figsize=(10, 10))
    # plt.title("net=LeNet channel=32 base_lr=0.0001 batch=50 func=None iters=2000 dropout=None")
    # plt.rcParams["font.sans-serif"] = ["SimHei"]  # 设置显示字体
    # plt.rcParams["axes.unicode_minus"] = False  # 设置正常显示符号
    # plt.xlabel("iters", size=12)
    # plt.ylabel("accuracy", size=12)
    # plt.plot(index, c, color="g")
    # plt.show()
    
    
    
    # =======================================================6通道
    import tensorflow as tf
    import numpy as np
    from tensorflow.examples.tutorials.mnist import input_data
    
    
    def max_pool_2x2(x):
        return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding="SAME")
    
    
    def avg_pool_2x2(x):
        return tf.nn.avg_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding="SAME")
    
    
    def variable_summaries(var, name):
        """
        生成变量监控信息并定义生成监控信息日志的操作
        """
        with tf.name_scope('summaries'):
            tf.summary.histogram(name, var)
            mean = tf.reduce_mean(var)
            tf.summary.scalar('mean/' + name, mean)
            stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean)))
            tf.summary.scalar('stddev/' + name, stddev)
    
    
    def nn_layer(input_tensor, input_dim, output_dim, layer_name, act=tf.nn.relu):
        """
        全连接层操作
        """
        with tf.name_scope(layer_name):
            with tf.name_scope('weights'):
                weights = tf.Variable(tf.truncated_normal([input_dim, output_dim], stddev=0.1))
                variable_summaries(weights, layer_name + '/weights')
            with tf.name_scope('biases'):
                biases = tf.Variable(tf.constant(0.0, shape=[output_dim]))
                variable_summaries(biases, layer_name + '/biases')
            with tf.name_scope('Wx_plus_b'):
                preactivate = tf.matmul(input_tensor, weights) + biases
                tf.summary.histogram(layer_name + '/pre_activations', preactivate)
            activations = act(preactivate, name='activation')
    
            # 记录神经网络节点输出在经过激活函数之后的分布。
            tf.summary.histogram(layer_name + '/activations', activations)
            return activations
    
    
    def cc_layer(input_tensor, kernel_size, input_dim, output_dim, layer_name):
        """
        卷积层操作
        """
        with tf.name_scope(layer_name):
            with tf.name_scope("weight"):
                weights = tf.Variable(tf.truncated_normal([kernel_size, kernel_size, input_dim, output_dim], stddev=0.1))
                variable_summaries(weights, layer_name+"/weights")
            with tf.name_scope("biases"):
                biases = tf.Variable(tf.truncated_normal([output_dim], stddev=0.1))
                variable_summaries(biases, layer_name+"/biases")
            with tf.name_scope("result"):
                res = tf.nn.conv2d(input_tensor, weights, strides=[1, 1, 1, 1], padding="SAME")+biases
    
            tf.summary.histogram(layer_name, res)
            return res
    
    
    def main():
        x = tf.placeholder("float32", [None, 784])
        y_ = tf.placeholder("float32", [None, 10])
    
        x_image = tf.reshape(x, [-1, 28, 28, 1])
    
        # 卷积层1
        h_conv1 = cc_layer(x_image, 5, 1, 6, "conv1")
    
        # 池化层1
        h_pool1 = max_pool_2x2(h_conv1)
    
        # 卷积层2
        h_conv2 = cc_layer(h_pool1, 5, 6, 16, "conv2")
    
        # 池化层2
        h_pool2 = max_pool_2x2(h_conv2)
    
        # 全连接层
        h_pool2_flat = tf.reshape(h_pool2, [-1, 7 * 7 * 16])
        h_fc1 = nn_layer(h_pool2_flat, 7*7*16, 120, "fc1")
    
        # 输出层
        y_conv = nn_layer(h_fc1, 120, 10, "fc2", act=tf.identity)
    
        cross_entropy = -tf.reduce_sum(y_ * tf.log(y_conv))
        train_step = tf.train.GradientDescentOptimizer(0.0001).minimize(cross_entropy)
    
        # 模型评价
        correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
    
        # TODO
        merged = tf.summary.merge_all()
    
        init = tf.global_variables_initializer()
        with tf.Session() as sess:
            sess.run(init)
            mnist_data_set = input_data.read_data_sets("mnist", one_hot=True)
    
            # TODO
            writer = tf.summary.FileWriter("./log", sess.graph)
    
            iters = 10
            for i in range(iters):
                batch_xs, batch_ys = mnist_data_set.train.next_batch(200)
    
                # TODO
                summary, _ = sess.run([merged, train_step], feed_dict={x: batch_xs, y_: batch_ys})
                writer.add_summary(summary, i)
    
    
    
    if "__main__" == __name__:
        main()

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FkbWluX21heGlu_size_16_color_FFFFFF_t_70 4][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FkbWluX21heGlu_size_16_color_FFFFFF_t_70 5][]

## 3.4 参考文章 ##

> [https://blog.csdn.net/qq\_33039859/article/details/80277379\#4%E9%AB%98%E7%BB%B4%E6%95%B0%E6%8D%AE%E5%8F%AF%E8%A7%86%E5%8C%96][https_blog.csdn.net_qq_33039859_article_details_80277379_4_E9_AB_98_E7_BB_B4_E6_95_B0_E6_8D_AE_E5_8F_AF_E8_A7_86_E5_8C_96]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FkbWluX21heGlu_size_16_color_FFFFFF_t_70]: /images/20220217/3c2b8e4bf6ba4ac28b236e059bd15750.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FkbWluX21heGlu_size_16_color_FFFFFF_t_70 1]: /images/20220217/94e8d3d3de11425698f2f64c2c51fcbb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FkbWluX21heGlu_size_16_color_FFFFFF_t_70 2]: /images/20220217/c0f79dc1a3064325a304500a1ab3cfff.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FkbWluX21heGlu_size_16_color_FFFFFF_t_70 3]: /images/20220217/c8f7bfc71e0e468d8edb25f42c1f7cdb.png
[70]: /images/20220217/98dc8fc2e2d049998677551928059d09.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FkbWluX21heGlu_size_16_color_FFFFFF_t_70 4]: /images/20220217/4487f6b9830f4d21b409a130a8dce10f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FkbWluX21heGlu_size_16_color_FFFFFF_t_70 5]: /images/20220217/60415cc4996e429f8ac5f2123c5b8d94.png
[https_blog.csdn.net_qq_33039859_article_details_80277379_4_E9_AB_98_E7_BB_B4_E6_95_B0_E6_8D_AE_E5_8F_AF_E8_A7_86_E5_8C_96]: https://blog.csdn.net/qq_33039859/article/details/80277379#4%E9%AB%98%E7%BB%B4%E6%95%B0%E6%8D%AE%E5%8F%AF%E8%A7%86%E5%8C%96