keras data fit generator 数据生成器多输入数据生成器图片分割逐像素label 直观调试生成图片

小咪咪 2022-01-29 08:31 274阅读 0赞

此处以keras为例。

多数的深度学习方法都是支持批量yeild数据，而不是直接加载所有数据到内存。

给一个对比：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNDk4NTM_size_16_color_FFFFFF_t_70][]

25512是未使用数据生成器的

30777是使用数据生成器的。

但是必须要说明的，GPU的内存是不会减少的，都是有多少用多少。给的多就多用，给的少就少用。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNDk4NTM_size_16_color_FFFFFF_t_70 1][]

那么首先给出原始教程：

[https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly][https_stanford.edu_shervine_blog_keras-how-to-generate-data-on-the-fly]

本教程就是教你怎么改的适合自己的数据。

源代码：（复制到你的文件中）

import numpy as np
    import keras
    
    class DataGenerator(keras.utils.Sequence):
        'Generates data for Keras'
        def __init__(self, list_IDs, labels, batch_size=32, dim=(32,32,32), n_channels=1,
                     n_classes=10, shuffle=True):
            'Initialization'
            self.dim = dim
            self.batch_size = batch_size
            self.labels = labels
            self.list_IDs = list_IDs
            self.n_channels = n_channels
            self.n_classes = n_classes
            self.shuffle = shuffle
            self.on_epoch_end()
    
        def __len__(self):
            'Denotes the number of batches per epoch'
            return int(np.floor(len(self.list_IDs) / self.batch_size))
    
        def __getitem__(self, index):
            'Generate one batch of data'
            # Generate indexes of the batch
            indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]
    
            # Find list of IDs
            list_IDs_temp = [self.list_IDs[k] for k in indexes]
    
            # Generate data
            X, y = self.__data_generation(list_IDs_temp)
    
            return X, y
    
        def on_epoch_end(self):
            'Updates indexes after each epoch'
            self.indexes = np.arange(len(self.list_IDs))
            if self.shuffle == True:
                np.random.shuffle(self.indexes)
    
        def __data_generation(self, list_IDs_temp):
            'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)
            # Initialization
            X = np.empty((self.batch_size, *self.dim, self.n_channels))
            y = np.empty((self.batch_size), dtype=int)
    
            # Generate data
            for i, ID in enumerate(list_IDs_temp):
                # Store sample
                X[i,] = np.load('data/' + ID + '.npy')
    
                # Store class
                y[i] = self.labels[ID]
    
            return X, keras.utils.to_categorical(y, num_classes=self.n_classes)

配套的fit\_generator源代码：（选择的复制，这代码这么简单，你看得懂吧，复制相关部分）

import numpy as np
    
    from keras.models import Sequential
    from my_classes import DataGenerator
    
    # Parameters
    params = {'dim': (32,32,32),
              'batch_size': 64,
              'n_classes': 6,
              'n_channels': 1,
              'shuffle': True}
    
    # Datasets
    partition = # IDs
    labels = # Labels
    
    # Generators
    training_generator = DataGenerator(partition['train'], labels, **params)
    validation_generator = DataGenerator(partition['validation'], labels, **params)
    
    # Design model
    model = Sequential()
    [...] # Architecture
    model.compile()
    
    # Train model on dataset
    model.fit_generator(generator=training_generator,
                        validation_data=validation_generator,
                        use_multiprocessing=True,
                        workers=6)

我的代码：（别复制，贼丑，还没优化呢，就是给你个成功事例）

os.environ['CUDA_VISIBLE_DEVICES'] = '2'
    imgDir = '/home/DATA/database/nyuv1/'
    seg_UCM_path = '/home/liutian/nyuv1/ucm/segmentation/ucm'
    
    class DataGenerator(keras.utils.Sequence):
        'Generates data for Keras'
    
        def __init__(self, list_IDs, batch_size=2, dim=(480, 640), n_channels=3,
                     shuffle=True):
            'Initialization'
            self.dim = dim
            self.batch_size = batch_size
            self.list_IDs = list_IDs
            self.n_channels = n_channels
    
            self.shuffle = shuffle
    
            self.on_epoch_end()
    
        def __len__(self):
            'Denotes the number of batches per epoch'
            return int(np.floor(len(self.list_IDs) / self.batch_size))
    
        def __getitem__(self, index):
            'Generate one batch of data'
            # Generate indexes of the batch
            indexes = self.indexes[index * self.batch_size:(index + 1) * self.batch_size]
    
            # Find list of IDs
            list_IDs_temp = [self.list_IDs[k] for k in indexes]
    
            # Generate data
            X, y, seg = self.__data_generation(list_IDs_temp)
    
            return [X,seg],y
    
        def on_epoch_end(self):
            'Updates indexes after each epoch'
            self.indexes = np.arange(len(self.list_IDs))
            if self.shuffle == True:
                np.random.shuffle(self.indexes)
    
        def __data_generation(self, list_IDs_temp):
            # 'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)
            # Initialization
    
            threshold = 60
            segFoldName = str(threshold).zfill(3)
            X = np.empty((self.batch_size, *self.dim, self.n_channels))
            Y = np.empty((self.batch_size, *self.dim))
            seg = np.empty((self.batch_size, *self.dim))
    
            # Generate data
            for i, ind in enumerate(list_IDs_temp):
                # Store sample
                img_file_dir = imgDir + '/allpic/' + str(ind + 1).zfill(8) + '.jpg'
                img_array = array(Image.open(img_file_dir))
                X[i, :, :, :] = img_array
                X = X.astype(np.float32)/255
                seg_file_dir = seg_UCM_path + segFoldName + '/' + str(ind + 1).zfill(8) + '.mat'
                seg_mat = scio.loadmat(seg_file_dir)
                seg_piece = seg_mat["seg"]
                seg[i, :] = seg_piece
                lab_file_dir = imgDir + '/label/' + str(ind + 1).zfill(8) + '.png'
                lab_array = array(Image.open(lab_file_dir))
                Y[i, :, :] = lab_array
                perform_shape = list(np.shape(Y))
                sum = np.zeros(shape=perform_shape + [14, ], dtype=np.int8)
                # TODO:change this to a function
                # I reshape my GT label through this
                # sum = np.zeros(shape=(2284, 480, 640, 14))
     
                pic_lab = Y[i, :, :]
                # pic_flatten = np.reshape(pic_lab, (1, 1200))
                for j in range(perform_shape[1]):
                    pic_line = pic_lab[j, :]
                    pic_onehot = keras.utils.to_categorical(pic_line, 14)
                    sum[i, j] = pic_onehot
    
            return X, sum, seg

注意，我这里自定义了数据读取路径，并且有两个输入，两个输入尺寸还不一样。那么配套的训练代码就是：

params = {'dim': (480, 640),
                      'batch_size': 1,
                      'n_channels': 3,
                      'shuffle': True}
    
            # Datasets
    
            partition = {'train': img.trainval_list,
                         'validation': img.test_list}  # IDs
    
            # Generators
            training_generator = DataGenerator(partition['train'], **params)
            validation_generator = DataGenerator(partition['validation'], **params)
    
            # self.model_simple1.compile(loss=loss,
            #                      optimizer=optimizers, metrics=metric)
            model_simple1.fit_generator(generator=training_generator,
                                        validation_data=validation_generator,
                                        use_multiprocessing=True,
                                        callbacks=config.callbacks_list,
                                        verbose=1,
                                        epochs=100,
                                        workers=6)

你需要修改：

1.**param字典**，改成你的图片尺寸，数据尺寸，等，就是很简单的函数传参

2.**partition字典**，大概张这样：关键字:list

partition
    {'train': [1,2,34,5,6,1024], 'validation': [99,88,77]}

这里是为了要在\_\_data\_generation这个方法里面读。你看我的读数据，用到了这个字典。

3.**改数据为onehot编码**。我这里由于是图片的dense prediction，所以不能使用默认的onehot一键转换函数，自己写了一个。你也要注意自己的监督信号。这部分我卸载了\_\_data\_generation中，其实应该写个函数，更好看

4.**\_\_data\_generation**:这个函数改起来非常简单，你只需要让该方法能够根据传入的序号，去读取文件即可。我的传入序号\[97\]这样。我的数据名称你跑一下就知道了。

***注意看我的多输入只不过就是把两个输入组成了一个List。***

**总结**

关键在于把控数据流。根据提供的train，test的id列表，从中随机取出batch\_size大小的id个数，\_\_data\_generation根据提供的随机id，读取若干数据，处理监督信号成为onehot编码。然后如果多输入，使用list框柱两个输入。返回。

下一步目标是数据扩增。

调试：[https://blog.csdn.net/u013249853/article/details/90602180][https_blog.csdn.net_u013249853_article_details_90602180]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNDk4NTM_size_16_color_FFFFFF_t_70 2][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNDk4NTM_size_16_color_FFFFFF_t_70]: /images/20220129/6f25413f199445dfae46d3993e5eafdb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNDk4NTM_size_16_color_FFFFFF_t_70 1]: /images/20220129/69b52c1aba4d45609b4d35cfaa283ff2.png
[https_stanford.edu_shervine_blog_keras-how-to-generate-data-on-the-fly]: https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly
[https_blog.csdn.net_u013249853_article_details_90602180]: https://blog.csdn.net/u013249853/article/details/90602180
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNDk4NTM_size_16_color_FFFFFF_t_70 2]: /images/20220129/d29e8651f0624ff7a9d39f38983fab3b.png