如何在Keras和Tensorflow中使用发生器来使用验证集?

fdx2calv  于 2022-11-16  发布在  其他
关注(0)|答案(1)|浏览(142)

大家好,我正在训练一个模型,其中有大量的数据不适合GPU内存,但我可以将其作为NumPy数组放入内存RAM中。因此,我需要将数据从RAM传递到GPU。为此,我使用tf.data.Dataset.from_generator,当我们只传递训练集时,效果非常好。当我给予验证集时,它不会计算验证的指标。此外,它不考虑我在www.example.com函数中设置的epoch数model.fit。下面是我使用的代码:

checkpoint_fine_tune = keras.callbacks.ModelCheckpoint(os.path.join(
    model_save_path, 'fineTune_'+model_name + '_' + '{epoch:08d}.hdf5'), period=1)

# history_tune = model.fit(X_train_reduced_norm, Y_train_reduced, batch_size=16,
# epochs=15, validation_data=( X_test_reduced_norm, Y_test ), callbacks=[checkpoint_fine_tune], shuffle=True)

BATCH_SIZE = 32
STEPS_PER_EPOCH = len(X_train_reduced_norm) // BATCH_SIZE

# Create the argument-free generator as function inside a function with arguments. 
def create_dataset_generator(inputs, labels):
    def argument_free_generator():
        for inp, label in zip(inputs, labels):
            yield inp, label
    return argument_free_generator

generator = create_dataset_generator(X_train_reduced_norm, Y_train_reduced) # Create the generator object from the function
# Create the tf.data.Dataset from this generator and specify the types and shapes of the data. 
dataset_train = tf.data.Dataset.from_generator(generator, (tf.float32, tf.uint8), ((128,128,1), (128,128))) # Create the tf.data.Dataset from this generator and specify the types and shapes of the data.
dataset_train = dataset_train.shuffle(buffer_size=len(X_train_reduced_norm), seed=1234,  
                          reshuffle_each_iteration=True) # Shuffle the data 
dataset_train = dataset_train.prefetch(tf.data.AUTOTUNE) # Prefetch the data for faster processing
dataset_train = dataset_train.repeat().batch(BATCH_SIZE) # Repeat the data for multiple epochs and batch it 

generator_test  = create_dataset_generator(X_test_reduced_norm, Y_test) # Create the generator object from the function
data_test = tf.data.Dataset.from_generator(generator_test, (tf.float32, tf.uint8), ((128,128,1), (128,128))) # Create the tf.data.Dataset from this generator and specify the types and shapes of the data.
data_test = data_test.shuffle(buffer_size=len(X_test_reduced_norm), seed=1234,  
                          reshuffle_each_iteration=True) # Shuffle the data 
data_test = data_test.prefetch(tf.data.AUTOTUNE) # Prefetch the data for faster processing 
data_test = data_test.repeat().batch(BATCH_SIZE) # Repeat the data for multiple epochs and batch it 

history_tune = model.fit(dataset_train, steps_per_epoch=STEPS_PER_EPOCH, epochs=10, validation_data = data_test, 
          validation_steps=len(X_test_reduced_norm)//BATCH_SIZE, callbacks=[checkpoint_fine_tune])


正如你在这里看到的,它没有显示任何来自验证集的指标,而且它只训练了1个历元。如果你对如何解决这个问题有什么想法,请告诉我。
最好的,

l0oc07j2

l0oc07j21#

我认为您没有在**model.compile()方法中使用metric请在您的model.compile()**方法中使用此行

#in case you are using sparecategoricalAccuracy() or simply use accuracy in the metrics...

model.comiple(..., metrics=[tf.keras.metrics.SparseCategoricalAccuracy() , ...]

还有一件事要做,就是将**model.fit()方法中的verbose**设置为2。

model.fit(...,verbose=2,...)

相关问题