大家好,我正在训练一个模型,其中有大量的数据不适合GPU内存,但我可以将其作为NumPy数组放入内存RAM中。因此,我需要将数据从RAM传递到GPU。为此,我使用tf.data.Dataset.from_generator,当我们只传递训练集时,效果非常好。当我给予验证集时,它不会计算验证的指标。此外,它不考虑我在www.example.com函数中设置的epoch数model.fit。下面是我使用的代码:
checkpoint_fine_tune = keras.callbacks.ModelCheckpoint(os.path.join(
model_save_path, 'fineTune_'+model_name + '_' + '{epoch:08d}.hdf5'), period=1)
# history_tune = model.fit(X_train_reduced_norm, Y_train_reduced, batch_size=16,
# epochs=15, validation_data=( X_test_reduced_norm, Y_test ), callbacks=[checkpoint_fine_tune], shuffle=True)
BATCH_SIZE = 32
STEPS_PER_EPOCH = len(X_train_reduced_norm) // BATCH_SIZE
# Create the argument-free generator as function inside a function with arguments.
def create_dataset_generator(inputs, labels):
def argument_free_generator():
for inp, label in zip(inputs, labels):
yield inp, label
return argument_free_generator
generator = create_dataset_generator(X_train_reduced_norm, Y_train_reduced) # Create the generator object from the function
# Create the tf.data.Dataset from this generator and specify the types and shapes of the data.
dataset_train = tf.data.Dataset.from_generator(generator, (tf.float32, tf.uint8), ((128,128,1), (128,128))) # Create the tf.data.Dataset from this generator and specify the types and shapes of the data.
dataset_train = dataset_train.shuffle(buffer_size=len(X_train_reduced_norm), seed=1234,
reshuffle_each_iteration=True) # Shuffle the data
dataset_train = dataset_train.prefetch(tf.data.AUTOTUNE) # Prefetch the data for faster processing
dataset_train = dataset_train.repeat().batch(BATCH_SIZE) # Repeat the data for multiple epochs and batch it
generator_test = create_dataset_generator(X_test_reduced_norm, Y_test) # Create the generator object from the function
data_test = tf.data.Dataset.from_generator(generator_test, (tf.float32, tf.uint8), ((128,128,1), (128,128))) # Create the tf.data.Dataset from this generator and specify the types and shapes of the data.
data_test = data_test.shuffle(buffer_size=len(X_test_reduced_norm), seed=1234,
reshuffle_each_iteration=True) # Shuffle the data
data_test = data_test.prefetch(tf.data.AUTOTUNE) # Prefetch the data for faster processing
data_test = data_test.repeat().batch(BATCH_SIZE) # Repeat the data for multiple epochs and batch it
history_tune = model.fit(dataset_train, steps_per_epoch=STEPS_PER_EPOCH, epochs=10, validation_data = data_test,
validation_steps=len(X_test_reduced_norm)//BATCH_SIZE, callbacks=[checkpoint_fine_tune])
正如你在这里看到的,它没有显示任何来自验证集的指标,而且它只训练了1个历元。如果你对如何解决这个问题有什么想法,请告诉我。
最好的,
1条答案
按热度按时间l0oc07j21#
我认为您没有在**
model.compile()
方法中使用metric
请在您的model.compile()
**方法中使用此行还有一件事要做,就是将**
model.fit()
方法中的verbose
**设置为2。