如何使用Tensorflow创建一个包含一小部分图像的数据集

a64a0gku  于 2023-05-01  发布在  其他
关注(0)|答案(1)|浏览(188)

我在一个文件夹中有9.9万张图像,但我只需要10000张图像作为数据集。我试着先随机选择图像,然后创建数据集,但似乎不起作用。我需要使用Tensorlfow。
我被这个问题困扰了一个多星期。如果任何人有另一种方法来创建一个数据集,它只包含与我完全不同的图像总数的一小部分,那对我来说没问题。
谢谢大家!
现在我正在尝试让下面的代码工作:

import tensorflow as tf
import numpy as np
import os

# Define the directory where your images are located
data_dir = '/home/clau/Escritorio/cropped/'

# Load the list of image filenames
filenames = tf.constant(os.listdir(data_dir))
num_images = len(filenames)

# Shuffle the filenames
filenames = tf.random.shuffle(filenames)

# Select the first 12000 filenames
filenames = filenames[:10000]

# Define a function to load and preprocess each image
def load_and_preprocess_image(filename):
    # Convert the filename tensor to a string
    filename = filename.numpy().decode('utf-8')

    # Load the image file
    file_path = os.path.join(data_dir, filename)
    img = tf.io.read_file(file_path)

    # Decode the image and convert to floating-point tensor
    img = tf.image.decode_jpeg(img, channels=3)
    img = tf.image.convert_image_dtype(img, tf.float32)

    # Resize the image to 256x256
    img = tf.image.resize(img, [256, 256])

    # Normalize the pixel values to the range [0,1]
    img = img / 255.0

    return img

# Create an empty dataset
dataset = tf.data.Dataset.from_tensor_slices(tf.constant([], dtype=tf.string))

for imgfile in filenames:
    processed_img = load_and_preprocess_image(imgfile)
    
    # Add the image
    dataset = dataset.concatenate(tf.data.Dataset.from_tensor_slices(processed_img))

它给出了以下错误:
TypeError:不兼容的数据集元素:TensorSpec(shape=(),dtype=tf.string,name=None)与TensorSpec(shape=(256,3),dtype=tf.float32,name=None)

a1o7rhls

a1o7rhls1#

您可以使用所需的特定数量的图像创建Tensorflow记录,并在训练模型之前解析它们。
下面是创建TF记录文件并将其解析为模型的示例代码。我假设图像文件名是你的标签,这些标签是整数,因为你的代码不清楚你使用的是什么标签。

import tensorflow as tf
import numpy as np
import random
import os

def image_feature(value):
    """
    Returns a bytes_list from a string / byte.
    """

    return tf.train.Feature(
        bytes_list=tf.train.BytesList(value=[tf.io.encode_jpeg(value).numpy()])
    )

def bytes_feature(value):
    """Returns a bytes_list from a string / byte."""
    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))

def float_feature(value):
    """Returns a float_list from a float / double."""
    return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))

def int64_feature(value):
    """Returns an int64_list from a bool / enum / int / uint."""
    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))

def create_example(image:np.ndarray, label:int):
    feature = {
        "label" : int64_feature(label),
        "image" : image_feature(image),        
    }
    return tf.train.Example(features=tf.train.Features(feature=feature))

path_to_your_images = "/path/to/your/images"

#Here we sample 10000 random images from your image folder
images_path_list = random.sample(os.listdir(path_to_your_images), 10000)

with tf.io.TFRecordWriter("train.tfrec") as writer:
    for file_path in images_path_list:
        img = tf.io.read_file(os.path.join(path_to_your_images, file_path))
        label = int(file_path.strip(".jpg"))
        """ 
        Here we decode the image bytes first and encode it again using encode_jpeg(). 
        This encoding reduces the memory occupied by the TF records file. 
        You can also directly write the raw image bytes to TF records but that would make 
        the file size of TF records to be huge as it saves the actual bytes of the image
        """
        example = create_example(tf.image.decode_jpeg(img, channels=3), label) 
        writer.write(example.SerializeToString())

上面的代码将创建具有10000个图像的TF记录文件。现在您可以使用下面的代码来解析图像和标签,并创建数据集句柄来训练模型

def _parse_dataset(example_proto):
    _feature_description = {
        "image" : tf.io.FixedLenFeature([], tf.string),        
        "label" : tf.io.FixedLenFeature([], tf.int64)
    }
    
    example = tf.io.parse_single_example(example_proto, _feature_description)

    image = tf.io.decode_jpeg(example["image"])    
    
    image = tf.cast(image, tf.float32) / 255.0
    label = example["label"]

def create_dataset(tfrecord_paths, batch_size):
    raw_dataset = tf.data.TFRecordDataset(tfrecord_paths)
    dataset = raw_dataset.map(_parse_dataset)
   
    dataset = dataset.batch(batch_size)
    return dataset

train_dataset = create_dataset("/path/to/train.tfrec",8)

现在,您可以直接在model.fit(train_dataset)中使用train_dataset来训练模型。要从数据集中访问一个示例,可以执行train_dataset.take(1)。这将给予你两个值一个是你的形象形状(batch_size, height, width, channels)和其他是你的label

***注意:***您不仅限于您的图像和标签,您还可以通过feature_description字典将其他信息写入TF记录文件。例如,如果要为每个示例添加图像文件名、图像的宽度和高度,则可以按如下方式修改要素描述

feature = {
        "file_name" : bytes_feature(filename.encode("utf8")),
        "width" : int64_feature(width),
        "height" : int64_feature(height)
        "label" : int64_feature(label),
        "image" : image_feature(image),        
    }

您需要确保在解析数据集时使用相同的特征描述。
希望这能帮上忙。Cheers!!!

**编辑:**如果不使用TF记录,可以执行以下操作

num_images = 10000

path_to_your_images = "/path/to/your/images"

all_image_paths = [os.path.join(path_to_your_images, file) for file in os.listdir(path_to_your_images)]

np.random.shuffle(all_image_paths)

image_path_list = all_image_paths[:num_images]

def process_images(image_file_path):

    image = tf.io.read_file(image_file_path)
    image = tf.image.decode_jpeg(image, channels=3)
    image = tf.image.resize(image,(resize_height, resize_width))
    image = tf.cast(image, tf.float32) / 255.0
    return image

image_dataset = tf.data.Dataset.from_tensor_slices(image_path_list)

image_dataset = image_dataset.map(process_images)

labels = np.zeros(num_images)

label_dataset = tf.data.Dataset.from_tensor_slices(labels)

dataset = tf.data.Dataset.zip((image_dataset, label_dataset))

dataset = dataset.shuffle(num_images).batch(8)

现在您可以使用dataset来训练模型

相关问题