pytorch 关于SubsetRandomSampler

92dk7w1h 于 2023-03-23 发布在其他

关注(0)|答案(2)|浏览(132)

我正在使用SubsetRandomSampler将分类数据集拆分为测试和验证。我们可以为每个类拆分数据集吗？

import numpy as np
import torch
from torchvision import transforms
from torch.utils.data.sampler import SubsetRandomSampler

train_transforms = transforms.Compose([transforms.ToTensor(),
                                       transforms.Normalize([0.485, 0.456, 0.406],
                                                            [0.229, 0.224, 0.225])])
dataset = datasets.ImageFolder( '/data/images/train', transform=train_transforms )

validation_split = .2
shuffle_dataset = True
random_seed= 42
batch_size = 20

dataset_size = len(dataset) #4996
indices = list(range(dataset_size))
split = int(np.floor(validation_split * dataset_size))

if shuffle_dataset :
    np.random.seed(random_seed)
    np.random.shuffle(indices)
train_indices, val_indices = indices[split:], indices[:split]

train_sampler = SubsetRandomSampler(train_indices)
valid_sampler = SubsetRandomSampler(val_indices)

train_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=train_sampler)
validation_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=valid_sampler)

pytorch

来源：https://stackoverflow.com/questions/67250023/related-to-subsetrandomsampler

2条答案

按热度按时间

vqlkdk9b1#

你的意思是培训和验证而不是测试和验证吗？
如果是这样，SubsetRandomSampler使用从索引中随机选择的样本。因此，您可以在将它们放入train_indices和val_indices之前随机拆分每个类的索引。
喜欢

indexs = [[] for _ in range(len(dataset.classes))]  # you can't use `[[]] * len(dataset.classes)`. Although there might be better ways but I don't know
for idx, (_, class_idx) in enumerate(dataset):
    indexs[class_idx].append(idx)
train_indices, val_indices = [], []
for cl_idx in indexs:
    size = len(cl_idx)
    split = int(np.floor(validation_split * size))
    np.random.shuffle(cl_idx)
    train_indices.extend(cl_idx[split:])
    val_indices.extend(cl_idx[:split])
train_sampler = SubsetRandomSampler(train_indices)
valid_sampler = SubsetRandomSampler(val_indices)

赞(0）回复(0）举报 2023-03-23

7qhs6swi2#

“#你不能使用[[]] * len(dataset.classes) .虽然可能有更好的方法，但我不知道”
[[]*len（dataset.classes）]工作。

赞(0）回复(0）举报 2023-03-23