AttributeError: Can't pickle local object 'pre_datasets.< locals>.< lambda>' when implementing Pytorch framework

nimxete2  于 2022-12-29  发布在  其他
关注(0)|答案(2)|浏览(542)

我试着在CNN上实现一个pytorch框架。
我确信代码是正确的,因为它来自一个教程,当我在GoogleDrive上的Jupyter Notebook上运行它时,它工作正常。
但是当我尝试将其本地化为.py文件时,它提示了一个错误:
AttributeError: Can't pickle local object 'pre_datasets.<locals>.<lambda>'我知道这是关于在函数外部推理对象的,但是这个错误的确切原因是什么呢?
我该怎么补救呢?
下面是代码的主要部分。

def pre_datasets():
    TRAIN_TFM = transforms.Compose(
        [
            transforms.Resize(size=(128, 128)),
            # TODO
            transforms.ToTensor(),
        ]
    )
    train_set = DatasetFolder(
        root=CONFIG["train_set_path"],
        loader=lambda x: Image.open(x),
        extensions="jpg",
        transform=TRAIN_TFM,
    )
    train_loader = DataLoader(
        dataset=train_set,
        batch_size=CONFIG["batch_size"],
        shuffle=True,
        num_workers=CONFIG["num_workers"],
        pin_memory=True,
    )
    return train_loader

def train(train_loader):
    ...
    for epoch in range(CONFIG["num_epochs"]):
    ...
        for batch in train_loader: # error happened here
    ...

if __name__ == "__main__":
    train_loader = pre_datasets()
    train(train_loader)

下面是错误消息:

Traceback (most recent call last):
  File "HW03_byCRZ.py", line 197, in <module>
    train(train_loader, valid_loader)
  File "HW03_byCRZ.py", line 157, in train
    for batch in train_loader:
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 355, in __iter__
    return self._get_iterator()
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 301, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 914, in __init__
    w.start()
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'pre_datasets.<locals>.<lambda>'
kb5ga3dv

kb5ga3dv1#

我也遇到过类似的问题,我使用dill时是这样的:

import dill as pickle

而且开箱即用!

z31licg0

z31licg02#

Pickle'ing一个lambda函数需要额外的修改(序列化函数)更多关于如何做的信息请参见Can Python pickle lambda functions?

相关问题