Paddle GPU内存不足

wkyowqbh  于 2022-10-20  发布在  其他
关注(0)|答案(3)|浏览(336)

/home/wu/venv/lib/python3.6/site-packages/paddle/fluid/executor.py:789: UserWarning: The following exception is not an EOF exception.
"The following exception is not an EOF exception.")
Traceback (most recent call last):
File "slim/distillation/distill.py", line 383, in
main()
File "slim/distillation/distill.py", line 224, in main
exe.run(teacher_startup_program)
File "/home/wu/venv/lib/python3.6/site-packages/paddle/fluid/executor.py", line 790, in run
six.reraise(*sys.exc_info())
File "/home/wu/venv/lib/python3.6/site-packages/six.py", line 696, in reraise
raise value
File "/home/wu/venv/lib/python3.6/site-packages/paddle/fluid/executor.py", line 785, in run
use_program_cache=use_program_cache)
File "/home/wu/venv/lib/python3.6/site-packages/paddle/fluid/executor.py", line 838, in _run_impl
use_program_cache=use_program_cache)
File "/home/wu/venv/lib/python3.6/site-packages/paddle/fluid/executor.py", line 912, in _run_program
fetch_var_name)
paddle.fluid.core_avx.EnforceNotMet:

C++ Call Stacks (More useful to developers):

0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int)
1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int)
2 paddle::platform::CUDADeviceContext::CUDADeviceContext(paddle::platform::CUDAPlace)
3 std::_Function_handler<std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > (), std::reference_wrapper<std::_Bind_simple<void paddle::platform::EmplaceDeviceContext<paddle::platform::CUDADeviceContext, paddle::platform::CUDAPlace>(std::map<paddle::platform::Place, std::shared_future<std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > >, std::lesspaddle::platform::Place, std::allocator<std::pair<paddle::platform::Place const, std::shared_future<std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > > > > >, paddle::platform::Place)::{lambda() #1 } ()> > >::_M_invoke(std::_Any_data const&)
4 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > >, std::__future_base::_Result_base::_Deleter>, std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > > >::_M_invoke(std::_Any_data const&)
5 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&)
6 std::__future_base::_Deferred_state<std::_Bind_simple<void paddle::platform::EmplaceDeviceContext<paddle::platform::CUDADeviceContext, paddle::platform::CUDAPlace>(std::map<paddle::platform::Place, std::shared_future<std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > >, std::lesspaddle::platform::Place, std::allocator<std::pair<paddle::platform::Place const, std::shared_future<std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > > > > >
, paddle::platform::Place)::{lambda() #1 } ()>, std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > >::_M_run_deferred()
7 paddle::platform::DeviceContextPool::Get(paddle::platform::Place const&)
8 paddle::framework::GarbageCollector::GarbageCollector(paddle::platform::Place const&, unsigned long)
9 paddle::framework::UnsafeFastGPUGarbageCollector::UnsafeFastGPUGarbageCollector(paddle::platform::CUDAPlace const&, unsigned long)
10 paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool)
11 paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocatorstd::string > const&, bool, bool)
我已经把batch_size改为1了

3j86kqsm

3j86kqsm1#

你好,还请详细描述一下你的使用场景,训练任务,以及调用代码,方便我们进行跟进

pgccezyw

pgccezyw2#

@nosoldier PaddleSlim的问题可以去 https://github.com/PaddlePaddle/PaddleSlim/issues 下面提issue。 另外,错误需要更详细一些,你上面贴的错误,还没看到GPU显存不足。 另外,注意nvidia-smi查看下,显存是否被占用。

jgwigjjp

jgwigjjp3#

好的…

---原始邮件--- 发件人: "qingqing01"<notifications@github.com> 发送时间: 2020年4月29日 17:40:49 收件人: "PaddlePaddle/Paddle"<Paddle@noreply.github.com>; 抄送: "Mention"<mention@noreply.github.com>;"nosoldier"<1430578977@qq.com>; 主题: Re: [PaddlePaddle/Paddle] GPU内存不足 (#24228) @nosoldier PaddleSlim的问题可以去 https://github.com/PaddlePaddle/PaddleSlim/issues 下面提issue。 另外,错误需要更详细一些,你上面贴的错误,还没看到GPU不足。 另外,注意nvidia-smi查看下,显存是否被占用。 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

相关问题