Paddle ExternalError: CUBLAS error(1). gpu下運行一段時間,程序崩潰退出

x7yiwoj4  于 2022-10-22  发布在  其他
关注(0)|答案(6)|浏览(702)

bug描述 Describe the Bug

cuda:10.2
cudnn:8.3
tensortRT:7.2.3.4
paddle預測庫:manylinux_cuda10.2_cudnn8.1_avx_mkl_trt7_gcc5.4
系統:unbuntu16

問題描述:運行一段時間,檢測幾張圖片後,程序就退出崩潰。
崩潰信息:

C++ Traceback (most recent call last):

0 PaddleOCR::DBDetector::Run(cv::Mat&, std::vector<std::vector<std::vector<int, std::allocator >, std::allocator<std::vector<int, std::allocator > > >, std::allocator<std::vector<std::vector<int, std::allocator >, std::allocator<std::vector<int, std::allocator > > > > >&, std::vector<double, std::allocator >*)
1 void paddle_infer::Tensor::CopyFromCpu(float const*)
2 paddle::platform::DeviceContextPool::Get(paddle::platform::Place const&)
3 std::__future_base::_Deferred_state<std::_Bind_simple<void paddle::platform::EmplaceDeviceContext<paddle::platform::CUDADeviceContext, paddle::platform::CUDAPlace>(std::map<paddle::platform::Place, std::shared_future<std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > >, std::lesspaddle::platform::Place, std::allocator<std::pair<paddle::platform::Place const, std::shared_future<std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > > > > >, paddle::platform::Place)::{lambda() #1 } ()>, std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > >::_M_complete_async()
4 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > >, std::__future_base::_Result_base::_Deleter>, std::_Bind_simple<void paddle::platform::EmplaceDeviceContext<paddle::platform::CUDADeviceContext, paddle::platform::CUDAPlace>(std::map<paddle::platform::Place, std::shared_future<std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > >, std::lesspaddle::platform::Place, std::allocator<std::pair<paddle::platform::Place const, std::shared_future<std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > > > > >
, paddle::platform::Place)::{lambda() #1 } ()>, std::unique_ptr<paddle::platform::DeviceContext, std::default_deletepaddle::platform::DeviceContext > > >::_M_invoke(std::_Any_data const&)
5 paddle::platform::CUDADeviceContext::CUDADeviceContext(paddle::platform::CUDAPlace)
6 paddle::platform::CUDAContext::CUDAContext(paddle::platform::CUDAPlace const&, paddle::platform::stream::Priority const&, paddle::platform::stream::StreamFlag const&)
7 paddle::platform::CublasHandleHolder::CublasHandleHolder(CUstream_st*, cublasMath_t)
8 paddle::platform::EnforceNotMet::EnforceNotMet(paddle::platform::ErrorSummary const&, char const*, int)
9 paddle::platform::GetCurrentTraceBackString abi:cxx11

Error Message Summary:

ExternalError: CUBLAS error(1).
[Hint: Please search for the error code(1) on website (https://docs.nvidia.com/cuda/cublas/index.html#cublasstatus_t) to get Nvidia's official solution and advice about CUBLAS Error.] (at /paddle/paddle/fluid/platform/cuda_helper.h:98)

其他补充信息 Additional Supplementary Information

  • No response*
brjng4g3

brjng4g31#

您好,我们已经收到了您的问题,会安排技术人员尽快解答您的问题,请耐心等待。请您再次检查是否提供了清晰的问题描述、复现代码、环境&版本、报错信息等。同时,您也可以通过查看 官网API文档常见问题历史IssueAI社区 来寻求解答。祝您生活愉快~

Hi! We've received your issue and please be patient to get responded. We will arrange technicians to answer your questions as soon as possible. Please make sure that you have posted enough message to demo your request. You may also check out the APIFAQGithub Issue and AI community to get the answer.Have a nice day!

ut6juiuv

ut6juiuv2#

您好,请补充一下Paddle的版本和模型信息

uajslkp6

uajslkp63#

很可能是 cudnn:8.3 的版本不匹配,https://www.paddlepaddle.org.cn/documentation/docs/zh/install/compile/linux-compile.html#cpu-gpu

cdmah0mi

cdmah0mi4#

您好,请补充一下Paddle的版本和模型信息

paddle预测库版本version信息:
GIT COMMIT ID: a5cf2e3
WITH_MKL: ON
WITH_MKLDNN: ON
WITH_GPU: ON
WITH_ROCM: OFF
CUDA version: 10.2
CUDNN version: v8.1
CXX compiler version: 5.4.0
WITH_TENSORRT: ON
TensorRT version: v7.2.3.4

模型版本:2.4

dddzy1tm

dddzy1tm5#

很可能是 cudnn:8.3 的版本不匹配,https://www.paddlepaddle.org.cn/documentation/docs/zh/install/compile/linux-compile.html#cpu-gpu

尝试cudnn版本从8.3替换为8.1后,问题仍然存在

3duebb1j

3duebb1j6#

CUDA 工具包10.1/10.2配合cuDNN 7

相关问题