现在我们使用tensorflow在hdfs中加载模型。模型大于1g,当达到堆大小时抛出oom。
2018-05-21 13:26:34.339483: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
18/05/21 13:26:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
INFO:tensorflow:Restoring parameters from hdfs://namenode:port/saved_model/0/variables/variables
INFO:tensorflow:Restoring parameters from hdfs://namenode:port/saved_model/0/variables/variables
2018-05-21 13:26:40.479367: W tensorflow/core/framework/allocator.cc:101] Allocation of 411041792 exceeds 10% of system memory.
hdfsPread: NewByteArray error:
java.lang.OutOfMemoryError: Java heap space
tensorflow正在使用 libhdfs
现在它可能会暴露出一些设置 JAVA_OPTS
使用参数或环境变量。
1条答案
按热度按时间b1zrtrql1#
最后,这将适用于libhdfs:
export LIBHDFS_OPTS="-Xms2048m -Xmx2048m"
.