有没有办法为tensorflow进程设置java选项

oaxa6hgo  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(524)

现在我们使用tensorflow在hdfs中加载模型。模型大于1g,当达到堆大小时抛出oom。

2018-05-21 13:26:34.339483: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
18/05/21 13:26:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
INFO:tensorflow:Restoring parameters from hdfs://namenode:port/saved_model/0/variables/variables
INFO:tensorflow:Restoring parameters from hdfs://namenode:port/saved_model/0/variables/variables
2018-05-21 13:26:40.479367: W tensorflow/core/framework/allocator.cc:101] Allocation of 411041792 exceeds 10% of system memory.
hdfsPread: NewByteArray error:
java.lang.OutOfMemoryError: Java heap space

tensorflow正在使用 libhdfs 现在它可能会暴露出一些设置 JAVA_OPTS 使用参数或环境变量。

b1zrtrql

b1zrtrql1#

最后,这将适用于libhdfs: export LIBHDFS_OPTS="-Xms2048m -Xmx2048m" .

相关问题