在rapidminer上运行sparkrm:提交或启动spark作业时出错

b1uwtaje  于 2021-05-31  发布在  Hadoop
关注(0)|答案(1)|浏览(475)

我正在和rapidminer一起从一个大数据集中提取规则。radoop是hadoop生态系统的扩展,sparkrm操作符允许进行fp增长,从从从hive检索数据到探索分析。我正在工作:-windows 8.1-hadoop 6.2-spark 1.5-hive 2.1我已将spark默认配置如下:


# spark.master                     yarn

# spark.eventLog.enabled           true

# spark.eventLog.dir               hdfs://namenode:8021/directory

# spark.serializer                 org.apache.spark.serializer.KryoSerializer

# spark.driver.memory              2G

# spark.driver.cores                    1

# spark.yarn.driver.memoryOverhead  384MB

# spark.yarn.am.memory             1G

# spark.yarn.am.cores               1

# spark.yarn.am.memoryOverhead      384MB

# spark.executor.memory            1G

# spark.executor.instances          1

# spark.executor.cores              1

# spark.yarn.executor.memoryOverhead    384MB

# spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

我拥有的站点xml文件:

<property>
    <name>yarn.resourcemanager.schedular.address</name>
    <value>localhost:8030</value>
</property>

<property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>localhost:8033</value>
</property>

<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>localhost:8031</value>
</property>

<property>
    <name>yarn.resourcemanager.resource.cpu-vcores</name>
    <value>2</value>
</property>

<property>
    <name>yarn.resourcemanager.resource.memory-mb</name>
    <value>2048</value>
</property>

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>localhost</value>
</property>

<property>
    <name>yarn.resourcemanager.address</name>
    <value>localhost:8032</value>
</property>

<property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>localhost:8088</value>
</property>

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

<property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/E:/tweets/hadoopConf/userlog</value>
    <final>true</final>
</property>

<property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/E:/tweets/hadoopConf/temp/nm-localdir</value>
</property>

<property>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>600</value>
</property>

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>2048</value>
</property>

<property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>512</value>
</property>

<property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>2048</value>
</property>

<property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>1</value>
</property>     

<property>
    <name>yarn.scheduler.minimum-allocation-vcores</name>
    <value>1</value>
</property>

<property>
    <name>yarn.scheduler.maximum-allocation-vcores</name>
    <value>3</value>
</property>

<property>
<name>yarn.application.classpath</name>
<value>
/tweets/hadoop/,
/tweets/hadoop/share/hadoop/common/*,
/tweets/hadoop/share/hadoop/common/lib/*,
/tweets/hadoop/share/hadoop/hdfs/*,
/tweets/hadoop/share/hadoop/hdfs/lib/*,
/tweets/hadoop/share/hadoop/mapreduce/*,
/tweets/hadoop/share/hadoop/mapreduce/lib/*,
/tweets/hadoop/share/hadoop/yarn/*,
/tweets/hadoop/share/hadoop/yarn/lib/*
/C:/spark/lib/spark-assembly-1.5.0-hadoop2.6.0.jar
</value>
</property>
</configuration>

hadoop的快速连接测试成功完成。当我运行rapidminer进程时,它由一个错误完成:

Process failed before getting into running state. this indicates that an error occurred during submitting or starting the spark job or writing the process output or the exception to the disc. Please check the logs of the spark job on the YARN Resource Manager interface for more information about the error.

在localhost:8088 i 在此处输入诊断信息的图像描述
这是作业的调度程序,请在此处输入图像描述
我是hadoop和spark的新手,无法有效地配置内存。

wribegjk

wribegjk1#

此错误消息说明提交的作业在超时之前无法分配所需的群集资源(vcore、内存),因此无法运行(请求的数量可能超过可用的总量,否则可能会一直等待)。我假设基于部署集群的yarn-site.xml的内容 localhost . 在这种情况下,您可以在 http://localhost:8088/cluster/scheduler 页面(资源管理器界面)。在radoop进程执行期间,您可以查看相应的yarn/spark应用程序日志,以获取有关请求的资源量和类型的更多信息。有了这些信息,您可以对集群进行微调,可能是按照允许应用程序使用更多资源的思路。
我还建议在radoop文档中查看一下哪些资源分配既适合您的用例,也适合您的系统。radoop能够使用不同的资源分配策略执行spark作业。这些策略描述了radoop从yarn请求spark作业执行资源的方式。通过调整此设置,您可能能够适应集群端的可用资源。您可以在这里阅读有关这些政策的更多信息。

相关问题