使用spring数据hadoop写入hdfs时出现的问题

ltqd579y  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(826)

我试图用springdatahadoop向hdfs编写一个简单的文本。但我在写作时遇到了一个未知的问题。
线程“main”org.springframework.data.hadoop.store.storeexception异常:存储输出上下文尚未初始化;嵌套异常为java.io.ioexception:意外的http响应:code=404!=200,op=getfilestatus,message=未在org.springframework.data.hadoop.store.support.outputstoreobjectsupport.getoutputcontext(outputstoreobjectsupport)中找到。java:135)在org.springframework.data.hadoop.store.output.abstractdatastreamwriter.getoutput(abstractdatastreamwriter。java:131)在org.springframework.data.hadoop.store.output.textfilewriter.write(textfilewriter。java:132)在com.mstack.app.mainapp.somemethod(mainapp。java:37)在com.mstack.app.mainapp.main(mainapp。java:32)原因:java.io.ioexception:意外的http响应:code=404!=200,op=getfilestatus,message=未在org.apache.hadoop.hdfs.web.webhdfsfilesystem.validateresponse(webhdfsfilesystem)中找到。java:347)在org.apache.hadoop.hdfs.web.webhdfsfilesystem.access$200(webhdfsfilesystem。java:90)位于org.apache.hadoop.hdfs.web.webhdfsfilesystem$abstractrunner.runwithretry(webhdfsfilesystem)。java:613)在org.apache.hadoop.hdfs.web.webhdfsfilesystem$abstractrunner.access$100(webhdfsfilesystem。java:463)在org.apache.hadoop.hdfs.web.webhdfsfilesystem$abstractrunner$1.run(webhdfsfilesystem。java:492)位于javax.security.auth.subject.doas(subject)的java.security.accesscontroller.doprivileged(本机方法)。java:422)在org.apache.hadoop.security.usergroupinformation.doas(用户组信息。java:1657)在org.apache.hadoop.hdfs.web.webhdfsfilesystem$abstractrunner.run(webhdfsfilesystem。java:488)在org.apache.hadoop.hdfs.web.webhdfsfilesystem.gethdfsfilestatus(webhdfsfilesystem。java:848)在org.apache.hadoop.hdfs.web.webhdfsfilesystem.getfilestatus(webhdfsfilesystem)。java:858)位于org.apache.hadoop.fs.filesystem.exists(filesystem。java:1424)在org.springframework.data.hadoop.store.support.outputstoreobjectsupport.findinitfiles(outputstoreobjectsupport。java:111)在org.springframework.data.hadoop.store.support.outputstoreobjectsupport.initoutputcontext(outputstoreobjectsupport)。java:93)位于org.springframework.data.hadoop.store.support.outputstoreobjectsupport.getoutputcontext(outputstoreobjectsupport)。java:133) ... 4更多原因:java.io.ioexception:内容类型“text/plain”与org.apache.hadoop.hdfs.web.webhdfsfilesystem.jsonparse(webhdfsfilesystem)中的“application/json”(parsed=“text/plain”)不兼容。java:320)在org.apache.hadoop.hdfs.web.webhdfsfilesystem.validateresponse(webhdfsfilesystem。java:343) ... 还有18个
my application-context.xml:-

<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/hadoop"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:beans="http://www.springframework.org/schema/beans"
    xmlns:hdp="http://www.springframework.org/schema/hadoop" xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
    http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">

    <hdp:configuration id="hadoopConfigBean">
        fs.defaultFS=${hdp.fs}
    </hdp:configuration>

    <context:annotation-config />
    <beans:bean id="textFileWriter"
        class="org.springframework.data.hadoop.store.output.TextFileWriter">
        <beans:constructor-arg index="0" ref="hadoopConfigBean"></beans:constructor-arg>
        <beans:constructor-arg index="1"
            type="org.apache.hadoop.fs.Path" value="/user/mhduser"></beans:constructor-arg>
        <beans:constructor-arg index="2" type="org.springframework.data.hadoop.store.codec.CodecInfo" >
        <beans:null></beans:null>
        </beans:constructor-arg>
    </beans:bean>

    <context:property-placeholder location="hadoop-configs.properties" />
</beans:beans>

主要类别:-

public class MainApp {

    @Autowired
    TextFileWriter textFileWriter;

    public static void main(String[] args) {
        ApplicationContext context = new ClassPathXmlApplicationContext("/META-INF/spring/application-context.xml",
                MainApp.class);
        System.out.println("Context loaded...");
        MainApp obj = new MainApp();
        context.getAutowireCapableBeanFactory().autowireBean(obj);
        obj.someMethod();
    }

    private void someMethod() {
        try {
            textFileWriter.write("Something");
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

任何帮助都将不胜感激!谢谢

qcuzuvrc

qcuzuvrc1#

我们这里有一些样品https://github.com/spring-projects/spring-hadoop-samples. 我认为没有特定的存储,所以我创建了一个简单的基于springboot的gist,可以通过它的cli运行。https://gist.github.com/jvalkeal/8145f0618f25c1d25d19f4e1e89de1e6
我们在单元测试中所做的工作也值得一看https://github.com/spring-projects/spring-hadoop/tree/master/spring-hadoop-store/src/test/java/org/springframework/data/hadoop/store

相关问题