由于springdatahadoop还没有发布,很难找到一个运行示例配置来与cloudera一起使用它。
在hadoop和cdh4(hadoop2.0.0-cdh4.1.3)一起运行spring数据时,我需要选择哪些依赖项?
通过选择不同的职位,我得到了以下例外:
空指针
Exception in thread "SimpleAsyncTaskExecutor-1" java.lang.ExceptionInInitializerError
at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:183)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
at org.springframework.util.ReflectionUtils.makeAccessible(ReflectionUtils.java:405)
at org.springframework.data.hadoop.mapreduce.JobUtils.<clinit>(JobUtils.java:123)
... 2 more
版本不匹配7到4
Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
at org.apache.hadoop.ipc.Client.call(Client.java:1070)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:238)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:372)
at org.springframework.data.hadoop.mapreduce.JobFactoryBean.afterPropertiesSet(JobFactoryBean.java:208)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1545)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1483)
... 12 more
2条答案
按热度按时间uxh89sit1#
这是一个如何配置它的示例。
maven设置:
笔记:
(可选)从spring数据hadoop中排除hadoop流和hadoop工具
添加hadoopcommon和hadoophdfs,通用版本为:2.0.0-cdhx.x.x
在mr1版本中添加hadoop工具和hadoop流:2.0.0-mr1-cdhx.x.x
spring数据hadoop目前只支持mr1。所以确保你没有把mr2包含在其他依赖项中。检查一下这个
mvn dependency:tree
!pom.xml文件:
spring设置(applicationcontext.xml):
更换
fs.default.name
使用namenode域这样您就可以访问集群了。
一些参考资料:
springsource论坛-空指针异常
cloudera maven存储库
8yparm6h2#
嘿,你可以从https://github.com/spring-projects/spring-data-book.
构建和运行它在readme文档中给出。