我正在尝试执行/运行一个web应用程序(使用spring工具套件),通过它可以向hadoop提交作业。我使用的是apache-tomcat-6.0.36服务器和mapr客户端。但是,当我运行应用程序时,会出现以下错误:
java.io.IOException: No FileSystem for scheme: maprfs
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1534)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:69)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1571)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1553)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:232)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:115)
at com.aexp.lookalike.PreInitialize.createVarFiles(PreInitialize.java:50)
at com.aexp.lookalike.PreInitialize.run(PreInitialize.java:98)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.springframework.data.hadoop.mapreduce.ToolExecutor.invokeTargetObject(ToolExecutor.java:31)
at org.springframework.data.hadoop.mapreduce.ToolExecutor.invokeTargetObject(ToolExecutor.java:27)
at org.springframework.data.hadoop.mapreduce.HadoopCodeExecutor.invokeTarget(HadoopCodeExecutor.java:185)
at org.springframework.data.hadoop.mapreduce.HadoopCodeExecutor.runCode(HadoopCodeExecutor.java:102)
at org.springframework.data.hadoop.mapreduce.ToolTasklet.execute(ToolTasklet.java:33)
at org.springframework.data.hadoop.mapreduce.ToolTasklet$$FastClassByCGLIB$$317c6986.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at org.springframework.data.hadoop.mapreduce.ToolTasklet$$EnhancerByCGLIB$$4b232693.execute(<generated>)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135)
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:281)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
配置时是否遗漏了某些内容?这个错误的原因是什么?我是否错过了向应用程序的类路径添加任何jar?
2条答案
按热度按时间qgzx9mmu1#
我得到了类似的例外,但“hdfs”。我发现问题出在我使用maven创建的应用程序jar中,它没有hadoop hdfs jar附带的hdfs的“服务”规范。我使用maven shade plugin(maven shade plugin)解决了这个问题,它使用了内置的转换器“org.apache.maven.plugins.shade.resource.servicesresourcetransformer”。
我认为没有shade插件的maven不会为hdfs文件系统所需的“org.apache.hadoop.fs.filesystem”提供hdfs服务规范。我猜“maprfs”是另一个需要在metainf/services文件夹中指定的文件系统。尝试手动提供。您可以从hadoop hdfs jar获得它。
希望这有帮助。
btxsgosb2#
问题是因为类路径没有mapr jar。确保类路径和库路径设置正确
类路径应该包含$mapr\u home/lib/maprfs-1.0.3-mapr-3.0.0.jar“lib\u路径应该包含/opt/mapr/lib