我正在尝试对存储在amazons3中的文件运行一些map reduce作业。我看见了http://wiki.apache.org/hadoop/amazons3 然后按照它进行集成。下面是我的代码,它为map reduce作业设置输入目录
FileInputFormat.setInputPaths(job, "s3n://myAccessKey:mySecretKey@myS3Bucket/dir1/dir2/*.txt");
当我运行mapreduce作业时,我得到了这个异常
Exception in thread "main" java.lang.IllegalArgumentException:
Wrong FS: s3n://myAccessKey:mySecretKey@myS3Bucket/dir1/dir2/*.txt,
expected: s3n://myAccessKey:mySecretKey@myS3Bucket
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:294)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:352)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:321)
at com.appdynamics.blitz.hadoop.migration.DataMigrationManager.convertAndLoadData(DataMigrationManager.java:340)
at com.appdynamics.blitz.hadoop.migration.DataMigrationManager.migrateData(DataMigrationManager.java:300)
at com.appdynamics.blitz.hadoop.migration.DataMigrationManager.migrate(DataMigrationManager.java:166)
at com.appdynamics.blitz.command.DataMigrationCommand.run(DataMigrationCommand.java:53)
at com.appdynamics.blitz.command.DataMigrationCommand.run(DataMigrationCommand.java:21)
at com.yammer.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:58)
at com.yammer.dropwizard.cli.Cli.run(Cli.java:53)
at com.yammer.dropwizard.Service.run(Service.java:61)
at com.appdynamics.blitz.service.BlitzService.main(BlitzService.java:84)
我找不到资源来帮助我。任何一个指针都将被深深地感激。
2条答案
按热度按时间yfwxisqw1#
你得继续玩下去
错误的fs:s3n://myaccesskey:mysecretkey@mys3bucket/dir1/dir2/*.txt文件
你给hadoop的路径是不正确的,除非它能访问正确的文件,否则它就不能工作。
nbysray52#
所以我发现了问题。它是由这个虫子引起的https://issues.apache.org/jira/browse/hadoop-3733
尽管我用“%2f”替换了“/”,但它仍然给出相同的问题。我重新生成了密钥,并在密钥中放置了一个不带“/”的密钥,它解决了这个问题。