String fileLocation;//set this to file absolute location
Configuration conf; //job Configuration
DistributedCache.addLocalFiles(conf,fileLocation);
conf.set("fileLocation",fileLocation);
在Map器设置方法中检索:
Configuration mapConf = context.getConfiguration();
URI[] cacheURIArray = DistributedCache.getCacheFiles();
String file2Location = mapConf.get("file2Location");
List<String> fileWords = new ArrayList<String>(); //set this as a clas variable so it can be accessed outside of the setup method of the mapper
for(URI uri: cacheURIArray){
if( uri.toString().matches(".*"+fileLocation)){
BufferedReader br = new BufferedReader(new InputStream(cacheFileSystem.open(new Path(uri.toString()))));
String line = "";
line = br.readLine();
while(line != null){
fileWords.add(line);
line = br.readLine();
}
}
}
1条答案
按热度按时间wsxa1bj11#
是的,事实上这是可能的。如果在作业开始前设置分布式缓存并将文件加载到其中,则会自动将其发送到Map程序。
分布式缓存设置示例:
在Map器设置方法中检索:
您的检索方法可能至少与我提供的示例有所不同,但是它用于说明如何使用分布式缓存。有关更多信息,请参阅分布式缓存