If you want to move files from directory it is little bit tricky below code done same task for me !!
val conf = new org.apache.hadoop.conf.Configuration()
val src:Path = new org.apache.hadoop.fs.Path(hdfsDirectory)
val fs = FileSystem.get(src.toUri,conf)
val srcPath: Path = new Path("hdfs://sourcePath/")
val srcFs =FileSystem.get(srcPath.toUri,conf)
val dstPath:Path =new Path("hdfs://targetPath/")
val dstFs =FileSystem.get(dstPath.toUri,conf)
val exists = fs.exists(new org.apache.hadoop.fs.Path(hdfsDirectory))
val status:Array[FileStatus] = fs.listStatus(new Path(hdfsDirectory))
if (status.length>0) {
status.foreach(x => {
println("My files: " + x.getPath)
FileUtil.copy(srcFs, x.getPath, dstFs, dstPath, true, conf)
println("Files moved !!" +x.getPath)
}
)}
else{
println("No Files Found !!")
}
2条答案
按热度按时间t8e9dugd1#
fileutil提供了一种复制文件的方法。
如果您需要将它复制到另一个集群,只需创建一个新的
Configuration
设置和新建FileSystem
.t98cgbkg2#