org.apache.hadoop.hive.common.FileUtils.listStatusRecursively()方法的使用及代码示例

x33g5p2x  于2022-01-19 转载在 其他  
字(11.2k)|赞(0)|评价(0)|浏览(155)

本文整理了Java中org.apache.hadoop.hive.common.FileUtils.listStatusRecursively()方法的一些代码示例，展示了FileUtils.listStatusRecursively()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。FileUtils.listStatusRecursively()方法的具体详情如下：
包路径：org.apache.hadoop.hive.common.FileUtils
类名称：FileUtils
方法名：listStatusRecursively

FileUtils.listStatusRecursively介绍

[英]Recursively lists status for all files starting from a particular directory (or individual file as base case).
[中]

代码示例

代码示例来源：origin: apache/hive

/**
 * Recursively lists status for all files starting from a particular directory (or individual file
 * as base case).
 *
 * @param fs
 *          file system
 *
 * @param fileStatus
 *          starting point in file system
 *
 * @param results
 *          receives enumeration of all files found
 */
public static void listStatusRecursively(FileSystem fs, FileStatus fileStatus,
  List<FileStatus> results) throws IOException {
 listStatusRecursively(fs, fileStatus, HIDDEN_FILES_PATH_FILTER, results);
}

代码示例来源：origin: apache/hive

public static void listStatusRecursively(FileSystem fs, FileStatus fileStatus,
  PathFilter filter, List<FileStatus> results) throws IOException {
 if (fileStatus.isDir()) {
  for (FileStatus stat : fs.listStatus(fileStatus.getPath(), filter)) {
   listStatusRecursively(fs, stat, results);
  }
 } else {
  results.add(fileStatus);
 }
}

代码示例来源：origin: apache/hive

/**
 * Recursively lists status for all files starting from the directory dir
 * @param job
 * @param dir
 * @return
 * @throws IOException
 */
protected FileStatus[] listStatus(JobConf job, Path dir) throws IOException {
 ArrayList<FileStatus> result = new ArrayList<FileStatus>();
 List<IOException> errors = new ArrayList<IOException>();
 FileSystem fs = dir.getFileSystem(job);
 FileStatus[] matches = fs.globStatus(dir, FileUtils.HIDDEN_FILES_PATH_FILTER);
 if (matches == null) {
  errors.add(new IOException("Input path does not exist: " + dir));
 } else if (matches.length == 0) {
  errors.add(new IOException("Input Pattern " + dir + " matches 0 files"));
 } else {
  for (FileStatus globStat : matches) {
   FileUtils.listStatusRecursively(fs, globStat, result);
  }
 }
 if (!errors.isEmpty()) {
  throw new InvalidInputException(errors);
 }
 LOG.debug("Matches for " + dir + ": " + result);
 LOG.info("Total input paths to process : " + result.size() + " from dir " + dir);
 return result.toArray(new FileStatus[result.size()]);
}

代码示例来源：origin: apache/drill

/**
 * Recursively lists status for all files starting from the directory dir
 * @param job
 * @param dir
 * @return
 * @throws IOException
 */
protected FileStatus[] listStatus(JobConf job, Path dir) throws IOException {
 ArrayList<FileStatus> result = new ArrayList<FileStatus>();
 List<IOException> errors = new ArrayList<IOException>();
 FileSystem fs = dir.getFileSystem(job);
 FileStatus[] matches = fs.globStatus(dir, FileUtils.HIDDEN_FILES_PATH_FILTER);
 if (matches == null) {
  errors.add(new IOException("Input path does not exist: " + dir));
 } else if (matches.length == 0) {
  errors.add(new IOException("Input Pattern " + dir + " matches 0 files"));
 } else {
  for (FileStatus globStat : matches) {
   FileUtils.listStatusRecursively(fs, globStat, result);
  }
 }
 if (!errors.isEmpty()) {
  throw new InvalidInputException(errors);
 }
 LOG.debug("Matches for " + dir + ": " + result);
 LOG.info("Total input paths to process : " + result.size() + " from dir " + dir);
 return result.toArray(new FileStatus[result.size()]);
}

代码示例来源：origin: apache/hive

/**
 * Lists status for all files under a given path. Whether or not this is recursive depends on the
 * setting of job configuration parameter mapred.input.dir.recursive.
 *
 * @param fs
 *          file system
 *
 * @param p
 *          path in file system
 *
 * @return list of file status entries
 */
private FileStatus[] listStatusUnderPath(FileSystem fs, Path p) throws IOException {
 boolean recursive = job.getBoolean(FileInputFormat.INPUT_DIR_RECURSIVE, false);
 // If this is in acid format always read it recursively regardless of what the jobconf says.
 if (!recursive && !AcidUtils.isAcid(p, job)) {
  return fs.listStatus(p, FileUtils.HIDDEN_FILES_PATH_FILTER);
 }
 List<FileStatus> results = new ArrayList<FileStatus>();
 for (FileStatus stat : fs.listStatus(p, FileUtils.HIDDEN_FILES_PATH_FILTER)) {
  FileUtils.listStatusRecursively(fs, stat, results);
 }
 return results.toArray(new FileStatus[results.size()]);
}

代码示例来源：origin: apache/drill

/**
 * Lists status for all files under a given path. Whether or not this is recursive depends on the
 * setting of job configuration parameter mapred.input.dir.recursive.
 *
 * @param fs
 *          file system
 *
 * @param p
 *          path in file system
 *
 * @return list of file status entries
 */
private FileStatus[] listStatusUnderPath(FileSystem fs, Path p) throws IOException {
 boolean recursive = job.getBoolean(FileInputFormat.INPUT_DIR_RECURSIVE, false);
 // If this is in acid format always read it recursively regardless of what the jobconf says.
 if (!recursive && !AcidUtils.isAcid(p, job)) {
  return fs.listStatus(p, FileUtils.HIDDEN_FILES_PATH_FILTER);
 }
 List<FileStatus> results = new ArrayList<FileStatus>();
 for (FileStatus stat : fs.listStatus(p, FileUtils.HIDDEN_FILES_PATH_FILTER)) {
  FileUtils.listStatusRecursively(fs, stat, results);
 }
 return results.toArray(new FileStatus[results.size()]);
}

代码示例来源：origin: apache/hive

try {
 FileStatus fileStatus = fs.getFileStatus(path);
 FileUtils.listStatusRecursively(fs, fileStatus, filter, result);
} catch (IOException e) {

代码示例来源：origin: org.apache.hive/hive-common

/**
 * Recursively lists status for all files starting from a particular directory (or individual file
 * as base case).
 *
 * @param fs
 *          file system
 *
 * @param fileStatus
 *          starting point in file system
 *
 * @param results
 *          receives enumeration of all files found
 */
public static void listStatusRecursively(FileSystem fs, FileStatus fileStatus,
  List<FileStatus> results) throws IOException {
 listStatusRecursively(fs, fileStatus, HIDDEN_FILES_PATH_FILTER, results);
}

代码示例来源：origin: org.apache.hive/hive-common

public static void listStatusRecursively(FileSystem fs, FileStatus fileStatus,
  PathFilter filter, List<FileStatus> results) throws IOException {
 if (fileStatus.isDir()) {
  for (FileStatus stat : fs.listStatus(fileStatus.getPath(), filter)) {
   listStatusRecursively(fs, stat, results);
  }
 } else {
  results.add(fileStatus);
 }
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

/**
 * Recursively lists status for all files starting from a particular directory (or individual file
 * as base case).
 *
 * @param fs
 *          file system
 *
 * @param fileStatus
 *          starting point in file system
 *
 * @param results
 *          receives enumeration of all files found
 */
public static void listStatusRecursively(FileSystem fs, FileStatus fileStatus,
  List<FileStatus> results) throws IOException {
 if (fileStatus.isDir()) {
  for (FileStatus stat : fs.listStatus(fileStatus.getPath(), HIDDEN_FILES_PATH_FILTER)) {
   listStatusRecursively(fs, stat, results);
  }
 } else {
  results.add(fileStatus);
 }
}

代码示例来源：origin: org.apache.hadoop.hive/hive-exec

protected FileStatus[] listStatus(JobConf job, Path path) throws IOException {
 ArrayList<FileStatus> result = new ArrayList<FileStatus>();
 List<IOException> errors = new ArrayList<IOException>();
 FileSystem fs = path.getFileSystem(job);
 FileStatus[] matches = fs.globStatus(path);
 if (matches == null) {
  errors.add(new IOException("Input path does not exist: " + path));
 } else if (matches.length == 0) {
  errors.add(new IOException("Input Pattern " + path + " matches 0 files"));
 } else {
  for (FileStatus globStat : matches) {
   FileUtils.listStatusRecursively(fs, globStat, result);
  }
 }
 if (!errors.isEmpty()) {
  throw new InvalidInputException(errors);
 }
 LOG.info("Total input paths to process : " + result.size());
 return result.toArray(new FileStatus[result.size()]);
}

代码示例来源：origin: edu.berkeley.cs.shark/hive-common

/**
 * Recursively lists status for all files starting from a particular directory (or individual file
 * as base case).
 *
 * @param fs
 *          file system
 *
 * @param fileStatus
 *          starting point in file system
 *
 * @param results
 *          receives enumeration of all files found
 */
public static void listStatusRecursively(FileSystem fs, FileStatus fileStatus,
  List<FileStatus> results) throws IOException {
 if (fileStatus.isDir()) {
  for (FileStatus stat : fs.listStatus(fileStatus.getPath())) {
   listStatusRecursively(fs, stat, results);
  }
 } else {
  results.add(fileStatus);
 }
}

代码示例来源：origin: org.apache.hadoop.hive/hive-common

/**
 * Recursively lists status for all files starting from a particular directory (or individual file
 * as base case).
 *
 * @param fs
 *          file system
 *
 * @param fileStatus
 *          starting point in file system
 *
 * @param results
 *          receives enumeration of all files found
 */
public static void listStatusRecursively(FileSystem fs, FileStatus fileStatus,
  List<FileStatus> results) throws IOException {
 if (fileStatus.isDir()) {
  for (FileStatus stat : fs.listStatus(fileStatus.getPath())) {
   listStatusRecursively(fs, stat, results);
  }
 } else {
  results.add(fileStatus);
 }
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

try {
 FileStatus fileStatus = fs.getFileStatus(path);
 FileUtils.listStatusRecursively(fs, fileStatus, result);
} catch (IOException e) {

代码示例来源：origin: com.facebook.presto.hive/hive-apache

/**
 * Lists status for all files under a given path. Whether or not this is recursive depends on the
 * setting of job configuration parameter mapred.input.dir.recursive.
 *
 * @param fs
 *          file system
 *
 * @param p
 *          path in file system
 *
 * @return list of file status entries
 */
private FileStatus[] listStatusUnderPath(FileSystem fs, Path p) throws IOException {
 boolean recursive = HiveConf.getBoolVar(job, HiveConf.ConfVars.HADOOPMAPREDINPUTDIRRECURSIVE);
 // If this is in acid format always read it recursively regardless of what the jobconf says.
 if (!recursive && !AcidUtils.isAcid(p, job)) {
  return fs.listStatus(p, FileUtils.HIDDEN_FILES_PATH_FILTER);
 }
 List<FileStatus> results = new ArrayList<FileStatus>();
 for (FileStatus stat : fs.listStatus(p, FileUtils.HIDDEN_FILES_PATH_FILTER)) {
  FileUtils.listStatusRecursively(fs, stat, results);
 }
 return results.toArray(new FileStatus[results.size()]);
}

代码示例来源：origin: org.apache.hadoop.hive/hive-exec

/**
  * Lists status for all files under a given path. Whether or not this is recursive depends on the
  * setting of job configuration parameter mapred.input.dir.recursive.
  *
  * @param fs
  *          file system
  *
  * @param p
  *          path in file system
  *
  * @return list of file status entries
  */
 private FileStatus[] listStatusUnderPath(FileSystem fs, Path p) throws IOException {
  HiveConf hiveConf = new HiveConf(job, FetchOperator.class);
  boolean recursive = hiveConf.getBoolVar(HiveConf.ConfVars.HADOOPMAPREDINPUTDIRRECURSIVE);
  if (!recursive) {
   return fs.listStatus(p);
  }
  List<FileStatus> results = new ArrayList<FileStatus>();
  for (FileStatus stat : fs.listStatus(p)) {
   FileUtils.listStatusRecursively(fs, stat, results);
  }
  return results.toArray(new FileStatus[results.size()]);
 }
}

代码示例来源：origin: com.facebook.presto.hive/hive-apache

/**
 * Recursively lists status for all files starting from the directory dir
 * @param job
 * @param dir
 * @return
 * @throws IOException
 */
protected FileStatus[] listStatus(JobConf job, Path dir) throws IOException {
 ArrayList<FileStatus> result = new ArrayList<FileStatus>();
 List<IOException> errors = new ArrayList<IOException>();
 FileSystem fs = dir.getFileSystem(job);
 FileStatus[] matches = fs.globStatus(dir, FileUtils.HIDDEN_FILES_PATH_FILTER);
 if (matches == null) {
  errors.add(new IOException("Input path does not exist: " + dir));
 } else if (matches.length == 0) {
  errors.add(new IOException("Input Pattern " + dir + " matches 0 files"));
 } else {
  for (FileStatus globStat : matches) {
   FileUtils.listStatusRecursively(fs, globStat, result);
  }
 }
 if (!errors.isEmpty()) {
  throw new InvalidInputException(errors);
 }
 LOG.debug("Matches for " + dir + ": " + result);
 LOG.info("Total input paths to process : " + result.size() + " from dir " + dir);
 return result.toArray(new FileStatus[result.size()]);
}

代码示例来源：origin: org.apache.hive/hive-common

try {
 FileStatus fileStatus = fs.getFileStatus(path);
 FileUtils.listStatusRecursively(fs, fileStatus, filter, result);
} catch (IOException e) {

内容来源于网络，如有侵权，请联系作者删除！

FileUtils

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多

    FileUtils类方法

    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com