我希望从中组织/移动数据hdfs://data/backup 到正确的hdfs文件夹。

e.g: hdfs://data/backup/a/1 to hdfs://data/a/1
hdfs://data/backup/b/1 to hdfs://data/b/2
hdfs://data/backup/c/1 to hdfs://data/c/3
hdfs://data/backup/d/1 to hdfs://data/d/4

这些文件夹下的数据以日期和时间为准
如果目标文件夹中已存在该小时的数据，请使用ecbackup中的数据覆盖它。
例如

hdfs://data/backup/a/1/merged/20190608/1 -> hdfs://data/a/1/merged/20190608/01
......
hdfs://data/backup/a/1/merged/20190608/17 -> hdfs://data/a/1/merged/20190608/17
....
hdfs://data/backup/a/1/merged/20190608/23 -> hdfs://data/a/1/merged/20190608/23

这是我的剧本

# !/usr/bin/env bash

SOURCE='hdfs://data/backup/audio_archive'
DEST='hdfs://data/a'
source_files=( $(hdfs dfs -ls -r "${SOURCE}" | awk '/^-/ {print $NF}') )
SOURCE="${SOURCE%/}"
SOURCE_DIR="${SOURCE//hdfs:\//}"
DEST="${DEST%/}"
for file_ in "${source_files[@]}"; do
    if [[ ! $file_ =~ /[[:digit:]]+/[[:digit:]]+ ]]; then
    continue
    fi
    dest_="${file_#${SOURCE_DIR}}"
    dest="${DEST}${dest_}"
    dest_dir="${dest%/*}"
    # NOTE: Remove `echo` after checking the output to
    # perform the real actions
    echo hdfs dfs -mkdir -p "${dest_dir}"
    echo hdfs dfs -mv -f "hdfs:/${file_}" "${dest}"
    echo
done

我在犯错误

20/01/31 18:16:10 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
20/01/31 18:16:10 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
20/01/31 18:16:10 INFO retry.RetryInvocationHandler: java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: ; destination host is:  , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over ... after 1 failover attempts. Trying to failover after sleeping for 1423ms.