如何将直线命令自动化到各种配置单元查询？

o4hqfura 于 2021-06-27 发布在 Hive

关注(0)|答案(1)|浏览(306)

我必须以直线方式自动化一些活动，就像我们在hivecli中所做的一样。下面是配置单元自动化任务的运行示例shell脚本：

echo "using new_db HIVE database..!!"
hive -e "use new_db;"

echo "truncating the staging table test..."
hive -e "TRUNCATE TABLE new_db.test;"

echo "Loading the data into the staging table test"
hive -e "LOAD DATA LOCAL INPATH '/<path>/IDI.txt' INTO TABLE new_db.test;"

echo "Appending the data into history table hist_test.."
hive -e "insert into table new_db.test select *, '$unix_time' from new_db.test;"

我想做一个类似的事情直线。我是直线的初学者。所以我想出了如下的办法。


# ! /bin/bash

timestamp=$(date +%Y-%m-%d-%H:%M:%S:%N)
unix_time=$(date +%Y-%m-%d-%H:%M:%S)

echo "Login to BeeLine..!!"

# beeline << EOF

beeline -u jdbc:hive2://server:port,server:port,server:port/;serviceDiscoveryMode=zookeeper;zookeeperNamespace=hiveserver2;principal=hive/server@hello.COM<< EOF -n <username> -p <password>

echo "using new_db HIVE database..!!"
beeline -e "use new_db;"

echo "truncating the staging table test..."
beeline -e "TRUNCATE TABLE new_db.test;"

echo "Loading the data into the staging table test"
beeline -e "LOAD DATA LOCAL INPATH '/<path>/IDI.txt' INTO TABLE new_db.test;"

echo "Appending the data into history table hist_test.."
beeline -e "insert into table new_db.test select *, '$unix_time' from new_db.test;"

EOF

++++++++++++++++++++++++++

新语法


# ! /bin/bash

export BEELINE_PREFIX="jdbc:hive2://server:port,server:port,server:port/;serviceDiscoveryMode=zookeeper;zookeeperNamespace=hiveserver2;principal=hive/server@hello.COM"

export FILE_PATH="path_toFile/staging_hive_tables.hql"
beeline -u $BEELINE_PREFIX -f $FILE_PATH

Hive shell beeline

来源：https://stackoverflow.com/questions/52697431/how-to-automate-beeline-commands-to-various-hive-queries

1条答案

按热度按时间

vptzau2j1#

像这样试试。

export BEELINE_PREFIX='beeline -u "jdbc:hive2://bigdataserver-xxxxxx.net:10000/;principal=hive/bigdataplatform-xxxxx.net@XYZ.NET;ssl=true" --silent=true --verbose=false --showHeader=false --outputformat=csv2 -e  ' # set hive.cli.print.header=false;
export PARTY="xxxxx"
export load_date="20181003"

### Get the table count  ###

######################################################################## 

export BEELINE_COMMAND="select count(*) from ${DMAR_HIVE_TABLE} where load='${PARTY}' and load_date=${yyymmdd} ; ";

echo "Hive command=$BEELINE_COMMAND"
${BEELINE_PREFIX}"${BEELINE_COMMAND}" | read table_count
check_count=$?
echo "Check count = $check_count"
echo "Table count = $table_count"

编辑1：
双引号不见了。。用单引号将直线前缀括起来试试这个


# ! /bin/bash

export BEELINE_PREFIX='"jdbc:hive2://server:port,server:port,server:port/;serviceDiscoveryMode=zookeeper;zookeeperNamespace=hiveserver2;principal=hive/server@hello.COM"'

编辑2：
我在命令提示符下运行了这些

> export BEELINE_PREFIX='"jdbc:hive2://xxxx:10000/;principal=hive/yyyyy@zzzzz;ssl=true" --silent=true --verbose=false --showHeader=false --outputformat=csv2 '
> cat /tmp/hive1.ql
select current_date;
>  echo "beeline -u ${BEELINE_PREFIX} -f /tmp/hive1.ql " | sh -x
+ beeline -u 'jdbc:hive2://xxxx:10000/;principal=hive/yyyyy@zzzzz;ssl=true' --silent=true --verbose=false --showHeader=false --outputformat=csv2 -f /tmp/hive1.ql
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release

2018-10-12
>

注意，您需要回显命令并将其传递给shell（sh）
echo“beeline-u${beeline_prefix}-f/tmp/hive1.ql”| sh-x

赞(0）回复(0）举报 2021-06-27

我来回答

如何将直线命令自动化到各种配置单元查询？

新语法

1条答案

相关问题

热门标签

最新问答