r—在尝试查询时,是否可以通过配置单元odbc连接发送配置单元conf变量?

4ioopgfo  于 2021-06-24  发布在  Hive
关注(0)|答案(1)|浏览(372)

我有一个配置单元脚本,上面有一些配置单元配置变量。当我在我们的emr集群上运行这个查询时,它工作得很好,返回了预期的数据。例如

set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.dynamic.partition=true;
set hive.exec.max.dynamic.partitions=10000;
set mapreduce.map.memory.mb=7168;
set mapreduce.reduce.memory.mb=7168;
set hive.exec.max.dynamic.partitions.pernode=10000;
set hive.exec.compress.output=true;
set mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
set hive.execution.engine=mr;
select 
  fruits,
  count(1) as n
from table
group by fruits;

我想在另一台与配置单元有odbc连接的服务器上运行此查询。
(我在r)

hive_conn <- DBI::dbConnect(odbc(), dsn = "Hive")
results <- DBI::dbGetQuery(hive_conn,  "select fruits, count(1) as n from table group by fruits")

这运行正常,并按预期返回Dataframe。
但是,如果我想设置一些配置单元配置,我不知道如何用odbc发送它们。
如何通过odbc告诉配置单元使用所选配置单元配置文件设置运行查询?

set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.dynamic.partition=true;
set hive.exec.max.dynamic.partitions=10000;
set mapreduce.map.memory.mb=7168;
set mapreduce.reduce.memory.mb=7168;
set hive.exec.max.dynamic.partitions.pernode=10000;
set hive.exec.compress.output=true;
set mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
set hive.execution.engine=mr;
gv8xihay

gv8xihay1#

我在驱动程序的文档中找到了解决方案:https://www.simba.com/products/hive/doc/odbc_installguide/linux/content/odbc/hi/configuring/serverside.htm
我需要在创建连接时添加这些“服务器端属性”。在前面加上字符串“ssp\”(服务器端属性),然后将它们作为名称-值对添加,例如:

hive_conn <- dbConnect(odbc(), 
                       dsn = "Hive", 
                       SSP_hive.execution.engine = "mr",
                       SSP_hive.exec.dynamic.partition.mode = "nonstrict",
                       SSP_hive.exec.dynamic.partition = "true",
                       SSP_hive.exec.max.dynamic.partitions = 10000,
                       SSP_mapreduce.map.memory.mb = 7168,
                       SSP_mapreduce.reduce.memory.mb = 7168,
                       SSP_hive.exec.max.dynamic.partitions.pernode = 10000,
                       SSP_hive.exec.compress.output = "true",
                       SSP_mapred.output.compression.codec = "org.apache.hadoop.io.compress.SnappyCodec"
                       )

相关问题