我目前只尝试执行一个从hive到elasticsearch的“select*from table”。我正在使用cloudera cdh6.0.1。我已经将elasticsearch-hadoop-hive-7.1.1jar添加到我的hive路径中。我有elasticsearch 7.1.1的cloudera堆栈和elastic运行在不同的服务器,但在同一个网络。
CREATE EXTERNAL TABLE ctrl_rater_resumen_lla_es
(
fecha_registro string,
direccion string,
linea_b_codigo_prestadora string,
linea_b_tipo_numero string,
es_roaming string,
call_duration string,
linea_b_routing_number string,
minutos string, fecha string
)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES (
'es.resource' = 'ctrl_rater_resumen_lla/hb',
'es.node' = 'http://10.129.x.xxx',
'es.port' = '9200',
'es.index.auto.create' = 'true',
'es.index.read.missing.as.empty' = 'true',
'es.nodes.discovery'='true',
'es.net.ssl'='false'
'es.nodes.client.only'='false',
'es.nodes.wan.only' = 'true'
'es.net.http.auth.user'='xxxxx',
'es.net.http.auth.pass' = 'xxxxx'
);
创建成功
SELECT * FROM ctrl_rater_resumen_lla_es;
请求tfetchresultsreq(fetchtype=0,operationhandle=toperationhandle(hasresultset=true,modifiedrowcount=none,operationtype=0,operationid=thandleidentifier(secret='\xbayg*\xd4wi\xc0\xb8\xf6\x94q\xa3\xa4iy',guid='\xff\xca\xdb\xb5\x040e\x0e\x8ee\xe4\xf7?t\x1b\x01'))的状态不正确,方向=4,maxrows=100):tfetchresultsresp(status=tstatus(errorcode=0,errormessage=“java.io.ioexception:org.elasticsearch.hadoop.eshadoopillegalargumentexception:cannot detect es version-通常情况下,如果网络/elasticsearch群集不可访问,或者在没有正确设置“es.nodes.wan.only”的情况下以wan/cloud示例为目标时,会发生这种情况,sqlstate=无,infomessages=[“*org.apache.hive.service.cli.hivesqlexception:java.io.ioexception:org.elasticsearch.hadoop.eshadopillegalargumentexception:无法检测es版本-通常情况下,如果无法访问网络/elasticsearch群集,或者在没有正确设置“es.nodes.wan.only”的情况下以wan/cloud示例为目标时,会发生这种情况:25:24“,'org.apache.hive.service.cli.operation.sqloperation:getnextrowset:sqloperation。java:492','org.apache.hive.service.cli.operation.operationmanager:getoperationnextrowset:operationmanager。java:297','org.apache.hive.service.cli.session.hivesessionimpl:fetchresults:hivesessionimpl。java:852','sun.reflect.generatedmethodaccessor24:调用::-1','sun.reflect.delegatingmethodacc'essorimpl:invoke:delegatingmethodaccessorimpl。java:43','java.lang.reflect。method:invoke:方法。java:498','org.apache.hive.service.cli.session.hivesessionproxy:invoke:hivesessionproxy。java:78','org.apache.hive.service.cli.session.hivesessionproxy:access$000:hivesessionproxy。java:36','org.apache.hive.service.cli.session.hivesessionproxy$1:run:hivesessionproxy.java:63',java.security.accesscontroller:doprivileged:访问控制器。java:-2','javax.security.auth。subject:doas:主题。java:422','org.apache.hadoop.security.usergroupinformation:doas:usergroupinformation。java:1726','org.apache.hive.service.cli.session.hivesessionproxy:invoke:hivesessionproxy。java:59','com.sun.proxy.$proxy38:fetchresults::-1','org.apache.hive.service.cli.cliservice:fetchresults:cliservice。java:505','org.apache.hive.service.cli.thrift.thriftcliservice:fetchresults:节俭服务。java:702','org.apache.hive.service.rpc.thrift.tcliservice$处理器$fetchresults:getresult:t仪器。java:1717','org.apache.hive.service.rpc.thrift.tcliservice$处理器$fetchresults:getresult:t仪器。java:1702','org.apache.thrift.processfunction:process:processfunction。java:39','org.apache.thrift.tbaseprocessor:process:t基础处理器。java:39','org.apache.hive.service.auth.tsetipaddress'地址processor:process:t提示处理器。java:56','org.apache.thrift.server.tthreadpoolserver$工作erprocess:run:t线程池服务器。java:286','java.util.concurrent.threadpoolexecutor:runworker:threadpoolexecutor。java:1149','java.util.concurrent.threadpoolexecutor$worker:run:threadpoolexecutor。java:624','java.lang。thread:run:线程。java:748',“*java.io.ioexception:org.elasticsearch.hadoop.eshadoopillegalargumentexception:无法检测es版本-通常情况下,如果无法访问network/elasticsearch群集,或者在没有正确设置“es.nodes.wan.only”的情况下以wan/cloud示例为目标时,会发生这种情况:29:4“,'org.apache.hadoop.hive.ql.exec.fetchoperator:getnextrow:获取运算符。java:521','org.apache.hadoop.hive.ql.exec.fetchoperator:pushrow:获取运算符。java:428','org.apache.hadoop.hive.ql.exec。fetchtask:fetch:获取任务。java:146','org.apache.hadoop.hive.ql。driver:getresults:驱动器。java:2196','org.apache.hive.service.cli.operation.sqloperation:getnextrowset:sqloperation。java:487',“*org.elasticsearch.hadoop.eshadoopillegalargumentexception:cannot detect es版本-如果无法访问network/elasticsearch群集,或者在没有正确设置“es.nodes.wan.only”的情况下以wan/cloud示例为目标时,通常会发生这种情况:35:6“,'org.elasticsearch.hadoop.rest.initializationutils:discoverclusterinfo:initializationutils。java:340','org.elasticsearch.hadoop.hive。hiveutils:init:hiveutils。java:197','org.elasticsearch.hadoop.hive.eshiveinputformat:getsplits:eshiveinputformat。java:112','org.elasticsearch.hadoop.hive.eshiveinputformat:getsplits:eshiveinputformat。java:51','org.apache.hadoop.hive.ql.exec.fetchoperator:getnextsplits:获取运算符。java:372','org.apache.hadoop.hive.ql.exec.fetchoperator:getrecordreader:获取运算符。java:304','org.apache.hadoop.hive.ql.exec.fetchoperator:getnextrow:获取运算符。java:459','*org.elasticsearch.hadoop.rest.eshadoopnonodes左exception:connection error (检查网络和/或代理设置)-所有节点失败;尝试localhost:9200:41:6“,”org.elasticsearch.hadoop.rest.networkclient:execute:networkclient。java:152','org.elasticsearch.hadoop.rest.restclient:execute:rest客户端。java:424','org.elasticsearch.hadoop.rest.restclient:execute:rest客户端。java:388','org.elasticsearch.hadoop.rest.restclient:execute:rest客户端。java:392','org.elasticsearch.hadoop.rest.restclient:get:rest客户端。java:168','org.elasticsearch.hadoop.rest.restclient:maininfo:rest客户端。java:735','org.elasticsearch.hadoop.rest.initializationutils:discoverclusterinfo:initializationutils。java:330'],statuscode=3),results=none,hasmorerows=none)
1条答案
按热度按时间cbjzeqam1#
正确的属性是“es.nodes”,而不是“es.node”。默认值为“localhost”,因此您尝试连接到localhost,而不是要连接到的节点。有关更多信息,请参阅文档。
如果您没有连接到cloude环境,您可能还需要考虑是否需要将属性“es.nodes.wan.only”设置为true,因为这将禁用网络中其他节点的自动发现,如文档中所述(您需要向下滚动一点)。它将强制系统使用“es.nodes”属性,默认情况下,该属性尝试连接到localhost。这就是为什么你会出现错误,但即使你成功了,这种设置也会对你的表现产生影响(最初的重点):
注意,在这种模式下,性能受到很大影响。