执行错误,从org.apache.hadoop.hive.ql.exec.mapredtask返回代码2

dm7nw8vv  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(422)

我在s3存储桶中写入数据时遇到了一个奇怪的错误。我不经常遇到这种错误。所以,我不知道到底是什么问题。仅供参考,我保持我的电子病历每次配置相同。另外,s3 bucket中的文件夹是 NOT 写保护。

insert overwrite directory 's3://logs/apr'
select f.cookie,sum(f.pgvw) as pageview, count(distinct(f.cookie)) as visits from
(
SELECT a.cookie,a.session,count(distinct(a.date_time)) as pgvw from
(
SELECT extcookie(cs_Cookie) as cookie,extsession(cs_cookie) as session,concat(logdate,' ',time) as date_time FROM apr_1 where (uri like '%.aspx%' or uri like '%.html%') and (not(uri like '/lts%')) 
and (extcookie(cs_Cookie)!='-' and extcookie(cs_Cookie)!='  ')
and (extsession(cs_Cookie)!='-' and extsession(cs_Cookie)!='  ')
group by extcookie(cs_cookie),extsession(cs_cookie),logdate,time
)a
group by a.cookie,a.session
)f
where f.pgvw>1
group by f.cookie;

失败作业的日志:

FINISH_TIME="1373886754825" HOSTNAME="10\.144\.95\.241" ERROR="java\.lang\.RuntimeException: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: Hive Runtime Error while processing row {\"_col0\":\"CWC\=4LD8UplOIb7rD5X3uinVAWd7H\",\"_col1\":7,\"_col2\":1}
        at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.map(ExecMapper\.java:166)
        at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:50)
        at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:441)
        at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:377)
        at org\.apache\.hadoop\.mapred\.Child$4\.run(Child\.java:255)
        at java\.security\.AccessController\.doPrivileged(Native Method)
        at javax\.security\.auth\.Subject\.doAs(Subject\.java:396)
        at org\.apache\.hadoop\.security\.UserGroupInformation\.doAs(UserGroupInformation\.java:1132)
        at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:249)
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: Hive Runtime Error while processing row {\"_col0\":\"CWC\=4LD8UplOIb7rD5X3uinVAWd7H\",\"_col1\":7,\"_col2\":1}
        at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.process(MapOperator\.java:550)
        at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.map(ExecMapper\.java:148)
        \.\.\. 8 more
    Caused by: java\.lang\.IndexOutOfBoundsException: Index: 1, Size: 1
        at java\.util\.ArrayList\.RangeCheck(ArrayList\.java:547)
        at java\.util\.ArrayList\.get(ArrayList\.java:322)
        at org\.apache\.hadoop\.hive\.serde2\.objectinspector\.StandardStructObjectInspector\.init(StandardStructObjectInspector\.java:118)
        at org\.apache\.hadoop\.hive\.serde2\.objectinspector\.StandardStructObjectInspector\.<init>(StandardStructObjectInspector\.java:106)
        at org\.apache\.hadoop\.hive\.serde2\.objectinspector\.ObjectInspectorFactory\.getStandardStructObjectInspector(ObjectInspectorFactory\.java:274)
        at org\.apache\.hadoop\.hive\.serde2\.objectinspector\.ObjectInspectorFactory\.getStandardStructObjectInspector(ObjectInspectorFactory\.java:259)
        at org\.apache\.hadoop\.hive\.ql\.exec\.ReduceSinkOperator\.initEvaluatorsAndReturnStruct(ReduceSinkOperator\.java:188)
        at org\.apache\.hadoop\.hive\.ql\.exec\.ReduceSinkOperator\.processOp(ReduceSinkOperator\.java:197)
        at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.process(Operator\.java:471)
        at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.forward(Operator\.java:762)
        at org\.apache\.hadoop\.hive\.ql\.exec\.TableScanOperator\.processOp(TableScanOperator\.java:83)
        at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.process(Operator\.java:471)
        at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.forward(Operator\.java:762)
        at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.process(MapOperator\.java:531)
        \.\.\. 9 more
cgyqldqp

cgyqldqp1#

如果select或group by子句包含两个或多个仅大小写不同的字段或聚合(例如,extsession(cs\u cookie)和extsession(cs\u cookie)),则即使execmapper不这样做,优化器也会尝试合并字段。这会导致您看到的错误。
您可以通过将“cs\u cookie”的所有示例转换为“cs\u cookie”(反之亦然)并再次尝试相同的查询来确认或否认这是问题的原因。如果你不再得到错误,这是由于这个优化问题。

相关问题