我在s3存储桶中写入数据时遇到了一个奇怪的错误。我不经常遇到这种错误。所以,我不知道到底是什么问题。仅供参考,我保持我的电子病历每次配置相同。另外,s3 bucket中的文件夹是 NOT
写保护。
insert overwrite directory 's3://logs/apr'
select f.cookie,sum(f.pgvw) as pageview, count(distinct(f.cookie)) as visits from
(
SELECT a.cookie,a.session,count(distinct(a.date_time)) as pgvw from
(
SELECT extcookie(cs_Cookie) as cookie,extsession(cs_cookie) as session,concat(logdate,' ',time) as date_time FROM apr_1 where (uri like '%.aspx%' or uri like '%.html%') and (not(uri like '/lts%'))
and (extcookie(cs_Cookie)!='-' and extcookie(cs_Cookie)!=' ')
and (extsession(cs_Cookie)!='-' and extsession(cs_Cookie)!=' ')
group by extcookie(cs_cookie),extsession(cs_cookie),logdate,time
)a
group by a.cookie,a.session
)f
where f.pgvw>1
group by f.cookie;
失败作业的日志:
FINISH_TIME="1373886754825" HOSTNAME="10\.144\.95\.241" ERROR="java\.lang\.RuntimeException: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: Hive Runtime Error while processing row {\"_col0\":\"CWC\=4LD8UplOIb7rD5X3uinVAWd7H\",\"_col1\":7,\"_col2\":1}
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.map(ExecMapper\.java:166)
at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:50)
at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:441)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:377)
at org\.apache\.hadoop\.mapred\.Child$4\.run(Child\.java:255)
at java\.security\.AccessController\.doPrivileged(Native Method)
at javax\.security\.auth\.Subject\.doAs(Subject\.java:396)
at org\.apache\.hadoop\.security\.UserGroupInformation\.doAs(UserGroupInformation\.java:1132)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:249)
Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: Hive Runtime Error while processing row {\"_col0\":\"CWC\=4LD8UplOIb7rD5X3uinVAWd7H\",\"_col1\":7,\"_col2\":1}
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.process(MapOperator\.java:550)
at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.map(ExecMapper\.java:148)
\.\.\. 8 more
Caused by: java\.lang\.IndexOutOfBoundsException: Index: 1, Size: 1
at java\.util\.ArrayList\.RangeCheck(ArrayList\.java:547)
at java\.util\.ArrayList\.get(ArrayList\.java:322)
at org\.apache\.hadoop\.hive\.serde2\.objectinspector\.StandardStructObjectInspector\.init(StandardStructObjectInspector\.java:118)
at org\.apache\.hadoop\.hive\.serde2\.objectinspector\.StandardStructObjectInspector\.<init>(StandardStructObjectInspector\.java:106)
at org\.apache\.hadoop\.hive\.serde2\.objectinspector\.ObjectInspectorFactory\.getStandardStructObjectInspector(ObjectInspectorFactory\.java:274)
at org\.apache\.hadoop\.hive\.serde2\.objectinspector\.ObjectInspectorFactory\.getStandardStructObjectInspector(ObjectInspectorFactory\.java:259)
at org\.apache\.hadoop\.hive\.ql\.exec\.ReduceSinkOperator\.initEvaluatorsAndReturnStruct(ReduceSinkOperator\.java:188)
at org\.apache\.hadoop\.hive\.ql\.exec\.ReduceSinkOperator\.processOp(ReduceSinkOperator\.java:197)
at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.process(Operator\.java:471)
at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.forward(Operator\.java:762)
at org\.apache\.hadoop\.hive\.ql\.exec\.TableScanOperator\.processOp(TableScanOperator\.java:83)
at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.process(Operator\.java:471)
at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.forward(Operator\.java:762)
at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.process(MapOperator\.java:531)
\.\.\. 9 more
1条答案
按热度按时间cgyqldqp1#
如果select或group by子句包含两个或多个仅大小写不同的字段或聚合(例如,extsession(cs\u cookie)和extsession(cs\u cookie)),则即使execmapper不这样做,优化器也会尝试合并字段。这会导致您看到的错误。
您可以通过将“cs\u cookie”的所有示例转换为“cs\u cookie”(反之亦然)并再次尝试相同的查询来确认或否认这是问题的原因。如果你不再得到错误,这是由于这个优化问题。