使用python的配置单元udf中出现运行时错误

vwkv1x7d  于 2021-06-26  发布在  Hive
关注(0)|答案(1)|浏览(325)

我正试图用python编写一个定制的hiveudf来计算员工的经验,但是我遇到了运行时错误,请解决这个问题。这是python脚本empexp.py

import sys
from datetime import datetime
from  dateutil.relativedelta import relativedelta

cdate = datetime.now().date()

for line in sys.stdin:
    line = line.strip()
    eno,ename,hiredate = line.split('\t')

    hdate = datetime.strptime(hiredate,'%Y-%m-%d').date()

    year = str(relativedelta(current_date,hdate).years)
    month = str(relativedelta(current_date,hdate).months)
    day = str(relativedelta(current_date,hdate).days)

    e = year+'-'+month+'-'+day
    list1 = [str(eno),str(ename),str(hiredate),str(cdate),str(e)]

    print '\t'.join(list1)

在下面我给出了Hive查询,

select transform(eno,ename,hiredate) 
using 'python empexp.py'
as (eno,ename,hiredate,cdate,e)
from emp;

但是我得到了运行时错误,这是错误日志,

Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: Hive Runtime Error while closing operators
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]:    An error occurred when trying to close the Operator running your custom script.
at org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:585)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
... 8 more

 FAILED: Execution Error, return code 20003 from   
 org.apache.hadoop.hive.ql.exec.mr.MapRedTask. 
 An error occurred when trying  to close the Operator running your custom    

 script.

请帮帮我。提前谢谢。

cczfrluj

cczfrluj1#

在调用脚本中的函数之前,需要将文件添加到分布式缓存中。

add file  /path/empexp.py;

然后

select transform(eno,ename,hiredate) 
using 'python empexp.py'
as (eno,ename,hiredate,cdate,e)
from emp;

相关问题