配置单元中配置对象时出错

ie3xauqp  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(332)

我正在尝试用hadoop中的tweets和hive来做一个情感分析项目。我已经在tweets json格式上创建了一个表,在查询selectfrom tweets之后,我可以查看tweets;然后我创建了以下视图,但这不是从tweet\u simple中检索查询select的数据;

hive> CREATE VIEW tweets_simple AS
> SELECT
>   id,
>   cast ( from_unixtime( unix_timestamp(concat( '2014 ', substring(created_at,5,15)), 'yyyy MMM dd hh:mm:ss')) as timestamp) ts,
>   text,
>   user.time_zone
> FROM tweets;
OK
Time taken: 0.949 seconds

hive> Select * from tweets_simple limit 10;

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201503010134_0001, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201503010134_0001
Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill     job_201503010134_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2015-03-01 02:21:28,124 Stage-1 map = 0%,  reduce = 0%
2015-03-01 02:22:25,083 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201503010134_0001 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://master:50030/jobdetails.jsp?jobid=job_201503010134_0001
Examining task ID: task_201503010134_0001_m_000002 (and more) from job     job_201503010134_0001

Task with the most failures(4):
-----
Task ID:
  task_201503010134_0001_m_000000

URL:
  http://master:50030/taskdetails.jsp?jobid=job_201503010134_0001&    tipid=task_201503010134_0001_m_000000
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at         org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.jav    a:1438)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at             sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at     sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja

FAILED: Execution Error, return code 2 from     org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
70gysomp

70gysomp1#

数据是json格式的。您已经在数据的顶部创建了表,因此每次查询数据时,都需要反序列化程序将josn数据解释为配置单元支持的数据格式。因此,在运行查询之前,请将jar添加为:
添加jar/usr/lib/hive/lib/hive-serdes-1.0-snapshot.jar;
这个jar有一个jsonserde.class来反序列化json对象。

相关问题