sparkr中空的avro格式警告

wko9yo5t  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(242)

我在rstudio中使用sparkr。我想用 hiveContext 从中提取数据 Hive/HDFS . 数据看起来有一个 AVRO 格式与声明不正确似乎有冲突 NULL 在数据库中。我在尝试加载数据时收到这样的警告 SparkR::sql 功能:
有人能提出克服这个错误的解决方案吗?

> library(SparkR, lib.loc = "/opt/spark-1.5.0-bin-hadoop2.4/R/lib")

Attaching package: ‘SparkR’

The following objects are masked from ‘package:stats’:

    filter, na.omit

The following objects are masked from ‘package:base’:

    intersect, rbind, sample, subset, summary, table, transform

Warning message:
package ‘SparkR’ was built under R version 3.2.1 
> sc <- sparkR.init(sparkHome = "/opt/spark-1.5.0-bin-hadoop2.4/")
Launching java with spark-submit command /opt/spark-1.5.0-bin-hadoop2.4//bin/spark-submit   sparkr-shell /tmp/RtmpgO9iNq/backend_portb16a2bf909d3 
Warning message:
In Filter(function(x) { : bytecode version mismatch; using eval
> sqlContext <- sparkRSQL.init(sc)
> hiveContext <- sparkRHive.init(sc)
> statki_pvki_datetime <- sql(hiveContext, select)
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
> str(statki_pvki_datetime)
Formal class 'DataFrame' [package "SparkR"] with 2 slots
  ..@ env:<environment: 0x3c6cfcf8> 
  ..@ sdf:Class 'jobj' <environment: 0x3c6cb250> 
> head(statki_pvki_datetime)
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
Error in invokeJava(isStatic = TRUE, className, methodName, ...) : 
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 0.0 failed 1 times, most recent failure: Lost task 2.0 in stage 0.0 (TID 2, localhost): org.apache.avro.AvroTypeException: Found pl.wp.avro.server.Server, expecting union
    at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:292)
    at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
    at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
    at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
    at org.apache.hadoop.hive.serde2.avro.AvroDeserializer$SchemaReEncoder.reencode(AvroDeserializer.java:11
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题