jsonpath 2.3.0与hadoop2.7环境json-smart1.2.0.jar冲突

at0kjp5o 于 2021-06-24 发布在 Pig

关注(0)|答案(2)|浏览(476)

在hdfs位置中输入json
需要解析json并聚合结果
为此，我使用了pig udf，它使用json路径库
在hadoop2.7环境jar中：json-smart1.2、json-path1.2是硬绑定的
每当我执行pig mapreduce时，它将我抛出以下异常

java.lang.NoSuchFieldError: defaultReader
at com.jayway.jsonpath.spi.json.JsonSmartJsonProvider.<init>(JsonSmartJsonProvider.java:39)
at com.jayway.jsonpath.internal.DefaultsImpl.jsonProvider(DefaultsImpl.java:21)
at com.jayway.jsonpath.Configuration.defaultConfiguration(Configuration.java:174)
at com.jayway.jsonpath.internal.JsonContext.<init>(JsonContext.java:52)
at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:596)

为了解决这个问题，尝试了以下几种选择
选项1：尝试设置注册json-smart2.3.jar&json-path2.3.0.jar，但没有很好的结果（因为它引用的jar是json-path1.2.jar）
选项2：将我的模块依赖关系降级到json-path1.2.jar没有结果
选项3：使用自定义类加载器尝试加载json-path2.3.0jar的jar，它加载的类出现org.slf4j绑定问题
标识了多个绑定路径，但sun.misc classloader出现问题

Failed to instantiate SLF4J LoggerFactory
Reported exception:
java.lang.NullPointerException
    at sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)
    at sun.misc.URLClassPath.getLoader(URLClassPath.java:485)
    at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
    at sun.misc.URLClassPath.access$100(URLClassPath.java:64)
    at sun.misc.URLClassPath$1.next(URLClassPath.java:239)
    at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:250)
    at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
    at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
    at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
    at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
    at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
    at org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)
    at com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)
    at com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)
    at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)
    at com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)
    at com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)
Caused by: java.lang.ExceptionInInitializerError
    at com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)
    at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)
    ... 6 more
Caused by: java.lang.IllegalStateException: Unexpected initialization failure
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:167)
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)
    at com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)
    ... 8 more
Caused by: java.lang.NullPointerException
    at sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)
    at sun.misc.URLClassPath.getLoader(URLClassPath.java:485)
    at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
    at sun.misc.URLClassPath.access$100(URLClassPath.java:64)
    at sun.misc.URLClassPath$1.next(URLClassPath.java:239)
    at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:250)
    at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
    at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
    at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
    at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
    at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
    at org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)
    ... 13 more

有人能建议我解决这个问题吗？有人能找到一个stackoverflow链接来说明weblogic，而不是一个同样适用于hadoop2.7的通用解决方案（json解析器-java.lang.nosuchfielderror:defaultreader）

Java JSON hadoop2 apache-pig JsonPath

来源：https://stackoverflow.com/questions/56945580/json-path-2-3-0-conflicts-with-hadoop-2-7-environment-json-smart1-2-0-jar

2条答案

按热度按时间

pn9klfpd1#

提供自定义实现来加载/解析json，这样我们就可以避免使用hadoop/lib的json-smart-2.x/1.x

public static void changeJsonPathConfig() {

    if (!configChanged) {

        Configuration.setDefaults(new Configuration.Defaults() {

            private final JsonProvider jsonProvider = new GsonJsonProvider(

                    new GsonBuilder().serializeNulls().create());

            private final MappingProvider mappingProvider = new GsonMappingProvider();

            @Override

            public JsonProvider jsonProvider() {

                return jsonProvider;

            }

            @Override

            public MappingProvider mappingProvider() {

                return mappingProvider;

            }

            @Override

            public Set<Option> options() {

                return EnumSet.noneOf(Option.class);

            }

        });

        configChanged = true;

    }

}

赞(0）回复(0）举报 2021-06-24

qpgpyjmq2#

由于hadoop环境（pig、hdfs、hive等）使用的是json-path-2.3.0，它更好的用户Map器逻辑应该使用另一个版本“jsonpath-1.0.jar”来解决这个问题

赞(0）回复(0）举报 2021-06-24