多类加载器和hadoop API

00jrzges  于 2021-05-27  发布在  Hadoop
关注(0)|答案(0)|浏览(267)

我们的应用程序有多个类加载器。这是为了避免软件访问的各种api之间的依赖冲突,包括但不限于hadoop。如果不使用多类加载器策略(或java9模块,我们还不支持),我们会在hadoopjar和其他东西之间遇到很多依赖冲突。所以,可以说我们为所有hadoop jar提供了一个urlclassloader,为其他库集提供了其他类装入器。
问题是:系统类加载器(jvm启动的类加载器)没有hadoopjar。这意味着线程上下文类加载器(默认情况下是系统类加载器)没有hadoopjar。不幸的是,各种各样的hadoop API似乎都隐式地使用了线程上下文类加载器,这导致了各种各样的classnotfound错误、ServiceLoader返回空列表、工厂找不到实现等等。这让我们慢慢发疯,因为当我们的客户发现新的安全选项和其他配置变体时,新的“找不到”问题偶尔会出现在现场,这会导致代码路径无法找到接口的类或实现。
您可以建议一个显而易见的方法:始终将线程上下文类加载器设置为hadoop类加载器。不幸的是,这似乎不是一个解决办法。首先,生成的线程具有默认的系统类加载器作为线程上下文类加载器,我不能控制所有这些线程。其次,在每个可能的hadoop入口点保存和恢复线程上下文类加载器很容易出错,而且成本可能太高。
我找到的唯一答案是通过反复试验了解哪些类是由线程上下文类加载器(或serviceloader)加载的,并强制它们在我设置线程上下文类加载器的地方预加载。有更好的办法吗?有这类课程的名单吗?
这是一个堆栈跟踪示例:

Error:ProcessNode(PSOutput16): org/apache/hadoop/fs/FileSystem.create:java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to load class: [org.apache.hadoop.yarn.api.records.impl.pb.ApplicationIdPBImpl]
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:468)
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:754)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283)
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:528)
org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1484)
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1586)
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1571)
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:440)
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:433)
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:433)
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:374)
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:926)
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:907)
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to load class: [org.apache.hadoop.yarn.api.records.impl.pb.ApplicationIdPBImpl]
org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java:58)
org.apache.hadoop.yarn.util.Records.newRecord(Records.java:36)
org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(ApplicationId.java:49)
org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:83)
org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:145)
org.apache.hadoop.security.token.Token.identifierToString(Token.java:349)
org.apache.hadoop.security.token.Token.toString(Token.java:369)
java.lang.String.valueOf(String.java:2994)
java.lang.StringBuilder.append(StringBuilder.java:131)
org.apache.hadoop.security.UserGroupInformation.logAllUserInfo(UserGroupInformation.java:1959)
org.apache.hadoop.crypto.key.kms.KMSClientProvider.getActualUgi(KMSClientProvider.java:1056)
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:451)
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:754)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123)
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283)
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:528)
org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1484)
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1586)
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1571)
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:440)
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:433)
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:433)
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:374)
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:926)
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:907)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.api.records.impl.pb.ApplicationIdPBImpl not found
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2255)
org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java:56)
org.apache.hadoop.yarn.util.Records.newRecord(Records.java:36)
org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(ApplicationId.java:49)
org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:83)

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题