如何在提交步骤时启用flink以访问emr上的/var/lib/flink

zdwk9cvp  于 2022-12-09  发布在  Apache
关注(0)|答案(1)|浏览(119)

I am on Release label:emr-6.7.0 Hadoop distribution:Amazon 3.2.1 Applications:Livy 0.7.1, ZooKeeper 3.5.7, Flink 1.14.2
I have also tried this with every emr-6.X version and I get the same error.
I am trying to submit a simple flink step using add a step on emr with args flink-yarn-session -d
and it fails with below error

2022-09-22 02:06:08,883 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting YarnSessionClusterEntrypoint down with application status FAILED. Diagnostics org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent.
at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:275)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:254)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:193)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:617)
at org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:90)
Caused by: java.nio.file.AccessDeniedException: /var/lib/flink
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:767)
at org.apache.flink.runtime.rest.RestServerEndpoint.checkAndCreateUploadDir(RestServerEndpoint.java:583)
at org.apache.flink.runtime.rest.RestServerEndpoint.createUploadDir(RestServerEndpoint.java:566)
at org.apache.flink.runtime.rest.RestServerEndpoint.<init>(RestServerEndpoint.java:126)
at org.apache.flink.runtime.webmonitor.WebMonitorEndpoint.<init>(WebMonitorEndpoint.java:211)
at org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint.<init>(DispatcherRestEndpoint.java:68)
at org.apache.flink.runtime.rest.SessionRestEndpointFactory.createRestEndpoint(SessionRestEndpointFactory.java:62)
at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:161)
... 9 more
.
2022-09-22 02:06:08,886 INFO  org.apache.flink.runtime.blob.BlobServer                 

    [] - Stopped BLOB server at 0.0.0.0:35111
2022-09-22 02:06:08,888 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Stopping Akka RPC service.
2022-09-22 02:06:08,891 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Stopping Akka RPC service.
2022-09-22 02:06:08,931 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator        [] - Shutting down remote daemon.
2022-09-22 02:06:08,932 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator        [] - Remote daemon shut down; proceeding with flushing remote transports.
2022-09-22 02:06:08,943 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator        [] - Shutting down remote daemon.
2022-09-22 02:06:08,943 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator        [] - Remote daemon shut down; proceeding with flushing remote transports.
2022-09-22 02:06:08,956 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator        [] - Remoting shut down.
2022-09-22 02:06:08,963 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator        [] - Remoting shut down.
2022-09-22 02:06:08,979 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Stopped Akka RPC service.
2022-09-22 02:06:08,987 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Stopped Akka RPC service.
2022-09-22 02:06:08,988 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Could not start cluster entrypoint YarnSessionClusterEntrypoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint YarnSessionClusterEntrypoint.
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:216) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:617) [flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:90) [flink-dist_2.12-1.14.2.jar:1.14.2]
Caused by: org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent.
    at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:275) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:254) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:193) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_342]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_342]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) ~[hadoop-common-3.2.1-amzn-7.jar:?]
    at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    ... 2 more
Caused by: java.nio.file.AccessDeniedException: /var/lib/flink
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_342]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_342]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_342]
    at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_342]
    at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_342]
    at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_342]
    at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_342]
    at org.apache.flink.runtime.rest.RestServerEndpoint.checkAndCreateUploadDir(RestServerEndpoint.java:583) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.runtime.rest.RestServerEndpoint.createUploadDir(RestServerEndpoint.java:566) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.runtime.rest.RestServerEndpoint.<init>(RestServerEndpoint.java:126) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.runtime.webmonitor.WebMonitorEndpoint.<init>(WebMonitorEndpoint.java:211) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint.<init>(DispatcherRestEndpoint.java:68) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.runtime.rest.SessionRestEndpointFactory.createRestEndpoint(SessionRestEndpointFactory.java:62) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:161) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:254) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:193) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_342]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_342]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) ~[hadoop-common-3.2.1-amzn-7.jar:?]
    at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
    ... 2 more

The main reason it says it is
Caused by: java.nio.file.AccessDeniedException: /var/lib/flink
How do I go about solving this? I am running out of ideas.

apache-flink

1条答案

按热度按时间
8yparm6h

8yparm6h1#

这可能是因为您正在为Flink群集使用任务节点。任务节点没有创建/var/lib/flink文件夹，因此当Flink作业提交到任务节点时，它将失败。
遗憾的是，这一点没有明确提到的文件。

赞(0）分享  回复(0）举报  2022-12-09
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com