我正在尝试将后端状态设置为hdfs
val stateUri = "hdfs/path_to_dir"
val backend: RocksDBStateBackend = new RocksDBStateBackend(stateUri, true)
env.setStateBackend(backend)
我使用flink 1.7.0运行,具有以下依赖项(我尝试了所有组合):
"org.apache.flink" %% "flink-connector-filesystem" % flinkV
"org.apache.flink" % "flink-hadoop-fs" % flinkV
"org.apache.hadoop" % "hadoop-hdfs" % hadoopVersion
"org.apache.hadoop" % "hadoop-common" % hadoopVersion
但是,在运行jar时,出现以下错误:
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:403)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:58)
at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:444)
at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createCheckpointStorage(RocksDBStateBackend.java:407)
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:249)
... 17 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.
at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:64)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399)
... 23 more
任何帮助都将不胜感激
1条答案
按热度按时间lkaoscv71#
用于访问
hdfs://
路径没有严格必要捆绑flink-hadoop-fs
尽你所能地完成你的工作flink-shaded-hadoop2-uber-1.8-SNAPSHOT.jar
在lib
flink安装的文件夹。如果你的生活中没有这种依赖
lib
文件夹,然后我建议使用flink-fs-hadoop-shaded
因为它还重新定位hadoop依赖项。此外,重要的是,这个依赖关系也包含在您得到的jobjar中。因此,请确保使用
sbt-assembly
插件。