我想从中复制文件 hdfs
文件系统到本地 file
系统。我在javadsl中尝试了一个简单的路由。
from("hdfs2://192.168.1.55:9000/user?owner=honey")
.to("file:/home/honey/Desktop/DATA");
但它不起作用。这些是日志。
2014-11-13 22:16:22,169 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] util.Shell (Shell.java:326) - setsid exited with exit code 0
2014-11-13 22:16:22,186 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] hdfs2.HdfsConsumer (HdfsConsumer.java:87) - Connected to hdfs file-system 192.168.1.55:9000/hdfs://192.168.1.55:9000/user
2014-11-13 22:16:22,189 INFO [main] hdfs2.HdfsConsumer (HdfsConsumer.java:84) - Connected to hdfs file-system 192.168.1.55:9000/hdfs://192.168.1.55:9000/user
2014-11-13 22:16:22,189 INFO [main] impl.DefaultCamelContext (DefaultCamelContext.java:2310) - Route: route1 started and consuming from: Endpoint[hdfs2://192.168.1.55:9000/user?owner=honey]
2014-11-13 22:16:22,190 DEBUG [main] management.DefaultManagementLifecycleStrategy$TimerListenerManagerStartupListener (DefaultManagementLifecycleStrategy.java:893) - Load performance statistics disabled
2014-11-13 22:16:22,190 INFO [main] impl.DefaultCamelContext (DefaultCamelContext.java:1654) - Total 1 routes, of which 1 is started.
2014-11-13 22:16:22,192 INFO [main] impl.DefaultCamelContext (DefaultCamelContext.java:1655) - Apache Camel 2.13.0 (CamelContext: camel-1) started in 1.521 seconds
2014-11-13 22:16:22,202 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] ipc.Client$Connection (Client.java:371) - The ping interval is 60000 ms.
2014-11-13 22:16:22,204 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] ipc.Client$Connection (Client.java:636) - Connecting to /192.168.1.55:9000
2014-11-13 22:16:22,218 DEBUG [IPC Client (1506043150) connection to /192.168.1.55:9000 from honey] ipc.Client$Connection (Client.java:886) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey: starting, having connections 1
2014-11-13 22:16:22,220 DEBUG [IPC Parameter Sending Thread #0] ipc.Client$Connection$3 (Client.java:948) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey sending #0
2014-11-13 22:16:22,227 DEBUG [IPC Client (1506043150) connection to /192.168.1.55:9000 from honey] ipc.Client$Connection (Client.java:1005) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey got value #0
2014-11-13 22:16:22,227 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] ipc.ProtobufRpcEngine$Invoker (ProtobufRpcEngine.java:221) - Call: getFileInfo took 36ms
2014-11-13 22:16:22,253 DEBUG [IPC Parameter Sending Thread #0] ipc.Client$Connection$3 (Client.java:948) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey sending #1
2014-11-13 22:16:22,255 DEBUG [IPC Client (1506043150) connection to /192.168.1.55:9000 from honey] ipc.Client$Connection (Client.java:1005) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey got value #1
2014-11-13 22:16:22,255 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] ipc.ProtobufRpcEngine$Invoker (ProtobufRpcEngine.java:221) - Call: getFileInfo took 2ms
2014-11-13 22:16:22,256 DEBUG [IPC Parameter Sending Thread #0] ipc.Client$Connection$3 (Client.java:948) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey sending #2
2014-11-13 22:16:22,257 DEBUG [IPC Client (1506043150) connection to /192.168.1.55:9000 from honey] ipc.Client$Connection (Client.java:1005) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey got value #2
2014-11-13 22:16:22,257 DEBUG [Camel (camel-1) thread #0 - hdfs2://192.168.1.55:9000/user] ipc.ProtobufRpcEngine$Invoker (ProtobufRpcEngine.java:221) - Call: getFileInfo took 1ms
2014-11-13 22:16:22,259 DEBUG [IPC Parameter Sending Thread #0] ipc.Client$Connection$3 (Client.java:948) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey sending #3
2014-11-13 22:16:22,261 DEBUG [IPC Client (1506043150) connection to /192.168.1.55:9000 from honey] ipc.Client$Connection (Client.java:1005) - IPC Client (1506043150) connection to /192.168.1.55:9000 from honey got value #3
so on.....
下面这些日志不断重复。但如果我从本地复制文件也是一样的 file system
至 hdfs
那么它的工作就完美无缺了。有人能给我提供任何工作链接或例子吗?这会很有帮助的。
编辑:现在对我有用了。 readSuffix was read by default
,则不再读取已读取的文件。现在我的问题是为什么 recursive option in consumer side.
我想以递归的方式使用文件。就像我们可以扫描进去一样 File2
使用递归选项。
1条答案
按热度按时间tpgth1q71#
看看camel的hdfs2组件的文档和源代码,我想这是不可能的。
您可以为此问题打开cr。或者更好的是,自己实现这一点(先看一下如何在HDFS中递归扫描目录),然后把礼物送给Apache基金会……