使用flume将网页数据流传输到hdfs

jhdbpxl9  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(543)

我有3个节点的集群,使用5.9版本的最新cloudera包。操作系统是centos 6.7的所有三个。我是第一次用Flume。
我的目的是将网页数据流到hdfs中。然而,这个网页是一个第三方网站,新闻网站在我的情况下,所以我不知道该使用哪个端口连接。
curl和telnet发生在端口80上,因此我使用了它。但是有个错误。
我的flume.conf是:

tier1.sources  = http-source
tier1.channels = mem-channel-1
tier1.sinks    = hdfs-sink
tier1.sources.http-source.type     = http
tier1.sources.http-source.handler = org.apache.flume.source.http.JSONHandler
tier1.sources.http-source.bind     = 132.247.1.32
tier1.sources.http-source.port     = 80
tier1.sources.http-source.channels = mem-channel-1
tier1.channels.mem-channel-1.type   = memory
tier1.sinks.hdfs-sink.type         = hdfs
tier1.sinks.hdfs-sink.channel      = mem-channel-1
tier1.sinks.hdfs-sink.hdfs.path    = /flume/events/%y-%m-%d/%H%M/%S

# Other properties are specific to each type of

# source, channel, or sink. In this case, we

# specify the capacity of the memory channel.

tier1.channels.mem-channel-1.capacity = 100

错误

2016-12-19 16:45:00,353 WARN org.mortbay.log: failed SelectChannelConnector@132.247.1.32:80: java.net.BindException: Cannot assign requested address
2016-12-19 16:45:00,353 WARN org.mortbay.log: failed Server@36772002: java.net.BindException: Cannot assign requested address
2016-12-19 16:45:00,353 ERROR org.apache.flume.source.http.HTTPSource: Error while starting HTTPSource. Exception follows.
java.net.BindException: Cannot assign requested address**
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at org.mortbay.jetty.nio.SelectChannelConnector.doStart2016-12-19 16:45:00,364 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{
name:http-source,state:IDLE} } - Exception follows.
java.lang.RuntimeException: java.net.BindException: Cannot assign requested address(SelectChannelConnector.java:315)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.flume.source.http.HTTPSource.start(HTTPSource.java:207)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
wgeznvg7

wgeznvg71#

尝试按以下方式更改源配置:httpagent.sources.http-source.port=80 httpagent.sources.http-source.bind=localhost httpagent.sources.http-source.url=132.247.1.32
注意:如果132.247.1.32不起作用,请尝试提供主机名。

yptwkmov

yptwkmov2#

flume的http源提供了一种使用get/post将数据发送到flume代理的方法。http源不会出去为您从网站获取数据,它会设置一个http服务器,等待get/post并接受来自该端点的数据。
我的建议是创建一个自定义源,获取您需要的网页。

相关问题