connecttimeoutexception连接到eclipse中的hortonworks vm以进行mapreduce作业？

eblbsuwk  于 2021-05-30  发布在  Hadoop
关注(0)|答案(2)|浏览(371)

我正在尝试在eclipse中运行mapreduce作业。
我正在尝试连接到hortonworks虚拟机并读取hdfs中的一个文件。以下是hdfs中文件的显示：

我正在使用以下代码访问该文件： FileInputFormat.setInputPaths(conf, new Path("hdfs://127.0.0.1:8020/user/hue/smallClaimData.txt")); 我很有信心这个路径是正确的，因为我第一次尝试运行它时遇到了一个错误：“文件不存在”。我添加了用户文件夹名称（我第一次省略了这个名称），错误就消失了。因此，我假设我在hdfs中正确地引用了这个文件，但是，当我运行mapreduce作业时，我得到了以下错误（警告：它很长而且很难看，但是我希望它能有帮助）：

[main] WARN  org.apache.hadoop.conf.Configuration  - file:/tmp/hadoop-user/mapred/local/localRunner/user/job_local1865934580_0001/job_local1865934580_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
[main] WARN  org.apache.hadoop.conf.Configuration  - file:/tmp/hadoop-user/mapred/local/localRunner/user/job_local1865934580_0001/job_local1865934580_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
[main] INFO  org.apache.hadoop.mapreduce.Job  - The url to track the job: http://localhost:8080/
[Thread-11] INFO  org.apache.hadoop.mapred.LocalJobRunner  - OutputCommitter set in config null
[Thread-11] INFO  org.apache.hadoop.mapred.LocalJobRunner  - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
[Thread-11] DEBUG org.apache.hadoop.mapred.LocalJobRunner  - Starting thread pool executor.
[Thread-11] DEBUG org.apache.hadoop.mapred.LocalJobRunner  - Max local threads: 1
[Thread-11] DEBUG org.apache.hadoop.mapred.LocalJobRunner  - Map tasks to process: 1
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.LocalJobRunner  - Starting task: attempt_local1865934580_0001_m_000000_0
[Thread-11] INFO  org.apache.hadoop.mapred.LocalJobRunner  - Waiting for map tasks
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.mapred.SortedRanges  - currentIndex 0   0:0
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.mapred.LocalJobRunner  - mapreduce.cluster.local.dir for child : /tmp/hadoop-user/mapred/local/localRunner//user/jobcache/job_local1865934580_0001/attempt_local1865934580_0001_m_000000_0
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.mapred.Task  - using new api for output committer
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.yarn.util.ProcfsBasedProcessTree  - ProcfsBasedProcessTree currently is supported only on Linux.
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.Task  -  Using ResourceCalculatorProcessTree : null
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.MapTask  - Processing split: hdfs://127.0.0.1:8020/user/hue/smallClaimData.txt:0+142
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.MapTask  - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.MapTask  - (EQUATOR) 0 kvi 26214396(104857584)
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.MapTask  - mapreduce.task.io.sort.mb: 100
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.MapTask  - soft limit at 83886080
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.MapTask  - bufstart = 0; bufvoid = 104857600
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.MapTask  - kvstart = 26214396; length = 6553600
[IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user sending #2
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user got value #2
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine  - Call: getBlockLocations took 6ms
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.hdfs.DFSClient  - newInfo = LocatedBlocks{
  fileLength=142
  underConstruction=false
  blocks=[LocatedBlock{BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629; getBlockSize()=142; corrupt=false; offset=0; locs=[10.0.2.15:50010]}]
  lastLocatedBlock=LocatedBlock{BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629; getBlockSize()=142; corrupt=false; offset=0; locs=[10.0.2.15:50010]}
  isLastBlockComplete=true}
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.hdfs.DFSClient  - Connecting to datanode 10.0.2.15:50010
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user: closed
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user: stopped, remaining connections 0
[LocalJobRunner Map Task Executor #0] WARN  org.apache.hadoop.hdfs.DFSClient  - Failed to connect to /10.0.2.15:50010 for block, add to deadNodes and continue. org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/10.0.2.15:50010]
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/10.0.2.15:50010]
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:532)
    at org.apache.hadoop.hdfs.DFSInputStream.newTcpPeer(DFSInputStream.java:955)
    at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1107)
    at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:533)
    at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749)
    at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
    at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:164)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.hdfs.DFSClient  - Could not obtain BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629 from any node: java.io.IOException: No live nodes contain current block. Will get new block locations from namenode and retry...
[LocalJobRunner Map Task Executor #0] WARN  org.apache.hadoop.hdfs.DFSClient  - DFS chooseDataNode: got # 1 IOException, will wait for 595.1956215159421 msec.
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.ipc.Client  - The ping interval is 60000 ms.
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.ipc.Client  - Connecting to /127.0.0.1:8020
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user: starting, having connections 1
[IPC Parameter Sending Thread #1] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user sending #3
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user got value #3
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine  - Call: getBlockLocations took 9ms
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.hdfs.DFSClient  - newInfo = LocatedBlocks{
  fileLength=142
  underConstruction=false
  blocks=[LocatedBlock{BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629; getBlockSize()=142; corrupt=false; offset=0; locs=[10.0.2.15:50010]}]
  lastLocatedBlock=LocatedBlock{BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629; getBlockSize()=142; corrupt=false; offset=0; locs=[10.0.2.15:50010]}
  isLastBlockComplete=true}
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.hdfs.DFSClient  - Connecting to datanode 10.0.2.15:50010
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user: closed
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user: stopped, remaining connections 0
[LocalJobRunner Map Task Executor #0] WARN  org.apache.hadoop.hdfs.DFSClient  - Failed to connect to /10.0.2.15:50010 for block, add to deadNodes and continue. org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/10.0.2.15:50010]
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/10.0.2.15:50010]
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:532)
    at org.apache.hadoop.hdfs.DFSInputStream.newTcpPeer(DFSInputStream.java:955)
    at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1107)
    at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:533)
    at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749)
    at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
    at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:164)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.hdfs.DFSClient  - Could not obtain BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629 from any node: java.io.IOException: No live nodes contain current block. Will get new block locations from namenode and retry...
[LocalJobRunner Map Task Executor #0] WARN  org.apache.hadoop.hdfs.DFSClient  - DFS chooseDataNode: got # 2 IOException, will wait for 3865.511256846443 msec.
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.ipc.Client  - The ping interval is 60000 ms.
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.ipc.Client  - Connecting to /127.0.0.1:8020
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user: starting, having connections 1
[IPC Parameter Sending Thread #2] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user sending #4
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user got value #4
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine  - Call: getBlockLocations took 9ms
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.hdfs.DFSClient  - newInfo = LocatedBlocks{
  fileLength=142
  underConstruction=false
  blocks=[LocatedBlock{BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629; getBlockSize()=142; corrupt=false; offset=0; locs=[10.0.2.15:50010]}]
  lastLocatedBlock=LocatedBlock{BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629; getBlockSize()=142; corrupt=false; offset=0; locs=[10.0.2.15:50010]}
  isLastBlockComplete=true}
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.hdfs.DFSClient  - Connecting to datanode 10.0.2.15:50010
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user: closed
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user: stopped, remaining connections 0
[LocalJobRunner Map Task Executor #0] WARN  org.apache.hadoop.hdfs.DFSClient  - Failed to connect to /10.0.2.15:50010 for block, add to deadNodes and continue. org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/10.0.2.15:50010]
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/10.0.2.15:50010]
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:532)
    at org.apache.hadoop.hdfs.DFSInputStream.newTcpPeer(DFSInputStream.java:955)
    at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1107)
    at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:533)
    at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749)
    at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
    at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:164)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.hdfs.DFSClient  - Could not obtain BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629 from any node: java.io.IOException: No live nodes contain current block. Will get new block locations from namenode and retry...
[LocalJobRunner Map Task Executor #0] WARN  org.apache.hadoop.hdfs.DFSClient  - DFS chooseDataNode: got # 3 IOException, will wait for 12531.690669475103 msec.
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.ipc.Client  - The ping interval is 60000 ms.
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.ipc.Client  - Connecting to /127.0.0.1:8020
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user: starting, having connections 1
[IPC Parameter Sending Thread #3] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user sending #5
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user got value #5
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine  - Call: getBlockLocations took 16ms
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.hdfs.DFSClient  - newInfo = LocatedBlocks{
  fileLength=142
  underConstruction=false
  blocks=[LocatedBlock{BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629; getBlockSize()=142; corrupt=false; offset=0; locs=[10.0.2.15:50010]}]
  lastLocatedBlock=LocatedBlock{BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629; getBlockSize()=142; corrupt=false; offset=0; locs=[10.0.2.15:50010]}
  isLastBlockComplete=true}
[LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.hdfs.DFSClient  - Connecting to datanode 10.0.2.15:50010
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user: closed
[IPC Client (1508440322) connection to /127.0.0.1:8020 from user] DEBUG org.apache.hadoop.ipc.Client  - IPC Client (1508440322) connection to /127.0.0.1:8020 from user: stopped, remaining connections 0
[LocalJobRunner Map Task Executor #0] WARN  org.apache.hadoop.hdfs.DFSClient  - Failed to connect to /10.0.2.15:50010 for block, add to deadNodes and continue. org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/10.0.2.15:50010]
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/10.0.2.15:50010]
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:532)
    at org.apache.hadoop.hdfs.DFSInputStream.newTcpPeer(DFSInputStream.java:955)
    at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1107)
    at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:533)
    at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749)
    at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
    at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:164)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[LocalJobRunner Map Task Executor #0] WARN  org.apache.hadoop.hdfs.DFSClient  - DFS Read
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629 file=/user/hue/smallClaimData.txt
    at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:838)
    at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:526)
    at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749)
    at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
    at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:164)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.MapTask  - Starting flush of map output
[Thread-11] INFO  org.apache.hadoop.mapred.LocalJobRunner  - Map task executor complete.
[Thread-11] WARN  org.apache.hadoop.mapred.LocalJobRunner  - job_local1865934580_0001
java.lang.Exception: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629 file=/user/hue/smallClaimData.txt
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:403)
Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1200952396-10.0.2.15-1398089695400:blk_1073742320_1629 file=/user/hue/smallClaimData.txt
    at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:838)
    at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:526)
    at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749)
    at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
    at java.io.DataInputStream.read(DataInputStream.java:100)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
    at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:164)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[Thread-11] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:user (auth:SIMPLE) from:org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:329)
[Thread-1] DEBUG org.apache.hadoop.ipc.Client  - Stopping client

我的假设是，它超时是因为hortonworks不允许它连接，可能是因为权限/用户问题？我已经调查了一段时间了，但进展不大。

hadoophdfsmapreducehortonworks-data-platformeclipse

2条答案

按热度按时间
fdbelqdn

fdbelqdn1#

类似的事情也发生在我身上。多亏了@zdenek的回答，我在vm的设置中禁用了所有其他网络，只启用了主机网络。
在我的机器的主机上我添加了

192.168.56.101 sandbox.hortonworks.com

在我使用的java代码中 192.168.56.101 连接。
不必在vm中编辑任何/etc/host文件。
更多细节在这里。

赞(0）分享  回复(0）举报  2021-05-30
qlckcl4x

qlckcl4x2#

我刚刚遇到了一个类似的问题，我的代码试图连接到 10.0.2.15 在我的网络里没有曝光。我正在os x 10.9.5上使用virtualbox 4.3.20。
我通过以下方法解决了这个问题：
打开virtualbox->首选项->网络->仅主机网络选项卡
添加 vboxnet0 （如果有，只需单击螺丝刀）
将ipv4地址设置为 192.168.56.1 （对我来说）面具 255.255.255.0 启用dhcp服务器并设置范围，即服务器地址 192.168.56.100 ，边界.101-.254
确认一切
打开沙盒虚拟机设置
转到网络
将适配器设置更改为仅主机网络，然后选择 vboxnet0 重新启动你的沙盒虚拟机
只是为了确认一切正常：
在mac终端中，运行ifconfig，您将看到 vboxnet0 那里
在您的沙盒vm cmd:cat/etc/hosts中应该有 192.168.56.101 而不是 10.0.2.15 现在我可以从mac连接到沙盒hdfs了。

赞(0）分享  回复(0）举报  2021-05-30
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com