mapreduce代码只在本地运行如何在分布式模式下执行我的代码？

ttcibm8c  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(330)

几天来，我开发了一个mapreduce代码，用phoenix在hbase中插入值。但是代码只在本地运行，并且会给机器多收费。无论我做了什么更改，我都注意到mapred.localjobrunner类被系统地使用。
你知道这个问题吗？
我把程序的执行日志附在这篇文章上。
提前感谢您的帮助。：）
如果有帮助的话，请随时向我询问更多细节。

[adviz@tdr-adviz-cdh-w-01 src]$ hadoop jar testDistributedAggregation.jar 
aggregation.retroactiveAggregation.RetroactiveAggregationApplication MEASURE FIVEMIN
16/05/25 16:29:42 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x33fb6021 connecting to ZooKeeperensemble=10.66.112.204:2181
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.4.0--1, built on 04/21/2015 19:12 GMT
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:host.name=tdr-adviz-cdh-w-01.eolas-services.com
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_67
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_67-cloudera/jre
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/etc/hadoop/conf:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.4.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/hadoop/lib/native
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.8.1.el6.x86_64
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:user.name=adviz
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/adviz
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/adviz/lucie/modules_test_distributedAggregation/modules/src
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.66.112.204:2181 sessionTimeout=90000watcher=hconnection-0x33fb60210x0, quorum=10.66.112.204:2181, baseZNode=/hbase
16/05/25 16:29:42 INFO zookeeper.ClientCnxn: Opening socket connection to server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181.Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:42 INFO zookeeper.ClientCnxn: Socket connection established to tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181,initiating session
16/05/25 16:29:42 INFO zookeeper.ClientCnxn: Session establishment complete on server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181, sessionid = 0x453ec7535016c89, negotiated timeout = 90000
16/05/25 16:29:43 INFO metrics.Metrics: Initializing metrics system: phoenix
16/05/25 16:29:43 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
16/05/25 16:29:43 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
16/05/25 16:29:43 INFO impl.MetricsSystemImpl: phoenix metrics system started
16/05/25 16:29:44 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
16/05/25 16:29:44 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2aa8b034 connecting to ZooKeeperensemble=10.66.112.204:2181
16/05/25 16:29:44 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.66.112.204:2181 sessionTimeout=90000watcher=hconnection-0x2aa8b0340x0, quorum=10.66.112.204:2181, baseZNode=/hbase
16/05/25 16:29:44 INFO zookeeper.ClientCnxn: Opening socket connection to server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181.Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:44 INFO zookeeper.ClientCnxn: Socket connection established to tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181,initiating session
16/05/25 16:29:44 INFO zookeeper.ClientCnxn: Session establishment complete on server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181, sessionid = 0x453ec7535016c8a, negotiated timeout = 90000
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c8a
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c8a closed
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:45 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x3d4e158b connecting to ZooKeeperensemble=10.66.112.204:2181
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.66.112.204:2181 sessionTimeout=90000watcher=hconnection-0x3d4e158b0x0, quorum=10.66.112.204:2181, baseZNode=/hbase
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Opening socket connection to server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181.Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Socket connection established to tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181,initiating session
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Session establishment complete on server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181, sessionid = 0x453ec7535016c8b, negotiated timeout = 90000
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c8b
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c8b closed
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:45 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6aff6bd5 connecting to ZooKeeperensemble=10.66.112.204:2181
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.66.112.204:2181 sessionTimeout=90000watcher=hconnection-0x6aff6bd50x0, quorum=10.66.112.204:2181, baseZNode=/hbase
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Opening socket connection to server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181.Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Socket connection established to tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181,initiating session
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Session establishment complete on server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181, sessionid = 0x453ec7535016c8c, negotiated timeout = 90000
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c8c
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c8c closed
16/05/25 16:29:45 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x4e42b9fd connecting to ZooKeeperensemble=10.66.112.204:2181
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.66.112.204:2181 sessionTimeout=90000watcher=hconnection-0x4e42b9fd0x0, quorum=10.66.112.204:2181, baseZNode=/hbase
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Opening socket connection to server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181.Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Socket connection established to tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181,initiating session
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Session establishment complete on server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181, sessionid = 0x453ec7535016c8d, negotiated timeout = 90000
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c8d
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c8d closed
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:50 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
16/05/25 16:29:50 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
16/05/25 16:29:50 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
16/05/25 16:29:50 INFO Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
16/05/25 16:29:50 INFO Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
16/05/25 16:29:50 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x4ed8d66f connecting to ZooKeeperensemble=localhost:2181
16/05/25 16:29:50 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000watcher=hconnection-0x4ed8d66f0x0, quorum=localhost:2181, baseZNode=/hbase
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid =0x453ec7535016c8f, negotiated timeout = 90000
16/05/25 16:29:50 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x506c1818 connecting to ZooKeeperensemble=localhost:2181
16/05/25 16:29:50 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000watcher=hconnection-0x506c18180x0, quorum=localhost:2181, baseZNode=/hbase
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid =0x453ec7535016c90, negotiated timeout = 90000
16/05/25 16:29:50 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:50 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c90
16/05/25 16:29:50 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c90 closed
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:51 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x77b18bb3 connecting to ZooKeeperensemble=localhost:2181
16/05/25 16:29:51 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000watcher=hconnection-0x77b18bb30x0, quorum=localhost:2181, baseZNode=/hbase
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid =0x453ec7535016c91, negotiated timeout = 90000
16/05/25 16:29:51 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:51 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c91
16/05/25 16:29:51 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c91 closed
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:51 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x57d5d80f connecting to ZooKeeperensemble=localhost:2181
16/05/25 16:29:51 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000watcher=hconnection-0x57d5d80f0x0, quorum=localhost:2181, baseZNode=/hbase
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid =0x453ec7535016c92, negotiated timeout = 90000
16/05/25 16:29:51 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:51 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c92
16/05/25 16:29:51 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c92 closed
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:51 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x3b30baff connecting to ZooKeeperensemble=localhost:2181
16/05/25 16:29:51 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000watcher=hconnection-0x3b30baff0x0, quorum=localhost:2181, baseZNode=/hbase
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid =0x453ec7535016c93, negotiated timeout = 90000
16/05/25 16:29:51 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:51 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c93
16/05/25 16:29:51 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c93 closed
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:52 INFO mapred.JobClient: Running job: job_local573864567_0001
16/05/25 16:29:52 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/05/25 16:29:52 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.phoenix.mapreduce.PhoenixOutputCommitter
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Waiting for map tasks
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Starting task: attempt_local573864567_0001_m_000000_0
16/05/25 16:29:52 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
16/05/25 16:29:52 INFO util.ProcessTree: setsid exited with exit code 0
16/05/25 16:29:52 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1309712b
16/05/25 16:29:52 INFO mapred.MapTask: Processing split: org.apache.phoenix.mapreduce.PhoenixInputSplit@b4eba526
16/05/25 16:29:52 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/05/25 16:29:52 INFO mapred.MapTask: io.sort.mb = 100
16/05/25 16:29:52 INFO mapred.MapTask: data buffer = 79691776/99614720
16/05/25 16:29:52 INFO mapred.MapTask: record buffer = 262144/327680
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.MapTask: Starting flush of map output
16/05/25 16:29:52 INFO mapred.MapTask: Finished spill 0
16/05/25 16:29:52 INFO mapred.Task: Task:attempt_local573864567_0001_m_000000_0 is done. And is in the process of commiting
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Task: Task attempt_local573864567_0001_m_000000_0 is allowed to commit now
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Task: Task 'attempt_local573864567_0001_m_000000_0' done.
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Finishing task: attempt_local573864567_0001_m_000000_0
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Starting task: attempt_local573864567_0001_m_000001_0
16/05/25 16:29:52 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
16/05/25 16:29:52 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7e6abaff
16/05/25 16:29:52 INFO mapred.MapTask: Processing split: org.apache.phoenix.mapreduce.PhoenixInputSplit@a313a8d1
16/05/25 16:29:52 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/05/25 16:29:52 INFO mapred.MapTask: io.sort.mb = 100
16/05/25 16:29:52 INFO mapred.MapTask: data buffer = 79691776/99614720
16/05/25 16:29:52 INFO mapred.MapTask: record buffer = 262144/327680
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.MapTask: Starting flush of map output
16/05/25 16:29:52 INFO mapred.MapTask: Finished spill 0
16/05/25 16:29:52 INFO mapred.Task: Task:attempt_local573864567_0001_m_000001_0 is done. And is in the process of commiting
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Task: Task attempt_local573864567_0001_m_000001_0 is allowed to commit now
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Task: Task 'attempt_local573864567_0001_m_000001_0' done.
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Finishing task: attempt_local573864567_0001_m_000001_0
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Starting task: attempt_local573864567_0001_m_000002_0
16/05/25 16:29:52 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
16/05/25 16:29:52 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4af8c887
16/05/25 16:29:52 INFO mapred.MapTask: Processing split: org.apache.phoenix.mapreduce.PhoenixInputSplit@b3945ee
16/05/25 16:29:52 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/05/25 16:29:52 INFO mapred.MapTask: io.sort.mb = 100
16/05/25 16:29:52 INFO mapred.MapTask: data buffer = 79691776/99614720
16/05/25 16:29:52 INFO mapred.MapTask: record buffer = 262144/327680
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.MapTask: Starting flush of map output
16/05/25 16:29:52 INFO mapred.MapTask: Finished spill 0
16/05/25 16:29:52 INFO mapred.Task: Task:attempt_local573864567_0001_m_000002_0 is done. And is in the process of commiting
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Task: Task attempt_local573864567_0001_m_000002_0 is allowed to commit now
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Task: Task 'attempt_local573864567_0001_m_000002_0' done.
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Finishing task: attempt_local573864567_0001_m_000002_0
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Map task executor complete.
16/05/25 16:29:52 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
16/05/25 16:29:52 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@217b10dd
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Merger: Merging 3 sorted segments
16/05/25 16:29:52 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 18192 bytes
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapreduce.PhoenixInputFormat: UseUpsertColumns=true, upsertColumnList.size()=9, upsertColumnList=id,startTimestamp,timestamp,first,last,min,max,avg,nb
16/05/25 16:29:52 INFO mapreduce.PhoenixInputFormat: Phoenix Custom Upsert Statement: UPSERT  INTO "mea_aggr_5minV2" ("id","startTimestamp", "0"."timestamp", "0"."first", "0"."last", "0"."min", "0"."max", "0"."avg", "0"."nb") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
16/05/25 16:29:57 INFO mapred.Task: Task:attempt_local573864567_0001_r_000000_0 is done. And is in the process of commiting
16/05/25 16:29:57 INFO mapred.LocalJobRunner:
16/05/25 16:29:57 INFO mapred.Task: Task attempt_local573864567_0001_r_000000_0 is allowed to commit now
16/05/25 16:29:57 INFO mapred.LocalJobRunner: reduce > reduce
16/05/25 16:29:57 INFO mapred.Task: Task 'attempt_local573864567_0001_r_000000_0' done.
16/05/25 16:29:58 INFO mapred.JobClient:  map 0% reduce 100%
16/05/25 16:29:58 INFO mapred.JobClient: Job complete: job_local573864567_0001
16/05/25 16:29:58 INFO mapred.JobClient: Counters: 25
16/05/25 16:29:58 INFO mapred.JobClient:   File System Counters
16/05/25 16:29:58 INFO mapred.JobClient:     FILE: Number of bytes read=222451
16/05/25 16:29:58 INFO mapred.JobClient:     FILE: Number of bytes written=1002250
16/05/25 16:29:58 INFO mapred.JobClient:     FILE: Number of read operations=0
16/05/25 16:29:58 INFO mapred.JobClient:     FILE: Number of large read operations=0
16/05/25 16:29:58 INFO mapred.JobClient:     FILE: Number of write operations=0
16/05/25 16:29:58 INFO mapred.JobClient:     HDFS: Number of bytes read=0
16/05/25 16:29:58 INFO mapred.JobClient:     HDFS: Number of bytes written=0
16/05/25 16:29:58 INFO mapred.JobClient:     HDFS: Number of read operations=0
16/05/25 16:29:58 INFO mapred.JobClient:     HDFS: Number of large read operations=0
16/05/25 16:29:58 INFO mapred.JobClient:     HDFS: Number of write operations=0
16/05/25 16:29:58 INFO mapred.JobClient:   Map-Reduce Framework
16/05/25 16:29:58 INFO mapred.JobClient:     Map input records=433
16/05/25 16:29:58 INFO mapred.JobClient:     Map output records=433
16/05/25 16:29:58 INFO mapred.JobClient:     Map output bytes=17320
16/05/25 16:29:58 INFO mapred.JobClient:     Input split bytes=2024
16/05/25 16:29:58 INFO mapred.JobClient:     Combine input records=0
16/05/25 16:29:58 INFO mapred.JobClient:     Combine output records=0
16/05/25 16:29:58 INFO mapred.JobClient:     Reduce input groups=40
16/05/25 16:29:58 INFO mapred.JobClient:     Reduce shuffle bytes=0
16/05/25 16:29:58 INFO mapred.JobClient:     Reduce input records=433
16/05/25 16:29:58 INFO mapred.JobClient:     Reduce output records=58
16/05/25 16:29:58 INFO mapred.JobClient:     Spilled Records=866
16/05/25 16:29:58 INFO mapred.JobClient:     CPU time spent (ms)=0
16/05/25 16:29:58 INFO mapred.JobClient:     Physical memory (bytes) snapshot=0
16/05/25 16:29:58 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
16/05/25 16:29:58 INFO mapred.JobClient:     Total committed heap usage (bytes)=1679818752
16/05/25 16:29:58 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c89
16/05/25 16:29:58 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c89 closed
16/05/25 16:29:58 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:58 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c8f
16/05/25 16:29:58 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:58 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c8f closed
hadoophbasemapreduceyarnphoenix

1条答案

按热度按时间
tgabmvqs

tgabmvqs1#

请看史蒂文·马加纳·祖克的回答
在yarn-site.xml中，设置如下

<property>
     <name>mapreduce.framework.name</name>
     <value>yarn</value>
    </property>

还可以看看Cloudera5.1下localjobrunner中的job keeps

赞(0）分享  回复(0）举报  2021-05-29
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com