为storm中的单个worker分配内存

cwdobuhd 于 2021-06-21 发布在 Storm

关注(0)|答案(2)|浏览(509)

我需要为每个新工人分配不同的内存值。所以我试着改变每一个插销的记忆。我目前也在使用一个自定义的调度程序。这是我解决这个问题的方法。
我的代码：

TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new EmailSpout(), 1).addConfiguration("node", "zoo1").setMemoryLoad(512.0);
builder.setBolt("increment1", new IncrementBolt(), PARALLELISM).shuffleGrouping("spout").addConfiguration("node", "zoo2").setMemoryLoad(2048.0);
builder.setBolt("increment2", new IncrementBolt(), PARALLELISM).shuffleGrouping("increment1").addConfiguration("node", "zoo3").setMemoryLoad(2048.0);
builder.setBolt("increment3", new IncrementBolt(), PARALLELISM).shuffleGrouping("increment2").addConfiguration("node", "zoo4").setMemoryLoad(2048.0);
builder.setBolt("output", new OutputBolt(), 1).globalGrouping("increment2").addConfiguration("node", "zoo1").setMemoryLoad(512.0);
Config conf = new Config();
conf.setDebug(false);
conf.setNumWorkers(4);
StormSubmitter.submitTopologyWithProgressBar("Microbenchmark", conf, builder.createTopology());

我的风暴.yaml：

storm.zookeeper.servers:
     - "zoo1"
 storm.zookeeper.port: 2181
 nimbus.seeds: ["zoo1"]
 storm.local.dir: "/home/ubuntu/eranga/storm-data"
 supervisor.slots.ports:
     - 6700
     - 6701
     - 6702
     - 6703
     - 6704
 storm.scheduler: "org.apache.storm.scheduler.NodeBasedCustomScheduler"
 supervisor.scheduler.meta:
     node: "zoo4"
 worker.profiler.enabled: true
 worker.profiler.childopts: "-XX:+UnlockCommercialFeatures -XX:+FlightRecorder"
 worker.profiler.command: "flight.bash"
 worker.heartbeat.frequency.secs: 1
 worker.childopts: "-Xmx2048m -Xms2048m -Djava.net.preferIPv4Stack=true -Dorg.xml.sax.driver=com.sun.org.apache.xerces.internal.parsers.SAXParser -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl"

当我提交拓扑时，我得到以下错误。
错误：
线程“main”java.lang.illegalargumentexception中出现异常：拓扑将无法成功调度：config topology\u worker\u max\u heap\u size\u mb=768.0<2048.0（拓扑中组件的最大内存需求）。可能将拓扑\u worker \u max \u heap size \u mb设置为更大的值
在org.apache.storm.stormsubmitter.validatetopologyworkermaxheapsizembconfigs（stormsubmitter。java:496)
有什么建议吗？

Java Configuration apache-storm topology

来源：https://stackoverflow.com/questions/40926092/assign-memory-for-individual-worker-in-storm

2条答案

按热度按时间

8fq7wneg1#

试着用这个。

import org.apache.storm.Config;

public class TopologyExecuter{
   for(List<StormTopology> StormTopologyObject : StormTopologyObjects ){
       Config topologyConf = new Config();
       topologyConf.put(Config.TOPOLOGY_WORKER_CHILDOPTS,"-Xmx512m -Xms256m");
       StormSubmitter.submitTopology("topology name", topologyConf, StormTopologyObject);
   }

}

赞(0）回复(0）举报 2021-06-21

4bbkushb2#

你有没有试着按照错误信息中的建议去做？
可能将拓扑\u worker \u max \u heap size \u mb设置为更大的值
尝试将此添加到storm.yaml：