在windows上使用“storm jar”命令远程部署storm拓扑

ma8fv8wu  于 2021-06-21  发布在  Storm
关注(0)|答案(1)|浏览(418)

我尝试在windows上使用storm jar命令将storm拓扑部署到远程计算机

storm jar jarName.jar mainClass

以下是我的storm.yaml文件中的内容：

storm.zookeeper.servers:
     - "HostName"

nimbus.host: "HostName"
nimbus.thrift.port: 6627

但每当我尝试部署时，它总是尝试部署到本地计算机。
编辑
我尝试了以下配置，但不起作用：

conf.put(Config.NIMBUS_HOST, "HostName");
conf.put(Config.NIMBUS_THRIFT_PORT,6627);
conf.put(Config.STORM_ZOOKEEPER_SERVERS,Arrays.asList(new String[]{"HostName"}));
conf.put(Config.STORM_ZOOKEEPER_PORT,2181);

我还在命令行中尝试了-c选项：

storm -c nimbus.host=HostName jar jarName.jar mainClass

但它不起作用；我得到：

Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

我试着把-c部分放在不同的地方，但似乎没有任何效果。
还有，我知道

The field Config.NIMBUS_HOST is deprecated

当我设置：

conf.put(Config.NIMBUS_HOST, "HostName");

我使用的是storm版本0.10.0-beta
编辑2
在我使用ide后，提交的工作非常好

System.setProperty("storm.jar", "path to jar");

但是，它不能从命令行工作。
编辑3
谢谢马特的帮助。我能够使用eclipse执行的java命令部署拓扑。但是，我仍然无法使用storm jar命令进行部署。它抱怨拓扑已经存在，即使我在storm ui上没有看到任何东西。另外，当我使用eclipse或命令行中的java命令进行部署时，拓扑部署成功。我应该如何比较执行java命令和stormjar命令的区别？我尝试回显cmd文件的这一部分中的变量

if %storm-command% == jar (
    set STORM_OPTS=%STORM_CLIENT_OPTS% %STORM_OPTS% -Dstorm.jar=%2
    echo %STORM_OPTS%
    set CLASSPATH=%CLASSPATH%;%2

    echo %CLASSPATH%

    set CLASS=%3
    set args=%4
    goto start
    :start
    shift
    if [%4] == [] goto done
    set args=%args% %4

    echo args: %args%

    goto start

    :done
    echo %args%
    set storm-command-arguments=%args%

  )

但我没有看到任何异常。另外，我在stacktrace中看到以下异常

Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])

通知

topologies:[]

然后它抱怨拓扑名称存在

Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
        at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
        at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)

        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)

        at mainClass.submitTopology(mainClass.java:127)
        at mainClass.main(mainClass.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
        at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
        at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:613)
        at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:549)
        at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)

        at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16546)
        at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16531)
        at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
        at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)

        at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
        at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
        at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
        ... 5 more

最后，这里是整个stacktrace。

X:\abc\xyz\target>ERROR StatusLogger Invalid URL C:/asd/ewt/apache-storm 0.10.0-beta/log4j2/cluster.xml java.net.MalformedURLException: unknown protocol: c
            at java.net.URL.<init>(URL.java:593)
            at java.net.URL.<init>(URL.java:483)
            at java.net.URL.<init>(URL.java:432)
            at java.net.URI.toURL(URI.java:1089)
            at org.apache.logging.log4j.core.config.ConfigurationFactory.getInputFromUri(ConfigurationFactory.java:296)
            at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:395)
            at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:254)
            at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:419)
            at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:138)
            at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:147)
            at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
            at org.apache.logging.log4j.LogManager.getContext(LogManager.java:175)
            at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:102)
            at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
            at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
            at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
            at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:277)
            at org.apache.log4j.Category.<init>(Category.java:56)
            at org.apache.log4j.Logger.<init>(Logger.java:35)
            at org.apache.log4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:59)
            at org.apache.log4j.Logger.getLogger(Logger.java:39)
            at mainClass.<clinit>(mainClass.java:32)
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field nimbus_uptime_secs' is unset!Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
                at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
                at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
                at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)

                at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)

                at mainClass.submitTopology(mainClass.java:127)
                at mainClass.main(mainClass.java:57)
        Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
                at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
                at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
        read(ClusterSummary.java:613)
                at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
        read(ClusterSummary.java:549)
                at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)

                at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
        resultStandardScheme.read(Nimbus.java:16546)
                at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
        resultStandardScheme.read(Nimbus.java:16531)
                at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
                at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)

                at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
                at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
                at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
                ... 5 more

顺便问一下，stacktrace顶部奇怪的日志异常/错误是什么？这可能与storm jar命令的怪异行为有关吗？
如果我这样做了

java -cp jarName.jar;C:\abc\asdasd\apache-storm-0.10.0-beta\lib\* mainClass

我明白了

513  [main] INFO  b.s.u.Utils - Using defaults.yaml from resources
582  [main] INFO  b.s.u.Utils - Using defaults.yaml from resources
590  [main] INFO  b.s.StormSubmitter - Generated ZooKeeper secret payload for MD
5-digest: -6100635753051151148:-8346710213172614614
591  [main] INFO  b.s.s.a.AuthUtils - Got AutoCreds []
599  [main] INFO  b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeM
s [2000] the maxSleepTimeMs [60000] the maxRetries [5]
626  [main] INFO  b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeM
s [2000] the maxSleepTimeMs [60000] the maxRetries [5]
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protoc
ol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName, uptime_secs:83775, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
        at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
        at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)

        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)

        at com.rbccm.th2015.storm.SampleTopologyX.submitTopology(SampleTopologyX.java:127)
        at com.rbccm.th2015.storm.SampleTopologyX.main(SampleTopologyX.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName,uptime_secs:83775, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
        at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
        at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:613)
        at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:549)
        at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)

        at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16546)
        at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16531)
        at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
        at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)

        at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
        at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
        at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
        ... 5 more

因此，它抱怨同样的事情，但通过不同的途径。这和我的yaml有关吗？nimbus的正常运行时间是什么？
最终编辑
好了，该休息了。至少现在看来，sotrmjar命令不适合windows。我可以清楚地看到storm.py在windows中尝试执行的命令与在linux中尝试执行的命令之间的区别。基本上，windows中的路径会出错，这一定是导致所有这些奇怪异常的原因。
谢谢马特的帮助和支持。我真的学到了很多。

Javaapache-storm

1条答案

按热度按时间
sqxo8psd

sqxo8psd1#

通过运行拓扑 storm jar 仅适用于linux（但不适用于windows）。“storm”命令是一个python脚本，它假定在linux上运行（storm官方不支持windows）。
然而，“风暴”命令只会启动一个常规的 java 具有适当设置的命令。如果拓扑是通过ide提交的，ide也会做同样的事情。如果您使用eclipse，则可以按照eclipse:export running configuration中的描述提取所使用的配置
注意，您需要设置jvm参数 -Dstorm.jar=path\to\myJar.jar ; 否则， StormSubmitter 找不到 jar 应该提交给风暴群。
在提取 java 命令，它应该允许您在windows上通过命令行提交拓扑。如果ide不支持提取命令，您仍然可以手动将其组合在一起（即使这是一个麻烦的解决方案）。
它将是这样的：

java -Dstorm.jar=yourJarFile.jar
     -classpath yourJarFile.jar:
                /home/mjsax/.m2/repository/org/apache/storm/storm-core/0.11.0-SNAPSHOT/storm-core-0.11.0-SNAPSHOT.jar:
                /home/mjsax/.m2/repository/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar:
                /home/mjsax/.m2/repository/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar:
                /home/mjsax/.m2/repository/org/ow2/asm/asm/4.0/asm-4.0.jar:
                /home/mjsax/.m2/repository/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar:
                /home/mjsax/.m2/repository/org/objenesis/objenesis/1.2/objenesis-1.2.jar:
                /home/mjsax/.m2/repository/org/clojure/clojure/1.6.0/clojure-1.6.0.jar:
                /home/mjsax/.m2/repository/clj-time/clj-time/0.8.0/clj-time-0.8.0.jar:
                /home/mjsax/.m2/repository/joda-time/joda-time/2.3/joda-time-2.3.jar:
                /home/mjsax/.m2/repository/compojure/compojure/1.1.3/compojure-1.1.3.jar:
                /home/mjsax/.m2/repository/org/clojure/core.incubator/0.1.0/core.incubator-0.1.0.jar:
                /home/mjsax/.m2/repository/org/clojure/tools.macro/0.1.0/tools.macro-0.1.0.jar:
                /home/mjsax/.m2/repository/clout/clout/1.0.1/clout-1.0.1.jar:
                /home/mjsax/.m2/repository/ring/ring-core/1.1.5/ring-core-1.1.5.jar:
                /home/mjsax/.m2/repository/commons-codec/commons-codec/1.6/commons-codec-1.6.jar:
                /home/mjsax/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:
                /home/mjsax/.m2/repository/commons-fileupload/commons-fileupload/1.2.1/commons-fileupload-1.2.1.jar:
                /home/mjsax/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:
                /home/mjsax/.m2/repository/hiccup/hiccup/0.3.6/hiccup-0.3.6.jar:
                /home/mjsax/.m2/repository/ring/ring-devel/1.3.0/ring-devel-1.3.0.jar:
                /home/mjsax/.m2/repository/clj-stacktrace/clj-stacktrace/0.2.7/clj-stacktrace-0.2.7.jar:
                /home/mjsax/.m2/repository/ns-tracker/ns-tracker/0.2.2/ns-tracker-0.2.2.jar:
                /home/mjsax/.m2/repository/org/clojure/tools.namespace/0.2.4/tools.namespace-0.2.4.jar:
                /home/mjsax/.m2/repository/org/clojure/java.classpath/0.2.2/java.classpath-0.2.2.jar:
                /home/mjsax/.m2/repository/ring/ring-jetty-adapter/1.3.0/ring-jetty-adapter-1.3.0.jar:
                /home/mjsax/.m2/repository/ring/ring-servlet/1.3.0/ring-servlet-1.3.0.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-server/7.6.13.v20130916/jetty-server-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/orbit/javax.servlet/2.5.0.v201103041518/javax.servlet-2.5.0.v201103041518.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-continuation/7.6.13.v20130916/jetty-continuation-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-http/7.6.13.v20130916/jetty-http-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-io/7.6.13.v20130916/jetty-io-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-util/7.6.13.v20130916/jetty-util-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/ring/ring-json/0.3.1/ring-json-0.3.1.jar:
                /home/mjsax/.m2/repository/cheshire/cheshire/5.3.1/cheshire-5.3.1.jar:
                /home/mjsax/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.3.1/jackson-core-2.3.1.jar:
                /home/mjsax/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.3.1/jackson-dataformat-smile-2.3.1.jar:
                /home/mjsax/.m2/repository/tigris/tigris/0.1.1/tigris-0.1.1.jar:
                /home/mjsax/.m2/repository/org/clojure/tools.logging/0.2.3/tools.logging-0.2.3.jar:
                /home/mjsax/.m2/repository/com/googlecode/disruptor/disruptor/2.10.4/disruptor-2.10.4.jar:
                /home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-api/2.1/log4j-api-2.1.jar:
                /home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-core/2.1/log4j-core-2.1.jar:
                /home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.1/log4j-slf4j-impl-2.1.jar:
                /home/mjsax/.m2/repository/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar:
                /home/mjsax/.m2/repository/org/slf4j/log4j-over-slf4j/1.6.6/log4j-over-slf4j-1.6.6.jar:
                /home/mjsax/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:
                /home/mjsax/.m2/repository/args4j/args4j/2.0.16/args4j-2.0.16.jar
    package.to.your.MainClass
赞(0）分享  回复(0）举报  2021-06-21
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com