我用的是apache spark的 spark-shell
使用mesos和zookeeper,似乎效果不错,只是我从zookeeper得到的日志比我想要的要多,这有点让人分心:
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.5.0
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_60)
Type in expressions to have them evaluated.
Type :help for more information.
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.6
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@716: Client environment:host.name=myhost
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-34-generic
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@725: Client environment:os.version=#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@733: Client environment:user.name=myname
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=localhost:9999 sessionTimeout=10000 watcher=0xffffffffff sessionId=0 sessionPasswd=<null> context=0x777777777777 flags=0
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:9999]
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:9999], sessionId=0x11111111111111, negotiated timeout=10000
Spark context available as sc.
SQL context available as sqlContext.
scala>
我试过换衣服 $SPARK_CONF_DIR/log4j.properties
```
log4j.logger.org.apache.zookeeper=WARN
这不起作用,但我并不惊讶,因为mesos文档提到zookeeper日志直接进入stderr:
注意:第三方日志消息(如zookeeper)仅写入stderr!
在spark的类路径中添加一个jar,这个jar有一个静态块,带有一个对zookeercapi函数的外部函数接口调用,以更改日志级别,这似乎是可行的,但我不确定,而且我没有任何从jvm调用c代码的经验。
2条答案
按热度按时间7cwmlq891#
在这方面也找不到任何东西,但是jni(java-to-c)可以提供这个。看起来zookeeper和mesos都没有可以使用的web服务或shell(zookeeper有zkshell,但在我们的例子中不可用)。
q7solyqu2#
除了,
还可以配置stderr日志级别以仅打印具有优先级的日志
ERROR
(及以上)。也是
LevelRangeFilter
如图所示,可能也很有用。