cassandra 3.11，nodetool重建失败

wfsdck30  于 2021-06-14  发布在  Cassandra
关注(0)|答案(0)|浏览(310)

我使用的是Cassandra3.11版本，由于java.net.socketexception:connection reset exception，重建失败。
现有节点上的异常：

INFO  [STREAM-INIT-/xx.xxx.xyz.xxx:35784] 2019-07-31 01:56:33,559 StreamResultFuture.java:116 - [Stream #52292810-b308-11e9-98f9-0fd8a171017f ID#0] Creating new streaming plan for Rebuild
INFO  [STREAM-INIT-/xx.xxx.xyz.xxx:35784] 2019-07-31 01:56:33,572 StreamResultFuture.java:123 - [Stream #52292810-b308-11e9-98f9-0fd8a171017f, ID#0] Received streaming plan for Rebuild
INFO  [STREAM-INIT-/xx.xxx.xyz.xxx:35786] 2019-07-31 01:56:33,774 StreamResultFuture.java:123 - [Stream #52292810-b308-11e9-98f9-0fd8a171017f, ID#0] Received streaming plan for Rebuild
INFO  [STREAM-IN-/xx.xxx.xyz.xxx:35786] 2019-07-31 01:56:34,060 StreamResultFuture.java:173 - [Stream #52292810-b308-11e9-98f9-0fd8a171017f ID#0] Prepare completed.

接收0个文件（0.000kib），发送60个文件（2.085mib）

ERROR [STREAM-IN-/xx.xxx.xyz.xxx:35786] 2019-07-31 01:56:34,576 StreamSession.java:706 - [Stream #52292810-b308-11e9-98f9-0fd8a171017f] Remote peer xx.xxx.xyz.xxx failed stream session.
ERROR [STREAM-OUT-/xx.xxx.xyz.xxx:35784] 2019-07-31 01:56:34,587 StreamSession.java:593 - [Stream #52292810-b308-11e9-98f9-0fd8a171017f] Streaming error occurred on session with peer xx.xxx.xyz.xxx
org.apache.cassandra.io.FSReadError: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset
        at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:145) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.compress.CompressedStreamWriter.lambda$write$0(CompressedStreamWriter.java:85) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.io.util.UnbufferedDataOutputStreamPlus.applyToChannel(UnbufferedDataOutputStreamPlus.java:383) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:85) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:101) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:52) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:50) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:408) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:380) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
Caused by: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset
        at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1533) ~[na:1.8.0_212]
        at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1545) ~[na:1.8.0_212]
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71) ~[na:1.8.0_212]
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) ~[na:1.8.0_212]
        at org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.write(WrappedDataOutputStreamPlus.java:48) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.io.util.DataOutputStreamPlus$2.write(DataOutputStreamPlus.java:101) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at sun.nio.ch.FileChannelImpl.transferToArbitraryChannel(FileChannelImpl.java:566) ~[na:1.8.0_212]
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:613) ~[na:1.8.0_212]
        at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:141) ~[apache-cassandra-3.11.3.jar:3.11.3]
        ... 10 common frames omitted
Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection reset
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[na:1.8.0_212]
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) ~[na:1.8.0_212]
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1903) ~[na:1.8.0_212]
        at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1867) ~[na:1.8.0_212]
        at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1812) ~[na:1.8.0_212]
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:128) ~[na:1.8.0_212]
        ... 16 common frames omitted
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115) ~[na:1.8.0_212]
        at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[na:1.8.0_212]
        at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431) ~[na:1.8.0_212]
        at sun.security.ssl.OutputRecord.write(OutputRecord.java:417) ~[na:1.8.0_212]
        at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:879) ~[na:1.8.0_212]
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:850) ~[na:1.8.0_212]
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[na:1.8.0_212]
        ... 16 common frames omitted

重建节点时出现异常：

INFO  [StreamReceiveTask:2] 2019-07-31 01:56:34,568 SecondaryIndexManager.java:386 - Index build of table_col1_idx,table_col2_idx,table_col3_idx,table_col4_idx,table_col5_idx,table_col6_idx,table_col7_idx,table_col8_idx complete
ERROR [CompactionExecutor:2] 2019-07-31 01:56:34,571 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:2,1,main]
java.util.NoSuchElementException: null
        at org.apache.cassandra.utils.AbstractIterator.next(AbstractIterator.java:64) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.index.SecondaryIndexManager.lambda$indexPartition$20(SecondaryIndexManager.java:619) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_212]
        at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:619) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:71) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1637) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_212]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3.jar:3.11.3]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]
ERROR [StreamReceiveTask:4] 2019-07-31 01:56:34,572 StreamSession.java:593 - [Stream #52292810-b308-11e9-98f9-0fd8a171017f] Streaming error occurred on session with peer xx.xxx.abc.xxx
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.NoSuchElementException
        at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:51) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:374) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.index.SecondaryIndexManager.buildIndexesBlocking(SecondaryIndexManager.java:383) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.index.SecondaryIndexManager.buildAllIndexesBlocking(SecondaryIndexManager.java:270) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:216) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3.jar:3.11.3]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]
Caused by: java.util.concurrent.ExecutionException: java.util.NoSuchElementException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_212]
        at java.util.concurrent.FutureTask.get(FutureTask.java:192) [na:1.8.0_212]
        at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:367) ~[apache-cassandra-3.11.3.jar:3.11.3]
        ... 9 common frames omitted
Caused by: java.util.NoSuchElementException: null
        at org.apache.cassandra.utils.AbstractIterator.next(AbstractIterator.java:64) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.index.SecondaryIndexManager.lambda$indexPartition$20(SecondaryIndexManager.java:619) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_212]
        at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:619) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:71) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1637) ~[apache-cassandra-3.11.3.jar:3.11.3]
        ... 6 common frames omitted
ERROR [STREAM-IN-/xx.xxx.abc.xxx:7001] 2019-07-31 01:56:34,579 StreamSession.java:593 - [Stream #52292810-b308-11e9-98f9-0fd8a171017f] Streaming error occurred on session with peer xx.xxx.abc.xxx
java.lang.RuntimeException: Outgoing stream handler has been closed
        at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:143) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:655) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:523) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
INFO  [StreamReceiveTask:4] 2019-07-31 01:56:34,580 StreamResultFuture.java:187 - [Stream #52292810-b308-11e9-98f9-0fd8a171017f] Session with /xx.xxx.abc.xxx is complete
WARN  [StreamReceiveTask:4] 2019-07-31 01:56:34,581 StreamResultFuture.java:214 - [Stream #52292810-b308-11e9-98f9-0fd8a171017f] Stream failed
ERROR [RMI TCP Connection(8)-127.0.0.1] 2019-07-31 01:56:34,582 StorageService.java:1281 - Error while rebuilding node
org.apache.cassandra.streaming.StreamException: Stream failed
        at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
        at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:571) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:250) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_212]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_212]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]
INFO  [StreamReceiveTask:1] 2019-07-31 01:56:34,589 SecondaryIndexManager.java:386 - Index build of tablebar_col1_index,tablebar_col2_index,tablebar_col3_index complete

请检查以下问题并提供提示：
这与网络和防火墙设置有关吗？
从重建节点上的堆栈跟踪和日志消息来看，secondaryindexmanager似乎有问题？
现有节点发送的大小是否有影响？
已检查的设置流\u保持\u活着\u期间\u在\u秒和phi \u罪犯\u阈值，但没有成功。
如果tablefoo有8个索引，那么文件大小会有什么影响，因为索引较少的表的其余部分不会面临问题。
流\u吞吐量\u出站\u兆比特每秒\u设置有任何影响吗？
相关参考文献：https://issues.apache.org/jira/browse/cassandra-12886httpshttp://issues.apache.org/jira/browse/cassandra-11841https://issues.apache.org/jira/browse/cassandra-12830 （不过这是一个多dc aws）

cassandra

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com