Redis背板的Asp.net核心SignalR一直得到超时?

wljmcqd8  于 2022-10-31  发布在  Redis
关注(0)|答案(1)|浏览(270)

我有下面的代码来通知SignalR客户端使用Redis(AWS Elasticache)背板。当组有两个以上的值时,它总是超时。什么导致了超时?

await Task.WhenAll(groups.Select(g => hubContext.Clients(g).SendAsync("DataUpdated", notification, ctoken)));

下面是其中的一条异常消息,
未处理的异常。等待响应超时(出站= 0 KiB,入站= 53 KiB,已用1009 ms,超时为1000 ms),命令=PUBLISH,下一个:发布xyz,安装:0,qu:0,qs:1,aw:假,rs:读取异步,ws:空闲,输入:33,服务器终结点:请输入您的网址:1/1/0,经理:10个可用,客户端名称:我的客户-...,IOCP:(忙碌=0,闲=1000,最小值=1,最大值=1000),工作者:(忙碌=4,闲=32763,最小值=1,最大值=32767),v:2.2.62.27853(请参阅本文,了解一些可能导致超时的常见客户端问题:(第10页)
顺便说一句,有一个超时设置为1000 ms。异常显示的时间通常只比1 s多一点点。我可以在哪里增加这个值?

Jul 27, 2022 @ 16:49:14.693    Aborted (core dumped)
Jul 27, 2022 @ 16:49:13.806       at System.Threading.Tasks.Task.c.b__128_1(Object state)
Jul 27, 2022 @ 16:49:13.806       at System.Threading.Thread.StartCallback()
Jul 27, 2022 @ 16:49:13.806       at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Jul 27, 2022 @ 16:49:13.806       at System.Threading.ThreadPoolWorkQueue.Dispatch()
Jul 27, 2022 @ 16:49:13.806       at Microsoft.AspNetCore.SignalR.StackExchangeRedis.RedisHubLifetimeManager`1.PublishAsync(String channel, Byte[] payload)
Jul 27, 2022 @ 16:49:13.806       at ....NotificationsHubProcessor.Notify(Family family, MessageNotification`1 notification, CancellationToken token) in /teamcity/work/.../NotificationsHubProcessor.cs:line 74
Jul 27, 2022 @ 16:49:13.806    Unhandled exception. StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=53KiB, 1000ms elapsed, timeout is 1000ms), command=PUBLISH, next: PUBLISH ....NotificationsHub:group:MWS:Groups:Family:All, inst: 0, qu: 0, qs: 1, aw: False, rs: ReadAsync, ws: Idle, in: 33, serverEndpoint: clustercfg.xxx.use1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: ..1, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=4,Free=32763,Min=1,Max=32767), v: 2.2.62.27853 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
Jul 27, 2022 @ 16:49:13.806       at ....NotificationsHubProcessor.b__9_0(MessageNotification`1 x) in /teamcity/work/.../NotificationsHubProcessor.cs:line 40
Jul 27, 2022 @ 16:49:13.806       at System.Threading.QueueUserWorkItemCallback.c.b__6_0(QueueUserWorkItemCallback quwi)
Jul 27, 2022 @ 16:49:13.806       at System.Threading.QueueUserWorkItemCallback.Execute()
Jul 27, 2022 @ 16:49:13.806       at ....NotificationsHubProcessor.Process(MessageNotification`1 notification, CancellationToken token) in /teamcity/work/.../NotificationsHubProcessor.cs:line 81

Jul 27, 2022 @ 16:45:04.163    Aborted (core dumped)
Jul 27, 2022 @ 16:45:02.494       at ....NotificationsHubProcessor.b__9_0(MessageNotification`1 x) in /teamcity/work/.../NotificationsHubProcessor.cs:line 40
Jul 27, 2022 @ 16:45:02.494       at ....NotificationsHubProcessor.Process(MessageNotification`1 notification, CancellationToken token) in /teamcity/work/.../NotificationsHubProcessor.cs:line 81
Jul 27, 2022 @ 16:45:02.494       at System.Threading.Tasks.Task.c.b__128_1(Object state)
Jul 27, 2022 @ 16:45:02.494       at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Jul 27, 2022 @ 16:45:02.494       at System.Threading.QueueUserWorkItemCallback.c.b__6_0(QueueUserWorkItemCallback quwi)
Jul 27, 2022 @ 16:45:02.494       at System.Threading.QueueUserWorkItemCallback.Execute()
Jul 27, 2022 @ 16:45:02.494       at ....NotificationsHubProcessor.Notify(Family family, MessageNotification`1 notification, CancellationToken token) in /teamcity/work/.../NotificationsHubProcessor.cs:line 74
Jul 27, 2022 @ 16:45:02.494    Unhandled exception. StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 1132ms elapsed, timeout is 1000ms), command=PUBLISH, next: PUBLISH ....NotificationsHub:group:MWS:Groups:Family:M, inst: 0, qu: 2, qs: 1, aw: True, bw: Starting, rs: ReadAsync, ws: Flushing, in: 0, serverEndpoint: clustercfg.xxx.use1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: ..1, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=3,Free=32764,Min=1,Max=32767), v: 2.2.62.27853 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
Jul 27, 2022 @ 16:45:02.494       at StackExchange.Redis.ConnectionMultiplexer.ExecuteAsyncImpl_Awaited[T](ConnectionMultiplexer this, ValueTask`1 write, TaskCompletionSource`1 tcs, Message message, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2778
Jul 27, 2022 @ 16:45:02.494       at System.Threading.ThreadPoolWorkQueue.Dispatch()
Jul 27, 2022 @ 16:45:02.494       at Microsoft.AspNetCore.SignalR.StackExchangeRedis.RedisHubLifetimeManager`1.PublishAsync(String channel, Byte[] payload)
     
Jul 27, 2022 @ 15:49:22.769    Aborted (core dumped)
Jul 27, 2022 @ 15:49:21.302    Unhandled exception. StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 1435ms elapsed, timeout is 1000ms), command=PUBLISH, next: PUBLISH ....NotificationsHub:group:MWS:Groups:Family:M, inst: 0, qu: 2, qs: 1, aw: True, bw: Starting, rs: ReadAsync, ws: Flushing, in: 0, serverEndpoint: tre...-0009-001.xxx.use1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: ..2, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=3,Free=32764,Min=1,Max=32767), v: 2.2.62.27853 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
Jul 27, 2022 @ 15:49:21.302       at System.Threading.ThreadPoolWorkQueue.Dispatch()
Jul 27, 2022 @ 15:49:21.302       at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Jul 27, 2022 @ 15:49:21.302       at ....NotificationsHubProcessor.Process(MessageNotification`1 notification, CancellationToken token) in /teamcity/work/.../NotificationsHubProcessor.cs:line 81
Jul 27, 2022 @ 15:49:21.302       at ....NotificationsHubProcessor.b__9_0(MessageNotification`1 x) in /teamcity/work/.../NotificationsHubProcessor.cs:line 40
Jul 27, 2022 @ 15:49:21.302       at System.Threading.Tasks.Task.c.b__128_1(Object state)
Jul 27, 2022 @ 15:49:21.302       at Microsoft.AspNetCore.SignalR.StackExchangeRedis.RedisHubLifetimeManager`1.PublishAsync(String channel, Byte[] payload)
Jul 27, 2022 @ 15:49:21.302       at System.Threading.QueueUserWorkItemCallback.Execute()
Jul 27, 2022 @ 15:49:21.302       at StackExchange.Redis.ConnectionMultiplexer.ExecuteAsyncImpl_Awaited[T](ConnectionMultiplexer this, ValueTask`1 write, TaskCompletionSource`1 tcs, Message message, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2778
Jul 27, 2022 @ 15:49:21.302       at ....NotificationsHubProcessor.Notify(Family family, MessageNotification`1 notification, CancellationToken token) in /teamcity/work/.../NotificationsHubProcessor.cs:line 74
Jul 27, 2022 @ 15:49:21.302       at System.Threading.QueueUserWorkItemCallback.c.b__6_0(QueueUserWorkItemCallback quwi)
cu6pst1q

cu6pst1q1#

解释错误

这个错误的根本原因是,从StackExchange.Redis(SignalR在这里依赖的底层redis提供程序)将命令排入队列到执行时间超过了1秒。
为什么会这样
为什么会发生这种情况可能是一个更棘手的问题,没有任何东西跳出来给我的文本错误,你有一个操作,已被发送等待响应。
可能是:
PUBLISH是具有线性复杂度的运算:

O(N+M) where N is the number of clients subscribed to the receiving channel and M is the total number of subscribed patterns (by any client).

所以如果你有很多订阅的客户端,每个发布都必须服务所有的客户端--这可能是很昂贵的。而且如果有很多其他的客户端正在发布或以其他方式占用你的Redis服务器,这可能会导致它自己的竞争形式,阻止这个客户端的发布尝试。这是不太可能的,因为Redis往往是超级快的

正在诊断

你可以检查一下你的Redis集群的流量,如果它看起来流量很大,也许它真的被锁住了。更有可能的是,你会想检查一下Redis的慢日志,运行一个SLOWLOG GET来检查Redis,它会告诉你是否有任何长时间运行的命令。

解决方法

为了回答你的最后一个问题,你可以通过在连接字符串或ConfigurationOptions中设置asyncTimeout配置选项来扩展允许的超时时间(无论你在设置Redis时做什么)

相关问题