Docker中的.NET Core高内存使用率(AWS ECS Fargate)

ego6inou  于 2023-08-08  发布在  .NET
关注(0)|答案(1)|浏览(118)

下面是应用程序的描述。我有一个运行Hangfire的控制台应用程序。有几个工作是经常性的。特别是一个作业，它解析几个大的XML文件。我将此控制台应用程序部署为AWS ECS Fargate ontainer。
每次循环作业运行时，容器内存使用量都会不断上升，直到应用程序无响应或AWS终止容器。这是ECS上的一个16 GB的容器（大），所以它应该能够处理我扔给它的任何东西。我可以告诉，只要内存使用率上升到~ 13 GB，它就注定要失败。
问题是，我在本地运行这个应用程序，它的内存使用率从来都不会很高。我将在下面的容器上发布几个来自我的工作站的示例日志。
感谢您的任何见解！
解析XML：

  • 使用XmlSerializer放入JSON格式的对象
  • 使用XDocument浏览文档
  • 使用XSD文件生成类，然后使用XmlSerializer进行填充

这三个都在本地工作，但在服务器上使用大量内存。这些是不变的XML文件，我在本地和容器中运行完全相同的文件。
仅供参考：我已经关闭了容器上的服务器垃圾收集。那没用

编辑3/5/2019

下面的日志来自以下代码。它没有应用程序的全部气息，这是相当复杂的。这段代码是在一个方法注册Hangfire每天运行一次。它从FTP站点提取XML文件，然后运行下面的代码来处理这些文件。每个文件的大小最多为10 MB。

XmlSerializer serializer = new XmlSerializer(Model);
foreach (var file in files)
{
    Console.WriteLine($"Processing file: {file.Name}");
    Model model;
    using (XmlReader reader = XmlReader.Create(file.FullName, new XmlReaderSettings { CheckCharacters = false }))
    {
        model = (Model)serializer.Deserialize(reader);
    }
    SaveToDatabase(model);
    Console.WriteLine($"Application Memory Usage - Working: {Process.GetCurrentProcess()?.WorkingSet64 ?? 0}, Private: {Process.GetCurrentProcess()?.PrivateMemorySize64 ?? 0}");
}

字符串
工作站日志：

[13:52:58 INF] Processing file: price_feed_supplemental_002981.xml
[13:52:58 INF] Application Memory Usage - Working: 1.4 GB, Private: 1.6 GB
[13:52:58 INF] Processing file: price_feed_002982.xml
[13:53:01 INF] Application Memory Usage - Working: 1.4 GB, Private: 1.6 GB
[13:53:01 INF] Processing file: price_feed_supplemental_002982.xml
[13:53:01 INF] Application Memory Usage - Working: 1.4 GB, Private: 1.6 GB
[13:53:01 INF] Processing file: price_feed_002983.xml
[13:53:01 INF] Application Memory Usage - Working: 1.4 GB, Private: 1.6 GB
[13:53:01 INF] Processing file: price_feed_supplemental_002983.xml
[13:53:02 INF] Application Memory Usage - Working: 1.4 GB, Private: 1.6 GB
[13:53:02 INF] Processing file: price_feed_002984.xml

型
容器日志：

[19:05:35 INF] Application Memory Usage - Working: 1.9 GB, Private: 0.0 bytes
[19:05:35 INF] Processing file: price_feed_supplemental_002987.xml
[19:05:35 INF] Application Memory Usage - Working: 1.9 GB, Private: 0.0 bytes
[19:05:35 INF] Processing file: price_feed_supplemental_002988.xml
[19:05:35 INF] Application Memory Usage - Working: 1.9 GB, Private: 0.0 bytes
[19:05:35 INF] Processing file: price_feed_002988.xml
[19:05:40 INF] Application Memory Usage - Working: 1.9 GB, Private: 0.0 bytes
[19:05:40 INF] Processing file: price_feed_002989.xml
[19:05:43 INF] Application Memory Usage - Working: 2.1 GB, Private: 0.0 bytes
[19:05:43 INF] Processing file: price_feed_supplemental_002989.xml
[19:05:43 INF] Application Memory Usage - Working: 2.1 GB, Private: 0.0 bytes
[19:05:43 INF] Processing file: price_feed_supplemental_002990.xml
[19:05:43 INF] Application Memory Usage - Working: 2.1 GB, Private: 0.0 bytes
[19:05:43 INF] Processing file: price_feed_002990.xml
[19:05:51 INF] Application Memory Usage - Working: 2.6 GB, Private: 0.0 bytes
[19:05:51 INF] Processing file: price_feed_002991.xml
[19:05:57 INF] Application Memory Usage - Working: 2.2 GB, Private: 0.0 bytes
[19:05:57 INF] Processing file: price_feed_supplemental_002991.xml
[19:06:00 INF] Application Memory Usage - Working: 2.2 GB, Private: 0.0 bytes
[19:06:00 INF] Processing file: price_feed_002992.xml
[19:06:19 INF] Application Memory Usage - Working: 2.1 GB, Private: 0.0 bytes
[19:06:19 INF] Processing file: price_feed_supplemental_002992.xml
[19:06:21 INF] Application Memory Usage - Working: 2.3 GB, Private: 0.0 bytes
[19:06:21 INF] Uncompressing file: usa_LTSF_20190228_002993_001_INC.zip
[19:06:22 INF] Source file: price_feed_002993_1.xml
[19:06:22 INF] Source file: price_feed_supplemental_002993_8.xml
[19:06:23 INF] Source file: price_feed_supplemental_002993_11.xml
[19:06:23 INF] Source file: price_feed_002993_2.xml
[19:06:23 INF] Source file: price_feed_supplemental_002993_5.xml
[19:06:23 INF] Source file: price_feed_supplemental_002993_4.xml
[19:06:24 INF] Source file: price_feed_supplemental_002993_17.xml
[19:06:24 INF] Source file: price_feed_supplemental_002993_15.xml
[19:06:24 INF] Source file: price_feed_supplemental_002993_3.xml
[19:06:25 INF] Source file: price_feed_supplemental_002993_1.xml
[19:06:25 INF] Source file: price_feed_supplemental_002993_9.xml
[19:06:25 INF] Source file: price_feed_supplemental_002993_7.xml
[19:06:26 INF] Source file: price_feed_supplemental_002993_6.xml
[19:06:26 INF] Source file: price_feed_supplemental_002993_14.xml
[19:06:26 INF] Source file: price_feed_supplemental_002993_13.xml
[19:06:27 INF] Source file: price_feed_supplemental_002993_16.xml
[19:06:27 INF] Source file: price_feed_supplemental_002993_12.xml
[19:06:27 INF] Source file: price_feed_supplemental_002993_2.xml
[19:06:28 INF] Source file: price_feed_supplemental_002993_10.xml
[19:06:28 INF] Processing file: price_feed_002984.xml
[19:06:28 INF] Application Memory Usage - Working: 2.5 GB, Private: 0.0 bytes
[19:06:28 INF] Processing file: price_feed_supplemental_002984.xml
[19:06:28 INF] Application Memory Usage - Working: 2.5 GB, Private: 0.0 bytes
[19:06:28 INF] Processing file: price_feed_supplemental_002985.xml
[19:06:28 INF] Application Memory Usage - Working: 2.5 GB, Private: 0.0 bytes
[19:06:28 INF] Processing file: price_feed_002985.xml
[19:06:28 INF] Application Memory Usage - Working: 2.5 GB, Private: 0.0 bytes
[19:06:28 INF] Processing file: price_feed_supplemental_002986.xml
[19:06:28 INF] Application Memory Usage - Working: 2.5 GB, Private: 0.0 bytes
[19:06:28 INF] Processing file: price_feed_002986.xml
[19:06:28 INF] Application Memory Usage - Working: 2.5 GB, Private: 0.0 bytes
[19:06:28 INF] Processing file: price_feed_002987.xml
[19:06:30 INF] Application Memory Usage - Working: 2.6 GB, Private: 0.0 bytes
[19:06:30 INF] Processing file: price_feed_supplemental_002987.xml
[19:06:30 INF] Application Memory Usage - Working: 2.6 GB, Private: 0.0 bytes
[19:06:30 INF] Processing file: price_feed_supplemental_002988.xml
[19:06:30 INF] Application Memory Usage - Working: 2.6 GB, Private: 0.0 bytes
[19:06:30 INF] Processing file: price_feed_002988.xml
[19:06:35 INF] Application Memory Usage - Working: 2.9 GB, Private: 0.0 bytes
[19:06:35 INF] Processing file: price_feed_002989.xml
[19:06:38 INF] Application Memory Usage - Working: 3.1 GB, Private: 0.0 bytes
[19:06:38 INF] Processing file: price_feed_supplemental_002989.xml
[19:06:38 INF] Application Memory Usage - Working: 3.1 GB, Private: 0.0 bytes
[19:06:38 INF] Processing file: price_feed_supplemental_002990.xml
[19:06:38 INF] Application Memory Usage - Working: 3.1 GB, Private: 0.0 bytes
[19:06:38 INF] Processing file: price_feed_002990.xml
[19:06:47 INF] Application Memory Usage - Working: 3.7 GB, Private: 0.0 bytes
[19:06:47 INF] Processing file: price_feed_002991.xml
[19:06:53 INF] Application Memory Usage - Working: 4.1 GB, Private: 0.0 bytes
[19:06:53 INF] Processing file: price_feed_supplemental_002991.xml
[19:06:55 INF] Application Memory Usage - Working: 4.3 GB, Private: 0.0 bytes
[19:06:55 INF] Processing file: price_feed_002992.xml
[19:07:05 INF] Application Memory Usage - Working: 4.9 GB, Private: 0.0 bytes
[19:07:05 INF] Processing file: price_feed_supplemental_002992.xml
[19:07:07 INF] Application Memory Usage - Working: 5.0 GB, Private: 0.0 bytes
[19:07:07 INF] Processing file: price_feed_002993_1.xml
[19:09:40 INF] 1 servers were removed due to timeout
[19:10:49 INF] Application Memory Usage - Working: 10.0 GB, Private: 0.0 bytes
[19:10:49 INF] Processing file: price_feed_supplemental_002993_8.xml
[19:12:10 INF] Application Memory Usage - Working: 12.5 GB, Private: 0.0 bytes
[19:12:10 INF] Processing file: price_feed_supplemental_002993_11.xml
[19:13:08 INF] Application Memory Usage - Working: 13.5 GB, Private: 0.0 bytes
[19:13:08 INF] Processing file: price_feed_002993_2.xml
[19:14:06 INF] Application Memory Usage - Working: 13.8 GB, Private: 0.0 bytes
[19:14:06 INF] Processing file: price_feed_supplemental_002993_5.xml

型

.net

1条答案

按热度按时间
gr8qqesn

gr8qqesn1#

通过将<ServerGarbageCollection>false</ServerGarbageCollection>添加到web应用程序的csproj文件来禁用服务器垃圾收集（强制工作站垃圾收集）似乎使GC更“急切”，从而有助于保持堆小，但非托管内存仍然缓慢上升。它最终成为一个内存泄漏-一个静态方法通过引用将它的一个参数传递回调用者，静态方法不是由GC sooo管理的，每次调用这个方法并给出更多的字符串来枚举，通过引用传递的对象变得越来越大。
我删除了pass-by-reference，方法现在正常返回对象（通过return），但我仍然必须在这个静态方法中调用GC.Collect()，以防止大约500 MB-1GB的瞬时峰值。通常情况下，即使在生产环境中，这也是可以的，因为GC确实会迅速将内存使用率降低到正常水平，但是如果这些内存峰值中的一个导致.NET应用程序超过内存配额，则会**杀死ECS示例。
不用说，我们正在使用O（N^2）枚举中的一些大字符串，我怀疑在重写这个方法之后可以删除对GC.Collect()的调用。在你的例子中，你可能在某个地方丢失了一个using，或者你正在使用事件（不太可能），或者像我一样，你有一个静态方法通过引用传递一些东西--或者没有--这个静态方法，不受GC的限制，正在慢慢泄漏。
至于禁用的“服务器”垃圾收集，这似乎是有效的，因为Fargate/ECS任务只运行在一个vCPU上。有关服务器与工作站GC的更多信息，请参阅here。
Visual Studio 2022的内置性能测量工具还可以，但是VS偶尔会崩溃（我的工作计算机也很弱），所以我最终使用了Jetbrains的dotMemory，它有更好的界面和更好的快照比较功能。

赞(0）分享  回复(0）举报  2023-08-08
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com