Impala -内存不足异常查询速度慢

rdrgkggo  于 2021-06-26  发布在  Impala
关注(0)|答案(1)|浏览(413)

有人能帮我吗。我正在为我的api运行一个由5个impala节点组成的集群。现在我在运行查询时遇到很多“内存不足”的异常。

Failed to get minimum memory reservation of 3.94 MB on daemon r5c3s4.colo.vm:22000 for query 924d155863398f6b:c4a3470300000000 because it would exceed an applicable memory limit. Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. Memory usage:
, node=[r4c3s2]
Process: Limit=55.00 GB Total=49.79 GB Peak=49.92 GB, node=[r4c3s2]
  Buffer Pool: Free Buffers: Total=0, node=[r4c3s2]
  Buffer Pool: Clean Pages: Total=4.21 GB, node=[r4c3s2]
  Buffer Pool: Unused Reservation: Total=-4.21 GB, node=[r4c3s2]
  Free Disk IO Buffers: Total=1.19 GB Peak=1.52 GB, node=[r4c3s2]

不过,该公司表示,150.00GB中只有23.83GB被使用。而且查询变得非常慢。这个问题不知从哪儿冒出来的。有人对此有什么解释吗?
以下是我从一个节点的“/memz?detailed=true”页面获得的所有内存信息:

Memory Usage
Memory consumption / limit: 23.83 GB / 150.00 GB
Breakdown`enter code here`

Process: Limit=150.00 GB Total=23.83 GB Peak=58.75 GB
  Buffer Pool: Free Buffers: Total=72.69 MB
  Buffer Pool: Clean Pages: Total=0
  Buffer Pool: Unused Reservation: Total=-71.94 MB
  Free Disk IO Buffers: Total=1.61 GB Peak=1.67 GB
  RequestPool=root.default: Total=20.77 GB Peak=59.92 GB
    Query(2647a4f63d37fdaa:690ad3b500000000): Reservation=20.67 GB ReservationLimit=120.00 GB OtherMemory=101.21 MB Total=20.77 GB Peak=20.77 GB
      Unclaimed reservations: Reservation=71.94 MB OtherMemory=0 Total=71.94 MB Peak=139.94 MB
      Fragment 2647a4f63d37fdaa:690ad3b50000001c: Reservation=0 OtherMemory=114.48 KB Total=114.48 KB Peak=855.48 KB
        AGGREGATION_NODE (id=9): Total=102.12 KB Peak=102.12 KB
          Exprs: Total=102.12 KB Peak=102.12 KB
        EXCHANGE_NODE (id=8): Total=0 Peak=0
        DataStreamRecvr: Total=0 Peak=0
        DataStreamSender (dst_id=10): Total=872.00 B Peak=872.00 B
        CodeGen: Total=3.50 KB Peak=744.50 KB
      Fragment 2647a4f63d37fdaa:690ad3b500000014: Reservation=0 OtherMemory=243.31 KB Total=243.31 KB Peak=1.57 MB
        AGGREGATION_NODE (id=3): Total=102.12 KB Peak=102.12 KB
          Exprs: Total=102.12 KB Peak=102.12 KB
        AGGREGATION_NODE (id=7): Total=119.12 KB Peak=119.12 KB
          Exprs: Total=119.12 KB Peak=119.12 KB
        EXCHANGE_NODE (id=6): Total=0 Peak=0
        DataStreamRecvr: Total=0 Peak=0
        DataStreamSender (dst_id=8): Total=6.81 KB Peak=6.81 KB
        CodeGen: Total=7.25 KB Peak=1.34 MB
      Fragment 2647a4f63d37fdaa:690ad3b50000000c: Reservation=2.32 GB OtherMemory=349.48 KB Total=2.32 GB Peak=2.32 GB
        AGGREGATION_NODE (id=2): Total=119.12 KB Peak=119.12 KB
          Exprs: Total=119.12 KB Peak=119.12 KB
        AGGREGATION_NODE (id=5): Reservation=2.32 GB OtherMemory=199.74 KB Total=2.32 GB Peak=2.32 GB
          Exprs: Total=120.12 KB Peak=120.12 KB
        EXCHANGE_NODE (id=4): Total=0 Peak=0
        DataStreamRecvr: Total=336.00 B Peak=549.14 KB
        DataStreamSender (dst_id=6): Total=6.44 KB Peak=6.44 KB
        CodeGen: Total=15.85 KB Peak=3.10 MB
      Fragment 2647a4f63d37fdaa:690ad3b500000004: Reservation=18.29 GB OtherMemory=100.52 MB Total=18.38 GB Peak=18.38 GB
        AGGREGATION_NODE (id=1): Reservation=18.29 GB OtherMemory=334.12 KB Total=18.29 GB Peak=18.29 GB
          Exprs: Total=148.12 KB Peak=148.12 KB
        HDFS_SCAN_NODE (id=0): Total=100.17 MB Peak=178.15 MB
          Exprs: Total=4.00 KB Peak=4.00 KB
        DataStreamSender (dst_id=4): Total=6.75 KB Peak=6.75 KB
        CodeGen: Total=9.72 KB Peak=2.92 MB
  RequestPool=fe-eval-exprs: Total=0 Peak=12.00 KB
  Untracked Memory: Total=1.44 GB

tcmalloc

------------------------------------------------
MALLOC:    24646559936 (23504.8 MiB) Bytes in use by application
MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
MALLOC: +    725840992 (  692.2 MiB) Bytes in central cache freelist
MALLOC: +      4726720 (    4.5 MiB) Bytes in transfer cache freelist
MALLOC: +    208077600 (  198.4 MiB) Bytes in thread cache freelists
MALLOC: +    105918656 (  101.0 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =  25691123904 (24501.0 MiB) Actual memory used (physical + swap)
MALLOC: +  53904392192 (51407.2 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =  79595516096 (75908.2 MiB) Virtual address space used
MALLOC:
MALLOC:         133041              Spans in use
MALLOC:            842              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

System

Physical Memory: 252.41 GB
Transparent Huge Pages Config:
  enabled: always [madvise] never
  defrag: [always] madvise never
  khugepaged defrag: 1

Process and system memory metrics
Name    Value   Description
memory.anon-huge-page-bytes     19.01 GB    Total bytes of anonymous (a.k.a. transparent) huge pages used by this process.
memory.mapped-bytes     113.09 GB   Total bytes of memory mappings in this process (the virtual memory size).
memory.num-maps     18092   Total number of memory mappings in this process.
memory.rss  24.51 GB    Resident set size (RSS) of this process, including TCMalloc, buffer pool and Jvm.
memory.thp.defrag   [always] madvise never  The system-wide 'defrag' setting for Transparent Huge Pages.
memory.thp.enabled  always [madvise] never  The system-wide 'enabled' setting for Transparent Huge Pages.
memory.thp.khugepaged-defrag    1   The system-wide 'defrag' setting for khugepaged.
memory.total-used   23.83 GB    Total memory currently used by TCMalloc and buffer pool.
Buffer pool memory metrics
Name    Value   Description
buffer-pool.clean-page-bytes    0   Total bytes of clean page memory cached in the buffer pool.
buffer-pool.clean-pages     0   Total number of clean pages cached in the buffer pool.
buffer-pool.clean-pages-limit   12.00 GB    Limit on number of clean pages cached in the buffer pool.
buffer-pool.free-buffer-bytes   72.69 MB    Total bytes of free buffer memory cached in the buffer pool.
buffer-pool.free-buffers    177     Total number of free buffers cached in the buffer pool.
buffer-pool.limit   120.00 GB   Maximum allowed bytes allocated by the buffer pool.
buffer-pool.reserved    20.67 GB    Total bytes of buffers reserved by Impala subsystems
buffer-pool.system-allocated    20.67 GB    Total buffer memory currently allocated by the buffer pool.
buffer-pool.unused-reservation-bytes    71.94 MB    Total bytes of buffer reservations by Impala subsystems that are currently unused
JVM aggregate memory metrics
Name    Value   Description
jvm.total.committed-usage-bytes     1.45 GB     Jvm total Committed Usage Bytes
jvm.total.current-usage-bytes   903.10 MB   Jvm total Current Usage Bytes
jvm.total.init-usage-bytes  1.92 GB     Jvm total Init Usage Bytes
jvm.total.max-usage-bytes   31.23 GB    Jvm total Max Usage Bytes
jvm.total.peak-committed-usage-bytes    2.09 GB     Jvm total Peak Committed Usage Bytes
jvm.total.peak-current-usage-bytes  1.48 GB     Jvm total Peak Current Usage Bytes
jvm.total.peak-init-usage-bytes     1.92 GB     Jvm total Peak Init Usage Bytes
jvm.total.peak-max-usage-bytes  31.41 GB    Jvm total Peak Max Usage Bytes
JVM heap memory metrics
Name    Value   Description
jvm.heap.committed-usage-bytes  1.37 GB     Jvm heap Committed Usage Bytes
jvm.heap.current-usage-bytes    827.25 MB   Jvm heap Current Usage Bytes
jvm.heap.init-usage-bytes   2.00 GB     Jvm heap Init Usage Bytes
jvm.heap.max-usage-bytes    26.67 GB    Jvm heap Max Usage Bytes
jvm.heap.peak-committed-usage-bytes     0   Jvm heap Peak Committed Usage Bytes
jvm.heap.peak-current-usage-bytes   0   Jvm heap Peak Current Usage Bytes
jvm.heap.peak-init-usage-bytes  0   Jvm heap Peak Init Usage Bytes
jvm.heap.peak-max-usage-bytes   0   Jvm heap Peak Max Usage Bytes
JVM non-heap memory metrics
Name    Value   Description
jvm.non-heap.committed-usage-bytes  76.90 MB    Jvm non-heap Committed Usage Bytes
jvm.non-heap.current-usage-bytes    75.68 MB    Jvm non-heap Current Usage Bytes
jvm.non-heap.init-usage-bytes   2.44 MB     Jvm non-heap Init Usage Bytes
jvm.non-heap.max-usage-bytes    -1.00 B     Jvm non-heap Max Usage Bytes
jvm.non-heap.peak-committed-usage-bytes     0   Jvm non-heap Peak Committed Usage Bytes
jvm.non-heap.peak-current-usage-bytes   0   Jvm non-heap Peak Current Usage Bytes
jvm.non-heap.peak-init-usage-bytes  0   Jvm non-heap Peak Init Usage Bytes
jvm.non-heap.peak-max-usage-bytes   0   Jvm non-heap Peak Max Usage Bytes
ego6inou

ego6inou1#

Process: Limit=150.00 GB Total=23.83 GB Peak=58.75 GB

这是由于
memory limit Memorylimit exceeded 更改这些设置 memory.soft_limit_in_bytes ,
memory.limit_in_bytes mem_limit , default_pool_mem_limit 价值 0 或者 -1 1或0表示无限制。

相关问题