我正在使用apachephoenix运行一些查询,但是它们的性能与我期望的相比看起来很差。例如,考虑如下表格:
CREATE TABLE MY_SHORT_TABLE (
MPK BIGINT not null,
... 38 other columns ...
CONSTRAINT pk PRIMARY KEY (MPK, 4 other columns))
SALT_BUCKETS = 4;
大约有46万行,查询如下:
select sum(MST.VALUES),
MST.III, MST.BBB, MST.DDD, MST.FFF,
MST.AAA, MST.CCC, MST.EEE, MST.HHH
from
MY_SHORT_TABLE MST
group by
MST.AAA, MST.BBB, MST.CCC, MST.DDD,
MST.EEE, MST.FFF, MST.HHH, MST.III
大约需要9-11秒才能完成。在具有类似结构但有近340000行的表中,完成查询大约需要45秒。
我在这个集群中有5个主机(1个主服务器和4个regionserver+PhoenixQ),有6个vcpu和32gb ram。
我在本例中使用的配置是:
HBase RegionServer Maximum Memory=8192(8GB)
HBase Master Maximum Memory=8192(8GB)
Number of Handlers per RegionServer=30
Memstore Flush Size=128MB
Maximum Record Size=1MB
Maximum Region File Size=10GB
% of RegionServer Allocated to Read Buffers=40%
% of RegionServer Allocated to Write Buffers=40%
HBase RPC Timeout=6min
Zookeeper Session Timeout=6min
Phoenix Query Timeout=6min
Number of Fetched Rows when Scanning from Disk=1000
dfs.client.read.shortcircuit=true
dfs.client.read.shortcircuit.buffer.size=131072
phoenix.coprocessor.maxServerCacheTimeToLiveMs=30000
我使用的是HDP2.4.0,所以是phoenix 4.4。
查询解释示例如下:
+------------------------------------------+
| PLAN |
+------------------------------------------+
| CLIENT 8-CHUNK PARALLEL 8-WAY FULL SCAN OVER MY_SHORT_TABLE |
| SERVER AGGREGATE INTO DISTINCT ROWS BY [AAA, BBB, CCC, DDD, EEE, FFF, HHH |
| CLIENT MERGE SORT |
+------------------------------------------+
此外,我还创建了一个索引:
CREATE INDEX i1DENORM2T1 ON MY_SHORT_TABLE (HHH)
INCLUDE ( AAA, BBB, CCC, DDD, EEE, FFF, HHH, VALUES ) ;
此索引将查询执行计划更改为:
+------------------------------------------+
| PLAN |
+------------------------------------------+
| CLIENT 4-CHUNK PARALLEL 4-WAY FULL SCAN OVER I1DENORM2T1 |
| SERVER AGGREGATE INTO DISTINCT ROWS BY ["AAA", "BBB", "DDD", "EEE", "FFF", "HHH |
| CLIENT MERGE SORT |
+------------------------------------------+
但是,性能与预期不符(大约3-4秒)。
上面的配置有什么问题?为了获得更好的性能,我应该更改什么?
提前谢谢。
暂无答案!
目前还没有任何答案,快来回答吧!