pig与cassandra集成:简单的分布式查询需要几分钟才能完成这正常吗?

cngwdvgl  于 2021-06-21  发布在  Pig
关注(0)|答案(1)|浏览(289)

我建立了一个cassandra+pig/hadoop的测试集成。8个节点是cassandra+tasktracker节点,1个节点是jobtracker/namenode。
我启动了cassandra客户端,并创建了一个简单的代码,其中包含cassandra发行版中readme.txt中列出的数据:

[default@unknown] create keyspace Keyspace1;
  [default@unknown] use Keyspace1;
  [default@Keyspace1] create column family Users with comparator=UTF8Type and default_validation_class=UTF8Type and key_validation_class=UTF8Type;
  [default@KS1] set Users[jsmith][first] = 'John';
  [default@KS1] set Users[jsmith][last] = 'Smith';
  [default@KS1] set Users[jsmith][age] = long(42)

然后我运行了cassandra\u home中列出的示例pig查询(使用pig\u cassandra):

grunt> rows = LOAD 'cassandra://Keyspace1/Users' USING CassandraStorage() AS (key, columns: bag {T: tuple(name, value)});
grunt> cols = FOREACH rows GENERATE flatten(columns);
grunt> colnames = FOREACH cols GENERATE $0;
grunt> namegroups = GROUP colnames BY (chararray) $0;
grunt> namecounts = FOREACH namegroups GENERATE COUNT($1), group;
grunt> orderednames = ORDER namecounts BY $0;
grunt> topnames = LIMIT orderednames 50;
grunt> dump topnames;

大约花了3分钟完成。

HadoopVersion   PigVersion      UserId  StartedAt                FinishedAt                            Features
    1.0.0             0.9.1          root    2012-01-12      22:16:53     2012-01-12 22:20:22     GROUP_BY,ORDER_BY,LIMIT
Success!

Job Stats (time in seconds):
JobId   Maps    Reduces MaxMapTime      MinMapTIme      AvgMapTime      MaxReduceTime   MinReduceTime   AvgReduceTime   Alias   Feature Outputs
job_201201121817_0010   8       1       12      6       9       21      21      21      colnames,cols,namecounts,namegroups,rows        GROUP_BY,COMBINER       
job_201201121817_0011   1       1       6       6       6       15      15      15      orderednames    SAMPLER 
job_201201121817_0012   1       1       9       9       9       15      15      15      orderednames    ORDER_BY,COMBINER       hdfs://xxxx/tmp/temp-744158198/tmp-1598279340,

Input(s):
Successfully read 1 records (3232 bytes) from: "cassandra://Keyspace1/Users"

Output(s):
Successfully stored 3 records (63 bytes) in: "hdfs://xxxx/tmp/temp-744158198/tmp-1598279340"

Counters:
Total records written : 3
Total bytes written : 63
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

日志记录中没有错误或警告。
这是正常的,还是有什么问题?

z6psavjg

z6psavjg1#

是的,这是正常的,因为在hadoop上运行map/reduce作业通常只需要1分钟就可以启动。pig根据脚本的复杂性生成多个map/reduce作业。

相关问题