处理64位数据集的大数据文件的修改wordcount java程序时出现异常

wj8zmpe1  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(319)

这是我的密码:

public class WordCount {

public static class TokenizerMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
   StringTokenizer itr = new StringTokenizer(value.toString());
    int j = 1, count = 0, i=0;
  while (itr.hasMoreTokens()) {

    if(j == 1)
    {
    itr.nextToken();
    j++;
    }
    else if (j == 2)
    {
    count = Integer.parseInt(itr.nextToken());
    j++;
    }
    else
    {
    for (i=1;i<=count;i++)
    {
    word.set(itr.nextToken());
    context.write(word, one);
    }
    j=1;
    }       
  }

}
}

public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
  int sum = 0;
  for (IntWritable val : values) {
    sum += val.get();
  }
  result.set(sum);
  context.write(key, result);
}
}

当我为一个包含64位大小数据集的35.4gb文件运行它时,它会显示以下错误:

Error: java.lang.NumberFormatException: For input string: "9267242563148306650"

此外,在添加biginteger之后,还会出现以下错误:
代码中所做的更改:

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
   StringTokenizer itr = new StringTokenizer(value.toString());
    int j = 1, i=0;
    int c;
    BigInteger count = new BigInteger("0");
  while (itr.hasMoreTokens()) {

    if(j == 1)
    {
    itr.nextToken();
    j++;
    }
    else if (j == 2)
    {
    count = new BigInteger(itr.nextToken());
    j++;
    }
    else
    {
    c=count.intValue();
    for (i=1;i<=c;i++)
    {
    word.set(itr.nextToken());
    context.write(word, one);
    }
    j=1;
    }       
  }

错误1:java.util.nosuchelementexception(主要由stringtokenizer执行)
错误2:java.lang.numberformatexception
我该如何着手解决这个问题?谢谢!

68bkxrlz

68bkxrlz1#

你有一个整数溢出( int 以及 long 变量是有界的,有关详细信息,请参见以下链接),您必须使用 BigInteger 类来处理如此庞大的数字。

nue99wik

nue99wik2#

java long是一个有符号的64位整数,因此它可以容纳63位的数字。int是32位的。
问题是9267242563148306650不适合63位,但适合31位。所以当你试图解析数字时,它就是不合适。您需要使用biginteger之类的工具来存储这些值。

相关问题