这是我的密码:
public class WordCount {
public static class TokenizerMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
int j = 1, count = 0, i=0;
while (itr.hasMoreTokens()) {
if(j == 1)
{
itr.nextToken();
j++;
}
else if (j == 2)
{
count = Integer.parseInt(itr.nextToken());
j++;
}
else
{
for (i=1;i<=count;i++)
{
word.set(itr.nextToken());
context.write(word, one);
}
j=1;
}
}
}
}
public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
当我为一个包含64位大小数据集的35.4gb文件运行它时,它会显示以下错误:
Error: java.lang.NumberFormatException: For input string: "9267242563148306650"
此外,在添加biginteger之后,还会出现以下错误:
代码中所做的更改:
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
int j = 1, i=0;
int c;
BigInteger count = new BigInteger("0");
while (itr.hasMoreTokens()) {
if(j == 1)
{
itr.nextToken();
j++;
}
else if (j == 2)
{
count = new BigInteger(itr.nextToken());
j++;
}
else
{
c=count.intValue();
for (i=1;i<=c;i++)
{
word.set(itr.nextToken());
context.write(word, one);
}
j=1;
}
}
错误1:java.util.nosuchelementexception(主要由stringtokenizer执行)
错误2:java.lang.numberformatexception
我该如何着手解决这个问题?谢谢!
2条答案
按热度按时间68bkxrlz1#
你有一个整数溢出(
int
以及long
变量是有界的,有关详细信息,请参见以下链接),您必须使用BigInteger
类来处理如此庞大的数字。nue99wik2#
java long是一个有符号的64位整数,因此它可以容纳63位的数字。int是32位的。
问题是9267242563148306650不适合63位,但适合31位。所以当你试图解析数字时,它就是不合适。您需要使用biginteger之类的工具来存储这些值。