首先从每行中删除重复的单词，然后运行单词计数。
清管器步骤：

REGISTER 'udf-1.0-SNAPSHOT.jar'
define tuple_set com.ts.pig.UniqueRecords();
data = load '<file>' using PigStorage();

删除每行中的重复单词

unique= foreach data generate tuple_set($0) as line;
words= foreach unique generate flatten(TOKENIZE(line,' ')) as word;
grouped = group words BY word;
count= foreach grouped GENERATE group, COUNT(words);
dump count;

pig自定义项示例代码：

/**
 * This udf removes duplicate words from line
 */
public class UniqueRecords extends EvalFunc<String> {
    @Override
    public String exec(Tuple tuple) throws IOException {
        if (tuple == null || tuple.size() == 0)
            return null;
        String[] splits=tuple.get(0).toString().split(" ");
        Set<String> elements = new HashSet<String>(Arrays.asList(splits));
        StringBuilder sb = new StringBuilder();
        for(String element:elements ){
            sb.append(element+" ");
        }
        return sb.toString();
    }
}