我有一个简短的python脚本:
import langid
import sys
for pig_tuple in sys.stdin:
cols = pig_tuple.split()
if len(cols) < 2:
sys.exit(0)
try:
id = int(cols[0])
text = " ".join(cols[1:])
except:
sys.exit(0)
(lang,prob) = langid.classify(text)
print "%s\t%s" %(id,lang)
sys.exit(0)
我想在Pig脚本中运行它。我试过:
define langid_cmd `python2.6 /data/test/compiled_python/langid_command_line.py` ship('/data/test/compiled_python/langid_command_line.py');
text = LOAD '$PIG_INPUT' USING PigStorage() as (text:chararray);
pythonDetect1 = STREAM text through langid_cmd AS (pid:chararray,planguage:chararray);
但我得到:
2013-03-29 15:53:22,290 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2013-03-29 15:53:22,303 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2999: Unexpected internal error. java.lang.String cannot be cast to org.apache.pig.data.Tuple
Details at logfile: /home/isl/ryan/src/main/pigScripts/pig_1364597410350.log
2013-03-29 15:53:22,306 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2999: Unexpected internal error. java.lang.String cannot be cast to org.apache.pig.data.Tuple
Details at logfile: /home/isl/ryan/src/main/pigScripts//src/main/pigScripts/pig_1364597410350.log
2013-03-29 15:53:22,308 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2999: Unexpected internal error. java.lang.String cannot be cast to org.apache.pig.data.Tuple
Details at logfile: /home/isl/ryan/src/main/pigScripts//src/main/pigScripts/pig_1364597410350.log
2013-03-29 15:53:22,311 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2999: Unexpected internal error. java.lang.String cannot be cast to org.apache.pig.data.Tuple
Details at logfile: /home/isl/ryan/src/main/pigScripts/src/main/pigScripts/pig_1364597410350.log
2013-03-29 15:53:22,313 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2999: Unexpected internal error. java.lang.String cannot be cast to org.apache.pig.data.Tuple
Details at logfile: /home/isl/ryan/src/main/pigScripts/src/main/pigScripts/pig_1364597410350.log
目录/data/test/compiled\u python是chmod'd到777的,当我从shell运行这个命令时:
-bash-3.2$ echo 14353 I can haz pigscriptz? | python /data/test/compiled_python/langid_command_line.py
14353 eu
??
1条答案
按热度按时间u3r8eeie1#
这个
AS (pid:chararray,planguage:chararray)
告诉pig预期的输出是字符串的元组,但返回制表符分隔的字符串。你应该把结果打印出来或者使用python-udf集成