我试图查看hdfs中的文件,并评估哪些文件比某个日期早。我想做一个 hdfs ls
把它的输出传给一头Pig LOAD
命令。
如何使用pig加载文件夹中的每个文件@donaldminer包含一个输出文件名的shell脚本;我借这个来传递一个文件名列表。但是,我不想加载文件的内容,我只想加载 ls
命令并将文件名视为文本。
这是myfirstscript.pig:
test = LOAD '$files' as (moddate:chararray, modtime:chararray, filename:chararray);
illustrate test;
我称之为:
pig -p files="`./filesysoutput.sh`" myfirstscript.pig
其中filesysoutput.sh包含:
hadoop fs -ls -R /hbase/imagestore | grep '\-rw' | awk 'BEGIN { FS = ",[ \t]*|[ \t]+" } {print $6, $7, $8}' | tr '\n' ','
这会产生如下输出:
2012-07-27 17:56 /hbase/imagestore/.tableinfo.0000000001,2012-04-23 19:27 /hbase/imagestore/08e36507d743367e1de57c359360b64c/.regioninfo,2012-05-10 12:13 /hbase/imagestore/08e36507d743367e1de57c359360b64c/0/7818124910159371133,2012-05-10 15:01 /hbase/imagestore/08e36507d743367e1de57c359360b64c/1/5537238047267916113,2012-05-09 19:40 /hbase/imagestore/08e36507d743367e1de57c359360b64c/2/6836317764645542272,2012-05-10 07:04 /hbase/imagestore/08e36507d743367e1de57c359360b64c/3/7276147895747401630,...
因为我想要的只是日期、时间和文件名,所以我只在输入到pig脚本的输出中包含这些字段。当我尝试运行它时,它肯定是在尝试将实际文件加载到 test
别名:
$ pig -p files="`./filesysoutput.sh`" myfirstscript.pig
2013-05-29 17:40:10.773 java[50830:1203] Unable to load realm info from SCDynamicStore
2013-05-29 17:40:10.827 java[50830:1203] Unable to load realm info from SCDynamicStore
2013-05-29 17:40:20,570 [main] INFO org.apache.pig.Main - Logging error messages to: /Users/username/Environment/pig-0.9.2-cdh4.0.1/scripts/test/pig_1369863620569.log
2013-05-29 17:40:20,769 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://stage-hadoop101.cluster:8020
2013-05-29 17:40:20,771 [main] WARN org.apache.hadoop.conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2013-05-29 17:40:20,773 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-05-29 17:40:20.836 java[50847:1203] Unable to load realm info from SCDynamicStore
2013-05-29 17:40:20.879 java[50847:1203] Unable to load realm info from SCDynamicStore
2013-05-29 17:40:21,138 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-05-29 17:40:21,452 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse:
<file myfirstscript.pig, line 3, column 7> pig script failed to validate: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: 2012-07-27 17:56%20/hbase/imagestore/.tableinfo.0000000001
Details at logfile: /Users/username/Environment/pig-0.9.2-cdh4.0.1/scripts/test/pig_1369863620569.log
2条答案
按热度按时间vshtjzan1#
使用基于java或python的嵌入式pig怎么样?
嵌入式清管器
uqzxnwby2#
您可以尝试另一种方法-使用dummy.txt输入文件(只有一行),然后使用
STREAM alias THROUGH
命令来处理hadoop fs -ls
你现在是:注意上面的内容没有经过测试-我模拟了一些与本地模式pig类似的东西,并且它工作正常(注意我在awk-ofs中添加了一些选项,并且不得不稍微更改grep):