我正在尝试使用python的parquet模块读入一些从本地mapr示例编写的parquet文件。
我用来输出这些Parquet文件的命令是:
df.sqlContext.sql("SQL HERE").write.format("parquet").option("mergeSchema", "true").save("/path/to/parquet/test.parquet")
这是我的linux主机上文件的外观:
drwxr-xr-x 2 mapr mapr 403 Oct 5 13:56 igayfvpwrs.parquet
不幸的是,当我在这里使用python时(https://pypi.python.org/pypi/parquet)-我收到以下异常:
IOError: [Errno 21] Is a directory: '/mnt/mapr/saw/user/mapr/igayfvpwrs.parquet'
你知道吗?这些文件在mapr中工作得很好。
编辑2:
我终于弄明白了。因为原始的.parquet“文件”是一个目录,所以只需使用glob在目录中循环所有内部的.parquet文件,python parquet的原始代码就可以在其中工作。
for filename in glob.glob("/mnt/mapr/saw/user/mapr/{0}.parquet/*.parquet".format(tempTableID)):
with open(filename) as foo:
for row in parquet.DictReader(foo, columns=["column"]):
print(json.dumps(row))
编辑:以下是Parquet文件中的内容:
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 _common_metadata
-rwxr-xr-x 1 mapr mapr 2.4K Oct 5 13:58 _metadata
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00000-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00001-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00002-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00003-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00004-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00005-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00006-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00007-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00008-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00009-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00010-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00011-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00012-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00013-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00014-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00015-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00016-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00017-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00018-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00019-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00020-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00021-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00022-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00023-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00024-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00025-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00026-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00027-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00028-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00029-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00030-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00031-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00032-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00033-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00034-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00035-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00036-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00037-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00038-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00039-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00040-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00041-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00042-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00043-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00044-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00045-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00046-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00047-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00048-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00049-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00050-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00051-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00052-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 1.2K Oct 5 13:58 part-r-00053-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00054-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00055-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00056-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00057-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00058-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00059-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00060-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00061-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00062-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00063-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00064-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00065-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00066-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00067-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00068-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00069-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00070-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00071-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00072-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00073-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00074-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00075-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00076-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00077-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00078-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00079-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00080-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00081-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00082-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00083-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00084-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00085-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00086-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00087-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00088-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00089-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00090-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00091-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00092-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00093-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00094-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00095-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00096-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00097-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00098-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00099-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00100-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00101-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00102-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00103-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00104-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 1.2K Oct 5 13:58 part-r-00105-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00106-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00107-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00108-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00109-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00110-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00111-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00112-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00113-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00114-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00115-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00116-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00117-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00118-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00119-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00120-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00121-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00122-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00123-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00124-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 1.2K Oct 5 13:58 part-r-00125-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00126-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00127-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00128-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00129-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00130-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00131-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00132-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00133-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00134-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00135-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00136-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00137-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00138-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00139-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00140-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00141-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00142-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00143-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00144-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00145-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00146-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00147-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00148-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00149-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00150-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00151-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00152-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00153-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00154-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00155-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00156-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00157-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00158-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00159-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00160-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00161-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00162-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00163-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00164-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00165-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00166-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00167-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00168-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00169-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00170-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00171-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00172-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00173-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00174-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00175-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00176-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00177-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00178-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00179-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00180-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00181-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00182-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00183-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00184-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00185-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00186-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00187-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00188-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00189-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00190-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00191-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00192-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00193-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00194-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00195-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00196-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00197-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00198-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00199-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00200-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00201-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00202-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00203-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00204-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00205-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00206-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00207-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00208-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00209-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00210-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00211-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00212-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00213-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00214-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00215-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00216-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00217-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00218-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00219-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00220-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00221-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00222-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00223-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00224-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00225-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00226-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00227-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00228-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00229-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00230-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00231-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00232-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00233-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00234-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00235-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00236-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00237-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00238-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00239-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00240-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00241-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00242-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00243-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00244-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet
-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00245-02f0b80b-2993-46f8-
[truncated]
-rwxr-xr-x 1 mapr mapr 0 Oct 5 13:58 _SUCCESS
2条答案
按热度按时间vm0i2vca1#
根据ls命令的输出,它看起来像igayfvpwrs.parquet实际上是一个目录。你能查一下里面的数据吗?
7gcisfzg2#
Spark上的 parquet 没有问题。dataframewriter将parquet格式写入一个目录,并根据它正在写入的dataframe的分区数对输出进行分区。
你得到的是绝对正常的。