我将一些数据从mysql导入hdfs,并以parquet格式保存:
drwxr-xr-x - root supergroup 0 2016-12-16 22:19 /wybdaily/test/zsqoop/.metadata-00000
drwxr-xr-x - root supergroup 0 2016-12-16 22:20 /wybdaily/test/zsqoop/.signals-00000
-rw-r--r-- 1 root supergroup 2056 2016-12-16 22:20 /wybdaily/test/zsqoop/19dfc82b-eb01-4395-a847-aa11390306fe.parquet
然后我尝试将这些数据导出回mysql,出现异常:
16/12/16 22:21:09 WARN spi.Registration: Not loading URI patterns in org.kitesdk.data.spi.hive.Loader
16/12/16 22:21:09 ERROR sqoop.Sqoop: Got exception running Sqoop: org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist: hdfs://iZbp1ig01gathmdv6uxz3wZ:8020/wybdaily/test/zsqoop/.metadata
org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist: hdfs://iZbp1ig01gathmdv6uxz3wZ:8020/wybdaily/test/zsqoop/.metadata
at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.checkExists(FileSystemMetadataProvider.java:562)
at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.find(FileSystemMetadataProvider.java:605)
at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.load(FileSystemMetadataProvider.java:114)
at org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.load(FileSystemDatasetRepository.java:197)
at org.kitesdk.data.Datasets.load(Datasets.java:108)
at org.kitesdk.data.Datasets.load(Datasets.java:140)
at org.kitesdk.data.mapreduce.DatasetKeyInputFormat$ConfigBuilder.readFrom(DatasetKeyInputFormat.java:92)
at org.kitesdk.data.mapreduce.DatasetKeyInputFormat$ConfigBuilder.readFrom(DatasetKeyInputFormat.java:139)
at org.apache.sqoop.mapreduce.JdbcUpdateExportJob.configureInputFormat(JdbcUpdateExportJob.java:192)
at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:432)
at org.apache.sqoop.manager.MySQLManager.upsertTable(MySQLManager.java:145)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:74)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
然后我将“.metadata-00000”重命名为“.metadata”,作业也失败了。
任何人都可以告诉我如何使用sqoop1将Parquet格式的数据导出到mysql?一个例子或一些文档。谢谢!
1条答案
按热度按时间qyuhtwio1#
通过使用hcatalog,我们可以将Parquet配置单元表导出到rdbms中。
sqoop导出--连接jdbc:oracle:thin:@//192.178.100.15:1521--用户名用户名--密码--表db.table\u name--hcatalog表“table\u name”--hcatalog数据库“db\u name”