使用sqoop将Parquet格式的数据导出到mysql

sh7euo9m  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(660)

我将一些数据从mysql导入hdfs,并以parquet格式保存:

drwxr-xr-x   - root supergroup          0 2016-12-16 22:19 /wybdaily/test/zsqoop/.metadata-00000
drwxr-xr-x   - root supergroup          0 2016-12-16 22:20 /wybdaily/test/zsqoop/.signals-00000
-rw-r--r--   1 root supergroup       2056 2016-12-16 22:20 /wybdaily/test/zsqoop/19dfc82b-eb01-4395-a847-aa11390306fe.parquet

然后我尝试将这些数据导出回mysql,出现异常:

16/12/16 22:21:09 WARN spi.Registration: Not loading URI patterns in org.kitesdk.data.spi.hive.Loader
16/12/16 22:21:09 ERROR sqoop.Sqoop: Got exception running Sqoop: org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist:     hdfs://iZbp1ig01gathmdv6uxz3wZ:8020/wybdaily/test/zsqoop/.metadata
org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist: hdfs://iZbp1ig01gathmdv6uxz3wZ:8020/wybdaily/test/zsqoop/.metadata
    at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.checkExists(FileSystemMetadataProvider.java:562)
    at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.find(FileSystemMetadataProvider.java:605)
    at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.load(FileSystemMetadataProvider.java:114)
    at org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.load(FileSystemDatasetRepository.java:197)
    at org.kitesdk.data.Datasets.load(Datasets.java:108)
    at org.kitesdk.data.Datasets.load(Datasets.java:140)
    at org.kitesdk.data.mapreduce.DatasetKeyInputFormat$ConfigBuilder.readFrom(DatasetKeyInputFormat.java:92)
    at org.kitesdk.data.mapreduce.DatasetKeyInputFormat$ConfigBuilder.readFrom(DatasetKeyInputFormat.java:139)
    at org.apache.sqoop.mapreduce.JdbcUpdateExportJob.configureInputFormat(JdbcUpdateExportJob.java:192)
    at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:432)
    at org.apache.sqoop.manager.MySQLManager.upsertTable(MySQLManager.java:145)
    at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:74)
    at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

然后我将“.metadata-00000”重命名为“.metadata”,作业也失败了。
任何人都可以告诉我如何使用sqoop1将Parquet格式的数据导出到mysql?一个例子或一些文档。谢谢!

qyuhtwio

qyuhtwio1#

通过使用hcatalog,我们可以将Parquet配置单元表导出到rdbms中。
sqoop导出--连接jdbc:oracle:thin:@//192.178.100.15:1521--用户名用户名--密码--表db.table\u name--hcatalog表“table\u name”--hcatalog数据库“db\u name”

相关问题