hbase exportsnapshot从0.94到0.98

dgjrabp2 于 2021-05-30 发布在 Hadoop

关注(0)|答案(3)|浏览(383)

我们有两个hbase为0.94的集群，hadoop 1.04和hbase 0.98，hadoop 2.4我已经从表中的0.94快照创建了一个快照，并希望将其迁移到hbase为0.98的集群。
在0.98群集上运行此命令后：

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshot-name -copy-from webhdfs://hadoops-master:9000/hbase -copy-to hdfs://solr1:8020/hbase

我懂了：

Exception in thread "main" org.apache.hadoop.hbase.snapshot.ExportSnapshotException: Failed to copy the snapshot directory: from=webhdfs://hadoops-master:9000/hbase/.hbase-snapshot/snapshot-name to=hdfs://solr1:8020/hbase/.hbase-snapshot/.tmp/snapshot-name
    at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:916)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:1000)
    at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:1004)
Caused by: java.net.SocketException: Unexpected end of file from server
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
    at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:596)
    at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:530)
    at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:417)
    at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:630)
    at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:641)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
    at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:914)
    ... 3 more

hadoop hbase snapshot

来源：https://stackoverflow.com/questions/27485811/hbase-exportsnapshot-from-0-94-to-0-98

3条答案

按热度按时间

ylamdve61#

我已经用jarjar重新打包了hadoop和hbase libshttps://code.google.com/p/jarjar/ 它需要修复一些类名问题。然后我写了一个mapreduce copytable作业。它从94 culster读取行，并写入98集群。代码如下：https://github.com/fiserro/copy-table-94to98
感谢github.com/falsecz的建议和帮助！

赞(0）回复(0）举报 2021-05-30

yqkkidmi2#

我正在深入研究这个问题，它与底层hdfs有更多的关系。
一旦流（在我的distcp示例中）被写入，就调用close：

public void close() throws IOException {
try {
      super.close();
    } finally {
      try {
        validateResponse(op, conn, true);
      } finally {
        conn.disconnect();
      }
    }
  }

其中，它在validate响应调用中失败（可能连接的另一端已关闭）。
这可能是由于hdfs 1.0和2.4之间的不兼容！

赞(0）回复(0）举报 2021-05-30

oknwwptz3#

lars hofhansl本人（hbase的主要提交人和0.94的维护者）曾表示，0.94和0.98不支持出口。所以你现在很可能带着这个去了。
以下是他今天下午的一篇文章摘录：

It's tricky from multiple angles:- replication between 0.94 and 0.98 does not work (there's a gateway process that supposedly does that, but it's not known to be very reliable)- snapshots cannot be exported from 0.94 and 0.98

来源：hbase用户的邮件列表今天12/15/14
在hbase邮件列表上更新有一个报告，其中一个用户能够找到进行导出的方法。其中一条信息是：

If exporting from an HBase 0.94 cluster to an HBase 0.98 cluster, you will need to use the webhdfs protocol (or possibly hftp, though I couldn’t get that to work). You also need to manually move some files around because snapshot layouts have changed. Based on the example above, on the 0.98 cluster do the following:

   check whether any imports already exist for the table:

  hadoop fs -ls /apps/hbase/data/archive/data/default

我在这里复制整个讨论是不正确的：包含全部血淋淋细节的nabble链接是：
http://apache-hbase.679495.n3.nabble.com/0-94-going-forward-td4066883.html

赞(0）回复(0）举报 2021-05-30

我来回答

hbase exportsnapshot从0.94到0.98

3条答案

相关问题

热门标签

最新问答