hadoopzlib和jdkgzip的多线程性能比较

aemubtdh  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(496)

我正在做一些单线程压缩编解码器的基准测试,我看到zlib的性能似乎比单线程的性能要高得多。我用过 org.apache.hadoop.io.compress.zlib.ZlibCompressor 对于zlib压缩机的实现,以及 java.util.zip.Deflate 用于gzip实现的比较。
在hadoop中,zlib compressor(wrapper)是以多线程的方式提供的吗,也许是通过jni接口提供的?
兹利布:

import org.apache.hadoop.io.compress.zlib.*;
protected final zlibCompressor = new ZlibCompressor(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION, ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY, ZlibCompressor.CompressionHeader.DEFAULT_HEADER, DEFAULT_BUFFER_SIZE);
protected final zlibDecompressor = new ZlibDecompressor(ZlibDecompressor.CompressionHeader.DEFAULT_HEADER, DEFAULT_BUFFER_SIZE);

//compress
zlibCompressor.setInput(uncompressed, 0, uncompressed.length);
zlibCompressor.finish();
int n = zlibCompressor.compress(compressBuffer, 0, compressBuffer.length);

//decompress
zlibCompressor.reset();
zlibDecompressor.setInput(compressed, 0, compressed.length);
int n = zlibDecompressor.decompress(uncompressBuffer, 0, uncompressBuffer.length);

gzip公司:

import java.util.zip.*;
protected final deflater = new Deflater(COMPRESSION_LEVEL, NO_WRAP);
protected final inflater = new Inflater(NO_WRAP);

//compress
int n = compressBlockUsingStream(uncompressed, compressBuffer);

//decompress
inflater.reset();
int n = uncompressBlockUsingStream(new InflaterInputStream(new ByteArrayInputStream(compressed), _inflater), uncompressBuffer);

gzip的助手功能:

protected int compressBlockUsingStream(byte[] uncompressed, byte[] compressBuffer) throws IOException
{
        ByteArrayOutputStream out = new ByteArrayOutputStream(compressBuffer);
        compressToStream(uncompressed, out);
        return out.length();
}    

protected int uncompressBlockUsingStream(InputStream in, byte[] uncompressBuffer) throws IOException
{
            ByteArrayOutputStream out = new ByteArrayOutputStream(uncompressBuffer);
            byte[] buffer = new byte[4096];
            int count;
            while ((count = in.read(buffer)) >= 0) {
                out.write(buffer, 0, count);
            }
            in.close();
            out.close();
            return out.length();
}

吞吐量:
zlib/块—143.902 mbps
gzip/jdk/stream—22.573 mbps
有人知道为什么zlib要快得多(在本地使用所有内核)?代码应运行单线程。有人能复制类似的结果吗?

2j4z5cfb

2j4z5cfb1#

java.util.zip使用zlib。
你确定这两个版本的压缩级别相同吗?是 COMPRESSION_LEVEL 等于 ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION ?

相关问题