我正在做一些单线程压缩编解码器的基准测试,我看到zlib的性能似乎比单线程的性能要高得多。我用过 org.apache.hadoop.io.compress.zlib.ZlibCompressor
对于zlib压缩机的实现,以及 java.util.zip.Deflate
用于gzip实现的比较。
在hadoop中,zlib compressor(wrapper)是以多线程的方式提供的吗,也许是通过jni接口提供的?
兹利布:
import org.apache.hadoop.io.compress.zlib.*;
protected final zlibCompressor = new ZlibCompressor(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION, ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY, ZlibCompressor.CompressionHeader.DEFAULT_HEADER, DEFAULT_BUFFER_SIZE);
protected final zlibDecompressor = new ZlibDecompressor(ZlibDecompressor.CompressionHeader.DEFAULT_HEADER, DEFAULT_BUFFER_SIZE);
//compress
zlibCompressor.setInput(uncompressed, 0, uncompressed.length);
zlibCompressor.finish();
int n = zlibCompressor.compress(compressBuffer, 0, compressBuffer.length);
//decompress
zlibCompressor.reset();
zlibDecompressor.setInput(compressed, 0, compressed.length);
int n = zlibDecompressor.decompress(uncompressBuffer, 0, uncompressBuffer.length);
gzip公司:
import java.util.zip.*;
protected final deflater = new Deflater(COMPRESSION_LEVEL, NO_WRAP);
protected final inflater = new Inflater(NO_WRAP);
//compress
int n = compressBlockUsingStream(uncompressed, compressBuffer);
//decompress
inflater.reset();
int n = uncompressBlockUsingStream(new InflaterInputStream(new ByteArrayInputStream(compressed), _inflater), uncompressBuffer);
gzip的助手功能:
protected int compressBlockUsingStream(byte[] uncompressed, byte[] compressBuffer) throws IOException
{
ByteArrayOutputStream out = new ByteArrayOutputStream(compressBuffer);
compressToStream(uncompressed, out);
return out.length();
}
protected int uncompressBlockUsingStream(InputStream in, byte[] uncompressBuffer) throws IOException
{
ByteArrayOutputStream out = new ByteArrayOutputStream(uncompressBuffer);
byte[] buffer = new byte[4096];
int count;
while ((count = in.read(buffer)) >= 0) {
out.write(buffer, 0, count);
}
in.close();
out.close();
return out.length();
}
吞吐量:
zlib/块—143.902 mbps
gzip/jdk/stream—22.573 mbps
有人知道为什么zlib要快得多(在本地使用所有内核)?代码应运行单线程。有人能复制类似的结果吗?
1条答案
按热度按时间2j4z5cfb1#
java.util.zip使用zlib。
你确定这两个版本的压缩级别相同吗?是
COMPRESSION_LEVEL
等于ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION
?