Python 3,从/向gzip文件读/写压缩的json对象

ehxuflar  于 2022-11-26  发布在  Python
关注(0)|答案(3)|浏览(171)

对于Python3,我按照@Martijn Pieters的代码编写了以下代码:

import gzip
import json

# writing
with gzip.GzipFile(jsonfilename, 'w') as fout:
    for i in range(N):
        uid = "whatever%i" % i
        dv = [1, 2, 3]
        data = json.dumps({
            'what': uid,
            'where': dv})

        fout.write(data + '\n')

但这会导致错误:

Traceback (most recent call last):
    ...
  File "C:\Users\Think\my_json.py", line 118, in write_json
    fout.write(data + '\n')
  File "C:\Users\Think\Anaconda3\lib\gzip.py", line 258, in write
    data = memoryview(data)
TypeError: memoryview: a bytes-like object is required, not 'str'

有什么想法是怎么回事?

jljoyd4f

jljoyd4f1#

这里有四个转变步骤。

  1. Python数据结构(嵌套指令、列表、字符串、数字、布尔值)
    1.包含该数据结构的序列化表示的Python字符串(“JSON”)
    1.包含该字符串表示形式的字节列表(“UTF-8”)
    1.包含该先前字节列表(“gzip”)的较短表示的字节列表
    让我们一步一步来。
import gzip
import json

data = []
for i in range(N):
    uid = "whatever%i" % i
    dv = [1, 2, 3]
    data.append({
        'what': uid,
        'where': dv
    })                                           # 1. data

json_str = json.dumps(data) + "\n"               # 2. string (i.e. JSON)
json_bytes = json_str.encode('utf-8')            # 3. bytes (i.e. UTF-8)

with gzip.open(jsonfilename, 'w') as fout:       # 4. fewer bytes (i.e. gzip)
    fout.write(json_bytes)

请注意,添加"\n"在这里是完全多余的。它不会破坏任何东西,但除此之外它没有任何用处。我添加它只是因为您的代码示例中有它。
阅读的工作方式正好相反:

with gzip.open(jsonfilename, 'r') as fin:        # 4. gzip
    json_bytes = fin.read()                      # 3. bytes (i.e. UTF-8)

json_str = json_bytes.decode('utf-8')            # 2. string (i.e. JSON)
data = json.loads(json_str)                      # 1. data

print(data)

当然,这些步骤可以合并:

with gzip.open(jsonfilename, 'w') as fout:
    fout.write(json.dumps(data).encode('utf-8'))

with gzip.open(jsonfilename, 'r') as fin:
    data = json.loads(fin.read().decode('utf-8'))
yeotifhr

yeotifhr2#

here(感谢@Rafe)提到的解决方案有一个很大的优势:因为编码是动态完成的,所以你不需要为生成的json创建两个完整的中间字符串对象。

with gzip.open(jsonfilename, 'wt', encoding='UTF-8') as zipfile:
    json.dump(data, zipfile)

此外,阅读和解码也很简单:

with gzip.open(jsonfilename, 'rt', encoding='UTF-8') as zipfile:
    my_object = json.load(zipfile)
mnowg1ta

mnowg1ta3#

要写入json.gz,可以使用以下代码段:

import json
import gzip

with gzip.open("file_path_to_write", "wt") as f:
        json.dump(expected_dict, f)

要读取json.gz,可以使用以下代码片段:

import json
import gzip

with gzip.open("file_path_to_read", "rt") as f:
        expected_dict = json.load(f)

相关问题