将JSON/dict转换为具有指示符标记的扁平字符串

1wnzp6jl 于 2023-05-02 发布在其他

关注(0)|答案(2)|浏览(178)

bounty还有7小时到期。回答此问题可获得+50声望奖励。alvas希望引起更多关注这个问题。

给定一个输入，如：

{'example_id': 0,
 'query': ' revent 80 cfm',
 'query_id': 0,
 'product_id': 'B000MOO21W',
 'product_locale': 'us',
 'esci_label': 'I',
 'small_version': 0,
 'large_version': 1,
 'split': 'train',
 'product_title': 'Panasonic FV-20VQ3 WhisperCeiling 190 CFM Ceiling Mounted Fan',
 'product_description': None,
 'product_bullet_point': 'WhisperCeiling fans feature a totally enclosed condenser motor and a double-tapered, dolphin-shaped bladed blower wheel to quietly move air\nDesigned to give you continuous, trouble-free operation for many years thanks in part to its high-quality components and permanently lubricated motors which wear at a slower pace\nDetachable adaptors, firmly secured duct ends, adjustable mounting brackets (up to 26-in), fan/motor units that detach easily from the housing and uncomplicated wiring all lend themselves to user-friendly installation\nThis Panasonic fan has a built-in damper to prevent backdraft, which helps to prevent outside air from coming through the fan\n0.35 amp',
 'product_brand': 'Panasonic',
 'product_color': 'White'}

目标是输出如下内容：

Panasonic FV-20VQ3 WhisperCeiling 190 CFM Ceiling Mounted Fan [TITLE] Panasonic [BRAND] White [COLOR] WhisperCeiling fans feature a totally enclosed condenser motor and a double-tapered, dolphin-shaped bladed blower wheel to quietly move air [SEP] Designed to give you continuous, trouble-free operation for many years thanks in part to its high-quality components and permanently lubricated motors which wear at a slower pace [SEP] Detachable adaptors, firmly secured duct ends, adjustable mounting brackets (up to 26-in), fan/motor units that detach easily from the housing and uncomplicated wiring all lend themselves to user-friendly installation [SEP] This Panasonic fan has a built-in damper to prevent backdraft, which helps to prevent outside air from coming through the fan [SEP] 0.35 amp [BULLETPOINT]

有几个操作可以按照规则生成所需的输出：

如果字典中的值为None，则不将内容添加到输出字符串中
如果值包含换行符\n，则将其替换为[SEP]标记
按照用户指定的顺序连接字符串。例如，上面的公式遵循["product_title", "product_brand", "product_color", "product_bullet_point", "product_description"]的顺序

我已经尝试过这种方法，但我写的函数看起来有点硬编码，以查看想要的键并连接和操作字符串。

item1 = {'example_id': 0,
 'query': ' revent 80 cfm',
 'query_id': 0,
 'product_id': 'B000MOO21W',
 'product_locale': 'us',
 'esci_label': 'I',
 'small_version': 0,
 'large_version': 1,
 'split': 'train',
 'product_title': 'Panasonic FV-20VQ3 WhisperCeiling 190 CFM Ceiling Mounted Fan',
 'product_description': None,
 'product_bullet_point': 'WhisperCeiling fans feature a totally enclosed condenser motor and a double-tapered, dolphin-shaped bladed blower wheel to quietly move air\nDesigned to give you continuous, trouble-free operation for many years thanks in part to its high-quality components and permanently lubricated motors which wear at a slower pace\nDetachable adaptors, firmly secured duct ends, adjustable mounting brackets (up to 26-in), fan/motor units that detach easily from the housing and uncomplicated wiring all lend themselves to user-friendly installation\nThis Panasonic fan has a built-in damper to prevent backdraft, which helps to prevent outside air from coming through the fan\n0.35 amp',
 'product_brand': 'Panasonic',
 'product_color': 'White'}

item2 = {'example_id': 198,
 'query': '# 2 pencils not sharpened',
 'query_id': 6,
 'product_id': 'B08KXRY4DG',
 'product_locale': 'us',
 'esci_label': 'S',
 'small_version': 1,
 'large_version': 1,
 'split': 'train',
 'product_title': 'AHXML#2 HB Wood Cased Graphite Pencils, Pre-Sharpened with Free Erasers, Smooth write for Exams, School, Office, Drawing and Sketching, Pack of 48',
 'product_description': "<b>AHXML#2 HB Wood Cased Graphite Pencils, Pack of 48</b><br><br>Perfect for Beginners experienced graphic designers and professionals, kids Ideal for art supplies, drawing supplies, sketchbook, sketch pad, shading pencil, artist pencil, school supplies. <br><br><b>Package Includes</b><br>- 48 x Sketching Pencil<br> - 1 x Paper Boxed packaging<br><br>Our high quality, hexagonal shape is super lightweight and textured, producing smooth marks that erase well, and do not break off when you're drawing.<br><br><b>If you have any question or suggestion during using, please feel free to contact us.</b>",
 'product_bullet_point': '#2 HB yellow, wood-cased pencils:Box of 48 count. Made from high quality real poplar wood and 100% genuine graphite pencil core. These No 2 pencils come with 100% Non-Toxic latex free pink top erasers.\nPRE-SHARPENED & EASY SHARPENING: All the 48 count pencils are pre-sharpened, ready to use when get it, saving your time of preparing.\nThese writing instruments are hexagonal in shape to ensure a comfortable grip when writing, scribbling, or doodling.\nThey are widely used in daily writhing, sketching, examination, marking, and more, especially for kids and teen writing in classroom and home.#2 HB wood-cased yellow pencils in bulk are ideal choice for school, office and home to maintain daily pencil consumption.\nCustomer service:If you are not satisfied with our product or have any questions, please feel free to contact us.',
 'product_brand': 'AHXML',
 'product_color': None}

def product2str(row, keys):
    key2token = {'product_title': '[TITLE]', 
     'product_brand': '[BRAND]', 
     'product_color': '[COLOR]',
     'product_bullet_point': '[BULLETPOINT]', 
     'product_description': '[DESCRIPTION]'}
    
    output = ""
    for k in keys:
        content = row[k]
        if content:
            output += content.replace('\n', ' [SEP] ') + f" {key2token[k]} "

    return output.strip()

product2str(item2, keys=['product_title', 'product_brand', 'product_color',
                        'product_bullet_point', 'product_description'])

问：是否有某种原生的CPython JSON来str flatten函数/配方，可以实现与product2str函数类似的结果？
问：或者在tokenizers库https://pypi.org/project/tokenizers/中已经有一些函数/管道可以将JSON/dict扁平化为令牌？

JSON

来源：https://stackoverflow.com/questions/76076286/converting-json-dict-to-flatten-string-with-indicator-tokens

2条答案

按热度按时间

7cjasjjr1#

所以我做了这个函数它可以完成你要求的

def flatten_dict(d, key_order):
    tokens = {
        "product_title": "[TITLE]",
        "product_brand": "[BRAND]",
        "product_color": "[COLOR]",
        "product_description": "[DESCRIPTION]",
        "product_bullet_point": "[BULLETPOINT]",
        # put your others token types here
    }
    parts = []
    for key in key_order:
        if key in d and d[key] is not None:
            parts.append(f"{d[key]} {tokens[key]}")
    return " ".join(parts)

item1 = {
    'example_id': 0,
    'query': ' revent 80 cfm',
    'query_id': 0,
    'product_id': 'B000MOO21W',
    'product_locale': 'us',
    'esci_label': 'I',
    'small_version': 0,
    'large_version': 1,
    'split': 'train',
    'product_title': 'Panasonic FV-20VQ3 WhisperCeiling 190 CFM Ceiling Mounted Fan',
    'product_description': None,
    'product_bullet_point': 'WhisperCeiling fans feature a totally enclosed condenser motor and a double-tapered, dolphin-shaped bladed blower wheel to quietly move air\nDesigned to give you continuous, trouble-free operation for many years thanks in part to its high-quality components and permanently lubricated motors which wear at a slower pace\nDetachable adaptors, firmly secured duct ends, adjustable mounting brackets (up to 26-in), fan/motor units that detach easily from the housing and uncomplicated wiring all lend themselves to user-friendly installation\nThis Panasonic fan has a built-in damper to prevent backdraft, which helps to prevent outside air from coming through the fan\n0.35 amp',
    'product_brand': 'Panasonic',
    'product_color': 'White'
}

keys = ["product_title", "product_brand", "product_color", "product_bullet_point", "product_description"]
output_str = flatten_dict(item1, keys)
print(output_str)

所以基本上我做了你做过的事情，但是，不是做一个字符串，我做了一个列表，然后我加入它。
输出：

Panasonic FV-20VQ3 WhisperCeiling 190 CFM Ceiling Mounted Fan [TITLE] Panasonic [BRAND] White [COLOR] WhisperCeiling fans feature a totally enclosed condenser motor and a double-tapered, dolphin-shaped bladed blower wheel to quietly move air
Designed to give you continuous, trouble-free operation for many years thanks in part to its high-quality components and permanently lubricated motors which wear at a slower pace
Detachable adaptors, firmly secured duct ends, adjustable mounting brackets (up to 26-in), fan/motor units that detach easily from the housing and uncomplicated wiring all lend themselves to user-friendly installation
This Panasonic fan has a built-in damper to prevent backdraft, which helps to prevent outside air from coming through the fan
0.35 amp [BULLETPOINT]

赞(0）回复(0）举报 2023-05-02

vojdkbi02#

对我来说，keys应该是一个全局变量，这似乎是非常清楚的，我猜你会用相同的keys参数反复调用函数，所以如果你把它设为全局变量，而不是不必要地把它作为参数传递，那会更好。
你的令牌遵循一个明确的模式，你正在删除'product_'前缀和删除下划线，然后转换为大写，为什么不做一个函数来做到这一点？
虽然您可以使用dict解析来预生成标记，但我建议不要使用它，因为它不会有任何显着的性能增益，并且每次查询dict时都会执行隐式循环。
我把你的代码缩短为：

KEYS=['product_title', 'product_brand', 'product_color', 'product_bullet_point', 'product_description']
def tokenize(key: str) -> str:
    return key.removeprefix('product_').replace('_', '').upper()

def product2str(item: dict) -> str:
    return ' '.join(
        '{} [{}]'.format(v.replace('\n', '[SEP]'), tokenize(key))
        for key in KEYS
        if (v := item.get(key, None))
    )

据我所知，恐怕没有别的办法了。
使用您的示例，我得到了以下输出：

Panasonic FV-20VQ3 WhisperCeiling 190 CFM Ceiling Mounted Fan [TITLE] Panasonic [BRAND] White [COLOR] WhisperCeiling fans feature a totally enclosed condenser motor and a double-tapered, dolphin-shaped bladed blower wheel to quietly move air[SEP]Designed to give you continuous, trouble-free operation for many years thanks in part to its high-quality components and permanently lubricated motors which wear at a slower pace[SEP]Detachable adaptors, firmly secured duct ends, adjustable mounting brackets (up to 26-in), fan/motor units that detach easily from the housing and uncomplicated wiring all lend themselves to user-friendly installation[SEP]This Panasonic fan has a built-in damper to prevent backdraft, which helps to prevent outside air from coming through the fan[SEP]0.35 amp [BULLETPOINT]

AHXML#2 HB Wood Cased Graphite Pencils, Pre-Sharpened with Free Erasers, Smooth write for Exams, School, Office, Drawing and Sketching, Pack of 48 [TITLE] AHXML [BRAND] #2 HB yellow, wood-cased pencils:Box of 48 count. Made from high quality real poplar wood and 100% genuine graphite pencil core. These No 2 pencils come with 100% Non-Toxic latex free pink top erasers.[SEP]PRE-SHARPENED & EASY SHARPENING: All the 48 count pencils are pre-sharpened, ready to use when get it, saving your time of preparing.[SEP]These writing instruments are hexagonal in shape to ensure a comfortable grip when writing, scribbling, or doodling.[SEP]They are widely used in daily writhing, sketching, examination, marking, and more, especially for kids and teen writing in classroom and home.#2 HB wood-cased yellow pencils in bulk are ideal choice for school, office and home to maintain daily pencil consumption.[SEP]Customer service:If you are not satisfied with our product or have any questions, please feel free to contact us. [BULLETPOINT] <b>AHXML#2 HB Wood Cased Graphite Pencils, Pack of 48</b><br><br>Perfect for Beginners experienced graphic designers and professionals, kids Ideal for art supplies, drawing supplies, sketchbook, sketch pad, shading pencil, artist pencil, school supplies. <br><br><b>Package Includes</b><br>- 48 x Sketching Pencil<br> - 1 x Paper Boxed packaging<br><br>Our high quality, hexagonal shape is super lightweight and textured, producing smooth marks that erase well, and do not break off when you're drawing.<br><br><b>If you have any question or suggestion during using, please feel free to contact us.</b> [DESCRIPTION]

赞(0）回复(0）举报 2023-05-02

我来回答

将JSON/dict转换为具有指示符标记的扁平字符串

2条答案

相关问题

热门标签

最新问答