使用bash修复多个文件中的json数据格式问题

pbgvytdp  于 2023-10-21  发布在  其他
关注(0)|答案(3)|浏览(157)

我使用一个app-store-scraper(https://github.com/facundoolano/app-store-scraper/)来获取评论并使用python阅读它们。问题是应用程序的输出不是json格式的,所以数据需要一些修复。
下面是生成输出的node.js代码:

var store = require('app-store-scraper');

store.reviews({
  id: 300238550,
  sort: store.sort.HELPFUL,
  page: 1
})
.then(console.log)
.catch(console.log);

下面是来自输出的示例记录:

{
id: '123456789',
userName: 'user123',
userUrl: 'https://itunes.apple.com/us/reviews/id1234567',
version: '150.71.0',
score: 1,
title: 'Difficulties and errors signing up',
text: 'The site asks you to put all your information and to link all your bank accounts and financial accounts accounts using your passwords.  \n' +
      'Multiple times reports errors.  "Sorry, our site is not working at this time".   For me, a waist of my time.',
url: 'https://itunes.apple.com/us/review?id=11111111111'
  },

其中“text”值被打印在输出文件中的两个单独的行上。
我使用的是pandas.read_json,当试图按原样读取文件时,它会给出很多错误。我认为数据应该采用以下格式:

{
"id": "123456789",
"userName": "user123",
"userUrl": "https://itunes.apple.com/us/reviews/id1234567",
"version": "150.71.0",
"score": 1,
"title": "Difficulties and errors signing up",
"text": "The site asks you to put all your information and to link all your bank accounts and financial accounts accounts using your passwords. Multiple times reports errors.  'Sorry, our site is not working at this time'.   For me, a waist of my time.",
"url": "https://itunes.apple.com/us/review?id=11111111111"
  },

这些问题是:
1.关键值周围缺少引号。
1.字符串周围缺少引号。这里的一个挑战是有些字符串同时包含撇号和引号。2a.某些字符串有多次出现的\n +,它将字符串分成多个部分,分别在不同的行上。
我正在使用macOS索诺马14.0,所以我使用BASH。以下是我解决问题1的方法:

find . -name '*.json' | xargs awk 'BEGIN {FS=OFS=":"} 
{
    if (NF > 1) {
        gsub(/^ *| *$/, "", $1);  # Trim leading and trailing spaces
        $1 = "\"" $1 "\"";
        print $0;
        next;
    }
    print;
}' >> output.txt

问题2有点复杂,所以我非常感谢任何帮助设计一个解决方案。我并不关心整个文本中的引号(比如例子中的“Sorry. time”部分),所以如果删除字符串中的所有撇号和引号更容易,只要文本最终是一个用引号 Package 的连续字符串,这对我来说就行了。
谢谢

7fyelxc5

7fyelxc51#

试试这个:

var store = require('app-store-scraper');

var raw = store.reviews({
  id: 300238550,
  sort: store.sort.HELPFUL,
  page: 1
});
var json = JSON.parse(raw);
console.log(JSON.stringify(json, null, 4));
lvmkulzt

lvmkulzt2#

TXR中的后处理解决方案:

@(do (defun esc-quote (str) (regsub "\"" "\\\"" str)))
@(repeat :gap 0)
@  (cases)
@prop: '@raw0'@{comma /,?/}
@    (bind out `    "@prop" : "@(esc-quote raw0)"@comma`)
@  (or)
@prop: '@raw0' +
@    (collect :vars (rawn))
 '@rawn' +
@    (last)
 '@rawl'@{comma /,?/}
@    (end)
@    (bind out `    "@prop" : "@(esc-quote (join raw0 rawn rawl))"@comma`)
@  (or)
@prop: @val
@    (bind out `    "@prop" : @val`)
@  (or)
@out
@  (end)
@  (do (put-line out))
@(end)
$ txr data.txr data
  {
    "id" : "123456789",
    "userName" : "user123",
    "userUrl" : "https://itunes.apple.com/us/reviews/id1234567",
    "version" : "150.71.0",
    "score" : 1,
    "title" : "Difficulties and errors signing up",
    "text" : "The site asks you to put all your information and to link all your bank accounts and financial accounts accounts using your passwords.  \nMultiple times reports errors.  \"Sorry, our site is not working at this time\".   For me, a waist of my time.",
    "url" : "https://itunes.apple.com/us/review?id=11111111111"
  },

由Vim着色:

jmp7cifd

jmp7cifd3#

我想出了一个非常接近使用bash的解决方案,但是由于一些未知的字符类型,“text”字符串永远不会以确切的json格式结束。为了解决这个问题,我想为什么这个应用程序的其他用户没有遇到同样的问题;为什么没有更多的人来讨论这个问题。原来我从README.txt中提取的应用程序调用取消了JSON格式。经过一些测试,这里是完整的解决方案,使用此应用程序拉评论:

var store = require('app-store-scraper');
var fs = require('fs');

// Dictionary to map sort keys to desired filename parts
const sortNameMapping = {
    'RECENT': 'recent',
    'HELPFUL': 'helpful',
    'FAVORITE': 'favorite',
    'CRITICAL': 'critical'
};

async function fetchReviews(appId, appName, sortKey, pageNum) {
  const reviews = await store.reviews({
    id: appId,
    sort: store.sort[sortKey],  // Access the sort method using store.sort[sortKey]
    page: pageNum
  });
  
  // Use the mapping to get the correct part for the filename
  const sortName = sortNameMapping[sortKey];

  // Construct the filename using the app name, sort name, and page number
  const fileName = `${appName.toLowerCase()}_${sortName}_${pageNum}.json`;

  // Write reviews to the constructed filename
  fs.writeFileSync(fileName, JSON.stringify(reviews, null, 2), 'utf-8');
}

// Dictionary of app names and their IDs
const appIds = {
  mint: 300238550,
  rocket: 1130616675,
  nerd: 1174471607,
  ynab: 1010865877
};

// Array of sort keys
const sortKeys = ['RECENT', 'HELPFUL', 'FAVORITE', 'CRITICAL'];

// Array of page numbers
const pageNumbers = [1,2,3,4,5,6,7,8,9,10];

// Loop over each combination of id, sort key, and page number
for (let appName in appIds) {
  for (let sortKey of sortKeys) {
    for (let pageNum of pageNumbers) {
      fetchReviews(appIds[appName], appName, sortKey, pageNum);
    }
  }
}

它被设置为每个应用程序、每个过滤器类型、每个页码创建一个文件。上面的例子为每个应用程序生成了40个json文件,尽管所有的记录可能不是唯一的。

相关问题