我有一个嵌套的字典的公共信息的就业历史的人,我想构建面板数据类似于下表。
这里是嵌套字典。
上表中人员1的嵌套字典如下所示。
{'basicInformation': {'individualId': 6092353,
'firstName': 'A','middleName': 'ANTHONY','lastName': 'OLIVETTI',
'otherNames': ['ALBERT A OLIVETTI',
'ALBERT ANTHONY OLIVETTI',
'ANTHONY A OLIVETTI',
'ANTHONY OLIVETTI'],
'bcScope': 'Active',
'iaScope': 'Active',
'daysInIndustryCalculatedDate': '10/16/2013'},
'currentEmployments': [{'firmId': 8174,
'firmName': 'UBS FINANCIAL SERVICES INC.',
'iaOnly': 'N',
'registrationBeginDate': '10/17/2013',
'firmBCScope': 'ACTIVE',
'firmIAScope': 'ACTIVE',
'iaSECNumber': '7163',
'iaSECNumberType': '801',
'bdSECNumber': '16267',
'branchOfficeLocations': [{'locatedAtFlag': 'Y',
'supervisedFromFlag': 'N',
'privateResidenceFlag': 'N',
'branchOfficeId': '88789',
'street1': '1251 AVE OF THE AMERICAS',
'street2': '2ND FLOOR',
'city': 'NEW YORK',
'cityAlias': ['MANHATTAN',
'NEW YORK',
'NEW YORK CITY',
'NY',
'NY CITY',
'NYC'],
'state': 'NY',
'country': 'United States',
'zipCode': '10020',
'latitude': '40.758908',
'longitude': '-73.97902',
'geoLocation': '40.758908,-73.97902',
'nonRegisteredOfficeFlag': 'N',
'elaBeginDate': '07/15/2013'}]}],
'currentIAEmployments': [{'firmId': 8174,
'firmName': 'UBS FINANCIAL SERVICES INC.',
'iaOnly': 'Y',
'registrationBeginDate': '2/24/2014',
'firmBCScope': 'ACTIVE',
'firmIAScope': 'ACTIVE',
'iaSECNumber': '7163',
'iaSECNumberType': '801',
'bdSECNumber': '16267',
'branchOfficeLocations': [{'locatedAtFlag': 'Y',
'supervisedFromFlag': 'N',
'privateResidenceFlag': 'N',
'branchOfficeId': '88789',
'street1': '1251 AVE OF THE AMERICAS',
'street2': '2ND FLOOR',
'city': 'NEW YORK',
'cityAlias': ['MANHATTAN',
'NEW YORK',
'NEW YORK CITY',
'NY',
'NY CITY',
'NYC'],
'state': 'NY',
'country': 'United States',
'zipCode': '10020',
'latitude': '40.758908',
'longitude': '-73.97902',
'geoLocation': '40.758908,-73.97902',
'nonRegisteredOfficeFlag': 'N',
'elaBeginDate': '07/15/2013'}]}],
'previousEmployments': [],
'previousIAEmployments': [],
'disclosureFlag': 'N',
'iaDisclosureFlag': 'N',
'disclosures': [],
'examsCount': {'stateExamCount': 1,
'principalExamCount': 0,
'productExamCount': 3},
'stateExamCategory': [{'examCategory': 'Series 66',
'examName': 'Uniform Combined State Law Examination',
'examTakenDate': '2/18/2014',
'examScope': 'BOTH'}],
'principalExamCategory': [],
'productExamCategory': [{'examCategory': 'SIE',
'examName': 'Securities Industry Essentials Examination',
'examTakenDate': '10/1/2018',
'examScope': 'BC'},
{'examCategory': 'Series 3',
'examName': 'National Commodity Futures Examination',
'examTakenDate': '10/27/2014',
'examScope': 'BC'},
{'examCategory': 'Series 7',
'examName': 'General Securities Representative Examination',
'examTakenDate': '10/17/2013',
'examScope': 'BC'}],
'registrationCount': {'approvedSRORegistrationCount': 10,
'approvedFinraRegistrationCount': 1,
'approvedStateRegistrationCount': 7,
'approvedIAStateRegistrationCount': 2},
'registeredStates': [{'state': 'California',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '5/31/2022'},
{'state': 'Connecticut',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '2/26/2014'},
{'state': 'Florida',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '2/26/2014'},
{'state': 'New Jersey',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/23/2014'},
{'state': 'New Jersey',
'regScope': 'IA',
'status': 'APPROVED',
'regDate': '2/24/2014'},
{'state': 'New York',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '2/18/2014'},
{'state': 'New York',
'regScope': 'IA',
'status': 'APPROVED',
'regDate': '10/26/2021'},
{'state': 'North Carolina',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '5/31/2022'},
{'state': 'Pennsylvania',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '2/26/2014'}],
'registeredSROs': [{'sro': 'BOX Exchange LLC', 'status': 'APPROVED'},
{'sro': 'Cboe Exchange, Inc.', 'status': 'APPROVED'},
{'sro': 'FINRA', 'status': 'APPROVED'},
{'sro': 'NYSE American LLC', 'status': 'APPROVED'},
{'sro': 'NYSE Arca, Inc.', 'status': 'APPROVED'},
{'sro': 'NYSE Chicago, Inc.', 'status': 'APPROVED'},
{'sro': 'Nasdaq ISE, LLC', 'status': 'APPROVED'},
{'sro': 'Nasdaq PHLX LLC', 'status': 'APPROVED'},
{'sro': 'Nasdaq Stock Market', 'status': 'APPROVED'},
{'sro': 'New York Stock Exchange', 'status': 'APPROVED'}],
'brokerDetails': {'hasBCComments': 'N',
'hasIAComments': 'N',
'legacyReportStatusDescription': 'Not Requested'}}
上表中人2的嵌套字典如下。
{'basicInformation': {'individualId': 2652161,
'firstName': 'ALBERT',
'middleName': 'B',
'lastName': 'HORMAN',
'otherNames': ['A B HORMAN', 'ALBERT WILLIAM HORMAN', 'BILL HORMAN'],
'bcScope': 'Active',
'iaScope': 'Active',
'daysInIndustryCalculatedDate': '9/17/1995'},
'currentEmployments': [{'firmId': 7784,
'firmName': 'FIDELITY BROKERAGE SERVICES LLC',
'iaOnly': 'N',
'registrationBeginDate': '1/1/2008',
'firmBCScope': 'ACTIVE',
'firmIAScope': 'NOTINSCOPE',
'bdSECNumber': '23292',
'branchOfficeLocations': [{'locatedAtFlag': 'Y',
'supervisedFromFlag': 'N',
'privateResidenceFlag': 'N',
'branchOfficeId': '369366',
'street1': '825 EAST 1180 SOUTH',
'city': 'AMERICAN FORK',
'cityAlias': ['AM FORK', 'AMERICAN FORK', 'HIGHLAND', 'TIMPANOGOS'],
'state': 'UT',
'country': 'United States',
'zipCode': '84003',
'latitude': '40.405984',
'longitude': '-111.82903',
'geoLocation': '40.405984,-111.82903',
'nonRegisteredOfficeFlag': 'N',
'elaBeginDate': '07/04/2022'}]}],
'currentIAEmployments': [{'firmId': 288590,
'firmName': 'FIDELITY PERSONAL AND WORKPLACE ADVISORS',
'iaOnly': 'Y',
'registrationBeginDate': '7/13/2018',
'firmBCScope': 'NOTINSCOPE',
'firmIAScope': 'ACTIVE',
'iaSECNumber': '112027',
'iaSECNumberType': '801',
'branchOfficeLocations': [{'locatedAtFlag': 'Y',
'supervisedFromFlag': 'N',
'privateResidenceFlag': 'N',
'street1': '245 SUMMER STREET, V2A',
'city': 'BOSTON',
'cityAlias': ['BOSTON'],
'state': 'MA',
'country': 'United States',
'zipCode': '02210',
'latitude': '42.346571',
'longitude': '-71.039563',
'geoLocation': '42.346571,-71.039563',
'nonRegisteredOfficeFlag': 'Y',
'elaBeginDate': '07/13/2018'}]}],
'previousEmployments': [{'iaOnly': 'N',
'bdSECNumber': '35097',
'firmId': 17507,
'firmName': 'FIDELITY INVESTMENTS INSTITUTIONAL SERVICES COMPANY, INC.',
'street1': '49 NORTH 400 WEST',
'city': 'SALT LAKE CITY',
'state': 'UT',
'zipCode': '84101',
'registrationBeginDate': '1/3/2001',
'registrationEndDate': '1/1/2008',
'firmBCScope': 'ACTIVE',
'firmIAScope': 'NOTINSCOPE'},
{'iaOnly': 'N',
'bdSECNumber': '23292',
'firmId': 7784,
'firmName': 'FIDELITY BROKERAGE SERVICES LLC',
'street1': '900 SALEM STREET',
'city': 'SMITHFIELD',
'state': 'RI',
'country': 'UNITED STATES',
'zipCode': '02917',
'registrationBeginDate': '9/18/1995',
'registrationEndDate': '1/4/2001',
'firmBCScope': 'ACTIVE',
'firmIAScope': 'NOTINSCOPE'}],
'previousIAEmployments': [{'iaOnly': 'Y',
'iaSECNumber': '13243',
'iaSECNumberType': '801',
'firmId': 104555,
'firmName': 'STRATEGIC ADVISERS LLC',
'street1': '49 NORTH 400 WEST',
'city': 'SALT LAKE CITY',
'state': 'UT',
'country': 'United States',
'zipCode': '84101',
'registrationBeginDate': '2/15/2008',
'registrationEndDate': '7/13/2018',
'firmBCScope': 'NOTINSCOPE',
'firmIAScope': 'ACTIVE'}],
'disclosureFlag': 'N',
'iaDisclosureFlag': 'N',
'disclosures': [],
'examsCount': {'stateExamCount': 2,
'principalExamCount': 0,
'productExamCount': 2},
'stateExamCategory': [{'examCategory': 'Series 66',
'examName': 'Uniform Combined State Law Examination',
'examTakenDate': '2/26/2008',
'examScope': 'BOTH'},
{'examCategory': 'Series 63',
'examName': 'Uniform Securities Agent State Law Examination',
'examTakenDate': '9/7/1995',
'examScope': 'BC'}],
'principalExamCategory': [],
'productExamCategory': [{'examCategory': 'SIE',
'examName': 'Securities Industry Essentials Examination',
'examTakenDate': '10/1/2018',
'examScope': 'BC'},
{'examCategory': 'Series 7',
'examName': 'General Securities Representative Examination',
'examTakenDate': '9/16/1995',
'examScope': 'BC'}],
'registrationCount': {'approvedSRORegistrationCount': 2,
'approvedFinraRegistrationCount': 1,
'approvedStateRegistrationCount': 52,
'approvedIAStateRegistrationCount': 2},
'registeredStates': [{'state': 'Alabama',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Alaska',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Arizona',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Arkansas',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'California',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Colorado',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Connecticut',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Delaware',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'District of Columbia',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Florida',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Georgia',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Hawaii',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Idaho',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Illinois',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Indiana',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Iowa',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Kansas',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Kentucky',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Louisiana',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Maine',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Maryland',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Massachusetts',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Michigan',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Minnesota',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Mississippi',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Missouri',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Montana',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Nebraska',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Nevada',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'New Hampshire',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'New Jersey',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'New Mexico',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'New York',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'North Carolina',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'North Dakota',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Ohio',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Oklahoma',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Oregon',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Pennsylvania',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Puerto Rico',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Rhode Island',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'South Carolina',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'South Dakota',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Tennessee',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Texas',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Texas',
'regScope': 'IA',
'status': 'APPROVED_RES',
'regDate': '7/13/2018'},
{'state': 'Utah',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Utah',
'regScope': 'IA',
'status': 'APPROVED',
'regDate': '7/13/2018'},
{'state': 'Vermont',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Virginia',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Washington',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'West Virginia',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Wisconsin',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'},
{'state': 'Wyoming',
'regScope': 'BC',
'status': 'APPROVED',
'regDate': '1/1/2008'}],
'registeredSROs': [{'sro': 'FINRA', 'status': 'APPROVED'},
{'sro': 'New York Stock Exchange', 'status': 'APPROVED'}],
'brokerDetails': {'hasBCComments': 'N',
'hasIAComments': 'N',
'legacyReportStatusDescription': 'Not Requested'}}
我尝试做的是实现JSON规范化和JSON扁平化。我已经修改了person1和person2的代码
import pandas as pds
from flatten_json import flatten
import json
#person_json is what I stored each person JSON. There are 2
#persons here. Thus, I do this two times to flatten the nested
#dictionary.
person_temp = pds.json_normalize(flatten(person_json))
# This line of the code is credited to Mr.Timeless
data_frame = (person_temp.set_axis(person_temp.columns.str.split("_", n=1,
expand=True), axis=1).stack(1).droplevel(0))
data_frame
编辑1:添加data_frame的捕获照片
示例data_frame看起来像这样。我只显示了data_frame的一部分,因为维度等于111行乘16列。
我从上面的代码中得到的是一个 Dataframe 。然而,我试图设法构建面板数据,就像我展示的第一张捕获的照片一样。我在这里发现的问题是提取“年份”和“城市”,并将它们构建到(不平衡)面板数据集。
我该怎么做?
欢迎任何建议/意见。
非常感谢
1条答案
按热度按时间cunj1qz11#
我建议采取不同的方法。
首先,定义以下helper函数:
然后:
它为您提供了
person1
字典中的所有数据作为展平的数据框架: