python ASW雅典娜boto 3冰山表-插页

hujrc8aj  于 2023-01-04  发布在  Python
关注(0)|答案(1)|浏览(65)

我正在使用AWS和Athena冰山。我试图通过使用boto 3向Athena冰山表插入新记录,但我收到了一个内部错误**GENERIC_INTERNAL_ERROR:**无法在没有事务冲突的情况下提交。如果数据清单文件是在 * a生成的,我猜boto 3试图像常规Athena表一样向s3添加文件。任何想法:
...

import boto3
import pandas as pd    
def athena_query_to_dataframe(db, s3Bucket, query):
    
    client = boto3.client('athena')
    listOfStatus = ['SUCCEEDED', 'FAILED', 'CANCELLED']
    listOfInitialStatus = ['RUNNING', 'QUEUED']
    
    print('Starting Query Execution:')
    
    tempS3Path = 's3://{}'.format(s3Bucket)
    
    response = client.start_query_execution(
        QueryString = query,
        QueryExecutionContext = {
            'Database': db
        },
        ResultConfiguration = {
            'OutputLocation': tempS3Path,
        }
    )

    queryExecutionId = response['QueryExecutionId']
    print(client.get_query_execution(QueryExecutionId = queryExecutionId))

    status = client.get_query_execution(QueryExecutionId = queryExecutionId)['QueryExecution']['Status']['State']

    while status in listOfInitialStatus:
        status = client.get_query_execution(QueryExecutionId = queryExecutionId)['QueryExecution']['Status']['State']
        if status in listOfStatus:
            if status == 'SUCCEEDED':
                print('Query Succeeded!')
                paginator = client.get_paginator('get_query_results')
                query_results = paginator.paginate(
                    QueryExecutionId = queryExecutionId,
                    PaginationConfig = {'PageSize': 1000}
                )
            elif status == 'FAILED':
                print('Query Failed!')
            elif status == 'CANCELLED':
                print('Query Cancelled!')
            break
    print(client.get_query_execution(QueryExecutionId = queryExecutionId))
    
    results = []
    rows = []
    
    print('Processing Response')

in_cmd = """ insert into my_iceberg_table values ('aaaaa','bbb');"""
athena_query_to_dataframe('my'db,'my-bck/athena/tables/my_iceberg_table/',in_cmd)

...

q8l4jmvw

q8l4jmvw1#

这是一个权限问题...因为这是一个冰山表,我还需要在表和架构中粘贴目录权限。所以我添加了这个IAM权限:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:CreateDatabase",
                "glue:UpdateTable",
                "glue:GetTable"
            ],
            "Resource": [
                "arn:aws:glue:*:<account>:table/<schema>/<table>",
                "arn:aws:glue:*:<account>:schema/AwsDataCatalog",
                "arn:aws:glue:*:<account>:database/<schema>",
                "arn:aws:glue:*:<account>:catalog"
            ]
        }
    ]
}

相关问题