python 使用AWS Lambda和boto3向S3中的文本文件对象追加新行

mwecs4sa  于 2023-01-04  发布在  Python
关注(0)|答案(3)|浏览(244)

我正在尝试使用python lambda函数在存储在S3中的对象上追加一个文本文件,并在其中添加一个新行。由于存储在S3中的对象是不可变的,因此您必须首先将该文件下载到“/tmp/”中,然后修改它,然后将新版本上载回S3。我的代码会追加数据,但不会追加新行。

BUCKET_NAME = 'mybucket' 
KEY = 'test.txt'
s3 = boto3.resource('s3')

def lambda_handler(event, context):
    try:
        s3.Object(BUCKET_NAME, KEY).download_file('/tmp/test.txt')
    except botocore.exceptions.ClientError as e:
        if e.response['Error']['Code'] == "404":
            print("The object does not exist.")
        else:
            raise
    with open('/tmp/test.txt', 'a') as fd:
        fd.write("this is a new string\n")

    s3.meta.client.upload_file('/tmp/test.txt', BUCKET_NAME, KEY)

文件总是附加新的字符串,但从来没有一个新的行。
更新:这个问题不会发生在Linux机器或Mac上。Lambda函数运行在Linux容器上,这意味着/tmp/中的文件保存为Unix的文本文件。一些Windows应用程序不会在Unix的文本文件上显示换行符,这是这里的情况。

2wnc66cl

2wnc66cl1#

您需要指定本地文件路径

import boto3
import botocore
from botocore.exceptions import ClientError

BUCKET_NAME = 'mybucket'
KEY = 'test.txt'
LOCAL_FILE = '/tmp/test.txt'
s3 = boto3.resource('s3')

def lambda_handler(event, context):
    try:
        obj=s3.Bucket(BUCKET_NAME).download_file(LOCAL_FILE, KEY)
    except ClientError as e:
        if e.response['Error']['Code'] == "404":
            print("The object does not exist.")
        else:
            raise
    with open('/tmp/test.txt', 'a') as fd:
        fd.write("this is a new string\n")

    s3.meta.client.upload_file(LOCAL_FILE, BUCKET_NAME, KEY)

Boto3文档参考:http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Bucket.download_file

xfb7svmp

xfb7svmp2#

不错的帖子!只是一个调整。您应该更改download_file方法的参数中LOCAL_FILE和KEY的顺序。正确的语法是:

obj=s3.Bucket(BUCKET_NAME).download_file(KEY,LOCAL_FILE)

此外,如果我们在桶中找不到文件的情况下删除本地文件也会很好。因为如果我们不删除本地文件(如果明显存在),我们可能会在已经存在的本地文件中添加一个新行。

def remove_local_file(filePath):
import os
     # As file at filePath is deleted now, so we should check if file exists or not not before deleting them
    if os.path.exists(filePath):
        os.remove(filePath)
    else:
        print("Can not delete the file as it doesn't exists")

以“try”开头的最终代码可能如下所示:

try:
    obj=s3.Bucket(BUCKET_NAME).download_file(KEY,LOCAL_FILE)
except ClientError as e:
    if e.response['Error']['Code'] == "404":
        print("The object does not exist.")
        remove_local_file(LOCAL_FILE)
    else:
        raise
with open(LOCAL_FILE, 'a') as fd:
    fd.write("this is a new string\n")

s3.meta.client.upload_file(LOCAL_FILE, BUCKET_NAME, KEY)
xoefb8l8

xoefb8l83#

您不需要下载和上传文件来覆盖S3中的文件;要覆盖现有对象,您只需上传同名文件,它将自动完成(reference)。请查看put_object函数(S3 doc)。
因此,您的代码将如下所示:

BUCKET_NAME = 'mybucket' 
KEY = 'test.txt'
# Use .client() instead of .resource()
s3 = boto3.client('s3')

def lambda_handler(event, context):
    try:
        # (Optional) Read the object
        obj = s3.get_object(Bucket=BUCKET_NAME, Key=KEY)
        file_content = obj['Body'].read().decode('utf-8')
        
        # (Optional) Update the file content
        new_file_content = file_content + "this is a new string\n"
        
        # Write to the object
        s3.put_object(Bucket=BUCKET_NAME, Key=KEY, Body=str(new_file_content))

    except botocore.exceptions.ClientError as e:
        if e.response['Error']['Code'] == "404":
            print("The object does not exist.")
        else:
            raise

相关问题