Serverless python打包numpy依赖错误

lawou6xi  于 2023-06-23  发布在  Python
关注(0)|答案(4)|浏览(134)

在从我部署的Python3.7 Lambda函数中调用函数时,我遇到了一些问题,从错误消息来看,这些函数与numpy相关。这个问题指出,无法导入软件包,尽管尝试了我读到的许多解决方案,但我没有任何成功,我想知道下一步要测试什么或如何进一步调试。

我试过以下几种方法:

1.安装Docker,添加插件serverless-python-requirements,在yml中配置
1.在要捆绑和部署的应用程序目录中安装软件包,pip install -t src/vendor -r requirements.txt --no-cache-dir
1.卸载setuptoolsnumpy并按此顺序重新安装

运行sls invoke -f auth后显示错误信息:

{
    "errorMessage": "Unable to import module 'data': Unable to import required dependencies:\nnumpy: \n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy c-extensions failed.\n- Try uninstalling and reinstalling numpy.\n- If you have already done that, then:\n  1. Check that you expected to use Python3.7 from \"/var/lang/bin/python3.7\",\n     and that you have no directories in your PATH or PYTHONPATH that can\n     interfere with the Python and numpy version \"1.18.1\" you're trying to use.\n  2. If (1) looks fine, you can open a new issue at\n     https://github.com/numpy/numpy/issues.  Please include details on:\n     - how you installed Python\n     - how you installed numpy\n     - your operating system\n     - whether or not you have multiple versions of Python installed\n     - if you built from source, your compiler versions and ideally a build log\n\n- If you're working with a numpy git repository, try `git clean -xdf`\n  (removes all files not under version control) and rebuild numpy.\n\nNote: this error has many possible causes, so please don't comment on\nan existing issue about this - open a new one instead.\n\nOriginal error was: No module named 'numpy.core._multiarray_umath'\n",
    "errorType": "Runtime.ImportModuleError"
}

提供的是我的设置:

OS: Mac OS X
Local Python: /Users/me/miniconda3/bin/python
Local Python version: Python 3.7.4

无服务器环境信息(Runtime = Python3.7):

Operating System:          darwin
 Node Version:              12.14.0
 Framework Version:         1.67.3
 Plugin Version:            3.6.6
 SDK Version:               2.3.0
 Components Version:        2.29.1

Docker:

Docker version 19.03.13, build 4484c46d9d

serverless.yml:

service: understand-your-sleep-api

plugins:
  - serverless-python-requirements
  - serverless-offline-python

custom:
  pythonRequirements:
    dockerizePip: true # non-linux
    slim: true
    useStaticCache: false
    useDownloadCache: false
    invalidateCaches: true

provider:
  name: aws
  runtime: python3.7
  stage: ${opt:stage, 'dev'}
  region: us-east-1
  iamRoleStatements:
    - Effect: Allow
      Action:
        - ssm:GetParameter
      Resource: "arn:aws:ssm:us-east-1:*id*:parameter/*"
  environment:
    STAGE: ${self:provider.stage}

functions:
  auth:
    handler: data.auth
    events:
      - http:
          path: /auth
          method: get
          cors: true

package:
  exclude:
    - env.yml
    - node_modules/**

requirements.txt:

pandas==1.0.0
fitbit==0.3.1
oauthlib==3.1.0
requests==2.22.0
requests-oauthlib==1.3.0

data.py:

import sys
sys.path.insert(0, 'src/vendor') # Location of packages that follow
import json
from datetime import timedelta, datetime, date
import math
import pandas as pd
from requests_oauthlib import OAuth2Session
from urllib.parse import urlparse, parse_qs
import fitbit
import requests
import webbrowser
import base64
import os 
import logging

def auth(event, context):
    ...
rm5edbpk

rm5edbpk1#

  • 使用lambda层来打包你所有的需求,确保你在requirements.txt文件中有numpy。试一次
  • 只有在插件部分列出了serverless-python-requirements插件时才有效。
  • 用这个替换你的自定义键,并给予函数一个使用该层的引用
custom:
  pythonRequirements:
    layer: true
functions:
  auth:
    handler: data.auth
    events:
      - http:
          path: /auth
          method: get
          cors: true
    layers:
      - { Ref: PythonRequirementsLambdaLayer}
vxf3dgd4

vxf3dgd42#

我用zipinfo .requirements.zip检查了一下,发现macos的dynlib加载的是linux的so文件。
我使用dockerizePip修复了这个问题:非Linux
请注意,如果工作目录中已存在. www.example.com,则不会触发此操作requirements.zip,因此在运行sls deploy之前,请执行git clean -xfd

iszxjhcz

iszxjhcz3#

由于您使用的是serverless-python-requirements插件,它会为您打包库。换句话说,您不需要手动执行pip install -t src/vendor -r requirements.txt --no-cache-dir所有这些操作。
要解决这个问题,请删除src/vendordata.py中的以下两行:

import sys
sys.path.insert(0, 'src/vendor') # Location of packages that follow

然后坐下来,让serverless-python-requirements为您做这项工作。

aemubtdh

aemubtdh4#

如果这对其他人有帮助:我在安装了serverless-python-requirements并部署了我的应用程序之后才在我的开发机器上安装了Docker,我认为这让我陷入了一种奇怪的状态。无论我做什么,我总是得到相同的numpy错误。
我的部署只有在我强制删除serverless-python-requirements缓存并重新部署后才开始工作。我运行serverless deploy --verbose来找出该高速缓存的位置:

$ serverless deploy --force --verbose

Deploying ...

Packaging
Generating requirements.txt from poetry.lock
Parsed requirements.txt from pyproject.toml in .../.serverless/requirements.txt
Using static cache of requirements found at /Users/danvk/Library/Caches/serverless-python-requirements/1434e..._x86_64_slspyc
^C

然后将其删除:

$ rm -rf /Users/danvk/Library/Caches

然后下一个serverless deploy工作!

相关问题