允许专有网络中的lambda访问同一专有网络中的elasticsearch域

8fsztsew  于 2021-06-14  发布在  ElasticSearch
关注(0)|答案(1)|浏览(404)

我正在学习使用amazon服务,特别是我现在想用一个云形成脚本创建一个简单的设置:一个vpc,有一个用js编写的lambda,可以访问同一个vpc中的elasticsearch服务。
不知怎的,我不能让它工作。所有从lambda到elasticsearch域的请求总是超时。但是,从同一vpc中运行amazon linux 2的ec2示例发出的来自同一js代码或curl(即使没有任何额外授权,也只是 curl es域端点)的相同请求可以正常工作,并且我可以从该ec2示例(被放入其中)与elasticsearch进行正常通信。
同时lambda能够访问专有网络中的aurora集群,因此lambda不能访问专有网络资源不是一般的问题。
请告诉我,我在云形成设置的描述中做错了什么?下面是我的云形成模板的相关摘录和js代码的示例,它可以从ec2示例访问es服务,但不能对lambda执行相同的操作:

AWSTemplateFormatVersion: 2010-09-09
Description: The AWS CloudFormation tutorial
Resources:

  SomeDeploymentBucket:
    Type: 'AWS::S3::Bucket'

  AppLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: /aws/lambda/some-lambda

    # ========= The Lambda Execution Role =========

  IamRoleLambdaExecution:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: !Join 
            - '-'
            - - dev
              - some-app
              - lambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 's3:*'
                  - 'rds-db:connect'
                  - 'rds:*'
                  - 'es:*'
                Resource: '*'
      Path: /
      RoleName: !Join 
        - '-'
        - - some-app
          - dev
          - eu-west-1
          - lambdaRole
      ManagedPolicyArns:
        - !Join 
          - ''
          - - 'arn:'
            - !Ref 'AWS::Partition'
            - ':iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'

    # ========= The Lambda =========

  AppLambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      Code:
        S3Bucket: !Ref SomeDeploymentBucket
        S3Key: >-
          tutorial/some-app/dev/1545610972669-2018-12-24T00:22:52.669Z/some-app.zip
      FunctionName: some-lambda
      Handler: app.server
      MemorySize: 1024
      Role: !GetAtt 
        - IamRoleLambdaExecution
        - Arn
      Runtime: nodejs8.10
      Timeout: 6
      VpcConfig:
        SecurityGroupIds:
          - !Ref xxxVPCSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet
    DependsOn:
      - AppLogGroup
      - IamRoleLambdaExecution

    # ========= VPC =========

  xxxVPC:
    Type: 'AWS::EC2::VPC'
    Properties:
      CidrBlock: 172.31.0.0/16
      InstanceTenancy: default
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'

  xxxVPCSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: VPC SG
      GroupDescription: VPC Security Group
      VpcId: !Ref xxxVPC

  xxxLambdaSubnet:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref xxxVPC
      CidrBlock: 172.31.32.0/20

    # ========= Elasticsearch =========

  xxxESSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: ES SG
      GroupDescription: ES Security group
      VpcId: !Ref xxxVPC
      SecurityGroupIngress:
        - IpProtocol: -1
          FromPort: 0
          ToPort: 65535
          SourceSecurityGroupId: !Ref xxxVPCSecurityGroup

  xxxElasticSearch:
    Type: 'AWS::Elasticsearch::Domain'
    Properties:
      AccessPolicies:
        Version: 2012-10-17
        Statement:
          - Action:
              - 'es:*'
              - 'ec2:*'
              - 's3:*'
            Principal:
              AWS:
                - '*'
            Resource: '*'
            Effect: Allow
      DomainName: es-xxx-domain
      AdvancedOptions:
        rest.action.multi.allow_explicit_index: 'true'
      ElasticsearchVersion: 6.3
      ElasticsearchClusterConfig:
        InstanceCount: 2
        InstanceType: m3.medium.elasticsearch
        DedicatedMasterEnabled: 'false'
      VPCOptions:
        SecurityGroupIds:
          - !Ref xxxESSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet

js代码(没有使用creds签名的版本,但签名时也不起作用):

var es = require('elasticsearch');
var client = new es.Client({
    host: 'vpc-es-domain-AMAZON.eu-west-1.es.amazonaws.com:80',
    log: 'trace'
});

client.ping({
    requestTimeout: 1000
}, function(error, res, status){
    if(error) {
        console.trace('es cluster error!');
        console.trace(error);
    } else {
        console.log('All is well');
        var response = {
            error: error,
            res: res,
            status: status
        }
        console.log(JSON.stringify(response));
    }
});

在同一vpc中的ec2示例的curl中执行此操作可以从es域获得响应,没有任何问题:

curl vpc-es-domain-AMAZON.eu-west-1.es.amazonaws.com:80

我真的很感激你的帮助,因为我已经被困在这个。

qgelzfjb

qgelzfjb1#

注意到设置中的两个问题
您只在堆栈中创建一个子网,并且只将一个子网分配给lambda。您需要为lambda分配多个子网
您需要修复es的访问策略。我在控制台中手动更新为“不需要iam凭据签名请求”。
我更新了cloudformation模板,创建了基于python的lambda处理程序来查询来自同一vpc的ElasticSearch。它是不完整的,但如果你确定上述问题,那么它应该工作。

AWSTemplateFormatVersion: 2010-09-09
Description: The AWS CloudFormation tutorial
Resources:

  SomeDeploymentBucket:
    Type: 'AWS::S3::Bucket'

  AppLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: /aws/lambda/some-lambda

    # ========= The Lambda Execution Role =========

  IamRoleLambdaExecution:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: !Join
            - '-'
            - - dev
              - some-app
              - lambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 's3:*'
                  - 'rds-db:connect'
                  - 'rds:*'
                  - 'es:*'
                Resource: '*'

      Path: /
      RoleName: !Join
        - '-'
        - - some-app
          - dev
          - eu-west-1
          - lambdaRole
      ManagedPolicyArns:
        - !Join
          - ''
          - - 'arn:'
            - !Ref 'AWS::Partition'
            - ':iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'

    # ========= The Lambda =========

  AppLambdaFunction:
    Type: AWS::Lambda::Function
    DependsOn:
      - AppLogGroup
      - IamRoleLambdaExecution
    Properties:
      FunctionName: some-lambda
      Handler: index.lambda_handler
      Runtime: python2.7
      Timeout: 60
      MemorySize: 1024
      Role: !GetAtt
        - IamRoleLambdaExecution
        - Arn
      VpcConfig:
        SecurityGroupIds:
          - !Ref xxxVPCSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet1
          - !Ref xxxLambdaSubnet2

      Code:
        ZipFile: !Sub |
          from __future__ import print_function
          import boto3
          iam = boto3.client('iam')

          def lambda_handler(event, context):
            print('called lambda_handler')

    # ========= VPC =========

  xxxVPC:
    Type: 'AWS::EC2::VPC'
    Properties:
      CidrBlock: 172.31.0.0/16
      InstanceTenancy: default
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'

  xxxVPCSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: VPC SG
      GroupDescription: VPC Security Group
      VpcId: !Ref xxxVPC

  xxxLambdaSubnet1:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref xxxVPC
      CidrBlock: 172.31.32.0/20
  xxxLambdaSubnet2:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref xxxVPC
      CidrBlock: 172.31.16.0/20

    # ========= Elasticsearch =========

  xxxESSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: ES SG
      GroupDescription: ES Security group
      VpcId: !Ref xxxVPC
      SecurityGroupIngress:
        - IpProtocol: -1
          FromPort: 0
          ToPort: 65535
          SourceSecurityGroupId: !Ref xxxVPCSecurityGroup

  xxxElasticSearch:
    Type: 'AWS::Elasticsearch::Domain'
    Properties:
      AccessPolicies:
        Version: 2012-10-17
        Statement:
          - Action:
              - 'es:*'
              - 'ec2:*'
              - 's3:*'
            Principal:
              AWS:
                - '*'
            Resource: '*'
            Effect: Allow
      DomainName: es-xxx-domain
      EBSOptions:
        EBSEnabled: true
        Iops: 0
        VolumeSize: 20
        VolumeType: "gp2"
      AdvancedOptions:
        rest.action.multi.allow_explicit_index: 'true'

      ElasticsearchClusterConfig:
        InstanceCount: 1
        InstanceType: m4.large.elasticsearch
        DedicatedMasterEnabled: 'false'
      VPCOptions:
        SecurityGroupIds:
          - !Ref xxxESSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet1

更新了lambda函数处理程序代码

import urllib2

def lambda_handler(event, context):
    print('called lambda_handler')
    data = ''
    url = 'https://vpc-es-xxx-domain-fixthis.es.amazonaws.com'
    req = urllib2.Request(url, data, {'Content-Type': 'application/json'})
    f = urllib2.urlopen(req)
    for x in f:
        print(x)
    f.close()

相关问题