如何使用运行在gcp之外的spark访问bigquery

x9ybnkn6 于 2021-05-16 发布在 Spark

关注(0)|答案(2)|浏览(559)

我正在尝试连接我的spark作业，它运行在私有数据中心和bigquery之间。我已经创建了服务帐户，获得了私有json密钥，并获得了对我想要查询的数据集的读取访问权。但是，当我尝试与spark集成时，我收到了 User does not have bigquery.tables.create permission for dataset :yyy. . 我们需要create table权限才能使用bigquery从表中读取数据吗？
下面是控制台上打印的响应，

{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Access Denied: Dataset xxx:yyy: User does not have bigquery.tables.create permission for dataset xxx:yyy.",
    "reason" : "accessDenied"
  } ],
  "message" : "Access Denied: Dataset xxx:yyy: User does not have bigquery.tables.create permission for dataset xxx:yyy.",
  "status" : "PERMISSION_DENIED"
}

下面是我试图访问bigquery的spark代码

object ConnectionTester extends App {

  val session = SparkSession.builder()
    .appName("big-query-connector")
    .config(getConf)
    .getOrCreate()

    session.read
      .format("bigquery")
      .option("viewsEnabled", true)
    .load("xxx.yyy.table1")
    .select("col1")
    .show(2)

  private def getConf : SparkConf = {
    val sparkConf = new SparkConf
    sparkConf.setAppName("biq-query-connector")
    sparkConf.setMaster("local[*]")
    sparkConf.set("parentProject", "my-gcp-project")
    sparkConf.set("credentialsFile", "<path to my credentialsFile>")

    sparkConf
  }
}

apache-spark apache-spark-sql google-bigquery google-cloud-platform

来源：https://stackoverflow.com/questions/65223535/how-to-access-bigquery-using-spark-which-is-running-outside-of-gcp

2条答案

按热度按时间

ryhaxcpt1#

检查以下代码。
凭据

val credentials = """
     | {
     |           "type": "service_account",
     |           "project_id": "your project id",
     |           "private_key_id": "your private_key_id",
     |           "private_key": "-----BEGIN PRIVATE KEY-----\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n-----END PRIVATE KEY-----\n",
     |           "client_email": "xxxxx@company.com",
     |           "client_id": "111111111111111111111111111",
     |           "auth_uri": "https://accounts.google.com/o/oauth2/auth",
     |           "token_uri": "https://oauth2.googleapis.com/token",
     |           "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
     |           "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/xxxxx40vvvvvv.iam.gserviceaccount.com"
     |         }
     | """

编码 base64 &把它传给spark conf。

def base64(data: String) = {
    import java.nio.charset.StandardCharsets
    import java.util.Base64
    Base64.getEncoder.encodeToString(data.getBytes(StandardCharsets.UTF_8))
}

spark.conf.set("credentials",base64(credentials))

spark
      .read
      .options("parentProject","parentProject")
      .option("table","dataset.table")
      .format("bigquery")
      .load()

赞(0）回复(0）举报 2021-05-17

iecba09b2#

对于阅读常规表格，没有必要 bigquery.tables.create 许可。但是，您提供的代码示例提示表实际上是一个bigquery视图。bigquery视图是逻辑引用，它们在服务器端没有具体化，为了让spark读取它们，首先需要将它们具体化为一个临时表。为了创建这个临时表 bigquery.tables.create 需要许可。

赞(0）回复(0）举报 2021-05-16

我来回答

如何使用运行在gcp之外的spark访问bigquery

2条答案

相关问题

热门标签

最新问答