spark shell错误:值生成器不是对象com.amazonaws.services.s3.model.putobjectrequest的成员

oxcyiej7  于 2021-05-27  发布在  Spark
关注(0)|答案(1)|浏览(843)

我刚刚开始使用emr hadoop/spark等,我正在尝试使用sparkshell运行scala代码,将文件上传到emrfs s3位置,但是我收到以下错误-
如果我运行=>

  1. val bucketName = "bucket"
  2. val outputPath = "test.txt"
  3. scala> val putRequest = PutObjectRequest.builder.bucket(bucketName).key(outputPath).build()
  4. <console>:27: error: not found: value PutObjectRequest
  5. val putRequest = PutObjectRequest.builder.bucket(bucketName).key(outputPath).build()
  6. ^

一旦我为putobjectrequest添加了导入包,我仍然会得到一个不同的错误。

  1. scala> import com.amazonaws.services.s3.model.PutObjectRequest

导入com.amazonaws.services.s3.model.putobjectrequest

  1. scala> val putRequest = PutObjectRequest.builder.bucket(bucketName).key(outputPath).build()
  2. <console>:28: error: value builder is not a member of object com.amazonaws.services.s3.model.PutObjectRequest
  3. val putRequest = PutObjectRequest.builder.bucket(bucketName).key(outputPath).build()
  4. ^

我不确定我错过了什么。任何帮助都将不胜感激!
注:spark版本为2.4.5

uidvcgyl

uidvcgyl1#

不使用生成器,而是通过合适的构造函数创建putobjectrequest的对象。另外,使用amazons3clientbuilder创建到s3的连接。

  1. import com.amazonaws.regions.Regions
  2. import com.amazonaws.services.s3.AmazonS3ClientBuilder
  3. import com.amazonaws.services.s3.model.ObjectMetadata
  4. import com.amazonaws.services.s3.model.PutObjectRequest
  5. import java.io.File
  6. val clientRegion = Regions.DEFAULT_REGION
  7. val bucketName = "***Bucket name***"
  8. val fileObjKeyName = "***File object key name***"
  9. val fileName = "***Path to file to upload***"
  10. val s3Client = AmazonS3ClientBuilder.standard.withRegion(clientRegion).build
  11. // Upload a file as a new object with ContentType and title specified.
  12. val request = new PutObjectRequest(bucketName, fileObjKeyName, new File(fileName))
  13. val metadata = new ObjectMetadata()
  14. metadata.setContentType("plain/text")
  15. metadata.addUserMetadata("title", "someTitle")
  16. request.setMetadata(metadata)
  17. s3Client.putObject(request)
展开查看全部

相关问题