如何使这些函数定义d.r.y？

bxfogqkk 于 2021-05-31 发布在 Hadoop

关注(0)|答案(2)|浏览(417)

我有多个函数定义来编写一个Parquet文件，但我宁愿有一个通用函数。我怎样才能使一个通用功能工作？问题是一个通用函数返回错误 could not find implicit value for parameter writerFactory: com.github.mjakubowski84.parquet4s.ParquetWriter.ParquetWriterFactory[Product] 多种功能：

import com.github.mjakubowski84.parquet4s.ParquetWriter
import org.apache.parquet.hadoop.metadata.CompressionCodecName
import org.apache.parquet.hadoop.ParquetFileWriter
import java.time.format.DateTimeFormatter
import java.time.LocalDate

def writeParquetUnsegmented(rows: List[ACaseClass], date: String) = {
    val outPath = s"s3a://redacted/path-$date.parquet"
    ParquetWriter.writeAndClose(outPath,
                                rows,
                                    ParquetWriter.Options(
                                                        writeMode = ParquetFileWriter.Mode.OVERWRITE,
                                                        compressionCodecName = CompressionCodecName.SNAPPY
                                                        ))
}

def writeParquetSegmented(rows: List[BCaseClass], date: String) = {
    val outPath = s"s3a://redacted/path-$date.parquet"
    ParquetWriter.writeAndClose(outPath,
                                rows,
                                    ParquetWriter.Options(
                                                        writeMode = ParquetFileWriter.Mode.OVERWRITE,
                                                        compressionCodecName = CompressionCodecName.SNAPPY
                                                        ))
}

一个功能：

def writeParquetSegmented(rows: List[Product], date: String) = {
    val outPath = s"s3a://redacted/path-$date.parquet"
    ParquetWriter.writeAndClose(outPath,
                                rows,
                                    ParquetWriter.Options(
                                                        writeMode = ParquetFileWriter.Mode.OVERWRITE,
                                                        compressionCodecName = CompressionCodecName.SNAPPY
                                                        ))
}

我也尝试了不同的函数签名，得到了相同的错误。

def writeParquetSegmented[A](rows: List[A], date: String)

def writeParquetSegmented[A :< RowParent](rows: List[A], date: String)

hadoop scala parquet amazon-web-services

来源：https://stackoverflow.com/questions/62544898/how-do-i-make-these-function-definitions-d-r-y

2条答案

按热度按时间

mbzjlibv1#

只需将函数设置为泛型：

def writeParquetUnsegmented[A](
  rows: List[A],
  date: String)(
  implicit writerFactory: ParquetWriterFactory[A]) = {
    val outPath = s"s3a://redacted/path-$date.parquet"
    ParquetWriter.writeAndClose(
      outPath,
      rows,
      ParquetWriter.Options(
        writeMode = ParquetFileWriter.Mode.OVERWRITE,
        compressionCodecName = CompressionCodecName.SNAPPY))
}

赞(0）回复(0）举报 2021-06-01

vdzxcuhz2#

我想试试这个签名：

def writeParquetSegmented[A : ParquetWriterFactory](rows: List[A], date: String)

编译器将打开上下文绑定 [A : ParquetWriterFactory] 错误消息所抱怨的隐式参数。

def writeParquetSegmented[A](rows: List[A], date: String)(implicit pwf: ParquetWriterFactory)

赞(0）回复(0）举报 2021-06-01

我来回答

如何使这些函数定义d.r.y？

2条答案

相关问题

热门标签

最新问答