jeudi 24 janvier 2019

Class.forName in Spark casting

I have a function that looks like this:

private def readDS[T <: Product : TypeTag](path: String): Dataset[T] = {
    spark
       .read
       .csv(path)
       .as[T]
}

T is always a case class

I am trying to get T using

CC = Class.forName("SomeCaseClass")

The issue I have is then using this class as a type parameter in readDS function.

I had an idea to do something like this:

CC match {
    case VALIDCC //  check if VALIDCC is a subtype of Product and TypeTag[VALIDCC] is available // => {
        readDS[VALIDCC]("/path/to/file")
    }
    case _ => logger.error("provided class can't be used to cast a DF")

But I have no idea how to match types in scala and also how to check if a type is subtype of Product and if implicit val tt = TypeTag[CC] is available in the scope to use CC in the readDS function.





Aucun commentaire:

Enregistrer un commentaire