lundi 18 janvier 2021

How to infer StructType schema for Spark Scala at run time given a Fully Qualified Name of a case class

Since a few days I was wondering if it is possible to infer a schema for Spark in Scala for a given case class, but unknown at compile time.

The only input is a string containing the FQN of the class (that could be used for example to create an instance of the case class at runtime via reflection)

I was thinking if it was possible to do something like:

package com.my.namespace

case class MyCaseClass (name: String, num: Int)

//Somewhere else in codebase

// coming from external configuration file, so unknown at compile time
val fqn = "com.my.namespace.MyCaseClass" 


val schema = Encoders.product [ getXYZ( fqn ) ].schema

Of course, any other techniques that is not using Encoders is fine (building StructType analysing an instance of the case class ? Is it even possible ?)

What is the best approach? Is it something feasible ?





Aucun commentaire:

Enregistrer un commentaire