mardi 16 juin 2020

Register UDF dynamically in spark using scala reflection

Problem: My program accepts a JSON string as inpu, which contains a udf name, function definition and type. eg.

{
  "udfList": [
   {
     "name": "add"
     "definition": "(x: Int, y:int) => x+y"
     "type": "(Int, Int) => Int"
   }
  ]
}

The JSON can contain any function and type. How can i use scala reflection to convert this string to actual function and cast to the given type then register it as udf with the give name.

Tried:

val myFunc = """(x: Int, y:int) => x+y"""
val tb = runtimeMirror(getClass.getClassLoader).mkToolBox()
val tree = tb.parse(myFunc)
val fun =  tb.eval(tree)

I am not able to cast it dynamically because the type is passed as String. So is there a way to do it dynamically?





Aucun commentaire:

Enregistrer un commentaire