I am building a spark application which has dependency on a java library. Java Interface exposed is as
String doSomething(String, Map<String,String>)
I have created a UDF as
def myfunc(properties: Map[String, String]) = udf((data: String) => {
...
doSomething(data,properties)
})
This function can be called as myfunc(properties)(data)
from spark shell where properties is a Map and data is of type Column.
The issue is I need to call this via reflection from a scala file. I need to do something like this:
val c = Class.forName("package.class")
val m = c.getMethod("myfunc",classOf[Map[String,String]])
m.invoke(c.newInstance, someMap)
m.invoke returns the function itself. How and where to pass the Column parameter? Or is there any other way to pass these properties map to spark UDF so that it can directly be called via reflection?
Aucun commentaire:
Enregistrer un commentaire