lundi 4 novembre 2019

Can you use scala reflection to find variables in the current scope?

We've got a fat jar that we use to store spark jobs and common code to run said spark jobs.

When we're working on new jobs, we like to use Zeppelin to develop and test our code, but the common code has grown useful enough that we want to allow our developers to call said functions in Zeppelin.

Unfortunately, the way we have it set up, we have a SparkSession variable we use buried in the code, and said SparkSession is not compatible with Zeppelin's spark session. So I'd like to write some magic reflection code that looks for the zeppelin variable, and creates a new SparkSession if it can't find it. (In this way, we use the production-version SparkSession when running the fat jar in prof, and use Zeppelins session when spinning up from Zeppelin)

Some basic research didn't turn up anything, and I suspect this particular problem is impossible, but I wanted to see if anyone had done something this weird before.

Any thoughts?





Aucun commentaire:

Enregistrer un commentaire