Prevent reflection without SecurityManager

I'm making a java program inside an environment that loads code from files put in a "loading directory". This environment launches my code after it has already set an unreplacable SecurityManager. I'm trying to hide a SecretKey from any malicious code that can also be loaded in the same way as mine. It seems very hard if not impossible to hide a field from reflection without a security manager. The purpose of the code is to protect the end user from any malicious code that is put in the "loading directory".

Here is my code:

public class KeyStore {
    private static SecretKey key = null;

    public static SecretKey getKey() {
        if (!Utils.getCallerClassName(1).startsWith("my.package.and.ClassName")) throw new SecurityException("Not allowed to get security key!");
        return key;
    }

    public static void setKey(SecretKey key) {
        if (!Utils.getCallerClassName(1).startsWith("my.package.and.ClassName")) throw new SecurityException("Not allowed to set security key!");
        if (KeyStore.key == null)
            KeyStore.key = key;
    }
}

Where Utils.getCallerClassName() is:

    public static String getCallerClassName(int howFarBack) {
        StackTraceElement[] stElements = Thread.currentThread().getStackTrace();
        if (stElements.length >= howFarBack + 3) return stElements[howFarBack + 2 /* One for getCallerClassName() and one for getStackTrace() */].getClassName();
        return stElements[stElements.length-1].getClassName();
    }

The loaded SecurityManager just prevents System.exit() and replacing it.

Is there any way to protect the key field from reflection? If not, what should I do to keep that data safe?