Block reflection field/method access from classloader

I'm doing a plugin system with a security manager to restrict the plugin's actions. My problem is, it will be easy to bypass the system when people are able to access every field using reflection.

I don't want to block reflection for everything, because it is useful to use reflection sometimes. Now my question is, is there a way to block reflection only for some fields/method when they are accessed from a certain classloader? (I want to access them myself, so I can't work with Reflection field filters)