Class.forName() takes too long to return a Class in JUnit

ClassLoader classLoader = getClass().getClassLoader();
ImmutableSet<ClassInfo> classInfoSet = ClassPath.from(classLoader)
          .getTopLevelClassesRecursive("com.example");
for (ClassInfo classInfo : classInfoSet) {
  Class<?> clazz = classInfo.load();
  doSomething();
}

I'm trying to process 20,00 classes to check if annotation usage is correct. The problem is classInfo.load() takes too much time for specific classes (approx. 20% of all classes).

I replaced this method by the following code for debug and confirmed that Class.forName() takes more than a few minutes for specific classes.

String className = classInfo.getName();
Class<?> clazz = Class.forName(className);

I've run this program many times and it always slows down at the same classes and speeds up at other classes. Those slow classes have normal amount of codes and couldn't find a problem.

I thought it's a memory issue and added the setting below to VM arguments, but nothing changed.

-XX:MaxPermSize=2048M
-XX:-UseGCOverheadLimit

Is there anything I can do to solve this problem?