Gradle Plugin Task Reflection Scans Plugin's Project Class Files Instead of Current Project Files

I've created a Gradle plugin which scans the class files in my project and find which ones have an annotation.

I got it to work and compiled it. Now when I run the task in my project, instead of scanning the classes of the project that I'm in and runs the task on it, it scans the class files of the plugin project itself. What am I doing wrong?

final ClassLoader loader = Thread.currentThread().getContextClassLoader();
    try {
        ClassPath classpath = ClassPath.from(loader); // scans the class path used by classloader
        log.info("classPath = {}", classpath);
        for (ClassPath.ClassInfo classInfo : classpath.getTopLevelClassesRecursive(packageName)) {
            log.info("classInfo={}", classInfo);
            Class<?> clazz = classInfo.load();
            if (clazz.isAnnotationPresent(RestController.class)) {
                Method[] methods = clazz.getMethods();
                for (Method method : methods) {
                    doSomething(clazz, method);
                }
            }
        }
    } catch (IOException | NoSuchFieldException | IllegalAccessException ex) {
        String errorMsg = new StringBuilder("Unable to generate ").append(propertiesFileName).append(" file.").toString();
        log.error(errorMsg, ex);
        throw ex;
    }