Annotation with public visibilty constraint

I'm creating a CSV annotation.

@Target({ElementType.FIELD})
@Retention(RetentionPolicy.RUNTIME)
public @interface CsvAttribute {
     String columnName();
     int position();
}

I want to enforce that the annotation can only be placed on public members, either field or method. Is this possible? My brief research on this topic said no. My goal is to achieve this without the try-catch block. Since I have access to the object can I do this without reflection?

protected Map<Integer, String> convertFieldsToMap(){
    final Method[] m = this.getClass().getDeclaredMethods();
    return new ArrayList<>(Arrays.asList(m)).stream()
            .filter(p -> p.isAnnotationPresent(CsvAttribute.class))
            .collect(Collectors.toMap(
                    p -> p.getAnnotation(CsvAttribute.class).position(),
                    p -> this.invokeGetter(p)));
}

private String invokeGetter(Method m){
    try {
        return Objects.toString(m.invoke(this), "");
    } catch (IllegalAccessException | InvocationTargetException e) {
        LOG.error("@CsvAttribute annotation must be placed on public getters!");
        e.printStackTrace();
    }
    return StringUtils.EMPTY;
}