Modifying the declared annotations of a field during runtime using Java Reflection API

From this answer it's possible to add annotations to Java classes during runtime by creating and installing a new internal AnnotationData object. I was curious if it would be possible for a Field. It seems like the way a Field handles annotations is pretty different from how a Class handles them.

I've been able to successfully add an annotation to the declaredAnnotations field of the Field class with the following class:

public class FieldRuntimeAnnotations {

  private static final Field DECLARED_ANNOTATIONS_FIELD;
  private static final Method DECLARED_ANNOTATIONS_METHOD;

  static {
    try {
      DECLARED_ANNOTATIONS_METHOD = Field.class.getDeclaredMethod("declaredAnnotations");
      DECLARED_ANNOTATIONS_METHOD.setAccessible(true);

      DECLARED_ANNOTATIONS_FIELD = Field.class.getDeclaredField("declaredAnnotations");
      DECLARED_ANNOTATIONS_FIELD.setAccessible(true);

    } catch (NoSuchMethodException | NoSuchFieldException | ClassNotFoundException e) {
            throw new IllegalStateException(e);
    }
  }

  // Public access method
  public static <T extends Annotation> void putAnnotationToField(Field f, Class<T> annotationClass, Map<String, Object> valuesMap) {
    T annotationValues = TypeRuntimeAnnotations.annotationForMap(annotationClass, valuesMap);

    try {

        Object annotationData = DECLARED_ANNOTATIONS_METHOD.invoke(f);

        // Get declared annotations
        Map<Class<? extends Annotation>, Annotation> declaredAnnotations =
                (Map<Class<? extends Annotation>, Annotation>) DECLARED_ANNOTATIONS_FIELD.get(f);

        // Essentially copy our original annotations to a new LinkedHashMap
        Map<Class<? extends Annotation>, Annotation> newDeclaredAnnotations = new LinkedHashMap<>(declaredAnnotations);

        newDeclaredAnnotations.put(annotationClass, annotationValues);

        DECLARED_ANNOTATIONS_FIELD.set(f, newDeclaredAnnotations);

    } catch (IllegalAccessException | InvocationTargetException e) {
            throw new IllegalStateException(e);
    }
  }
}

However, the field's declaring class does not get updated with the proper ReflectionData. So essentially I need to "install" the new field information with its declaring class, but I am having trouble of figuring out how.

To make it clearer what I'm asking, the 3rd assertion in my test here fails:

public class RuntimeAnnotationsTest {

  @Retention(RetentionPolicy.RUNTIME)
  @Target({ElementType.TYPE, ElementType.FIELD})
  public @interface TestAnnotation {}

  public static class TestEntity {
    private String test;
  }

  @Test
  public void testPutAnnotationToField() throws NoSuchFieldException {

    // Confirm class does not have annotation
    TestAnnotation annotation = TestEntity.class.getDeclaredField("test").getAnnotation(TestAnnotation.class);
    Assert.assertNull(annotation);

    Field f = TestEntity.class.getDeclaredField("test");
    f.setAccessible(true);

    FieldRuntimeAnnotations.putAnnotationToField(f, TestAnnotation.class, new HashMap<>());

    // Make sure field annotation gets set
    Assert.assertNotNull(f.getAnnotation(TestAnnotation.class));

    // Make sure the class that contains that field is also updated -- THIS FAILS
    Assert.assertNotNull(TestEntity.class.getDeclaredField("test").getAnnotation(TestAnnotation.class));
  }

} 

I understand what I'm trying to achieve is rather ridiculous, but I'm enjoying the exercise :D ... Any thoughts?