How to simplify type casting in Java

I have a reflection utility. This class is implemented to casting the values of certain fields obtained through reflection.

I give the code:

/**
 * Get the field values with the types already listed according to the field type
 *
 * @param clazz clazz
 * @param fieldName fieldName
 * @param fieldValue fieldValue
 * @return value cast to specific field type
 */
public static Object castFieldValue(Class<?> clazz, String fieldName, Object fieldValue) {
    Field field = getField(clazz, fieldName)
            .orElseThrow(() -> new IllegalArgumentException("Cannot find field name:" + fieldName));

    Class<?> fieldType = field.getType();

    if (fieldType.isAssignableFrom(Double.class)) {
        if (fieldValue instanceof String) {
            return Double.valueOf((String)fieldValue);
        }
        return ((Number) fieldValue).doubleValue();
    }

    if (fieldType.isAssignableFrom(Long.class)) {
        if (fieldValue instanceof String) {
            return Long.valueOf((String)fieldValue);
        }
        return ((Number) fieldValue).longValue();
    }

    if (fieldType.isAssignableFrom(Float.class)) {
        if (fieldValue instanceof String) {
            return Float.valueOf((String)fieldValue);
        }
        return ((Number) fieldValue).floatValue();
    }

    if (fieldType.isAssignableFrom(Integer.class)) {
        if (fieldValue instanceof String) {
            return Integer.valueOf((String)fieldValue);
        }
        return ((Number) fieldValue).intValue();
    }

    if (fieldType.isAssignableFrom(Short.class)) {
        if (fieldValue instanceof String) {
            return Short.valueOf((String)fieldValue);
        }
        return ((Number) fieldValue).shortValue();
    }

    return fieldValue;
}

This code is very verbose. Is it possible to simplify it somehow?