How to set a struct member that is a pointer to an arbitrary value using reflection in Go

Note: I want to do the same as How to set a struct member that is a pointer to a string using reflection in Go, but in a more generic way. The existing question does not solve my problem.

I have structs with different kinds of fields that I want to populate using reflection:

type MyStruct struct {
    SomeInt       int
    SomeString    string
    SomeIntPtr    *int
    SomeStringPtr *string
}

The value I want to write into the individual fields is retrieved from a configuration-store and parsed into the correct type, similar to this:

func getValueForField(fieldName string) interface{}

For int and *int types, the function returns an int (wrapped in an interface). For string and *string, the function returns string (behind an interface) and so on, for all types.

--> Note that it does NOT return *int/*string!

And now I want to assign the value to the struct fields:

var field reflect.Value  = reflect.ValueOf(ptrToMyStruct).Elem().Field(i)
var value interface{}    = getValueForField(....)
var isPointer bool       = field.Kind() == reflect.Ptr

// assign "value" to "field":
if isPointer {
    // ??
    field.Set(reflect.ValueOf(value).Addr()) // panic: reflect.Value.Addr of unaddressable value
} else {
    field.Set(reflect.ValueOf(value)) // works
}

Assigning those values to concrete types is easy and works as expected. But I can't assign an int type (returned from getValueForField) to an *int field without somehow getting an address. And since I only have an interface{}, this needs to be done via reflection.