Dereference struct pointer and access fields with reflection

I'm writing a recursive function that iterates through every primitive field in a struct.

I need to be able to support fields that are structs, pointers to structs, fields, and pointers to fields.

I've tried doing something like this, where for each field, I first do a check if it's a pointer. If it is, I switch on the type of that instead of just the field itself.

//Get reflect values and types
valOf := reflect.ValueOf(dest).Elem()
typeOf := valOf.Type()

//Iterate through each field
for i := 0; i < valOf.NumField(); i++ {

    var fieldValDeref reflect.Value

    //Get reflect value and type of single field
    fieldVal := valOf.Field(i)
    fieldTyp := typeOf.Field(i)

    //Check if field is a pointer. If so, dereference and switch on dereferenced type
    if fieldVal.Kind() == reflect.Ptr {
        fieldValDeref = fieldVal.Elem()
    } else {
        fieldValDeref = fieldVal
    }


    switch fieldValDeref.Kind() {
    case reflect.Array, reflect.Chan, reflect.Interface, reflect.Func, reflect.Map, reflect.UnsafePointer:
        return errors.New("invalid destination field: " + fieldTyp.Name)

    case reflect.Struct:
        //Recursive call
        break
    default:
        //Perform Action on Field
..................................

The issue I'm getting with this, is that the type of any pointer, struct or not, after calling .Elem() is reflect.Invalid.

How can I first dereference a field (if it is a pointer) and then perform actions accordingly, whether the field is a struct or a primitive?

Thanks