'System.Reflection.TargetException: Object does not match target type' exception being thrown for trying to use property.setvalue call

In trying to optimize my code which used to have a select block for multiple fields, like diamond_0_color, diamond_0_quantity, I thought I would optimize my code to use reflection instead. The below code fails every time I try to set a property value of the baseobject class to a value with an exception of 'System.Reflection.TargetException: Object does not match target type', such as the below code:

SetClassPropertyValue(xmlItem, $"diamond_{intCurrentDiamond}_quantity",
                      clsDiamond.DiamondQuantity)

in the above example, xmlItem is an XML class, intCurrentDiamond is an int32 value = 0, diamond_0_quantity is an int32 field, and clsDiamond.DiamondQuantity is an int32 property. My code for SetClassPropertyValue below. Anyone see what I am doing wrong?

        Private Function SetClassPropertyValue(baseObject As Object, propertyName As String, value As Object) As Boolean

            Try

                Dim baseObjectType As Type = baseObject.GetType()
                Dim propertyInfo As PropertyInfo = baseObjectType.GetProperty(propertyName)

                If propertyInfo IsNot Nothing Then

                    Dim propertyType as Type = propertyInfo.GetType()
                    dim valueType as Type = value.GetType()

                    propertyInfo.SetValue(baseObjectType, value)
                    Return True

                Else

                    Display.ShowMessageBox("Fatal value set", $"Unable to set property '{propertyName}' to '{value}'")
                    Return False

                End If

            Catch ex As Exception

                Display.ShowMessageBox("Fatal value set", $"Unable to set property '{propertyName}' to '{value}' due to exception '{ex}'")
                Return False

            End Try

        End Function