Cast a value and set a property using reflection

I am trying to set a property of an object dynamically. The method takes a parameter of type object and sets the property of the object using PropertyInfo.SetValue().

public void SetValue(object value)
{
    var valueType = value.GetType();

    // Passes
    if (!_propertyInfo.PropertyType.IsAssignableFrom(valueType)) return;
    var valueType = value.GetType(); 

    // Tried but this doesn't help
    // var value = (dynamic) value;     

    // Fails   
    _propertyInfo.SetValue(value, Element, null);        
}

Pretty basic, as you can see. However, I keep getting a TargetException because object has the wrong type.

Additional information: Object does not match target type.

I don't know if this refers to the value or the object whose property I want to set. Both definitely are of the expected types. However, value is passed in as object and Element is passed in as a base type of the DeclaringType of the property info.

Is that the cause for the problem? How can I workaround this limitation, if the concrete type of Element is unknown?