Check whether PropertyInfo.SetValue will throw an ArgumentException

I have inherited some code that tries to set a property:

try
{
    propertyInfo.SetValue(object, value, null);
}
catch (ArgumentException)
{
    // look for a TypeConverter to use instead
}

In current usage the exception is caught and thrown a lot, so I would like to make a check first:

if (propertyInfo.PropertyType.IsAssignableFrom(value.GetType())
{
    // Try/catch as above
}
else
{
    // look for a TypeConverter to use instead, as above
}

This runs much faster. However, my one concern is that IsAssignableFrom returns false in some cases where SetValue would in fact succeed. The spec for SetValue says

value cannot be converted to the type of PropertyType

which is not exactly the same as assignability.

(If IsAssignableFrom returns true in cases where SetValue fails, that's fine - the throw/catch will still do the job.)

Can someone confirm for me whether this is possible or not?