Dynamically convert double? to PropertyInfo.PropertyType?

I'm trying to dynamically convert a double? to the property type of a PropertyInfo, which may be int? or decimal?, and maybe other types down the road. An explicit cast works, but then I'd need something like:

if(prop.PropertyType == typeof(int?))
{
    prop.SetValue(obj, (int?)value);
}
else if(prop.PropertyType == typeof(decimal?))
{
    prop.SetValue(obj, (decimal?)value);
}
// etc...

I considered using a Dictionary<Type, Action<...>> containing actions to perform the casts, but I'm looking for something even more elegant. Is there a single line of code I can write to dynamically convert to the property type? Something like this, which throws an InvalidCastException:

prop.SetValue(obj, Convert.ChangeType(value, prop.PropertyType));