Restrict attribute to property type

Using the AttributeUsage attribute, I can restrict the valid targets for an attribute to certain entity types (class, property, etc.). However, how can restrict it further so that it can only be applied to properties of a certain type?

For example:

[MyAttribute] // Valid
public Foo Item { get; set; }

[MyAttribute] // Compilation error: "Attribute 'MyAttribute' is only valid on classes derived from Foo"
public int Value { get; set; }

I'm pretty sure this should be feasible, because the AttributeUsage attribute itself has this restriction if we try to apply it to a class that isn't derived from Attribute :

[AttributeUsage(AttributeTargets.Property)] // Attribute 'AttributeUsage' is only valid on classes derived from System.Attribute
public class Bar { }