Set FieldInfo value using SetValue vs Linq Expressions

I want to set private fields using Linq Expressions. I have this code:

 //parameter "target", the object on which to set the field `field`
 ParameterExpression targetExp = Expression.Parameter(typeof(object), "target");

 //parameter "value" the value to be set in the `field` on "target"
 ParameterExpression valueExp = Expression.Parameter(typeof(object), "value");

 //cast the target from object to its correct type
 Expression castTartgetExp = Expression.Convert(targetExp, type);

 //cast the value to its correct type
 Expression castValueExp = Expression.Convert(valueExp, field.FieldType);

 //the field `field` on "target"
 MemberExpression fieldExp = Expression.Field(castTartgetExp, field);

 //assign the "value" to the `field` 
 BinaryExpression assignExp = Expression.Assign(fieldExp, castValueExp);

 //compile the whole thing
 var setter = Expression.Lambda<Action<object, object>> (assignExp, targetExp, valueExp).Compile();

```

This compiles a delegate that takes an object - the subject and another object - the value.

setter(someObject,someValue);

The variable field contains what field is supposed to be set.

This works great for reference types, but if the subject is a struct, this thing will pass the subject as a copy to the setter delegate and set the value there. (at least that is what I think is going on)

while

field.SetValue(someObject,someValue);

Works just fine, even for structs.

Is there anything I can do about this in order to set the field of the subject using the compiled expressons?

Thoughts?