Field getter/setter with expression tree in base class

Following the examples on this post and its follow-up question, I am trying to create field getters / setters using compiled expressions.

The getter works just great, but I am stuck the setter, as I need the setter to assign any type of fields.

Here my setter-action builder:

public static Action<T1, T2> GetFieldSetter<T1, T2>(this FieldInfo fieldInfo) {
  if (typeof(T1) != fieldInfo.DeclaringType && !typeof(T1).IsSubclassOf(fieldInfo.DeclaringType)) {
    throw new ArgumentException();
  }
  ParameterExpression targetExp = Expression.Parameter(typeof(T1), "target");
  ParameterExpression valueExp = Expression.Parameter(typeof(T2), "value");
  //
  // Expression.Property can be used here as well
  MemberExpression fieldExp = Expression.Field(targetExp, fieldInfo);
  BinaryExpression assignExp = Expression.Assign(fieldExp, valueExp);
  //
  return Expression.Lambda<Action<T1, T2>> (assignExp, targetExp, valueExp).Compile();
}

Now, I store the generic setters into a cache list (because of course, building the setter each time is a performance killer), where I cast them as simple "objects":

 // initialization of the setters dictionary
 Dictionary<string, object> setters = new Dictionary(string, object)();
 Dictionary<string, FieldInfo> fldInfos = new Dictionary(string, FieldInfo)();
 FieldInfo f = this.GetType().GetField("my_int_field");
 setters.Add(f.Name, GetFieldSetter<object, int>(f); 
 fldInfos.Add(f.Name, f); 
 //
 f = this.GetType().GetField("my_string_field");
 setters.Add(f.Name, GetFieldSetter<object, string>(f); 
 fldInfos.Add(f.Name, f); 

Now I try to set a field value like this:

 void setFieldValue(string fieldName, object value) {
      var setterAction = setters[fieldName];
      // TODO: now the problem => how do I invoke "setterAction" with 
      // object and fldInfos[fieldName] as parameters...?
 }

I could simply call a generic method and cast each time, but I am worried about the performance overhead... Any suggestions?