How do I set value of field that is a generic interface through reflection?

I want to be able to use reflection to get a member of a class that is, say, of type IList and I want to create a delegate for it so that I pay less for reflection. All of this so that late on, I am able to call the delegate by passing in my instance and a new List. C# is giving me invalid cast exceptions whenever I try this. Am I missing something silly with generics?

I have tried using interface fields with no generics and those seem to work just fine. I only seem to encounter the issue when using generics. I also tried to use Convert.ChangeType to convert my input to the desired type and I get errors about List<> not implementing IConvertible. If I can set a value of type IList<> from a concrete List<> using reflection, I will be happy :)

using System;
using System.Collections.Generic;
using System.Linq.Expressions;
using System.Reflection;

public class MyObject
{
    public IList<int> MyField;
}

public class ReflectionPerfTesting
{

    public void Run()
    {
        var testObject = new MyObject();
        var field = testObject.GetType().GetField("MyField", BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public);
        InvokeUsingReflection(field, testObject);
    }

    private void InvokeUsingReflection(FieldInfo fieldInfo, object testObject)
    {
        var fieldType = fieldInfo.FieldType;
        var sourceType = fieldInfo.DeclaringType;

        // ZAS: create value parameter
        var valueParam = Expression.Parameter(fieldType);

        // ZAS: create targetParameter
        var sourceParam = Expression.Parameter(sourceType);

        var field = Expression.Field(sourceParam, fieldInfo);
        var returnExpression = (Expression)Expression.Assign(field, valueParam);
        if (!fieldType.IsClass)
        {
            returnExpression = Expression.Convert(returnExpression, fieldType);
        }

        // ZAS: create generic delegate from definition
        var delegateType = typeof(Action<,>);
        var genericType = delegateType.MakeGenericType(sourceType, fieldType);

        // ZAS: Create lambda for faster access to setter using reflection. This is meant to be cached!
        var lambdaExpression = Expression.Lambda(genericType, returnExpression, sourceParam, valueParam);
        var lambda = lambdaExpression.Compile();
        var lambdaInvokeMethod = lambda.GetType().GetMethod("Invoke");

        var list = new List<int>();
        lambdaInvokeMethod.Invoke(testObject, new object[] { list });
    }

}

Expected results are for me to be able to cache a "setter" to the property and support setting the value of any type including generics and interfaces. My current result is that setting values of any type work EXCEPT when there are generics... so I must be doing something incorrectly.