C# generating action with argument passed by ref

I am dynamically creating setters for fields of some classes and structs. Those setters work fine with classes but don't work with structs because of passing by value. How can I generate setter with object passed by ref? That's my setter generation (from other post on SO):

    public static Action<T, TField> BuildUntypedSetter<T,TField>(MemberInfo memberInfo)
    {
        var targetType = memberInfo.DeclaringType;
        var exInstance = Expression.Parameter(targetType, "t");

        var exMemberAccess = Expression.MakeMemberAccess(exInstance, memberInfo);


        var exValue = Expression.Parameter(typeof(TField), "p");
        var exConvertedValue = Expression.Convert(exValue, GetUnderlyingType(memberInfo));
        var exBody = Expression.Assign(exMemberAccess, exConvertedValue);

        var lambda = Expression.Lambda<Action<T, TField>>(exBody, exInstance, exValue);
        var action = lambda.Compile();
        return action;
    }

And here's simple use-case with struct:

public void SetterTest()
{
    IB ib = new A();
    var setter = FastInvoke.BuildUntypedSetter<A, double>(ib.GetType().GetField("field"));
    Print(ib.GetType().GetField("field").GetValue(ib));
    if(ib is A a)
    {
        setter(a, 5);
    }
    Print(ib.GetType().GetField("field").GetValue(ib));
}

private struct A : IB
{
    public double field;
}

private interface IB
{
    
}

How can I change BuildUntypedSetter method so it takes T by ref. Expression.Parameter doesn't let me to chose.