Adding custom attributes to ReturnParameter using reflection or IL

I am trying to create some types on runtime using IL. I have an original MethodInfo which I have to replicate.

var parameters = OriginalMethod.GetParameters();

MethodBuilder methodBuilder = Builder.DefineMethod(
    $"<{OriginalMethod.Name}>k__BackingMethod",
    OriginalMethod.Attributes,
    CallingConventions.HasThis,
    OriginalMethod.ReturnType,
    parameters.Select(i => i.ParameterType).ToArray()
);

foreach (var attribute in OriginalMethod.GetCustomAttributesData().ToCustomAttributeBuilder())
    methodBuilder.SetCustomAttribute(attribute);

foreach (var attribute in OriginalMethod.ReturnParameter.GetCustomAttributesData().ToCustomAttributeBuilder())
    methodBuilder.SetCustomAttribute(attribute); // <---- Problem here

ILGenerator il = methodBuilder.GetILGenerator();
// Some code removed for brevity

return methodBuilder;

As you can see, I can successfully copy the method level custom attributes. I can even get the attributes on ReturnParameter. However, I cannot seem to add new attribute to ReturnAttribute. Using the above code, the return attribute is applied as a regular method attribute. Any idea how I would do something like this?

EDIT: Added code for ToCustomAttributeBuilder

internal static CustomAttributeBuilder[] ToCustomAttributeBuilder(this IEnumerable<CustomAttributeData> CustomAttributes)
=> CustomAttributes.Select(attribute => attribute.ToCustomAttributeBuilder()).ToArray();

internal static CustomAttributeBuilder ToCustomAttributeBuilder(this CustomAttributeData CustomAttribute)
{
    var attributeArgs = CustomAttribute.ConstructorArguments.Select(a => a.Value).ToArray();

    var propertyArgs = CustomAttribute.NamedArguments.Where(i => i.MemberInfo is PropertyInfo);
    var propertyInfos = propertyArgs.Select(a => (PropertyInfo)a.MemberInfo).ToArray();
    var propertyValues = propertyArgs.Select(a => a.TypedValue.Value).ToArray();
    
    var fieldArgs = CustomAttribute.NamedArguments.Where(i => i.MemberInfo is FieldInfo);
    var namedFieldInfos = fieldArgs.Select(a => (FieldInfo)a.MemberInfo).ToArray();
    var namedFieldValues = fieldArgs.Select(a => a.TypedValue.Value).ToArray();
    
    return new CustomAttributeBuilder(CustomAttribute.Constructor, attributeArgs, propertyInfos, propertyValues, namedFieldInfos, namedFieldValues);
}