Is there a way to obtain a "reference" to a value type field through reflection?

im trying to make a tool for Unity, which will display values from fields which have my custom [ObserveField] attribute in the Editor window. I've got to a point where I have a list of needed FieldInfo's but here's where my problem comes - at the moment I have to use fieldInfo.GetValue(fieldObject) to obtain the values and if I do it for every field in the list, in a GUI loop, it's gonna slow down significantly and I assume its not the best idea.

I know that if a field would be of reference type, there would be no problem, but for now only value types will be possible to "observe". Is there a way to somehow obtain the "reference" to the actual value, not the copy? Maybe something like a c++ pointer to a memory address of a field value?