How to optimize using reflection SetValue in c#?

What I am trying to do: I am trying to make component based Objects which can be easly created with custom set type of rules for each component value.

How I am doing it: I have created IComponent interface which each component implements. All components need to be structs, example:

public struct Weight : IComponent
{
     int weight;
}

Each Object is defined by just list of components with their values. Then to make it custom set of rules I made ObjectSettings which holds list of generic class ComponentSetup<T> where T : IComponent. ComponentSetup is a class which by reflection gets list of fields in IComponent and pairs them in Dicionary as FieldName and GenerationType for field. For example: for Object "Car" :

Car:
    Weight:
         weight: 
            GenerationType: RandomRange
               Min: 1200
               Max: 1700

For Object "Human":

Human:
    Weight:
         weight: 
            GenerationType: NormalDistribution
               Expected Value: 70
               Variance: 4

For Object "1kg dumbbell":

1kgDumbbell:
    Weight:
         weight: 
            GenerationType: Fixed
               Value: 1

In order to get generated Objects I used reflection to set values of components compose in List and return as Object.

The problem with this approach: When I want to generate 5k-10k of those Objects it takes way too much time.

My solution so far: I generate semi filled objects(on startup) and store them as prefabs in PrefabManager. They are Objects with components' values set only if their GenerationType is "Fixed" and then only fill values with other types of Generation.

My question: How can I make setting values by reflection faster, if it's not possible then how can I get the same result but faster? I also would like to keep prefab generation on startup because they help me instantiating Objects because I don't need to create whole new object, just copy prefab and fill it, which is faster in my case.