Collecting Dynamic Assemblies: Moving from one assembly to multiple modules and assemblies has tripled memory usage without RunAndCollect

At our company we have several in house languages and compilers/decompilers, such that we aggressively utilise the C# dynamic code generation and reflection abilities. Most of the code we write is just to dynamically create code at runtime, or in our proprietary scripting language.

At a certain point, we found out that adding types to a dynamic assembly scales O(n^2), presumably because the whole assembly is rewritten? This meant that when over 10k types were created, our software became might slower. What we did then was to keep one single dynamic assembly and module, and add types to it whenever we generated dynamic code (with a few exceptions for DynamicMethods).

To migate this, we did a gradient descent on the optimal combination of types, modules and assemblies to dynamically generate so that we could get O(n) performance.

We implemented an abstraction layer, which automatically created the modules/assemblies as needed according to the optimal manner shown by the results, for when we create dynamic types. This had the desired performance benefits. These assemblies/modules/types reflection objects always have a strong reference kept to them behind the scenes through various dictionaries and hashsets.

However, a few months down the line some very strange problems started occuring. The first one was extremely intermittent and would happen every 3/4 runs. It was fatal ExecutionEngineException, with error code 80131506. This was very perplexing and we originally thought it was caused by upgrading from .NET 451 to .NET 472 (we very occasionally got this problem before). Eventually we tracked it down to setting the System.Reflection.Emit.AssemblyBuilderAccess to RunAndCollect. When we changed this to just Run, we never got this problem again. Bare in mind that we always keep strong references to these assemblies, so it's very confusing as to how this could solve the problem.

The problems did not stop here however, as we descended further into the abyss of .NET we found unspeakable horrors. It appeared the memory usage of our programs had tripled, and many were failing with out of memory exceptions. By limiting the 'Strength' of our programs (a setting which reduces memory usage), our programs would run, but taking as much as three times as long.

Given this behaviour, it seems like we are producing too many dynamic assemblies, and they need to be collected otherwise we run out of memory. However, I find it hard to believe that dynamic assemblies could use up much space, and I don't understand how they can be collected, given that we maintain objects which are instances of types in dynamic assemblies (which presumably hold reference to the Type objects ) and direct strong references to the reflection objects of these dynamic assemblies anyway.