Assembly.LoadFrom in C# is quick in the debugger, but very slow when deployed

As part of one of our products, we have a dynamically loaded plugin architecture which loads assemblies from a directory.

The system uses Assembly.LoadFrom to load the DLL from the path, and when running inside VS with the debugger attached, the LoadFrom operation is very quick.

When we deploy the application however, the LoadFrom operation takes upwards of 10 seconds to load each Assembly object, which is cripplingly slow as we need to load the Assembly object to check whether it is derived from the base plugin class.

Is there any known reason why loading a DLL in this way should be so much slower when running outside of the IDE?