Operation on "Assembly" is very slow when another application already use them

I've some operation that I'm doing with reflection(this will be changed end of year, but in between I've to find a solution), so we load the DLL of one directory( with AppDomain.CurrentDomain.GetAssemblies()), then we filter it to check it contains some DLL.

The filter is based on some specific attribute present in the AssemblyInfo, then we check each type of the assembly(assembly.GetTypes()) and check it implements a specific interface.

If it does we instantiate the object and execute a method on it:

if (type.GetInterface(typeof(IAssemblyInitializer).FullName) == null)
{
    return ;
}
bool isCompatible = !type.IsAbstract;
if (!type.IsClass)
{
    return
}
if (type.GetConstructor(new Type[0]) == null)
{
    return
}

IAssemblyInitializer initializer = Activator.CreateInstance(type, new object[0]) as IAssemblyInitializer;
if (initializer != null)
{
    initializer.InitializeAssembly();
}

Everything seems to work, but when we have another application already started(and having done this exact same procedure), the execution of this takes a lot longer(like 2 minutes instead of 20 seconds).

So my question: Why does it seems we are influenced on the fact that another application has already loaded thoses assemblies, and what can we do against ?

Thanks!