Assembly.DefinedType is loading dependencies from disk that are already loaded in the AppDomain

I've got an application that loads all assemblies out of a particular folder using this mechanism:

Assembly.Load(System.IO.File.ReadAllBytes(filename));

I do not have control over this loading mechanism as it's being handled by another component that I do not have the ability to change right now.

In my component I'm trying to get the types that are in those assemblies:

var appDomainAssemblies = appDomain.GetAssemblies().OrderBy(a => a.FullName).ToList();

foreach (var assembly in appDomainAssemblies)
{
  var types = assembly.DefinedTypes.Select(t => t.AsType());
  // Do something with those types
}

When that's done across all the assemblies in the app domain, I'm seeing the dependencies of those loaded dlls getting loaded from disk, even though they are already loaded.

This is what modules I'm seeing loaded after the DefinedTypes call:

AssemblyA
AssemblyA.dll
AssemblyB
AssemblyB.dll

The ones loaded with the .dll extension have a path that points to disk. The others don't.

What can I do to prevent the file's dependency from being loaded, when they're already loaded into memory?