How to load an assembly and get all types in it with C# (without ReflectionTypeLoadException)

I'm trying to build a plugin architecture where dlls are included in a zip file, unzipped, stored, then loaded so that a specific type can be constructed and worked with. In theory, it sounds doable. But I'm having an issue with it. I keep getting a

ReflectionTypeLoadException.

Here's what I have that isn't working:

        var dlls = pluginFiles.Where(p => p.Value.FileInfo.Extension.ToLower() == ".dll").ToList();
        int num = 0;
        foreach(var dll in dlls){
            var assembly = Assembly.LoadFile (dll.Value.FileInfo.FullName);
            var pluginTypes = assembly.GetTypes().Where (p => p.IsSubclassOf (typeof (Plugin)));
            foreach(var pluginType in pluginTypes){
                var ctor = pluginType.GetConstructors ().FirstOrDefault ();
                if (ctor == null) continue;
                var plugin = (Plugin)ctor.Invoke (new object [] { });
                if (plugin == null) continue;
                num++;
                _mgr.RegisterNew (plugin);
            }                      
        }

As I'm stepping through this in debug, the exception happens on the line where I'm running assembly.GetTypes().

What am I doing wrong here? Should I be doing something with a new AppDomain?

Further info: The assembly I'm loading is a test assembly. It does have a reference to FakeItEasy. It has a single class, which is the class I'm looking for.

I've looked at this. It seems like it might be applicable, but I can't fully tell how (and I don't quite understand why that answer works for that question).