Different type hash code with custom plugin architecture

I am building a custom plugin architecture where I want to store the implementations of interfaces in a Dictionnary.

I know there are frameworks that adress the plugin thing in .Net, but I do it to better understand the problematics under the concept of a plugin oriented application.

Now the problem:

I reference a common library in both the main application and the plugins, but the plugins have their own folder (different from the main application). Since the plugins use the common library, it is copied in that folder during the build. So I have 2 places where I find the common library (main application Debug folder, and plugins Debug folder).

When I dynamically load the interface type the plugins use, I then get the plugins folder for the fully qualified name of the module of the type. But when I try to get the plugin implementation using generics:

ServiceProvider.GetService<IPluginService>

I get the main application folder for the fully qualified name of the module of IPluginService.

Of course the hash code doesn't match, and I can't retrieve the plugins corresponding to that interface.

So my question: is there a way to tell the plugin AND the main application to use the same common library file? I tried to use the exact plugins folder library in the main application but it still gets copied into the main program folder.

I could use the namespace instead of the type for indexing in the dictionnary, but I'd like to avoid that.