Cannot load a certain assembly due to version conflict

One assembly (exe) depends on a nuget package (actually serilog.settings.configuration 3.1.0) This package depends on microsoft.extensions.options.configurationextensions (>=2.0.0) which in turn depends on microsoft.extensions.configuration.binder (>= 2.0.0) All this is ok.

However I have other projects (class libraries) in the solutions which are not referred to from the exe project (to be loaded at runtime) and one of these depends on a newer version of microsoft.extensions.configuration.binder (6.0.0) all is not ok. The new version cannot be loaded because the exe project has already loaded the old one.

I can solve this problem by explicitly add microsoft.extensions.configuration.binder (6.0.0) as dependency for the exe project, but I guess there is a better way to do this, is it? (If the dynamically loaded assembly which depends on the newer version for some reason e.g. users choice, is not loaded I will end up with unnecessary new assembly loaded in the process, perhaps not a big problem in this case, but could be in other cases).

I am using .Net 6 so loading into separate AppDomain is (afaik) not an option