Exception on opening DataSet Visualizer for .Net Core 3.1 in Visual Studio 2019

When trying to use the magnifing glass on one of my DataSet or DataTable in my .Net Core 3.1 WPF Project I get a System.IO.FileLoadException with following text:

Could not load file or assembly 'DataSetVisualizer.DebuggeeSide, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. General Exception (0x80131500)

Stack trace:

at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, RuntimeAssembly assemblyContext, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, AssemblyLoadContext assemblyLoadContext) at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext) at System.Reflection.Assembly.Load(AssemblyName assemblyRef, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext) at System.Reflection.Assembly.Load(AssemblyName assemblyRef) at Microsoft.VisualStudio.DebuggerVisualizers.DebuggeeSide.Impl.ClrCustomVisualizerDebuggeeHost..ctor(String debuggeeSideVisualizerTypeName, String debuggeeSideVisualizerAssemblyName, String[] probePaths) at Microsoft.VisualStudio.DebuggerVisualizers.DebuggeeSide.Impl.ClrCustomVisualizerDebuggeeHost.Create(String debuggeeSideVisualizerTypeName, String debuggeeSideVisualizerAssemblyName, String[] probePaths)

I narrowed the cause for this problem down to my recently implemented method to load assemblies from subfolders at runtime, which I wrote based on Reza Aghaei's answer on my last question.

In narrowed it down to the subscription on the AppDomain.CurrentDomain.AssemblyResolve event, but couldn't find a way to solve it yet.

AppDomain.CurrentDomain.AssemblyResolve += (obj, arg) =>
{
    var name = $"{new AssemblyName(arg.Name).Name}.dll";
    var assemblyFile = referenceFiles.Where(x => x.EndsWith(name))
        .FirstOrDefault();
    if (assemblyFile != null)
        return Assembly.LoadFrom(assemblyFile);
    throw new Exception($"'{name}' Not found");
};

It doesn't matter if I am trying to view the DataSet in one of the loaded assemblies or the startup application.

I would like to keep working with this method to load assemblies at runtime, but since I am working with a lot of DataSets being able to use the DataSet Visualizer is crucial for me.

Any suggestions?