Load assemblies with references from subfolders at runtime

I am currently working on a project which is supposed to work as a framework for several Add-Ons, which should be loaded at runtime.

I am tasked to have following structure in my application folder:

• 2 Directories with subfolders. One named "/addons" for Add-Ons and one named "/ref" for any additonal references these addons might use (like System.Windows.Interactivity.dll)
• When selecting one of the Add-Ons from a menu in the applicaiton, the .dll is supposed to be loaded at runtime and a pre-set entry point should be opened
• All references of the newly loaded assembly should be loaded aswell.

I know the subfolder and filename when an Add-On is being loaded, so I simply use Path.GetDirectoryName(System.Reflection.Assembly.GetEntryAssembly().Location)) and Path.Combine() to build a path to the .dll and then load it via Assembly.LoadFile() before using reflection with assembly.GetExportedTypes() to find the class that inherits for my 'EntryPointBase' and then create it with Activator.CreateInstance().

However, as soon as I have any references within my Add-On, an System.IO.FileNotFoundException targeting the reference will pop up at assembly.GetExportedTypes()

I built a method to load all referenced assemblies, even made it recursive to load all references from the references, like this:

public void LoadReferences(Assembly assembly)
{

  var loadedReferences = AppDomain.CurrentDomain.GetAssemblies();

  foreach (AssemblyName reference in assembly.GetReferencedAssemblies())
  {
    //only load when the reference has not already been loaded 
    if (loadedReferences.FirstOrDefault(a => a.FullName == reference.FullName) == null)
    {
      //search in all subfolders
      foreach (var location in Directory.GetDirectories(Path.GetDirectoryName(System.Reflection.Assembly.GetEntryAssembly().Location)))
      {
        //GetDirectoriesRecusrive searchs all subfolders and their subfolders recursive and 
        //returns a list of paths for all files found
        foreach (var dir in GetDirectoriesRecusrive(location))
        {

          var assemblyPath = Directory.GetFiles(dir, "*.dll").FirstOrDefault(f => Path.GetFileName(f) == reference.Name+".dll");
          if (assemblyPath != null)
          {
            Assembly.LoadFile(assemblyPath); 
            break; //as soon as you find a vald .dll, stop the search for this reference.
          }
        }
      }
    }
  }
}

and made sure all references are loaded in by checking AppDomain.CurrentDomain.GetAssemblies(), but the exception stays the same.

It works if either all assemblies are directly in the application folder, or if all references fro the addon are already referenced by the startup application itself. Both ways are not suitable for my case, because higher ups demand on this file system and Add-Ons with new references should be able to load without touching the appication itself.

Question:

How can I load assemblies from one subfolder and their references from another without a System.IO.FileNotFoundException?

Additional information:

• Application is in the new .csproj format and runs on <TargetFrameworks>netcoreapp3.1;net472</TargetFrameworks>, although support for net472 should be ceased soon (currently still debugging in net472)
• Most Add-Ons still have the old .csproj format on net472
• the ref subfolder is sturtured in subfolders aswell (devexpress, system, etc.), while the addon subfolder has no further subfolders.