How can I dynamically load assemblies referenced by project, but not referenced in code

Consider a .NET Core application that references a NuGet package.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.2</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="MyPackage" Version="1.0.0" />
  </ItemGroup>

</Project>

If my code references a type in MyPackage, then the MyPackage assembly will get loaded. If I print out all the referenced or loaded assemblies then it will appear.

static void Main(string[] args)
{
    // Because I have a reference to a type in MyPackage, the assembly 
    // is loaded and will be printed out by both foreach statements below.
    var throwaway = typeof(MyPackage.Cars);
    foreach (var an in Assembly.GetEntryAssembly().GetReferencedAssemblies())
    {
        WriteLine(an.Name);
    }

    foreach (var assembly in AppDomain.CurrentDomain.GetAssemblies())
    {
        WriteLine(assembly.FullName);
    }
}

However, if I get rid of that throwaway line, then then assembly is not loaded, and so is not available by either GetReferencedAssemblies or GetAssemblies.

The .NET Framework had this same problem, and the fix was typically to read all the assemblies in the executing folder and load them manually - something like:

Directory
    .GetFiles(executingFolder, "*.dll", SearchOption.TopDirectoryOnly)
    .Select(AssemblyLoadContext.Default.LoadFromAssemblyPath));

However, .NET Core will load assemblies from other places (such as the NuGet cache - I've not found a comprehensive description of the new binding process as of yet), so the above approach won't work.

Question

So, my question is: How can I dynamically load all DLLs that are referenced by my csproj file (as NuGet PackageReferences). My use-case is fairly esoteric so I don't think any other mechanism will do.

Background

Ok, so someones going to ask what my use-case is, so here it is.

We have a set of interfaces that define messages (IAuditEvent, IValidationEvent, that sort of thing). We also have various implementations of these interfaces for different serialization formats (Protobuf, XML, JSON, etc). Each of these is a separate NuGet package (MyMessages.Proto, MyMessages.Xml).

We have a factory that will create the appropriate implementation (factory.Create<IAuditEvent>()) but it does this using reflection - e.g. the proto factory finds a class that implements IAuditEvent but also is a Protobuf generated class. It can't work if the assembly hasn't been loaded in the first place...