How to read ONLY the metadata bytes from a .net assembly

Since Reflection only load no longer works in .NET Core, I'm implementing a wrapper around System.Reflection.Metadata to allow me to scan an assembly for the types it contains without a full assembly load.

Although the MetadataReader is basically working, I'm achieving it as follows:

 private unsafe MetadataReader LoadMetadataReader(
    string filename,
    MetadataReaderOptions options = MetadataReaderOptions.Default,
    MetadataStringDecoder decoder = null)
{
    buffer = File.ReadAllBytes(filename);
    pinnedHandle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
    var headers = new PEHeaders(new MemoryStream(buffer));
    var startOffset = headers.MetadataStartOffset;
    var metaDataStart = (byte*)pinnedHandle.AddrOfPinnedObject() + startOffset;
    return new MetadataReader(metaDataStart, headers.MetadataSize, options, decoder);
}

This means I am loading ALL the bytes for the assembly each time.

My question is: Is there a way of identifying and loading ONLY the subset of bytes from the file necessary to read the metadata?