Cannot read CustomAttributes in assembly using reflection

In this question I posted before, I had a problem inspecting an assembly because some of the types referenced there were defined in a separate assembly.

Trying to read the name of a custom attribute

This question is kinda related, however different as, even with that fix in place, I have problems reading the CustomAttributes of one type.

TypeInfo mytype = ...
IEnumerable<CustomAttributeData> customAttributes = null;
try
{
  customAttributes = mytype.CustomAttributes;
} catch (TypeLoadException e)
{
  // Could not access the attributes
  throw new Exception("Type not loaded for the attribute", e);
}

The point is that e has the name of the attribute class that I am using for that type, so I can read its full name in the exception:

Could not load type 'System.Runtime.CompilerServices.ScriptNamespaceAttribute' from assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.":"System.Runtime.CompilerServices.ScriptNamespaceAttribute

Assembly loaded in reflection only context

I have loaded the assembly by using:

Assembly.ReflectionOnlyLoadFrom("Path to my assembly")

So I was expecting to be able to access types and read at least their names. It happens with classes and other types, but for custom attributes I have this problem. But if I open the ILDASM (the .NET Disassembler on my assembly). I can see ScriptNamespaceAttribute being applied to some of the classes defined in the assemblies.

Question

How can I safely read the custom attributes in an assembly which references types defined externally? I do not need to have the implementation for these classes, just their names is what I need.