What's the most efficient way to find all other assemblies that reference a specific assembly?

I often find myself wanting to iterate over all types in the current environment for various reasons such as to find everything that inherits from a particular class or to find all types with a specific attribute.

I can do this using System.AppDomain.CurrentDomain.GetAssemblies() to get all the assemblies, then Assembly.GetTypes() on each of them and iterating through all the types. But this is needlessly inefficient; in the Unity editor in a project containing a single script, this method goes through 8590 types, most of which can't possibly meet the criteria I'm searching for anyway. For example, nothing in mscorlib.dll or UnityEngine.dll is ever going to have my custom attribute or inherit from one of my classes, so I should just skip those assemblies.

So now, I'm trying to find all assemblies that reference any given assembly, but I'm having trouble coming up with an efficient algorithm for finding them since I can only get an array of assemblies that the target references but not an array of assemblies that reference the target.

Also note that if assembly A references B and B references C, when searching for everything that references C I would need to get both A and B (in case something in B inherits from the class I'm looking for and something in A inherits from that class without A referencing C directly).