How to efficiently group all implementations by interface from multiple assemblies?

I want to create a grouping using Linq that gives me back all implementation types keyed by the interface being implemented.

Basically, given a number of types in a number of assemblies, I want to get all interfaces that have at least one implementation that matches a certain criteria, and then for those interfaces I need to then get all implementations.

For example, suppose I have these classes:

interface IDoX {}

interface IDoY {}

interface IDoZ {}

class Impl1 : IDoX, IDoY {}

class Impl2 : IDoY {}

class Impl3 : IDoY, IDoZ {}

class Impl4 : IDoX, IDoY, IDoZ {}

Grouping these classes and interfaces, I should get back a result like this (in dictionary form, for example):

{
    IDoX, {Impl1, Impl4},
    IDoY, {Impl2, Impl3, Impl4},
    IDoZ, {Impl3, Impl4}
}

I need this because I have to analyze all implementations of a given interface to take a decision, and I need to do this across all loaded assemblies.

I'm trying to come up with something but since the reflection classes all give me back the results in the other direction it's kinda weird to do. For example, to get all implementations of a given interface, I need to search all assemblies and then check for each type if they implement the interface or not.

A naive approach to this lookup building would result in an insane amount of iterations. Is there a clean way of achieving this grouping that I need, or will I have to resort to a crazy amount of repeated iteration over all types on all assemblies?