Merging activerecords :has_many associations with unique constraint

I'm trying to write a concern Mergeable for generic deep merging of activerecords. Code is rough draft, may not compile; Given:

class Faz < ApplicationRecord
  has_many :bar
end

class Bar < ApplicationRecord
  belongs_to :foo
  belongs_to :faz

  # has index :bars, [:foo_id, :faz_id], unique: true
end

class Foo < ApplicationRecord
  include Mergeable
  has_many :bar
  has_many :faz, through: :bar
end

I want to be able to call foo1.merge!(foo2)

Using class.reflect_on_all_associations, I'm grabbing a list of :has_many reflections to perform merging (regular columns, and has_one are handled already afaik)

has_manys = self.class.reflect_on_all_associations(:has_many).select do |has_many|
  exclude.exclude?(has_many.name) && !has_many.options[:through]
end

has_manys.each do |has_many|
   local = public_send(has_many.name)
   remote = other.public_send(has_many.name) - local
   remote.reject! { |r| r.try(:skip_merge_if, self, other) } # callback hook for some custom logic

   local << remote # ActiveRecord::RecordNotUnique
end

I'm stuck here, looking for advice on how to resolve ActiveRecord::RecordNotUnique. In this case I don't know the through relation is defined (from foo -> bar -> faz) since I'm working from Foo's side, so I'm not sure how to do the dedup.

I could add an option to specific through relations, but in product I have 20-30 associations to go through. If there is a cleaner soln, its greatly appreciated.