Increase reflection performance by caching delegates

There are a lot of answers about how to speed up dynamic access to properties. One is to create a delegate that will be cached and in subsequence calls the delegate is used.

If I have a lot of different object for which I want to cache the getter and setter delegate I create with reflection then how should I store the delegates in a cache? I have tried to store them in a dictionary like this

Dictionary<Type,Dictionary<string, ISetter>>

Where ISetter has the delegates as a property. I found out that this is not really a lot faster than pure reflection and I assume this is because of the two layers of accessing a dictionary.

How should I store the delegates such that I really benefit a lot from the caching?