Manually resolving assembly does not work as expected

Background:
I have an executable, which depends on a third party, let's call it library.dll. I have 2 separate installations of this dll: One for 32 bit and one for 64 bit. The library.dll needs to be loaded from a specific location, since it is not in GAC and neither in my executables folder. So I use the AppDomain.AssemblyResolve event to load library.dll from the correct path. My exe is compiled with "Any-CPU" flag set; note that this application can be started either in 32 bit environment or a 64 bit environment. In my VS project, I can only refer to one version of this library (either 32 bit or 64 bit) while doing the build.

Problem:
When I access the types within the library, it only works for the environment (32 or 64) of the referred library.dll. For example: If I have a reference to 64 bit version of the library.dll in my csproj, then my app loads the library.dll successfully and is able to access the types within that assembly, only in the 64 bit environment, but not in the 32 bit environment. In the 32 bit environment, I get a FileNotFound exception before accessing the types from library.dll. Also, my AssemblyResolve event does not get invoked in that case.

Here's what I understand so far:

1. If I do not use a reference in the csproj, it will work fine, but that is not feasible, since all my type access will have to be done using reflection.
2. When I add a reference (csc /reference) in my project, the compiler will auto-inject some code to load this dll before the usage. Apparently AppDomain.Current.AssemblyResolve event does not get fired while that happens.
3. If I assume that the library.dll will be in the folder with the executable, then I don't need to implement AppDomain.Current.AssemblyResolve event to manually load the library.dll. In that case, my executable works fine in both 32-bit and 64-bit environments.

Conclusion & Questions:
It appears that the Framework is able to load the dll correctly irrespective of 32/64 bit environment, but I am not - so I am not loading the library.dll correctly. I am aware that a simple workaround to this would be to build 2 different versions of my exe, one for 32 bit with the 32 bit reference and one for 64 bit with the 64 bit reference. A side question is, why does the AppDomain.AssemblyResolve event not get fired when the dll is loaded based on c# auto-generated code to load the referenced dll. I have tried experimenting with using AppDOmain.ReflectionOnlyAssemblyResolve event with not much luck. I also wonder if AppDomain.TypeResolve event somehow plays a role in this situation.

Can you folks help me fill in the blanks in my knowledge about this process

Below is my sample code to demonstrate the issue. In my project, I have reference to a library.dll. Remember that I have a 32 bit version and a 64 bit version of this library available. You will have to build the application as 32 bit or 64 bit to simulate the 32/64 bit environments

using System;
using System.Reflection;
using System.IO;

class Program {
    static void Main(string[] args) {
        AppDomain.CurrentDomain.AssemblyResolve += new ResolveEventHandler(AssemblyResolver);
        Console.WriteLine(string.Join(", ", Enum.GetNames(typeof(MyEnum)))); // access some type from within library.dll
    }


    public static string GetLibraryPath() {
        return @"C:\Program Files (x86)\library";
    }

    private static Assembly AssemblyResolver(object sender, ResolveEventArgs args) {
        Assembly asmb = null;
        string splitchar = ",";

        // args.Name is like: "Library, Version=1.0.0.26173, Culture=neutral, PublicKeyToken=null"
        string module = args.Name.Substring(0, args.Name.IndexOf(splitchar));

        string dllPath = GetLibraryPath() + @"\" + module + ".dll";

        Console.WriteLine(String.Format("AssemblyResolver: About to load '{0}'", dllPath));

        try {
            if (File.Exists(dllPath))
                asmb = Assembly.LoadFrom(dllPath);

            if (asmb == null)
                Console.WriteLine("Unable to load assembly " + dllPath);
            else
                Console.WriteLine("Successfully loaded " + dllPath);
        } catch (Exception ex) {
            Console.WriteLine("Unable to load assembly: " + dllPath + "\n" + ex);
        }

        return asmb;
    }

}