How can we replace a dynamically referred DLL

I have a requirement where I need to replace a dynamically referred DLL in a project.

The scenario is that I have an EXE and a DLL. The EXE contains a dynamic reference to the DLL. The EXE will be running on the client machine all the time. Without killing the application, I want to deploy the latest DLL on the client's machine.

I thought that I can achieve that by referencing the DLL dynamically in the EXE. However once the EXE is started and running, when the DLL is replaced with a new one, "The file is in use error" is thrown by the system.

It might be that the implementation that I have done will not help me achieve what I want. Is there another way by which I can achieve what I want? Or should I modify my approach some way to achieve it?

Below is the code that I am using to dynamically refer the DLL in my EXE project:

Assembly assembly = Assembly.Load("MyAssemblyName");
Type businessLogicType = assembly.GetType("ClassName");

object businessLogic = Activator.CreateInstance(businessLogicType);

MethodInfo initializeMethod = businessLogicType.GetMethod("Initialize");

initializeMethod?.Invoke(businessLogic, null);