Remoting exception when invoking code in an assembly loaded using reflection

We have an implementation assembly which contains remoting code. The said code works fine in the console application when there is a direct reference to the implementation assembly, but when the console application loads the implementation assembly using reflection (Assembly.LoadFrom or Assembly.UnsafeLoadFrom or Assembly.Load(AssemblyName)) a remoting exception occurs.

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Security.SecurityException: Type System.Runtime.Remoting.ObjRef and the types derived from it (such as System.Runtime.Remoting.ObjRef) are not permitted to be deserialized at this security level. Server stack trace: at System.Runtime.Serialization.FormatterServices.CheckTypeSecurity(Type t, TypeFilterLevel securityLevel) at System.Runtime.Serialization.Formatters.Binary.ObjectReader.CheckSecurity(ParseRecord pr) at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ParseObject(ParseRecord pr) at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Parse(ParseRecord pr) at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryObjectWithMapTyped record) at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryHeaderEnum binaryHeaderEnum) at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run() at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage) at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage) at System.Runtime.Remoting.Channels.CoreChannel.DeserializeBinaryRequestMessage(String objectUri, Stream inputStream, Boolean bStrictBinding, TypeFilterLevel securityLevel) at System.Runtime.Remoting.Channels.BinaryServerFormatterSink.ProcessMessage(IServerChannelSinkStack sinkStack, IMessage requestMsg, ITransportHeaders requestHeaders, Stream requestStream, IMessage& responseMsg, ITransportHeaders& responseHeaders, Stream& responseStream) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at