Difference between how Visual studio loads assemblies and Assembly.LoadFrom?

I have an assembly (2 actually) both are from SAP directory C:\Program Files (x86)\SAP\FrontEnd\SAPgui.

If I add the assemblies to my Visual Studio project via Add Reference the code works perfectly fine without issue. The problem is I need to (for reasons I'd rather not have to explain) be able to load the assembly on the fly via reflection and not directly reference it.

If I use either of the following though it gives the following error.

Assembly.ReflectionOnlyLoadFrom(@"C:\Program Files (x86)\SAP\FrontEnd\SAPgui\sapfewse.ocx");
Assembly.LoadFrom(@"C:\Program Files (x86)\SAP\FrontEnd\SAPgui\SapROTWr.dll");

Could not load file or assembly 'file:///C:\Program Files (x86)\SAP\FrontEnd\SAPgui\sapfewse.ocx' or one of its dependencies. The module was expected to contain an assembly manifest.

How can I load the assembly through reflection and have it actually work? It has no problem using the assembly when I add it as a reference so I imagine the problem is with my assembly loading and not the assembly itself.

After looking inside the project file itself, it has the two references listed below isntead of just <Reference Include="..." />.

<COMReference Include="SAPFEWSELib">
  <Guid>{5EA428A0-F2B8-45E7-99FA-0E994E82B5BC}</Guid>
  <VersionMajor>1</VersionMajor>
  <VersionMinor>0</VersionMinor>
  <Lcid>0</Lcid>
  <WrapperTool>tlbimp</WrapperTool>
  <Isolated>False</Isolated>
  <EmbedInteropTypes>True</EmbedInteropTypes>
</COMReference>