Read Binary File Using Reflection of Target Class

New to c# and looking for help on reading a binary file (generated by a non windows machine) into one of several possible classes...

The problem: The binary files all start with a generic header, which I can read in with no problems. The header tells me which version of data follows. I read this and then create an object based on this information. I then use reflection to get the field info of this object and understand what I'm reading next. Then read in the appropriate number of bytes, and here is the problem. I would then populate the field with the data. I've written the following...

The Code:

        Type type = myData.GetType();
        BindingFlags flags = BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.Static;
        FieldInfo[] fields = type.GetFields(flags);
        int readSize;
        int readNumber;
        foreach (FieldInfo field in fields)
        {
            if (field.FieldType.IsArray)
            {
                Array myArray = (Array)field.GetValue(myData);
                readSize = Marshal.SizeOf(myArray.GetType().GetElementType());
                readNumber = myArray.Rank * myArray.GetLength(myArray.Rank);
                field.SetValue(myData, br.ReadBytes(readSize*readNumber));
            }
            else
            {
                readSize = Marshal.SizeOf(field.FieldType);
                if (readSize > 0)
                {
                    field.SetValue(myData, br.ReadBytes(readSize)); // FAILS HERE
                }
            }
        }

When I run this, it falls over at the point indicated with: System.ArgumentException: 'Object of type 'System.Byte[]' cannot be converted to type 'System.Single'.'

Further to this, the binary data can and does contain two dimensional C style arrays, but don't want to use unsafe to crowbar in a solution.

Possible Solutions...

I understand why I'm getting this error, thanks to Convert byte[] to single [closed]

But I can't keep switching based on system Type information (System.Single etc...), as covering a reasonable number of possibilities would be huge, and worth avoiding after reading some other good stuff here on the stack.

I also want to avoid Marshalling my target classes, as that would require a lot work creating them. And then, whenever the binary structures change (new firmware on the non windows system) I would have to create a new set again, making the whole thing slightly unmanageable.

Any suggestions, or possible other solutions!?