There are physical differences between the sexes; however, they are not always apparent.
1) Males can have a longer second dorsal fin; however, in most cases this longer fin "thread" is absent because it has either been nipped off by other fish or damaged in transit or in the more crowded confines of a dealer tank. When I bought my 2 Banggais the length of this second dorsal "thread" on both fish was basically identical. Only later did the males thread grow back to be maybe 1/2" longer than the females...
2) Males will generally be larger; however, the age of any given fish specimen also plays a large role in its size. You could have a juvenile male and an older female - and, in that case, the female would actually be the larger of the 2...
3) Males will have a greatly pronounced jawline - but only when they are carrying eggs. At all other times the jawlines of males and females look very similar...
So, yes, under most circumstances the only way to pretty much guarantee you will end up with at least 1 fish of both sexes is to purchase 4 or more and let
them figure out who is a boy and who is a girl...