The Quirk
Using the SharePoint GUI to modify list or library settings with the goal of changing the ‘new button order’ or ‘default content type’ may yield unexpected results.

Confusingly (or perhaps to avoid confusion) the control which displays the ordering of the content types for the New button, along with their visibility, does not show the items in the order in which they have been set.

As an example of this, I will set a content type marked as not-visible to have an order of 1.

I press OK to save the form. Upon returning to the form it would be reasonable to expect to see the control rendered with the non-visible content type in position 1 as we have just set it to be this way.

However, this is not the case. You will see the content types listed in same order as before the change was made.

The control always displays the visible content types at the top of the list which are then sorted by order number. The content types which are not marked as visible are then shown in alphabetical order. Importantly, this is just how they are displayed. The actual content type order may be different depending on how you have set it.
So who cares?
Most likely there are very few people who will even notice this let alone care. In fact, in nearly all scenarios it makes perfect sense to render the control this way. That is, until you want to have a default content type that is not visible under the new button. At this point the control may become very misleading. There is really just one case where the control does not accurately display the functional impact of changing the content type order. When you set a non-visible content type as the default content type (position 1), it is still treated as the default content type, despite how the control may render it.
Why would you want a default content type that is not visible under the new button?
In my case it was because in the library’s root folder only document set content types and a particular document content type (A) were allowed. However, inside the document set only a different document content type was allowed (B). By setting the document set’s allowed content type you can have the new button display content types that are not listed as visible in the list settings. All good, you can only create documents of content type B inside the document set and only create document of content type A outside the document set.
My issue arose when a user uploaded a document to the document set. An uploaded document ignores the document set’s allowed content type and is created as the first content type listed on the New button order in the list settings. To workaround this issue I set content type B as position 1, which works as expected despite how the control may render it.
Paul, Thanks for the post, I had been dealing with a similar problem and this pointed me in the right direction, if it helps, I’d like to add that in that document set (default) scenario, if you make all other CTs not visible, they won’t change order, but if you make the one you want as default visible (inside a document set), then change the order to 1, it will display it as 2, under the document set, then you can change it back as not visible.
In short, it seems like you can only change order of CT if you make it visible, for me the default never changed when non visible.
I hope this makes sense 🙂
This made perfect sense and worked for me. Thanks Ricardo!
Thank you! That’s exactly what I was looking for!