I have come to the conclusion that checking explicitly for selinux contexts is not a good way to go, because it de-facto moves a little bit of the policy into the configuration.
To do selinux policy checking we should instead instruct admins to create a different socket and then use selinux policy facilities to lable the socket appropriately and have actual selinux policy decide on access.