It curious that so many who promote this right so ardently are so selective in its application.
Few people would argue against people being able to chose their partners on whatever basis they choose. While some might criticize those who would choose a wife, husband or lover on the basis of race, few would advocate laws to make this illegal. The thought of someone being able to sue because they weren't chosen as a marriage partner because of their race is anathema to all but total fascists. With regards to personal relationships we have freedom of association.
Likewise few would advocate laws to force people to choose business partners by certain criteria. It is your business partner - it's your choice. Within our business ventures we have freedom of association.
Employees are able to choose their employer on whatever basis they like - even if it is as asinine as race, religion or political persuasion. The mere mention of some third party dictating who someone can and can't work for conjures visions of feudalism, communism, or slavery. Hence employees have freedom of association.
But apparently an employer is not free to select their employees on any basis they choose.
Suppose I run a small business, and want to employ someone to help me run it - say to answer the phones. Do I have freedom of association? My employee and I may be working in the same room 8 or even 10 hours a day. They can refuse to work with me on whatever grounds they like (race, religion, marital status and so on), but as the employer I apparently do not have reciprocal rights.
Some may argue that the relationship is fundamentally different - that employees (sellers of labor) are less powerful that employers (buyers of labor).
However, when I buy other products I have the right to choose. I may choose not to shop at a particular fast food chain because they are American owned, or the local pizza shop because I will be served by Italians, or the local Chinese restaurant because it is run by Asians.
Freedom of association protects our rights to choose these actions (foolish as they may be), but the same arguments are also used to prevent us making these choices. We are also told that employees must be free to be members of trade unions and suchlike without discrimination from their employer. Freedom of association in nearly every context means the freedom to discriminate on whatever basis you choose, but in the case of an employer it means the obligation not to discriminate.
It is idiocy to suggest that everyone has a self-evident right to associate (or not associate) with whomever they like for whatever reason except employers.
Those who truly believe in freedom of association believe it in all cases. People have the right to associate with whomever they choose regardless of the reason - even if it is their association with others.