In the past, the most obstinate white opponents of integration (the Goldwater wing of the Republican Party and conservative Southern Democrats ) argued that people should have the right to associate--or not to associate--with whomever they wish.
White opponents of integration supported their argument on the basis of freedom of association or non-association more specifically.