B.E.T.’s co-founder Bob Johnson said he’s never going back to the Eau Palm Beach, a 5-star luxury hotel located in Florida. That’s because a staff person demanded that he remove his prescription sunglasses, but he refused.

According to Johnson, it happened on Aug. 24 when he came back from a fishing trip and tried to check in. Although he handed his driver’s license, passport and American Express card to the front desk staff person, he still wasn’t given access to his room.

“Mr. Johnson, you need to take off your sunglasses,” the 72-year-old said he was told.

At first he thought the staff person was just being humorous, but she explained that removing dark glasses is the hotel’s policy to protect guests from fraud. But Johnson refused and asked the woman to call the police, who eventually arrived.

After they spoke, the hotel workers said they wouldn’t change the rule, so Johnson left. The multimillionaire also said that he owns 165 hotels himself and considers the sunglasses rule ridiculous.

“I think it’s a silly rule, and I think it has overtones of racial profiling,” he said.

Eventually, a spokesperson for Eau Palm Beach released a statement about the incident, as well as the hotel’s policy.

“It is standard operating procedure in the industry to ask to make sure that the person who is presenting themselves to check-in and they present identification, that it does match,” said Nick Gold. “And if they’re wearing dark glasses with a ball cap low over their face, you want to make sure that it is the same person, much like what would happen at TSA rules going through the airport.”

“We’d love for him to come back,” he added. “All guests are welcome here at Eau Palm Beach Resort and Spa, and we pride ourselves on how we treat our guests.”

Johnson said he’s familiar with the policy but only outside of the U.S., not domestically.