How do you remove a favorite server?

I miss-clicked on the favorite button and now it sticks out like a sore thumb, it severally bothers me and i need help, the remove favorite button also refuses to work

The same button that you click to set a Favorite, removes the Favorite