How do you remove a favorite server?

1

I miss-clicked on the favorite button and now it sticks out like a sore thumb, it severally bothers me and i need help, the remove favorite button also refuses to work

2

The same button that you click to set a Favorite, removes the Favorite