2005 bonneville window control panel removal

I have a 2005 Bonneville. The control of the front passenger window no longer works from the drivers side control panel. From the passenger side control button the window will go up and down correctly. I can move the 3 other windows up and down from the drivers side window control panel . Is there a trick to removing the drivers side window control panel in the arm rest of the drivers side? Or, do I have to remove the door panel to access the switches? Any advice would be appreciated