This has been a very annoying issue for me, switching to the main menu to check chats and other games that are starting causes a bit of FPS loss. Each time you do it it gets worse.

I did a test by spamming my console key when the pick/ban interface was open and I got it down to 3 FPS. After the players finish picking the FPS goes back to normal.

See here..