I don't know, what App store is doing during the install, but using it to manage Xcode has always been extremely irritating.

It's a lot easier to manually download Xcode from Apple's dev portal. The download speed always maxes out my internet connection, and installing is just a question of unpacking the archive, which doesn't make more than a few minutes..