You are asking about this particular statement of Jesus from his final discourse to his closest disciples in the Gospel of John (14:2): "In my Father’s house are many rooms; if it were not so, would I have told you that I go to prepare a place for you?" (RSV)
You indicate that what bothers you about this is that you always thought of heaven as basically one big space. That is clearly not what Jesus is describing. Metaphysically we understand that he is not describing a specific distant place called heaven but a state of consciousness that we can achieve wherever we happen to be. At different times Jesus describes this kingdom of heaven as being "at hand" and as being "within." Here he’s assuring his disciples that whether he’s with them in human form or not, he will not be "gone." He will be exploring another of the "many rooms"—another dimension of the consciousness that is the kingdom of heaven. Jesus is always one with us in the Christ energy we share; and by moving our collective consciousness forward through his teachings and example, he is preparing a place for our own Christ consciousness to achieve and express.