J. K. Rowling describes Mrs. Dursley as having a long neck that was useful for spying on her neighbors over the fence. But she never tells us who those neighbors were or what was it they did that Mrs. Dursley was spying on. Rowling probably knows. She knows everything about her world. She once said she knew the names of all the players in all the quidditch teams.
Readers do not have the patience to read lengthy descriptions anymore (or so they say), so there is a limit to the number of words you can cram into a book. If you create an elaborate world, you will have to create a lot of detail that you know you are not going to write. This applies to every genre.
For example, in order to create convincing characters, you should write a short biography for each during the writing process. The information in the biographies may not appear in the book. It is just some background knowledge that the writer has.
If there is any kind of writing that epitomizes the idea that “writers write to write, not to be read” it must be the so called “fantasy” writing, which involves lots of world building. Much of what the writer writes is hidden from view. You might write the entire history of a kingdom and little of it will ever appear in the story. The more elaborate the fantasy world, the smaller the proportion the reader will see.
As a consequence, all fantasy worlds begin to look alike. All dystopian futures begin to look alike. All space travel stories begin to look alike. But in the mind of the writer, there is a distinct world with distinct images, landscapes, and inhabitants. How do you project that elaborate world to the reader? How do you explain the world without lengthy boring expositions?
First, you must grasp the correct definition of world building. You do not start your world building by saying “Barahir, son of Hador, was the eighth Ruling Steward of Gondor, and was succeeded by his son Dior” or “The Thermians are under attack from Sarris and need the help from Jason who will activate Omega 13”. You are just attaching names to characters, places and devices. That is not world building. World building is when you attach emotions and personality to places, customs, objects and characters.
A British royal wedding is a massive event resplendent with ornate carriages, marching soldiers in period uniform and church bells resonating through the city. A Japanese Imperial wedding is a somber and private series of ceremonies held within the walls of the Imperial Palace with little music and less fanfare but much sanctity and seriousness. Same event, different emotions. These emotions are what make your world.
When creating a stage, it is easy to be tempted into making up fantastical names for everything and describe strange landscapes under the assumption that these things, when amassed in sufficient quantity, will explain the world we created. They do not. But whether or not Hobbits drink and dance during weddings or young wizards-to-be suffer peer pressure on the Hogwarts Express are important elements of world building. Ron Weasley feels a little bit inferior because he does not have the money to buy treats on the Hogwarts Express. The fact that such emotions exist in the wizarding world is an important element.
World building entails the creation of large amounts of information that will not be included into the pages of the novel. But it is ultimately about emotions. A sacred cave does not have a sign outside that says “Sacred Cave”. You have to make it sacred through the attitudes of your characters. And in the modern novel, in which lengthy expositions are shunned, you have to cut straight to the chase and describe what people are doing there and how they are feeling.
And that is why we never learn what Mrs. Dursley was looking at when she spied over the fences, or the names of all the quidditch teams. We do not need those to understand the world around them.