World-building is a little like character-building. You want it there where it supports, folds into, and enriches the story; you don't want to go so far that it becomes its own tangent and distracts or takes away from the narrative.
Regarding "support", it's sometimes what the reader doesn't see that makes the world vivid and real. Imagine, for example, that there are five different castes in the world of your creation, and your story engages that Lia's caste regarded as so wretched and Waat's so exalted that he can, and does, rape her in the middle of the street and no one, including poor Lia's family, thinks to intervene.
Now, it is genuinely wonderful that you, the author, know what every caste is called, how they inter-relate, the kinds of work each does, the resentments that exist between them, which districts of the city-state they live in, the most prominent families of each caste, and so on, and so on.
Do you need to include all that detail in the story? Should you put all that detail into the story?
Hell, no! Much less should you extrude and contort your story to pack in all that detail.
Having that detail, however, even if it's only in some little file folder tucked into the back of your own mind, can help make your writing stronger and more confident. The world feels more real because you, the writer, know how it works, the various gears and cogs whirring away in the background while the focus of your action takes place.
From there, the writer can pick and choose what enriches the particular "slice of life" they're choosing to reveal to the reader. Going back to the earlier example, the biggest hindrance for Waat is that raping (which he, delusionally, considers "seducing") a young woman of Lia's caste is considered stooping; he chooses to do this thing in the midst of Lia's caste because it makes it less likely that word of his dalliance with a lower caste will follow him. Lia's father, looking on, remembers a similar event that happened to his sister and wonders if things will ever change; he looks on in a combination of fascination and embarrassment, wanting to look away, telling himself that he's only waiting for the moment that he can help her, but horrified to find that he's aroused. Other members of Lia's caste try to continue their business, knowing that failing to get their work done will bring trouble to the neighborhood as surely as trying to intervene.
I don't need to know that Waat's caste deals in spices, but it might be an interesting detail that his fingers smell of cardamom and cinnamon. Lia isn't going to be thinking about the amount of money Waat's caste brings into the city-state, but she might fleetingly wonder what it might be like to live such a privileged life. Let the reader recognize that there is life beyond the piece of action that they get to see, and the story will have life beyond the confines of the narrative.
It is fun and useful for an author to be able to "interrogate" a setting or a character and realize that the answers come easily, slotting together with precision in a way that suggests the place or the person has life. Just don't imagine that that life becomes richer for being scrutinized under a microscope. When it comes to what gets onto the page, less is often more.
