Yellow jackets are wasps, belonging to the same order as honey bees, ants,
horntails, and hornets.  They can develop pretty large colonies, especially
certain species.  They are cavity-nesters and will often nest in a burrow
they find in the ground, or in man-made structures.  Again this varies with
different species.  Some tend to build aerial nests under eaves or in tree
branches.  They can often find space, too, behind or inside old
railroad-ties used for landscaping.  This year a neighbor "discovered" them
in his compost heap!
 
You are very observant to notice the queen-like ones.  Normally the colony
is established in springtime by a single, fertilized female (queen). She
lays a small patch of eggs in a tiny comb, which she constructs from wood
pulp.  Eventually the first batch of younsters hatches and they continue
with the foraging and nest construction.  Then the queen eventually settles
down to egg-laying only, and the subsequent cycles of brood continue to
populate and enlarge the colony.  Some species will grow to a few thousand
wasps; the population peaks in late summer.  This is when most stings are
encountered.
 
In late summer/early autumn, the colony begins rearing "reproductives."
These are fertile females (queens) and males.  These are probably what you
have seen around the nest.  Each new queens will mate and find a place to
overwinter, which she does individually.  Sometimes you can find a young
queen in winter, under some loose bark, hiding in a wall, or under a log.
She endures the winter, all alone, and in spring searches for a new nest
site and starts the process all over again.
 
The non-reproductives (workers) and old queen perish soon after frost.  So
they won't be around much longer to bug you.  However, towards the end of
the season they get pretty aggressive, so be careful!