Are you sure that the bees dive bombing you are honey bees from your hives?
As you are gardening some distance from the hives could they be ground
nesting solitary bees whose nests you are disturbing? If the bees were
aggressive you would be stung not just buzzed.
Chris