Dear Mary:
I have no specific personal knowledge regarding poisonous pollen.
However, a visiting friend from Northern Virginia once told me that they have
a
"japanese" flowering tree in their yard that attracts insects by the hundreds
and
that they think that the tree is poisonous because they see "piles" of insects
that drop dead under the tree while it is in bloom.  The insects stop dieting
soon as there are no more flowers. I think that it is logic to assume that
such tree is toxic to insects!
Sincerely.
Dr. Rodriguez
Virginia Beach, VA