Deryk Barker mentioned Messiaen in connection with this subject and James
Tobin mentioned synesthesia.

I'll elaborate a bit on this.

"Synesthesia" is strictly spoken a cerebral disorder where the person
in question perceives different sensations (light/color and sound) as
intertwined.  Especially sounds with a relatively high degree of order with
respect to frequencies (like musical chords) trigger a cerebral sensation
of color.

I was explained once that the effect is remotely similar to when you close
your eyes, press your fingers against your eyeballs and turn towards an
intense light source, where you'll see all kinds of colored circles and
spots (The similarity ends there!)

Actually, in it's medical sense, synesthesia is really a disorder; people
with this are seriously hampered in their perception of the world around
them.

Apart from that, there are people, and Messiaen was one of them, who
seem to experience the effect on a more intellectual level where especially
musical chords are related to colors.  Although there seems to be a
relation between pitches and colors, this is really not straightforward.
What seems to be important is not so much the actual pitch, but rather the
grouping and relative distance of the tones in a chord.

An interesting text on this can be found on
http://members.dencity.com/anand-s/synes.html

Messiaen declared that his experiences also were very complex because
almost always the colors he saw associated with chords are rarely "pure"
monochrome colors, but almost always colors with a lot of subtle nuances.
In his scores you will find descriptions like "red, almost crimson with a
shade of gray and gold".  Very generally speaking, the more complex the
chord, the more nuances.  He actually kept a little book where he took
down his experiences so that they would be with him more easily when he
composed.  E.g.  the last part of "La Transfiguration" is a very slowly
moving study of harmony revolving round the center key of E major.  It is
only in the last heard harmony of the work that all "shades" are removed.
Here the chord is very broad; if you play it on the piano, you'll really
have to stretch to the limit.  But, actually, because of the tuning of the
piano (well tempered since papa Bach) something was always lost to him when
the part is played on the piano.

Although you'll probably (like I) will not experience colors in the way
Messiaen did when he composed the part, anybody with ears to his head
will have to admit that there is something tremendous exhilarating in this
last chord.  It was very special to Messiaen anyway.  It is therefore no
surprise that a lot of his works end in E major (like St. Francis, the
opera).

Together with the birds and the Eastern rythms, another big source of
inspiration to Messiaen were stained-glass windows.  Obviously he must have
seen parallels between the light falling through Gothic church windows like
those of the Sainte Chapelle in paris and his music.

Furthermore: Messiaen was always intrigued by "the charm of
impossibilities".  This can be interpreted in the way that he devised
several musical tools (non-retrogadable rhythms, modes of limited
transpositions in harmony, "language musicale" in melody, etc.) that
served as self-imposed limitations and perhaps even disciplinary tools
for composition.  On the other hand there were devices of freedom (valeurs
ajoutes in rhythm, birdsong in melody) that served as the needed tools for
variation, juxtaposition and ornamentation.  In this light, his playing
with musical colors might perhaps be seen as the freedom tool for the
harmony component of music.

Here are some Messiaen pieces where "color" plays a major role:

1.  the afore mentioned last part of "La Transfiguration"

2.  part 7 ("Bryce Canyon") and also other parts of "des canyons aux
etoiles"; see http://www.soton.ac.uk/~modsem/saw-can2.htm

3.  the last movement of "Eclairs sur l'au dela"

4.  Colours de la Cite Celeste

Regards, Jos