Ed Zubrow writes of Mahler's fifth:

>The stylistic evolution from Brahms' music was cited and this is
>where my question arises.  Certainly the "melody riding on a cushion of
>accompaniment" describes homophonic music.  And, in Brahms and those who
>inspired him, we see the beauty of polyphonic voices: multiple thoughts
>but all related to each other and joined by formal properties.  So, if
>Mahler represents something new; if it is neither homophonic or polyphonic;
>what is it?

It's polyphonic, but with a greater degree of independence in each line
and, characteristically, more lines.  Melody with accompaniment is not
necessarily homophonic, but you do usually know which is which.  Mahler
has this in the 5th symphony as well, just as Brahms has great independence
of line in such passages as the culminating passacaglia of the Haydn
Variations.  But in general, greater independence of voices tends to blur
background/foreground distinctions.  One thing Zander might be getting at
is the proportions of such passages in each composer.  Occurrences highly
independent counterpoint in Mahler probably take up more of the piece.

>How do we characterize this style? indeed what keeps it from disintegrating
>into cacaphony? Is Mahler truly an innovator here or are there others
>moving simultaneously towards this?

"Cacaphony" is often in the ear of the beholder.  It generally indicates
that the listener cannot make enough sense out of what he's hearing, as
opposed to the contention that there is no inherent sense to be made.  It's
hard for me to conceive of a completely meaningless piece of music, but I
can't rule it out.  Composers try to work against such muddle by keeping
"planes of sound" distinct.  This is often Mahler's tactic - separating
the orchestra into registers or timbres and assigning a particular musical
idea to a specific division.  One also notes this in Ravel, Vaughan
Williams, Britten, Prokofiev, and Rubbra.  However, there's also those
miracles of orchestration - Hindemith's Mathis der Maler, the finale to
Mahler's 5th and Vaughan Williams's 4th, Webern's arrangement of the Bach
Ricercar from the Musical Offering - in which everything seems "jumbled
up" coloristically, yet contrapuntally clear.  We must also mention the
performer's role, since none of this happens by itself.  A bad performance
can give a completely erroneous impression of a work.

At any rate, many other composers move toward this kind of contrapuntal
independence - Taneyev has been mentioned, but also Reger and others.  In
fact, it seems a characteristic of Wagnerian and post-Wagnerian music in
general to load a passage with several counter-lines - in Wagner's case,
often to increase the amount of dramatic allusion of a passage by combining
several different motifs.

Steve Schwartz