This is an extremely complicated and controversial topic. Paul Grobstein in the department of biology at Bryn Mawr College offers the following thoughts:
"These are fascinating questions, but I have to start with a warning. ' Intelligence' and 'cognitive ability' are concepts that are probably too ill defined to use in relation to either evolutionary trends or mechanisms. It is increasingly apparent that both words are umbrella terms covering a large variety of distinguishable skills, processing abilities and attributes, which different organisms possess to different degrees. Antonio Damasio's recent book, Descartes' Error (Avon Books, New York, 1994), provides a particularly clear example of the underlying complexity of these concepts in the case of humans, showing compellingly that ' intelligence,' as we usually use the term, depends fundamentally on both rationality and emotion. Whether it will ever be possible to define the meanings of intelligence and cognitive ability sufficiently clearly to use them for the kinds of questions you are asking is uncertain at best.
It is, however, possible to talk about both evolutionary trends and evolutionary advantages in relation to a better-defined variable: brain size. For exactly the reasons mentioned above, it is not possible to equate brain size with either intelligence or cognitive abilities. But studying brain size yields insights that may be relevant nonetheless. Useful observations were documented and discussed in an article by Harry J. Jerison in Scientific American ('Paleoneurology and the Evolution of Mind", January 1976). Brain size--or, more accurately, brain size in relation to body size--has clearly increased over evolutionary time, not only in human lineages but in those of many other groups of organisms as well. In general, however, what has occurred is not the replacement of smaller- brained organisms by larger-brained ones but rather an expansion of the range of brain sizes: larger-brained organisms appear later in time, but smaller-brained organisms continue to persist. The only really valid criterion for evolutionary success is current existence, so the conclusion would seem to be that large brains do not, in general, confer an evolutionary advantage over small brains. Instead large and small brains seem to represent different but equally good evolutionary outcomes.
"It is helpful to bear in mind that trends in evolution do not necessarily mean that later-appearing organisms have an evolutionary advantage (that is, are in some way 'better than') over earlier ones. This important general principle most likely applies in the cases of intelligence and cognitive ability, however one comes to understand those terms. Evolution is the continuing exploration of the viability of randomly created variants of preexisting life-forms. Hence, a feature that appears later in evolution means that it is a novel alternative, but not necessarily a ' better' one.
"That much is reasonably clear, but it leaves an intriguing, nagging question, along with some additional observations about brain size. There certainly seem to be some meaningful mental attributes that humans (and at least some other contemporary organisms) have more of than did prior ancestors. Furthermore, the recent human lineage (and at least some other animal lineages) does seem to show not only the emergence of creatures having larger brains but in addition their replacement of creatures having smaller brains.
"These two issues are brought together in an interesting way by Christopher Wills in his book The Runaway Brain (BasicBooks, 1993). Wills suggests that larger brain size is associated with increased capacity to handle sensory information and hence with increased flexibility as well as decreased predictability in behavior. The latter innovation would then, in turn, exert selection pressure for increased brain size on other animals, whose survival relates to, among other things, their ability to predict behavior. These resulting increases in brain size would, of course, generate pressure for further increases in brain size. It is because of this positive feedback loop that Wills refers to the 'runaway brain.'