The condition of having no more than five fingers or toes--in this context, 'most species' means a subgroup of jawed vertebrates--probably evolved before the evolutionary divergence of amphibians (frogs, toads, salamanders and caecilians) and amniotes (birds, mammals, and reptiles in the loosest sense of the term). This event dates to approximately 340 million years ago in the Lower Carboniferous Period. Prior to this split, there is evidence of tetrapods from about 360 million years ago having limbs bearing arrays of six, seven and eight digits. Reduction from these polydactylous patterns to the more familiar arrangements of five or fewer digits accompanied the evolution of sophisticated wrist and ankle joints--both in terms of the number of bones present and the complex articulations among the constituent parts.
Early evolutionary experiments in hexa- or octodactyly (that is, creatures having six or eight digits) were associated with rather simple limb skeletons, much like those present in the flippers of modern whales and dolphins. This might provide a functional clue about one of the reasons for digit number reduction, which is related to the functional demands of simple "walking" limbs. Unlike paddles, such limbs have to provide purchase on a range of substrates, provide the platform for an efficient push-off and allow some rotation relative to the lower and upper limb bones as the rest of the body travels onward. In the very few instances of secondarily evolved polydactylous limbs from the fossil record, the phenomenon is associated with aquatic taxa. The classic instance of this is in the paddles of ichthyosaurs, extinct fishlike marine reptiles that lived more than 65 million years ago.
Is there really any good evidence that five, rather than, say, four or six, digits was biomechanically preferable for the common ancestor of modern tetrapods? The answer has to be "No," in part because a whole range of tetrapods have reduced their numbers of digits further still. In addition, we lack any six-digit examples to investigate. This leads to the second part of the answer, which is to note that although digit numbers can be reduced, they very rarely increase. In a general sense this trait reflects the developmental-evolutionary rule that it is easier to lose something than it is to regain it. Even so, given the immensity of evolutionary time and the extraordinary variety of vertebrate bodies, the striking absence of truly six-digit limbs in today's fauna highlights some sort of constraint. Moles' paws and pandas' thumbs are classic instances in which strangely re-modeled wrist bones serve as sixth digits and represent rather baroque solutions to the apparently straightforward task of growing an extra finger. Patterns of six (or more) digits can be achieved by laboratory-based developmental manipulations, some of which concern changes in gene activity that probably reflect transformations involved in the fin-to-limb evolutionary transition. Here might lie another part of the reason for the prevalence of five: pleiotropy, or the multiple effects of genes upon more than one physical characteristic. For instance, Hand-Foot-Genital syndrome is a rare condition in which, as the name implies, the genito-urinary tract and the limbs are malformed. Crucially, the genes responsible are within the set of those involved in digit number and patterning. Therefore, although this tells us nothing directly about the significance of digit number, it indicates something important about developmental stability: the mechanisms involved in patterning the tips of our limbs include those involved in our reproductive success. Thus, tweak at your peril.