Aldo Palmisano is a research chemist at the Western Fisheries Research Center of the U.S. Geological Survey Biological Resources Division and is affiliated with the University of Washington in Seattle. Here is his answer.
The reason some fish normally live in freshwater and others live in seawater is that one or the other environment provides them with opportunities that have traditionally contributed to their survival. An obvious difference between the two habitats is salt concentration. Freshwater fish maintain the physiological mechanisms that permit them to concentrate salts within their bodies in a salt-deficient environment; marine fish, on the other hand, excrete excess salts in a hypertonic environment. Fish that live in both environments retain both mechanisms.
Life began evolving several billion years ago in the oceans and since that time, living things have maintained an internal environment closely resembling the ionic composition of those primeval seas. Presumably, the ionic conditions in which life began are uniquely appropriate to its continuation. Laboratory studies support the view that the various chemical phenomena on which life depends--including the interactions of nucleic acids with each other and with proteins, the folding and performance of proteins such as enzymes, the functioning of intracellular machines such as ribosomes, and the maintenance of cellular compartments--are critically dependent on the ionic milieu in which the reactions take place.
Given time, ocean-dwelling creatures took advantage of untapped resources, such as relatively safe spawning habitats or new food sources, that were available to them only by colonizing other environments, like freshwater and land. Colonization was facilitated, if not necessitated, by geological events, such as the movements and collisions of land masses (plate tectonics) and volcanic activity, which served to isolate portions of very similar populations of a single species from one another. Such geological change forced some populations to either adapt or face extinction. Time and natural selection due to physical and environmental variation worked in concert with isolation to foster adaptations. In some cases, these adaptations became permanent and led to species differentiation.
One important aspect of environmental variation is the ionic composition of bodies of water utilized as habitat. Chloride cells in the gills of marine fish produce an enzyme, called gill Na+/K+ ATPase, that enables them to rid their plasma of excess salt, which builds up when they drink seawater. They use the enzyme to pump sodium out of their gills at the cost of energy. Additionally, their kidneys selectively filter out divalent ions, which they then excrete. An alternative set of physiological mechanisms allows freshwater fish to concentrate salts to compensate for their low salinity environment. They produce very dilute, copious urine (up to a third of their body weight a day) to rid themselves of excess water, while conducting active uptake of ions at the gill.
Certainly, other adaptations contributed to the capability of isolated populations to adapt more fully to their circumstances. With different sets of predator and prey organisms present in the differing habitats, and different physical ranges available to them, behavioral changes would be required; perhaps a smaller or larger body size or body part would be favored. The accumulation of these kinds of physiological, behavioral and physical changes ultimately led to new species. Isolation may have forced them to conserve their newly developed adaptations among their own descendants, rather than distribute them more broadly. For some, the rift eventually became complete and there could no longer be any cross-breeding between populations that once interbred.