Heinrich M. Jaeger, a professor of physics at the University of Chicago, explains.
The phenomenon by which large grains move up and small ones move down when shaking the can is called granular size separation. It is often referred to as "the Brazil nut effect," since the same result occurs in a shaken can of mixed nuts. There are several physical mechanisms that can give rise to size separation. Obviously, the very finest (dust-like) grains might just fall down through the cracks left between the larger particles. The more interesting case concerns mixtures of particles that do not differ all that much in size, perhaps by as little as 10 percent. Surprisingly, in this case the larger (and thus heavier) ones still end up near the top.
Two main mechanisms give rise to this separation. Firstly, if during a shaking cycle (as the material lifts off the bottom of the can and then collides with the bottom again) the large particles briefly separate from the surrounding smaller ones and leave a gap underneath, small grains can move into this opening. When the shaking cycle is finished the large particles are prevented from getting back to their original positions. Thus, the bigger particles are slowly "ratcheted" upward.
Secondly, the granular material rubs against the sidewalls of the can when it is shaken. This friction causes a net downward motion of grains along the walls. This downward flow occurs in a narrow region only a few particle diameters wide close to the walls. In the center of the can, meanwhile, the particles move up, completing a convection roll. Large grains, just like any other grains, are moved up along the center (similar to being on an escalator). Once they reach the top they move toward the side wall and try to enter the downward flow. But if they are too large, they cannot fit into the narrow region that contains the downward flow and they get stuck near the top. After a few shaking cycles, this leads to an enrichment of large particles near the top.
Both mechanisms can apply simultaneously, in principle, and both will lead to the same net effect: large particles will end up near the top. Differences between the two mechanisms are somewhat subtle. For example, the speed with which the larger particles rise to the top surface is different in the two scenarios. In practice, the second mechanism, known as a convective mechanism, dominates the first mechanism as long as the sidewalls are not frictionless (which is hard to achieve), and as long as you are considering particles not too deep below the surface.
One further remark: if the particle size is very small (smaller than, say, one millimeter) the presence of air can modify both mechanisms. How this occurs in detail is still an open question and the focus of much research at the moment.
Answer originally posted on March 24, 2003.