Barry Lienert, a geophysicist at the University of Hawaii, provides the following explanation.
We start by determining the mass of the Earth. Issac Newton's Law of Universal Gravitation tells us that the force of attraction between two objects is proportional the product of their masses divided by the square of the distance between their centers of mass. To obtain a reasonable approximation, we assume their geographical centers are their centers of mass.
Because we know the radius of the Earth, we can use the Law of Universal Gravitation to calculate the mass of the Earth in terms of the gravitational force on an object (its weight) at the Earth's surface, using the radius of the Earth as the distance. We also need the Constant of Proportionality in the Law of Universal Gravitation, G. This value was experimentally determined by Henry Cavendish in the 18th century to be the extemely small force of 6.67 x 10-11 Newtons between two objects weighing one kilogram each and separated by one meter. Cavendish determined this constant by accurately measuring the horizontal force between metal spheres in an experiment sometimes referred to as "weighing the earth."
Knowing the mass and radius of the Earth and the distance of the Earth from the sun, we can calculate the mass of the sun (right), again by using the law of universal gravitation. The gravitational attraction between the Earth and the sun is G times the sun's mass times the Earth's mass, divided by the distance between the Earth and the sun squared. This attraction must be equal to the centripetal force needed to keep the earth in its (almost circular) orbit around the sun. The centripetal force is the Earth's mass times the square of its speed divided by its distance from the sun. By astronomically determining the distance to the sun, we can calculate the earth's speed around the sun and hence the sun's mass.
Once we have the sun's mass, we can similarly determine the mass of any planet by astronomically determining the planet's orbital radius and period, calculating the required centripetal force and equating this force to the force predicted by the law of universal gravitation using the sun's mass.
Additional details are provided by Gregory A. Lyzenga, a physicist at Harvey Mudd College in Claremont, Calif.
The weight (or the mass) of a planet is determined by its gravitational effect on other bodies. Newton's Law of Gravitation states that every bit of matter in the universe attracts every other with a gravitational force that is proportional to its mass. For objects of the size we encounter in everyday life, this force is so minuscule that we don't notice it. However for objects the size of planets or stars, it is of great importance.
In order to use gravity to find the mass of a planet, we must somehow measure the strength of its "tug" on another object. If the planet in question has a moon (a natural satellite), then nature has already done the work for us. By observing the time it takes for the satellite to orbit its primary planet, we can utilize Newton's equations to infer what the mass of the planet must be.
For planets without observable natural satellites, we must be more clever. Although Mercury and Venus (for example) do not have moons, they do exert a small pull on one another, and on the other planets of the solar system. As a result, the planets follow paths that are subtly different than they would be without this perturbing effect. Although the mathematics is a bit more difficult, and the uncertainties are greater, astronomers can use these small deviations to determine how massive the moonless planets are.