William Beaty is an electrical engineer who has designed numerous science exhibits for museums. He is the creator of the Science Hobbyist web page and is a member of the Mad Science Network. He offers the following explanation:
St. Elmo's Fire is a type of continuous electric spark called a "glow discharge." You've seen it many times before, since it is almost exactly the same as the glows found inside fluorescent tubes, mercury vapor streetlights, old orange-display calculators and in "eye of the storm" plasma globes. When it occurs naturally, we call it St. Elmo's Fire, but when it occurs inside a glass tube, we call it a neon sign.
St. Elmo's Fire and normal sparks both can appear when high electrical voltage affects a gas. St. Elmo's fire is seen during thunderstorms when the ground below the storm is electrically charged, and there is high voltage in the air between the cloud and the ground. The voltage tears apart the air molecules and the gas begins to glow. It takes about 30,000 volts per centimeter of space to start a St. Elmo's fire (although sharp points can trigger it at somewhat lower voltage levels.)
St. Elmo's Fire is plasma. A normal gas is composed of molecules. The molecules are composed of atoms, which in turn are composed of electrons and clusters of proton particles. If the electric force applied to each bit of gas is greater than a certain level of voltage, it causes the electrons and protons of the gas molecules to be pulled away from each other. High voltage transforms the gas into a glowing mixture of separate proton clusters and electrons. We call this mixture of particles by the name "plasma," and it is conductive. It also fluoresces with light.
The color of the glow depends on the type of gas involved. If we lived in an atmosphere of neon gas, then St. Elmo's fire would be red/orange, and lightning would be white with orange edges. Our atmosphere is nitrogen and oxygen, and this mixture glows blue/violet when exposed to high voltage fields. If a neon sign tube was filled with nitrogen/oxygen instead of neon, it would light up blue/violet rather than red/orange.
Is this phenomenon related to ball lightning? No one knows, because no one knows what ball lightning is, and it might not be a spark at all. St. Elmo's fire is sometimes mistaken for ball lightning. Among other differences, ball lightning can drift around like a soap bubble, while St. Elmo's Fire always remains attached to an object.
A more literary analysis of St. Elmo's Fire is provided by Steven A. Ackerman, an assistant professor of atmospheric and oceanic science at the University of Wisconsin-Madison:
I boarded the Kings' ship; now in the beak,
Now in the waist, the deck, in every cabin,
I flamed amazement; sometime I'd divide
And burn in many places; on the topmast
The yards and bowsprit, would I flame distinctly
Then meet and join.
--The Tempest (Act I, Scene 2)
Air is a good electric insulator. However, if an electric field at the end of a metal rod is strong, air molecules in its vicinity are ionized and charges stream off the end of the rod. Since an electric field is intensified in regions of high curvature, discharges are intensified at the end of pointed metal rods.
These "corona discharges" give a faint light that is visible in the dark. St. Elmo's fire is an example of a continuous corona discharge. It is not a form of lightning. The luminous glow of St. Elmo's fire is sometimes observed at night from the tops of ships' masts when electrified clouds are overhead. The mast appears to be on fire but does not burn. St. Elmo's' fire tends to occur toward the end of a thunderstorm. Mariners have traditionally interpreted it as a good omen.
St. Elmo is the patron saint of sailors but St. Elmo's fire is not bound to the sea. It has been observed atop tall steeples. My brother once saw it envelop metal pipes outside his home.