Atoms emit specific wavelengths that can be used like a unique fingerprint to identify them. Getting a pure sample of an element is the primary limitation.
During a flame test a sample of material is burnt and the spectrum of light it produces is analysed (I think with a spectrometer or similar equipment). The spectrum produced is called an emission spectrum (because it is emitted from the sample rather than the sample absorbing light from a continuous spectrum). A typical emission spectrum has only some wavelengths in it. It is called a discrete or line spectrum, meaning that only specific wavelengths are in it, rather than all wavelengths from red to violet.
The different types of spectra are shown in the images below.
Each element produces a unique spectrum, i.e. the specific wavelengths in it are unique to that element. If we had a pure sample of mercury and burnt it we would get a set of wavelengths that we knew belonged only to mercury. We could then use that spectrum (set of wavelengths) to identify mercury in other samples.
In fact, this method of using specific wavelengths associated with elements is still used in modern astronomy to identify elements in present in distant stars. It's pretty handy seeing as we cannot send mass spectrometers up to them currently!
Here is an image with a few different emission spectra:
The primary limitation with flame tests is ensuring that you have a pure sample of an element. Impurities will introduce wavelengths into the spectrum that are not related to the element. Usually scientists will use line spectra that measure intensity as well as identifying the wavelengths present. Wavelengths introduced by impurities will likely have a small intensity which would not be considered significant.
Here's a picture of a line spectrum available at Wikipedia: