Although a section in the bibliography is devoted to the refractivity of air, it's buried down near the end of the monster file. Besides, a little more detailed discussion of the dispersion formulae for air seems to be called for.
The main problem is that the refractivity of air is difficult to measure accurately, so that there have been many re-measurements, and several different formulae have been used to represent the dispersion curve of air by different authors. Worse yet, the older formulae, which have long been known to be incorrect, have become fossilized in handbooks, and copied by more recent handbooks from the older ones, so that obsolete and inaccurate formulae are often cited and used.
Note that the refractivity is usually given for 15° C, and refers to dry air, not air with some water vapor in it. The argument is usually vacuum wavenumber, not just the reciprocal of the wavelength in air. Sometimes the formula given refers to air free of CO2 as well as water vapor. Because the refractivity, (n−1), is generally less than 3×10−4, it's quite common to have the formula give either 106 or 108 times the actual refractivity. Bear all these facts in mind when using any refractivity formula. Read all the fine print before using any formula!
Finally, there is more than one “refractive index” of interest, and the right one to use depends on the kind of measurement being made. For angular refraction (which is what I deal with on these Web pages), the appropriate index is the “phase index,” which is the one usually meant if no such distinction is made. However, for radar and laser geodimetry, where an electromagnetic signal is modulated, it is usually the group velocity rather than the phase velocity that is of interest; then the “group index” is required. If observations are made in or near absorption lines, things become still more complex. Fortunately, these niceties are mainly of concern for the geodesists, and will be ignored here.
In its day, the discussion by
H. Barrell, J. E. Sears
The Refraction and Dispersion of Air for the Visible Spectrum
Phil. Trans. Roy. Soc. A, 238, (1939)
was quite influential, though it is now quite obsolete. Its refractivity error approaches a part in 1000 near the atmospheric transmission cutoff just below 300 nm, and is about 1 part in 5000 or 6000 through most of the visible spectrum. (However, the discussion of the physical effects that must be taken into account is still worth reading, even though the numerical values have been superseded.)
Unfortunately, one occasionally still finds papers that use it. (The tip-off that this obsolete formula is being used is that it has the form of a 2-term Cauchy dispersion formula: a constant, a term in λ−2, and a term in λ−4.)
A recent example is the paper by J. Gubler and D. Tytler in PASP 110, 738 (1998). They got the B&S formula from A. T. Sinclair's NAO Technical Note (1982), which doesn't sound too obsolete. But Sinclair got it from the IAG's Bulletin Géodésique (1963), p. 390 — which of course got it from Barrell & Sears (1939). By the time Gubler & Tytler picked it up, the formula was out of date by almost 60 years!
Unfortunately, this incorrect and obsolete formula was also used by Hohenkerk & Sinclair (1985), from which it was copied in the new (1992) edition of the Explanatory Supplement (see the middle of p. 142 for the obsolete formula there); so we can expect to continue to see it crop up again in the future.
B. Edlén
Dispersion of standard air
J. O. S. A. 43, 339–344 (1953)
This is the formula with a 41 in the denominator of the last term. (A modified form of this formula was also included in the IAG's Bulletin Géodésique (1963), p. 390. Recently, the IAG finally got clued in and dumped these obsolete formulae.) Though it's still often cited, it's known to be wrong. Don't use it!
It's inexcusable that it was still given as “the” refractive index of air on p. 262 of the 4th edition of Allen's Astrophysical Quantities in the year 2000 — 36 years after it was superseded. (Worse yet, this handbook even misprints it with an egregious typo that reduces the formula to rubbish, on p. 257. They're charging $99 for this?)
B. Edlén
The refractive index of air
Metrologia 2, 71–80 (1966)
J. C. Owens
Optical refractive index of air: dependence on pressure, temperature and composition
Appl. Opt. 6, 51–59 (1967)
Owens, like Barrell & Sears, separated out the contribution from carbon dioxide. He claimed an accuracy of 1 part in 109 for the refractive index, or 3 parts in 106 for the refractivity. Like Barrell & Sears, Owens considered compressibility effects (i.e., deviations from the ideal-gas law); he found them to be about 10−4 at 245 K, increasing rapidly at lower T. (From his figure, it looks like 2 × 10−4 at 200K).
Owens showed that Edlén (1966) is off by 7×10−8 at −30°C, and by 4×10−7 at 45°C, 100% RH. Unfortunately, the 1966 Edlén formula is still widely quoted and used.
E. R. Peck, K. Reeder
Dispersion of air
JOSA 62, 958–962 (1972)
made new measurements in the infrared. They showed that Edlén was certainly wrong in the IR, and that one could obtain accuracy as good as the data (except at wavelengths below 0.23 µm) with a 2-term Sellmeier dispersion formula, which involves only 4 parameters instead of 5. This accuracy is a little better than 2 parts in 109 for the refractive index, from 0.23µm in the UV to 1.69µm in the near IR.
Personally, I like to use their simple formula for calculating atmospheric refraction. It's “good enough” for most purposes, as long as you don't need to worry about water vapor, or the middle infrared.
F. E. Jones
The refractivity of air
J. Res. NBS 86, 27–32 (1981)
reconsidered the real-gas effects, and performed a careful error analysis:
The major contributors to the uncertainty in refractivity are the uncertainties in the measurements of temperature and pressure. The magnitude of the uncertainty due to variation in CO2 concentration can approach that of the uncertainties due to the pressure and temperature measurements. Therefore, the CO2 concentration should be treated as a variable and should be observed.
Unfortunately, he seems to have overlooked the Peck – Reeder work. He produced a formula good for the visible region with an error in the refractivity of a few parts in 108.
H. Matsumoto
The refractive index of moist air in the 3-µm region
Metrologia 18, 49–52 (1982)
measured the effects of water vapor, and found considerable effects due to its near-IR absorptions.
Then additional errors due to water vapor, even in the visible, were found by
K. P. Birch, M. J. Downs
The results of a comparison between calculated and measured values of the refractive index of air
J. Phys. E: Sci. Instrum. 21, 694–695 (1988)
and confirmed by
J. Beers, T. Doiron
Verification of revised water vapour correction to the refractive index of air
Metrologia 29, 315–316 (1992)
P. E. Ciddor
Refractive index of air: new equations for the visible and near infrared
Appl. Opt. 35, 1566–1573 (1996)
We can regard Ciddor's results as definitive, at least for the present. They have been adopted as the basis of a new standard by the International Association of Geodesy (IAG). Geodetic applications often require the group rather than the phase index; see
P. E. Ciddor, R. J. Hill
Refractive Index of Air. 2. Group Index.
Applied Optics (Lasers, Photonics and Environmental Optics), 38, 1663–1667 (1999)
Soon after Ciddor's first paper, a new set of highly accurate measurements at 4 wavelengths in the visible spectrum was published by
G. Bönsch and E. PotulskiTheir work focuses primarily on determining the effects of water vapor and CO2 content, and takes account of a slight adjustment between older practical temperature scales and the ITS-90 scale, which is almost a hundredth of a degree.
Measurement of the refractive index of air and comparison with modified Edlén's formulae
Metrologia 35, 133–139 (1998)
They start with the “new” Edlén formula of 1966. The aim is to provide the highest possible accuracy for laboratory conditions, so they consider only a small temperature range about 20°C and a CO2 mixing ratio near 400 ppm, where linear formulae are adequate; their results are not sufficiently general for field use in the actual atmosphere. They give a nice practical summary of their results in an Appendix.
As they essentially confirm the results of previous recent studies — in particular, the water-vapor corrections of Birch & Downs and more recent workers — I would regard their paper as indirectly supporting Ciddor's results. The uncertainty in the refractive index of air is about 1 unit in the 8th decimal place.
For further discussion of these matters, see the Web pages of NIST and the IAG.
The above discussion refers to the effects of water vapor in the visible spectrum. But water vapor also has strong absorptions in the near infrared, which (by classical dispersion theory) produce considerable effects on the dispersion curve there. The following reference gives an example of the size of these effects out to 25 microns (in spite of the narrower scope suggested by its title):
R. J. Mathar
Calculated refractivity of water vapor and moist air in the atmospheric window at 10 μm
Appl. Opt. 43, 928-932 (2004)
Similarly, a change of 0.1 mm of mercury changes the density and hence the refractivity of air by about a part in 104, so that still smaller pressure changes might be significant. This is also the level at which deviations from the ideal gas law become appreciable.
Stone says that, for accurate positional astrometry, errors should be kept below 0.05 arc seconds at zenith distances up to 70°, where the sea-level refraction is about 147 arc seconds. This refraction is nearly 3000 times the allowed error.
For most purposes, we can neglect effects smaller than 1 or 2 seconds of arc at the horizon, corresponding to relative errors in density of 1 part in 1000, or perhaps 1 part in 2000. It appears that an error of 1 part in a few thousand is small enough for astronomers.
This means an accuracy of 1 part in 3000 for the refractivity, or 1 part in 107 for the refractive index, is adequate for most astronomical purposes. That would make the Edlén formulae acceptable. But, as the more recent formulae — in particular, the Peck – Reeder formula — are more reliable, as well as less expensive to calculate, I think one of them should be used.
Note that 1 part in 3000 requires measuring temperatures to 0.1°C or better accuracy (not just precision); pressures must be accurate to about 1/3 of a millibar; and the water-vapor mixing ratio should be known to a relative accuracy of about 10%.
For astronomical refraction calculations in the visible spectrum, the Peck – Reeder formula seems a good compromise between accuracy and economy of calculation. I recommend it.
For other purposes, such as geodesy, more accuracy may well be required. Then one should adopt Ciddor's results.
Copyright © 2003 – 2007 Andrew T. Young
or the
GF home page
or the Overview page