Integer Ambiguity Resolution & LAMBDA
How to find the whole number of wavelengths — and know when you're sure
Why it matters
The integer ambiguity is the single gate between metre-level and millimetre-level positioning. Every high-precision technique — RTK, network RTK, and PPP with ambiguity resolution — depends on fixing it correctly, and a wrong fix is far more dangerous than no fix at all because it looks precise while being decimetres off. LAMBDA is the method the entire industry uses.
Prerequisites
The question RTK left hanging
When we built RTK, we reached the centimetre prize by fixing the integer ambiguities — and then committed a small crime. The solver's Stage 3 simply rounded each float ambiguity to its nearest integer and moved on, with a comment admitting that "real resolvers (e.g. LAMBDA) decorrelate and search a lattice." We waved at the hard part. This lesson is that hard part.
Here is why it matters enough to earn its own concept. The carrier phase measures range to a few millimetres plus an unknown whole number of wavelengths — the integer ambiguity . A float solution (from RTK, or from a Kalman filter, or from PPP) does not give you . It gives you a real-valued estimate of — say — together with how uncertain that estimate is. The whole game is turning into the correct integer. Get it right and the range snaps to millimetres. Get it wrong by even one cycle and you have injected a 19 cm blunder that the solution will wear as if it were truth.
So the question is deceptively simple: given a real-valued estimate of a set of integers, and how uncertain it is, which integers are they? And the first answer everyone reaches for — round to the nearest — is, surprisingly, usually wrong.
The idea, before any mathematics
Start with a single ambiguity. The float says ; rounding to is obviously right. The trouble begins when you have several ambiguities at once, because they are not independent. The same satellite geometry that couples your position estimates couples the ambiguities too: an error in one tends to come with a matching error in another. Their uncertainty is not a tidy circle around the float point — it is a long, tilted ellipse.
Picture the ambiguities as a point on a grid of integers. The truth is one of the grid intersections. The float solution is a point somewhere between them, and its uncertainty is that tilted ellipse draped over the grid. Independent rounding asks: which grid point is nearest if I measure distance the ordinary way, square by square? But that is the wrong ruler. The ellipse is the ruler: a direction in which the float is very uncertain costs little to move along, while a direction in which it is sharply determined is expensive. The right question is: which grid point is nearest measured by the ellipse's own distorted sense of distance?
When the ellipse is a tilted cigar — which strong correlation always produces — those two questions have different answers. The ordinary-nearest grid point can sit outside the long thin ellipse, while a point that looks farther by eye sits comfortably inside it. The second one is the correct fix. Rounding, blind to the tilt, confidently chooses the first.
The float ambiguities live in a skewed ellipse, not a circle. Rounding each one on its own measures distance with the wrong ruler; the honest question is which integer point is closest in the metric the covariance defines. That question is called integer least squares.
So we have the real problem. But searching a tilted, elongated ellipse for the closest lattice point is slow and error-prone — the cigar can be so long and thin that candidate points stretch far along its axis. LAMBDA's insight is to first change coordinates so the cigar becomes almost a sphere, using a transformation that is careful to keep the grid a grid of integers. In the round version the ordinary ruler and the ellipse ruler nearly agree, the search is quick, and the answer maps exactly back. Finally, having found the best integer point, we ask one more question before trusting it: how much better is it than the runner-up? If it barely wins, two integer solutions are effectively tied, and a wise receiver declines to commit.
Seeing it
Hold three pictures in mind. First, a grid of integers with a tilted cigar-shaped ellipse lying across it — the float solution and its uncertainty. Second, the two candidate answers: the grid point nearest by eye (rounding) and the grid point nearest by the ellipse's ruler (the fix), which part company as the cigar tilts. Third, the same scene after decorrelation: the cigar has been rounded into a near-circle and the grid gently sheared, and now the nearest point by eye is the nearest point by the ruler. That third picture is the entire reason LAMBDA works.
Try it yourself
The float solution is the cross; its covariance is the ellipse. The square is independent rounding; the circle is the integer-least-squares fix.
- Naive rounding
- [1, 0]
- ILS fix
- [1, 1]
- Agree?
- no — rounding is wrong
- Best / naive norm
- 0.68 / 3.47
- Ratio (2nd / best)
- 1.10
The × is the float solution; the ellipse is its covariance. Independent rounding (the square) snaps to the nearest grid point by eye. The ILS fix (the circle) snaps to the nearest point by the ellipse’s ruler. Crank ρ toward 1 and the ellipse becomes a tilted cigar: the two answers part ways, and rounding commits to the wrong integers. Now tick decorrelated view: LAMBDA’s integer change of basis Z rounds the cigar into a near-circle, where nearest-by-eye and nearest-by-ruler agree again — so the search is fast and correct. The ratio of the runner-up’s distance to the winner’s is the confidence check: above the threshold the fix is trusted (FIXED); near 1 two lattice points are tied and a wise receiver stays float.
Start with low correlation: the ellipse is nearly round, and rounding and the ILS fix agree. Now push ρ toward 1 and watch the ellipse stretch into a tilted cigar — at some point the square and the circle jump apart, and the read-out reports rounding is wrong: its integer vector sits on a larger covariance-distance than the fix. Slide the float point around and you will find whole regions where the honest answer is a diagonal neighbour of the rounded one. Then tick decorrelated view: the same problem, redrawn in LAMBDA's integer basis Z, shows a near-circular ellipse in which rounding and the fix coincide again. Finally watch the ratio: when it falls below the threshold — two lattice points nearly tied — the verdict turns to FLOAT, the receiver refusing to gamble on an integer it cannot distinguish from its rival.
Now the mathematics
A float solution hands us a vector of real-valued ambiguities and their covariance . The truth is an integer vector . The correct fix is the integer vector that is closest to the float estimate in the metric the covariance defines — the integer least squares problem:
The quadratic form is the squared Mahalanobis distance — the ellipse's ruler. If were diagonal (uncorrelated), this would split into independent one-dimensional roundings and plain rounding would be optimal. It is the off-diagonal correlation that makes the problem genuinely multidimensional and rounding wrong.
Decorrelation. LAMBDA seeks an integer matrix with — a unimodular transformation — that reshapes the covariance
to be as close to diagonal as possible. Because is integer and volume preserving, it maps the integer lattice onto itself: the transformed ambiguities are still sought as integers, and the final answer transforms back exactly via . Nothing about the solution is approximated; only the shape of the search changes. The construction uses an factorisation of the covariance, then alternates integer Gauss transformations (which shrink the off-diagonal entries) with permutations (which reorder the ambiguities to expose more shrinkage) until no swap helps — flattening the cigar toward a sphere.
Search. In the decorrelated frame the search region is compact, and a depth-first shrinking-ellipsoid search walks the lattice level by level, accumulating the Mahalanobis distance and pruning any branch that already exceeds the best complete candidate found so far. It returns not just the best integer vector but the second-best, because validation needs both.
Validation — the ratio test. Let and be the squared Mahalanobis residuals of the best and second-best integer candidates. The fix is accepted only when the runner-up is clearly worse:
with a threshold typically around to . A large ratio means the winner stands alone and the integers are trustworthy; a ratio near means two lattice points are nearly tied, so committing to either risks a wrong fix — and the receiver keeps the float solution instead. This is exactly the philosophy of integrity applied to the ambiguities: better to admit uncertainty than to broadcast a confident lie.
From the idea to working code
@gnerd/gnss-core implements the real algorithm, not the rounding shortcut. resolveAmbiguities
takes the float vector and its covariance and runs the four stages explicitly — factorisation,
decorrelation, search, ratio test — returning the fix, the runner-up, the naive rounding for
comparison, both squared norms, the ratio verdict, and the decorrelating itself:
import { resolveAmbiguities } from "@gnerd/gnss-core";
// A strongly correlated 2-ambiguity float solution (cycles) and its covariance (cycles^2).
const floatAmbiguities = [0.9, 0.45];
const covariance = [
[1.0, 0.95],
[0.95, 1.0],
];
const result = resolveAmbiguities(floatAmbiguities, covariance, { ratioThreshold: 3 });
// result.naiveRounding -> [1, 0] (what plain rounding would pick)
// result.fixedAmbiguities -> [1, 1] (the integer-least-squares answer)
// result.bestSquaredNorm < result.naiveSquaredNorm (the fix is genuinely closer)
// result.status -> "float" (the two candidates are nearly tied here)
This is the same engineering shape as every other solver in the course: the measurement model is separated from the estimation, intermediate quantities are explicit and inspectable, and the acceptance criterion (the ratio threshold) is a documented, tunable number rather than a hidden constant. A production resolver adds more — wide-laning (combining L1 and L2 into a longer effective wavelength so candidate cycles sit farther apart and resolve first), partial fixing (resolving the confident subset when the full set will not validate), and continuous quality monitoring — but every one of them is this same integer least squares with a validation gate.
Check your understanding
- Why does independent rounding fail for correlated ambiguities, and what is the correct question it should be asking instead? (Rounding uses the ordinary ruler; the correct fix is the lattice point nearest in the covariance metric — integer least squares.)
- What does it mean for to be unimodular, and why is that property essential? (Integer entries with : it maps the integer lattice onto itself, so decorrelation reshapes the search without changing which integer solutions are possible.)
- A receiver computes a best candidate far better than any other, but reports "float." What single number could still have blocked the fix, and why might that be the right call? (The ratio — if it were configured very high, or the second-best were unusually close; refusing a doubtful fix avoids a confident wrong answer.)
Where this goes next
Integer ambiguity resolution is the hinge the whole precision world turns on. The same machinery scales from a single baseline to network RTK, where a mesh of reference stations resolves ambiguities over a region, and to PPP with ambiguity resolution, which applies integer least squares to a single receiver once the satellite-side fractional biases are provided — closing much of the historic gap between PPP and RTK. The ratio test you met here is one member of a family of integer validation tests, and it rhymes directly with the integrity story: in both, the hard-won lesson is that a system earns trust not by always producing an answer, but by knowing — and saying — when it should not. That principle, as much as any equation, is what separates a navigation toy from a navigation system.
Common misconceptions
- You fix ambiguities by rounding each float estimate to its nearest integer. That ignores the correlation between ambiguities; the metric-nearest lattice point is often a diagonal neighbour, not the one independent rounding lands on.
- Decorrelation changes the answer. The Z-transformation is integer and volume preserving, so it only reshapes the search; the integer solution maps exactly back to the original ambiguities.
- A fix is always better than a float. A wrong integer fix is decimetres off while looking precise; the ratio test exists so the receiver can refuse to fix when candidates are tied.
Key takeaways
- Float solutions give real-valued ambiguities and a covariance; the truth is an integer lattice.
- Fixing is integer least squares - closest lattice point in the covariance metric, not by eye.
- Correlated ambiguities make independent rounding pick the wrong integers.
- LAMBDA decorrelates with an integer unimodular Z so the lattice search is fast and reliable.
- The ratio test validates the fix; a confident wrong fix is worse than staying float.