  We introduce an algorithm for the least squares solution of a rectangular
linear system $Ax=b$, in which $A$ may be arbitrarily ill-conditioned. We
assume that a complementary matrix $Z$ is known such that $A - AZ^*A$ is
numerically low rank. Loosely speaking, $Z^*$ acts like a generalized inverse
of $A$ up to a numerically low rank error. We give several examples of $(A,Z)$
combinations in function approximation, where we can achieve high-order
approximations in a number of non-standard settings: the approximation of
functions on domains with irregular shapes, weighted least squares problems
with highly skewed weights, and the spectral approximation of functions with
localized singularities. The algorithm is most efficient when $A$ and $Z^*$
have fast matrix-vector multiplication and when the numerical rank of $A -
AZ^*A$ is small.
