Accurate evaluation of layer potentials near boundaries and interfaces are needed in many applications, including fluid-structure interaction problems and near-field scattering problems. A classical method to approximate the solution everywhere in the domain consists of using the same quadrature rule (Nyström method) used to solve the underlying boundary integral equation. This method is problematic for evaluations close to boundaries and interfaces. For a fixed number, N, of quadrature points, this method incurs a non-uniform error with O(1) errors in a boundary layer of thickness O(1/N). Using an asymptotic expansion of the associated kernel, we remove this O(1) error without having to use high-order Nyström methods. To demonstrate this method, we consider the interior and exterior Laplace problems, and present results for acoustic scattering problems.