The eddy covariance technique is a widely used and accepted method to quantify ecosystem-scale mass and energy fluxes. Eddy covariance measurements of evaporation, also known as evapotranspiration, are used to determine local, regional and global water budgets, calibrate and validate land surface models, and acquire understanding of ecosystem processes. This paper assesses the accuracy of eddy covariance evaporation measurements by comparing them with those derived from small watershed water balances. Comparing thirteen years of data from shrubland, grassland and savanna sites in southern Arizona USA, the two independent measures agreed to within an average of 3% annually and differed from −10 to +17% in any given year, when an assumed 5% underestimation in precipitation due to gage undercatch was considered. The agreement between the two measures was generally better in drier years and at less topographically complex sites. Despite an indication of a systematic underestimate of evaporation by a commonly used assessment of the energy balance, forcing energy balance closure on evaporation led to worse results for nine of the thirteen annual periods but improved multiyear sums at two of the three sites.