(wiki-TODO: This page is old. It needs cleanup & improvement.)
In a nutshell
Sketched ideas about artificial atomically precise technology where first introduced to a wider public in Richard Feynman's famous talk "There is Plenty of Room at the Bottom". Back then the idea was simply called "nanotechnology".
The technologies potential was well perceived and lead to heavy funding of everything claiming to be "nanotechnology" in the US. Since at this point in time atomically precise material science was still quite far out of reach all things smaller than a micron where called nanotechnology. This mainly included non atomically precise nanoparticles and other structures with the atoms statistically distributed. Those structures can not by definition (non AP) play a central role in the achievement of APM.
Other parts of the world followed with the funding of "nanotechnology" without ever publicly perceiving the original idea of atomic precision. The place where atomic precision was already achieved at this time where the well established molecular sciences. People involved in those saw little incentive to switch their work under the banner of "nanotechnology". Also they where heavily focused in the scientific study of organic chemistry and biological systems. Taking the inverse route to engineer molecular systems for non biological applications was not on their schedule.
Small but parallel to the funding growth of "nanotechnology" APM was (under the same term) further advertised in the book "Engines of Creation"  ("Engines of Creation" is superseded by "Radical Abundance"  by now). To be sensationalistic scifi literature writers picked the most dystopic ideas out of there left the rest behind and packed everything under the "nanotechnology" label. The mental image of "nanotechnology" that spread in the non scientific but interested US public (a very flawed one) was about swarms of with high potential uncontrollable and unstoppable life-like nanobugs eating anything and everything until nothing is left.
Some material scientists working under the banner "nanotechnology" began to see themselve associated with this as very dangerous perceived goal when what they actually did was absolutely unrelated. This in combination with fundamental conceptual differences between biological systems and artificial advanced APM systems may be the reason why some highly recognized scientific minds initially put heavy criticism on the ideas of APM negating its feasibility. This criticism led to APM being perceived as more unsound than it is. The main critic points where straw man attacks discussing the impossibility of an illusionary design no one has proposed. Real tooltip chemistry for technology level III has been carefully analyzed: (Paper: A Minimal Toolset for Positional Diamond Mechanosynthesis) and led to very promising results.
This whole story was presented in the book "Radical Abundance"  (and several other places?). It concludes this episode with that now with the emergence systematic polypeptide design and structural DNA nanotechology the first steps of the "inverse route" in molecular sciences to less biological and more predictable systems are done. And that we need some further steps in this bottom up direction until we can tie it together with the available top down tools.
Citation: " ... Initially, NNI was funded to address atomically precise manufacturing, but by 2004 all reference to APM was removed from the NNI strategic plan and replaced instead by a concentration on phenomena at the nanoscale. ..." Source: 
A collection of other common misconceptions about APM can be found here.
- Foresight Institute news: A bird's-eye view of half a century of nanotechnology
- From Feynman To Funding from K. Eric Drexler - Foresight Institute
- Engines of Creation: The Coming Era of Nanotechnology - by K. Eric Drexler
- Radical Abundance: How a Revolution in Nanotechnology Will Change Civilization - by K. Eric Drexler