1670s, "a trick, a hoax, a fraud," also as a verb and an adjective, of uncertain origin; the words burst into use in 1677. Perhaps from sham, a northern dialectal variant of shame (n.); a derivation OED finds "not impossible." Sense of "something meant to be mistaken for something else" is from 1728. The meaning "false front" in pillow-sham (1721) is from the notion of "counterfeit." Related: Shammed; shamming; shammer. Shamateur "amateur sportsman who acts like a professional" is from 1896.
