The town of Pillaiyarpatti is named after Pillayar - the tamil name for Ganesha, Karpaka Vinayaka temple houses rock cut images of lord Shiva and others as well as several other shrines. Steeped in the tradition of Agamic texts the temple bears testimony to the vibrant temple culture of the Tamil people, passed down through centuries.

Karpaka Vinayakar or Desi Vinayaka Pillaiyar is the presiding deity here, Like the Karpaga Vriksham, which bestows all prosperity, praying to this idol will shower all good. Lord Ganesha faces north, which is said to be the direction of Kubera he is portrayed with two arms and a trunk curled towards his right in the valampuri mode. This 6 feet tall mammoth image of Ganesha is a bas relief in an excavated cave, off of a hill in the precincts of the temple. Tiruveesar is a Shiva Lingam carved in a similar manner. Other deities in the temple include Marudankudi Eesar and his consort Vaadaamalar Mangai. Over 15 inscriptions are also found within the temple, that help establish the age of the temple.

The Stalapuranam published by the temple classifies the growth of this temple into three distinct stages. The first stage goes back in time by about 1600 years. During this period, the innermost rock cut shrines housing Karpaka Vinayakar and Tiruveesar came into being. The uniqueness of the image of Ganesha is one factor testifying this date; the characters used in the temple inscriptions also help establish this date. The pillars within the shrine are of pre-Pallava origin.

As Vinayagar satisfies the wishes of his devotees like Karpagam tree, he is also known as "Karpaga Vinayagar". It is believed that by worshipping Valampuri Vinayagar who is seated facing north, devotees are blessed with prosperity.