The task is to build an intelligent sensor that can instruct a Lego robot to perform certain tasks. The sensor is mounted on the Lego robot and it contains a digital camera which takes continuous images of the front view of the robot. These images are received by an FPGA which simultaneously saves them in an external storage device (SDRAM). At one time only one image is saved and during the time it is being saved, FPGA processes the image to extract some meaningful information. In front of digital camera there are different objects. The sensor is made to classify various objects on the basis of their color. For the classification, the requirement is to implement color image segmentation based object tracking algorithm on a small Field Programmable Gate array (FPGA). For the color segmentation in the images, we are using RGB values of the pixels and with the comparison of their relative values we get the binary image which is processed to determine the shape of the object. A histogram is used to retrieve object‟s features and saves results inside the memory of FPGA which can be read by an external microcontroller with the help of serial port (RS-232).