This article is concerned with detection of invasive species---Prorocentrum minimum (P. minimum)---in phytoplankton images. The species is known to cause harmful blooms in many estuarine and coastal environments. A new technique, combining phase congruency-based detection of circular objects in images, stochastic optimization, image segmentation, and SVM and random forest-based classification of objects was developed to solve the task. The developed algorithms were tested using 114 images of 1280 x 960 pixels. There were 2088  P. minimum cells in the images in total. The algorithms were able to detect 93.25% of objects representing P. minimum cells and correctly classify 94.9% of all objects. The results are rather encouraging and will be used to develop an automated system for obtaining abundance estimates of the species.