This chapter addresses the problem of combining color and geometric invariants for object description by proposing a novel colored invariant local feature descriptor. The proposed approach uses scale-space theory to detect the most geometrically robust features in a physical-based color invariant space. Building a geometrical invariant feature descriptor in a color invariant space grants the built descriptor the stability to both geometric and color variations. The comparison between the proposed colored local invariant features and gray-based local invariant features with respect to stability and distinction supports the potential of the proposed approach. The proposed approach is applicable in any vision-based intelligent system that requires object recognition/retrieval. At the end of this chapter, we present a case study of a local features-based camera planning platform for smart vision systems.