Neural Semantic Video Analysis

Method

The core promise of machine learning is the automation and scaling of human intelligence. Neural networks, as the most powerful machine learning tool, are utilized to provide viable solutions for the challenges of SVA. Learning spatiotemporal patterns from sample videos enables neural SVA to recognize complex spatial and temporal patterns in noisy and long videos. Modern neural SVA architectures are capable of human-level performance in this regard. Furthermore, these models can be efficiently computed using modern graphical processing units (GPUs), so scalability is limited only by the available GPUs. Hence, neural networks are the preferred solution for SVA due to their accuracy and scalability.

For machine learning models to function properly, it is critical to understand how images are represented and stored on computers. RGB, short for red, green, and blue, is the most common format for representing images on computers. Typically, RGB images are stored as a matrix of pixel values in a 2D plane (or 3D, if RGB values are considered as dimensions). Each image is composed of pixels arranged in rows (across its height) and columns (across its width). The intensity of red, green, and blue colors is indicated by three values per pixel at each X (horizontal position) and Y (vertical position). Different combinations of red, green, and blue values produce a variety of colors. Essentially, videos are just extended versions of images. So, videos can be regarded as a succession of images across time. Video adds a third dimension to visual data by adding a temporal dimension. The added Z dimension indicates the temporal position of the pixel in the video. Accordingly, RGB values are assigned to a pixel in space and time with unique X, Y, and Z coordinates.