Flash-Based Storage Deduplication Techniques: A Survey

2. General Deduplication Process

The aim of deduplication is reducing data redundancy. The idea is to store only unique data pieces using references on existing ones instead. A general deduplication process consists of four stages (see Figure 1).

It starts with splitting the input data stream (e.g., a file) into data pieces called chunks. For the file-level deduplication, this stage is omitted, but chunk-level shows better deduplication ratio (Xia et al., 2016). Dynamic chunk size provides higher deduplication ratio compared to the fixed size. In SSD-based storage systems, the chunk size usually equals the SSD page size (4 KB): this improves the write performance, garbage collection, and wear levelling. However, other options of chunk size are considered, e.g.:

• •

  A page (in (Seo et al., 2015): Page-sized chunking is a base for a novel deduplicating Flash Translation Layer (FTL) scheme);

• •

  Several pages (in (Lee et al., 2011): Different sizes of chunks are compared from the point of view of performance);

• •

  A block (in (Ha et al., 2013): A new “block-level content-aware chunking” deduplication scheme is proposed to extend the lifetime of SSD).

Figure 1.

General deduplication process