Dictionary Training

Train dictionaries from sample data to improve compression of small files

Dictionary training analyzes sample data to create optimized dictionaries for compressing similar files. The resulting dictionary can dramatically improve compression ratios for small data.

ZDICT_trainFromBuffer()

Train a dictionary from an array of samples using the fast COVER algorithm.

size_t ZDICT_trainFromBuffer(
    void* dictBuffer,
    size_t dictBufferCapacity,
    const void* samplesBuffer,
    const size_t* samplesSizes,
    unsigned nbSamples
);

dictBuffer · void*

Output buffer where the trained dictionary will be stored.

dictBufferCapacity · size_t

Maximum size of the output dictionary buffer. Recommended: ~100 KB.

samplesBuffer · const void*

Input buffer containing all samples concatenated together.

samplesSizes · const size_t*

Array containing the size of each sample, in order.

nbSamples · unsigned

Number of samples provided. Recommended: provide ~100x the dictionary size in total samples.

Returns

Size of dictionary stored into dictBuffer (<= dictBufferCapacity), or an error code which can be tested with ZDICT_isError().

Notes

• This function redirects to ZDICT_optimizeTrainFromBuffer_fastCover() with default parameters (d=8, steps=4, f=20, accel=1)
• Memory usage is about 6 MB
• Training will fail if there are not enough samples or if samples are too small (< 8 bytes)
• Recommended to provide a few thousand samples totaling ~100x the target dictionary size

Example

#define DICT_SIZE (100 * 1024)  // 100 KB
#define NUM_SAMPLES 1000

size_t sampleSizes[NUM_SAMPLES];
void* samplesBuffer = /* load concatenated samples */;
void* dictBuffer = malloc(DICT_SIZE);

size_t dictSize = ZDICT_trainFromBuffer(
    dictBuffer, DICT_SIZE,
    samplesBuffer, sampleSizes, NUM_SAMPLES
);

if (ZDICT_isError(dictSize)) {
    fprintf(stderr, "Dictionary training failed: %s\n",
            ZDICT_getErrorName(dictSize));
}

---

ZDICT_finalizeDictionary()

Convert raw dictionary content into a zstd dictionary by adding headers and entropy tables.

size_t ZDICT_finalizeDictionary(
    void* dstDictBuffer,
    size_t maxDictSize,
    const void* dictContent,
    size_t dictContentSize,
    const void* samplesBuffer,
    const size_t* samplesSizes,
    unsigned nbSamples,
    ZDICT_params_t parameters
);

dstDictBuffer · void*

Output buffer for the finalized dictionary. Can overlap with dictContent.

maxDictSize · size_t

Maximum size of the output dictionary. Must be >= max(dictContentSize, ZDICT_DICTSIZE_MIN).

dictContent · const void*

Raw dictionary content (can be from any source, not just zstd training).

dictContentSize · size_t

Size of the raw dictionary content.

samplesBuffer · const void*

Buffer containing concatenated samples for building entropy tables.

samplesSizes · const size_t*

Array of sizes for each sample.

nbSamples · unsigned

Number of samples provided.

parameters · ZDICT_params_t

Dictionary parameters:

• compressionLevel: Optimize for specific compression level (0 = default)
• notificationLevel: Log verbosity (0-4, where 0 = none)
• dictID: Force specific dictionary ID (0 = auto-generate random ID)

Returns

Size of dictionary stored into dstDictBuffer (<= maxDictSize), or an error code which can be tested with ZDICT_isError().

Notes

• Adds zstd header with magic number, dictionary ID, and entropy tables
• Samples are used to construct statistics for the compression level specified
• If header + content doesn’t fit in maxDictSize, content is truncated from the beginning
• Most profitable content is presumed to be at the end of the dictionary
• May fail if not enough samples, samples are uncompressible, or all samples are identical

Example

// Convert raw content to zstd dictionary
char rawDict[1024] = /* custom dictionary content */;
void* samples = /* load samples */;
size_t sampleSizes[100];

ZDICT_params_t params;
memset(&params, 0, sizeof(params));
params.compressionLevel = 3;
params.notificationLevel = 2;  // Show progress
params.dictID = 0;  // Auto-generate

void* dictBuffer = malloc(110 * 1024);
size_t dictSize = ZDICT_finalizeDictionary(
    dictBuffer, 110 * 1024,
    rawDict, sizeof(rawDict),
    samples, sampleSizes, 100,
    params
);

if (ZDICT_isError(dictSize)) {
    fprintf(stderr, "Failed: %s\n", ZDICT_getErrorName(dictSize));
}

---

Helper Functions

ZDICT_getDictID()

Extract the dictionary ID from a dictionary buffer.

unsigned ZDICT_getDictID(const void* dictBuffer, size_t dictSize);

Returns the dictionary ID, or 0 if the buffer is not a valid zstd dictionary.

ZDICT_isError()

Test if a return value indicates an error.

unsigned ZDICT_isError(size_t errorCode);

Returns 1 if error, 0 otherwise.

ZDICT_getErrorName()

Get a human-readable error message.

const char* ZDICT_getErrorName(size_t errorCode);

Returns a string describing the error.