This document aims to describe the file format and database schema used by Denon DJ's Engine Prime software. The same format is also used by Denon DJ's standalone SC5000 Prime professional DJ players.

The purpose of the data written is to capture information about audio tracks that may be played through SC5000 Prime players. Either an SC5000 player itself, or the Engine Prime software running on a normal computer, can identify and write this analysis data.

The SC5000 player accepts either a USB stick or SD card as media.

On a USB stick, the filesystem structure is as follows:

/

/Engine Library

/Engine Library/m.db

/Engine Library/m.db-journal

/Engine Library/p.db

/Engine Library/p.db-journal

/Engine Library/sm.db

/Engine Library/sm.db-journal

/Engine Library/sp.db

/Engine Library/sp.db-journal

The .db files are SQLite database files. The .db-journal files appear to be rollback journals, which are supposed to be temporary files, but for some reason are retained. Further information about temporary files used by SQLite is available on the SQLite website.

The 'm' database is the main location for storing metadata about music/audio tracks.

Each row represents a single image that can be used as 'album art' for any number of audio tracks.

By default, there is always an entry with id equal 1, with hash and albumArt set to NULL, which represents 'no album art'. So, a track which does not have any art would have track.idAlbumArt set to 1.

As the image data is embedded in the database albumArt column as a BLOB directly, the hash field is used to avoid duplication of identical album art across multiple tracks. If multiple tracks have the same album art (as determined by hashing the image data), then only one entry should be written in AlbumArt and all such tracks should set their track.idAlbumArt column to point to it.

TBC - unsure how the hash is computed. It appears to be an SHA1 hash from the length, but not sure what it is a hash of.

TODO - unsure what a row represents. Perhaps when a track is played from a linked media device (e.g. USB stick on another SC5000 player connected via network cable), the track is copied into the local database? Is the file copied too (assume not, because of space reasons)?

Each row represents a 'Crate', i.e. a unique subset of unordered tracks in the overall collection, grouped together for DJ convenience.

Each row represents an optional relationship between a parent and child crate.

Each row represents a statement that a given track is part of a given crate. A track may belong to any number of crates

Each row represents an instance when the media containing the current database was played on a compatible player.

Each row represents a track that was loaded and played from the current database.

TBC - how much of a track needs to be played before it appears in this table?

TBC - if tracks from another database are played on the same player that the media holding this database is plugged into, are those foreign tracks recorded in this table?

Single-row table containing metadata about the current Engine Library database.

The schema is versioned according to a typical three-part version indicator (presumably according to Semantic Versioning). When the original author of this document tried this out on a newly-bought Denon SC5000 Prime purchased in August 2017, the schema version created by the player was v1.6.0.

Each row represents some kind of textual (non-numeric) track meta-data, as distinguished by a reference to the track and a reference to the type of metadata for that row.

Meanings of the type column:

1. Title
2. Artist
3. Album
4. Genre
5. Comment
6. Publisher
7. Composer
8. TBC - seems to always be set to NULL
9. TBC - seems to always be set to NULL
10. Duration in MM:SS
11. *Unused - no entries should be made in Metadata for this value of type*
12. Whether the track has ever been played and added to HistorylistTrackList; set to 1 if it has, or NULL if not
13. File extension, one of:

• wav
• aiff
• mp3
• flac
• m4a
• ogg

14. *Unused - no entries should be made in Metadata for this value of type*
15. Analysed flag. Is set to 1 if track is analyzed. Setting this to 0 triggers track analysis (when automatic is set in preferences)
16. TBC - seems to always be set to 1

Each row represents some kind of whole-number numeric track meta-data, as distinguished by a reference to the track and a reference to the type of metadata for that row.

Meanings of the type column:

1. Date/time, expressed as a UNIX timestamp, when the track was last played (will match the entry in HistorylistTrackList); set to NULL if not ever played
2. Date/time, expressed as a UNIX timestamp, of the file's last status change
3. Date, expressed as a UNIX timestamp, of the file's last load (rounded to midnight GMT) TBC - confirm exactly what file time property this refers to, i.e. is it load, play, prepare, something else?
4. The musical key of the track, as determined by track analysis; the values should be interpreted as follows:

• 0 => 8B / C major
• 1 => 8A / A minor
• 2 => 9B / G major
• 3 => 9A / E minor
• 4 => 10B / D major
• 5 => 10A / B minor
• 6 => 11B / A major
• 7 => 11A / F# minor
• 8 => 12B / E major
• 9 => 12A / Db minor
• 10 => 1B / B major
• 11 => 1A / Ab minor
• 12 => 2B / F# major
• 13 => 2A / Eb minor
• 14 => 3B / Db major
• 15 => 3A / Bb minor
• 16 => 4B / Ab major
• 17 => 4A / F minor
• 18 => 5B / Eb major
• 19 => 5A / C minor
• 20 => 6B / Bb major
• 21 => 6A / G minor
• 22 => 7B / F major
• 23 => 7A / D minor

5. Rating, a value in [0, 20, 40, 60, 80, 100, 120]. 100 and 120 both relate to 5 stars, so 120 is probably a bug from Engine Prime's user interface.
6. TBC - unsure, not populated for any sample tracks
7. TBC - unsure, not populated for any sample tracks
8. TBC - unsure, not populated for any sample tracks
9. TBC - unsure, not populated for any sample tracks
10. A 64-bit number acting as a hash that indicates which tracks have been played in the most recent 'playthrough' of songs from this database. See the currentPlayedIndicator column in the Information table. The field is not populated if a track hasn't ever been played (i.e. doesn't appear in the HistorylistTrackList table). Unsure if the number has any inherent meaning other than for hashing.
11. TBC - unsure, seems to be set to 1 for all sample tracks

Each row represents a playlist.

Each row represents a track in a particular playlist.

There is normally just one row in the Preparelist table, representing a list of tracks that are being 'prepared' for playback.

Each row represents a track that has been 'prepared' for later playing.

Note that after a track in the prepare list is loaded onto a player, the SC5000 Prime removes it from the prepare list.

Each row in the table represents a single audio track that is already stored as a file.

The 'p' database is the main location for persisting analysed performance information about tracks, such as cues, beat grids, and waveforms.

Single-row table containing metadata about the current Engine Library database.

The schema is versioned according to a typical three-part version indicator (presumably according to Semantic Versioning). When the original author of this document tried this out on a newly-bought Denon SC5000 Prime purchased in August 2017, the schema version created by the player was v1.6.0.

Note that the UUID in the p.db database will not be the same as that in the m.db database.

Each row represents information about a track that has been determined by track analysis and preparation, such as the location of hot cues, loops, the beat grid, as well as waveform data.

On a sample USB stick, the 'sm' database is apparently identical to the 'm' database in schema, but with no values populated in any tables apart from the Information table.

TBC - perhaps the 's' in 'sm' stands for Serato?

On a sample USB stick, the 'sp' database is apparently identical to the 'p' database in schema, but with no values populated in any tables apart from the Information table.

TBC - perhaps the 's' in 'sp' stands for Serato?

The PerformanceData table in the p.db database contains a number of BLOB columns, most of which are a compressed binary format. These are described below.

With the exception of the loops column, the BLOBs are compressed using zlib, but the compressed data payload is prefixed with an unsigned 32-bit integer containing the uncompressed data length. This is the format used by the qCompress and qUncompress convenience methods of the QByteArray class from the QT library, a library which is used by Engine Prime. If you extract compressed BLOB data directly from the SQLite database, you will need to uncompress it first to examine the contents.

An example C++ program which can do this is shown below:

#include <iostream>
#include <QtCore>

int main(int argc, char**argv)
{
    if (argc != 2)
    {
        std::cerr << "Usage:" << std::endl;
        std::cerr << argv[0] << " hexstring" << std::endl;
        return 1;
    }
    
    QByteArray blob = QByteArray::fromHex(argv[1]);

    // Uncompress
    QByteArray uncompressed = qUncompress(blob);
    if (uncompressed.isEmpty())
    {
        std::cerr << "Corrupt input, unable to uncompress" << std::endl;
        return 1;
    }

    // Print hexdump of the buffer
    std::cout << uncompressed.toHex().data() << std::endl;
}

Example usage (assuming the above source file is compiled to a binary called "ep_uncompress"):

$ sqlite3 p.db "select hex(trackData) from PerformanceData where id = 1;" | xargs ./ep_uncompress 
40e58880000000000000000001337a003fe2342ca00000000000000b
$

A number of fields in the PerformanceData BLOBs refer to positions within the track. These positions are usually measured in samples, rather than elapsed time. To convert to seconds, you will need to divide by the track's sample rate, which is stored in both trackData and beatData. The vast majority of tracks downloaded from the internet will have 44100Hz as their sample rate, although 48000Hz makes an appearance on occasion too.

Furthermore, some position fields (for example, in beatData and loops) are stored as doubles but in little-endian format, and so the endianness will need to be changed before working with the numbers. The below function can do this for a 64-bit integer (containing the bits that will subsequently be interpreted as a double):

uint64_t change_endianness(uint64_t v)
{
    uint64_t result = 0;
    result |= (v & 0x00000000000000ff) << 56;
    result |= (v & 0x000000000000ff00) << 40;
    result |= (v & 0x0000000000ff0000) << 24;
    result |= (v & 0x00000000ff000000) << 8;
    result |= (v & 0x000000ff00000000) >> 8;
    result |= (v & 0x0000ff0000000000) >> 24;
    result |= (v & 0x00ff000000000000) >> 40;
    result |= (v & 0xff00000000000000) >> 56;
    return result;
}

The standard cue/loop colours are shown below:

TODO - not documented yet

TODO - not documented yet

The beatData format has two sub-structures nested within it: the beat grid, and a beat grid marker. There are two beat grids in each field: one for the default beat grid (as analyzed by Engine Prime or an SC5000) and an adjusted beat grid (if you mess around with it by hand). Each beat grid contains a number of beat grid markers, of which there are a minimum of two.

The first beat marker in any beat grid is always "beat -4", i.e. four beats before the first usable beat in the track. Hence, its sample offset in the file is negative (before the start of the track!). Naturally, it is not possible to actually play any audio from a period in time before the track has actually begun, but the position is still useful to allow for the beat grid to be manually adjusted by up to 4 beats.

The last beat marker in any beat grid is always "beat N + 1", i.e. one beat past the last usable beat in the track. Hence, its sample offset in the file is beyond the last sample in the track.

Also note that when discussing the index/number of any given beat, the beatData format always assumes that the first beat in the file is beat 0, and the last is beat N. The index/number of a given beat is not shown to the end user in either Engine Prime or an SC5000's display.

Note that the BPM can be calculated from the information in beatData as follows:

BPM = SampleRate * 60 * (LastMarkerBeatIndex - FirstMarkerBeatIndex) / (LastMarkerSampleOffset - FirstMarkerSampleOffset)

Main beatData format:

beatgrid format:

marker format:

TODO - there is still an unknown value in the beat grid marker sub-structure. This is populated with varying values, but its exact form and function is not currently known.

As beatData is one of the more complex fields in the Engine Library format, an example always helps. The below is from an example track, where the beatgrid has been adjusted in Engine Prime to correct a mis-identified tempo (wrongly thought to be 97, but was actually 108.3):

Using the adjusted beatgrid values above, the BPM can be calculated as:

BPM = 44100 * 60 * (694 + 4) / (16995906.29 + 57722.04) = 108.3

Note that the quickCues format has the hot cue label length, label, position, and colour fields 8 times sequentially, for each of the 8 hot cues. This is shown in the below table as a 'repeating frame', for brevity. Also note that since cue labels can be of varying length, each repeating frame can also be of varying length.

Note that the loops format has the loop label length, label, position, and colour fields 8 times sequentially, for each of the 8 loops. This is shown in the below table as a 'repeating frame', for brevity. Also note that since labels can be of varying length, each repeating frame can also be of varying length.

Note also that, unlike the other fields in PerformanceData, loops is not compressed.