RAWcooked

RAWcooked encodes RAW audio-visual data into the Matroska container (MKV), using the video codec FFV1 for the image and audio codec FLAC for the sound. The metadata accompanying the RAW data are preserved, and sidecar files, like MD5, LUT or XML, can be added into the Matroska container as attachments. This allows for the management of these audio-visual file formats in an effective and transparent way (e.g. native playback in VLC), while saving typically between one to two thirds of the needed storage, and speeding up the back-up process on LTO cartridges.
When needed, the uncompressed source is retrieved bit-by-bit, in a manner faster than uncompressed sources directly stored on LTO cartridges.

A current reality of many archives

Many archives preserve their single-image based files as TIFF or DPX, grouped by entire films or single reels, and stored inside folders, or TAR or ZIP archives. Since the beginning of digital moving image preservation, having a single file format which is suitable for the preservation of both film and video has been a dream of many audio-visual archives, especially smaller ones.

Additional context:

The missing piece of software

The solution we propose is RAWcooked, a piece of software that encodes from and decodes to various so-called «raw» audio-visual file formats in a completely transparent way. The image content and/or sound content, including all the related metadata and sidecar files, is fully preserved at the bit level.
For instance, the unplayable ZIP files could be replaced by a playable file, with the full reversibility of the original package content.

Decoding the Matroska file back to the original RAW

Whenever needed, RAWcooked decodes back the encoded Matroska file to the original RAW files, including all the original metadata and sidecar files. It is important to stress clearly that the encoded files can be decoded, and the resulting files are bit-by-bit identical to the original ones. Not only is the image and/or sound content fully preserved, but also all enclosed metadata and the all of the file’s characteristics. Therefore, an encoded and decoded RAW file cannot be differentiated from its original.

A complete ecosystem

The Matroska files which have been encoded by RAWcooked can be used on wide range of popular software supporting Matroska/FFV1 and FLAC plackack, including:

  • VLC media player for playback
  • FFmpeg tools can transcode, playback and probe those files
  • QCTools can performs in-depth analysis
  • MediaInfo can display technical information
  • MediaConch can check the conformance of the file

This way an archive with limited resources can process the files encoded by RAWcooked on every current computer (e.g. for cataloging or access purposes).

Funding plan

MediaArea.net realizes the implementation of the basis module for anticipated costs of 10 000 € which are guaranteed by AV Preservation by reto.ch.

The archival community and the industry are invited to buy the flavors (components, bit depths, endianess, packing, etc.) of the formats they actually need. An example file of each flavor of a format will be tested by MediaArea.net and the exact specification implemented into the software. This way the contributing bodies and individuals are sure their files can be processed by the software. The price is 1 000 € for the first flavor of a format and 500 € for each following.

Status

RAWcooked is currently under development. Development snapshots are available (corresponding code), not to be used in production yet. As in January, some DPX flavors are supported, a sequence of DPX is detected, encode is delegated to FFmpeg and decode is native (from another MediaArea project), rawcooked yourdpx.dpx provides the FFmpeg command to run for encode, rawcooked yourdpx.mkv reverts bit-by-bit.
The first release is officially launched during the FIAF Congress 2018 (22–27 April 2018 in Prague, Czech Republic). Priority is given to formats and flavors from our sponsors. You can test with this file (or a sequence of files).

Supported (when officially launched) input flavors

D = Supported by default / K = Supported with a key, please contact us / N = File needed, please contact us

Video format Components Bit depth Packing Endianess Support
DPX
left to right
top to bottom
1 image element
unsigned
without encoding
RGB 8-bit D
10-bit Filled A BE D
LE D
12-bit Packed BE K
LE N
Filled A BE K
LE K
16-bit BE K
LE K
16-bit Float (out of FFV1 specs) BE K
LE N
RGBA 8-bit K
10-bit Filled A BE K
LE K
12-bit Packed BE K
LE N
Filled A BE K
LE K
16-bit BE K
LE K
Video format Components Bit depth Endianess Support
TIFF
Limitations to be defined
RGB 16-bit N
Video format Components Bit depth Endianess Support
EXR
Limitations to be defined
RGB 16-bit Float K
32-bit K
32-bit Float K
Audio format Sampling rate Bit depth Channels Support
PCM
In legacy WAV, BWF, RF64
48 kHz 16-bit 2-channel D
6-channel K
24-bit 2-channel K
6-channel K
32-bit 2-channel N
6-channel N
96 kHz 16-bit 2-channel K
6-channel K
24-bit 2-channel K
6-channel K
32-bit 2-channel N
6-channel N

Another format/flavor you want us to support (e.g. little endian…)? Contact us.

Supported (when officially launched) output formats

Container format Stream format Support
MKV FFV1 D
FLAC D
PCM K

Another format you want us to support (e.g. MOV…)? Contact us.

License:

It is Open-Source software which means that end users and developers have freedom to study, to improve and to redistribute the program (BSD 2-Clause license for the code developed by us).