RAWcooked

RAWcooked easily encodes RAW audio-visual sequences into a lossless video stream, reducing the file size by between one and two thirds. FFmpeg encodes the audio-visual data into a Matroska container (MKV) using the video codec FFV1, and audio codec FLAC. The metadata accompanying the RAW data is fully preserved, along with additional sidecar files such as MD5 checksums, LUT or XML if desired. This allows for the management of these audio-visual file formats in an effective and transparent way. The lossless Matroska audio and video streams can be played in VLC or MPV media players, and writing and retrieving from storage devices such as LTO is significantly quicker.
If you need to use the RAW source in its original form, one line of code will easily restore it bit-by-bit, faster than retrieving the same file from LTO tape storage.


You need to download and install FFmpeg to use RAWcooked.

The challenges of archival storage

Many archives preserve their scanned films as TIFF or DPX image sequences. These are grouped as multiple film reels, or individual reels in folders that are often TAR or ZIP wrapped before being copied to longterm storage. Since the beginning of digital moving image preservation it has been the dream of many archives to have a single file format that is suitable for the preservation of both film and video.

For further context:

The missing piece of software

The solution MediaArea have created to meet this need is RAWcooked. This software encodes from, and decodes to, «raw» audio-visual image sequences in a revolutionary and transparent way. The image and/or sound content, including all the embedded metadata and sidecar files, are fully preserved at the bit level.
An otherwise unviewable image sequence can be viewed easily with VLC or MPV when encoded as a Matroska file. Plus this process is fully reversible returning your image sequence back to its original package content.

Matroska file complete reversibility

Whenever necessary, RAWcooked can decode the Matroska file back to the original RAW image sequence, including restoration of the original metadata and sidecar files. It is important to stress that the encoded files can be fully decoded, and that this process will be created bit-by-bit identical files to the originals. Not only is the image and/or sound content fully preserved, but also all enclosed metadata and the all of the file’s characteristics. Therefore, an encoded and decoded RAW file cannot be differentiated from its original.

A complete ecosystem

The Matroska files that have been encoded by RAWcooked can be used with many popular open source softwares which support FFV1 and FLAC playback, including:

  • VLC media player for quick and simple review
  • FFmpeg for transcoding, playback, metadata viewing and extraction
  • QCTools advanced quality control software for audio-visual files
  • MediaInfo display and export technical metadata
  • MediaConch for checking the conformance of a file

Being open source, they’re low cost (and free for the many commonly used flavors) allowing archives with limited resources to process the files encoded by RAWcooked on any computer, such as for cataloging or access purposes.

Funding plan

MediaArea.net has achieved implementation of RAWcooked’s basic module with thanks to AV Preservation reto.ch, who has guaranteed software development with funding of 10 000 €.

The archive community or interested industry users are invited to buy additional flavors required by their institution - such as components, bit depths, endianness, packing, etc. in order to help the long-term sustainability of the project and its further development. Required flavors will be fully tested by MediaArea.net to exact specification before being implemented into the software.

This model allows contributors the assurance that their files will work seamlessly with the software from the outset.

The price is 1000 € for basic support including bug fixes and one additional flavor, then 500 € for each following flavor which can be added later if you are not sure of how many you need immediately.

Status

RAWcooked had its first official release October 2018, with support for a limited number of DPX flavors. To use RAWcooked simply type rawcooked /RAWsequenceDPX/ which begins the encoding using FFmpeg, creating an MKV encoded file in the same folder, leaving the original DPX sequence intact. You can also use this command rawcooked yourdpx.dpx using the first DPX file in a sequence within your folder. To return the MKV back to its RAW source format use rawcooked yourdpx.mkv which uses RAWcooked software to unpack your RAW sequence bit-by-bit, including all original metadata and sidecar files.
Developments are still ongoing for a future version that will implement checksum hash embedding and reversibility check functions. Development snapshots are available and you are welcome to contribute to this process by testing the code.

Supported input flavors

D = Supported by default / K = Supported with a key, please contact us / N = Need file & development if interest, please contact us

Video format Components Bit depth Packing Endianness Support
DPX
left to right
top to bottom
1 image element
unsigned
without encoding
RGB 8-bit D
10-bit Filled A BE D
LE D
12-bit Packed BE K
LE N
Filled A BE K
LE K
16-bit BE K
LE K
16-bit Float (out of FFV1 specs) BE N
LE N
RGBA 8-bit K
10-bit Filled A BE K
LE K
12-bit Packed BE K
LE N
Filled A BE K
LE K
16-bit BE K
LE K
Y 8-bit K
10-bit Filled A BE K
LE N
Filled B BE K
LE N
12-bit Packed BE K
LE N
Filled A BE N
LE N
16-bit BE K
LE K
Video format Components Bit depth Configuration Endianness Support
TIFF
Raw Compression
Continuous StripOffsets
RGB 8-bit Chunky K
16-bit Chunky BE K
LE K
RGBA 8-bit Chunky K
16-bit Chunky BE N
LE K
Y 8-bit Chunky K
16-bit Chunky BE K
LE K
Video format Components Bit depth Endianness Support
EXR
Limitations to be defined
RGB 16-bit Float (out of FFV1 specs) K
32-bit N
32-bit Float N
Video format Components Bit depth Endianness Support
AVI/v210+PCM RGB 10-bit K
Audio format Sampling rate Bit depth Channels Endianness Support
PCM
In WAV, BWF, RF64, AIFF, AVI
44.1/48/96 kHz 8/16/24-bit 2/4/6/8-channel BE/LE K
Except 48 kHz 16-bit 2-channel LE D
44.1/48/96 kHz 32-bit 2/4/6/8-channel BE/LE N

Another format/flavor you want us to support (e.g. little endian…)? Contact us.

Supported output formats

Container format Stream format Support
MKV FFV1 D
FLAC D
PCM K

Another format you want us to support (e.g. MOV…)? Contact us.

License:

It is Open-Source software which means that end users and developers have the freedom to study, improve and redistribute the program (BSD 2-Clause license for the code developed by MediaArea).