RAWcooked

The challenges of archival storage

Many archives preserve their scanned films as TIFF or DPX image sequences. These are grouped as multiple film reels, or individual reels in folders that are often TAR or ZIP wrapped before being copied to longterm storage. Since the beginning of digital moving image preservation it has been the dream of many archives to have a single file format that is suitable for the preservation of both film and video.

For further context:

• Reto Kromer: «Matroska and FFV1: One File Format for Film and Video Archiving?», in Journal of Film Preservation, n. 96 (April 2017)
• Kieran O’Leary: «Introduction to FFV1 and Matroska for Film Scans», 2016–10–07

The missing piece of software

The solution MediaArea have created to meet this need is RAWcooked. This software encodes from, and decodes to, «raw» audio-visual image sequences in a revolutionary and transparent way. The image and/or sound content, including all the embedded metadata and sidecar files, are fully preserved at the bit level.
An otherwise unviewable image sequence can be viewed easily with VLC or MPV when encoded as a Matroska file. Plus this process is fully reversible returning your image sequence back to its original package content.

Matroska file complete reversibility

Whenever necessary, RAWcooked can decode the Matroska file back to the original RAW image sequence, including restoration of the original metadata and sidecar files. It is important to stress that the encoded files can be fully decoded, and that this process will be created bit-by-bit identical files to the originals. Not only is the image and/or sound content fully preserved, but also all enclosed metadata and the all of the file’s characteristics. Therefore, an encoded and decoded RAW file cannot be differentiated from its original.

A complete ecosystem

The Matroska files that have been encoded by RAWcooked can be used with many popular open source softwares which support FFV1 and FLAC playback, including:

• VLC media player for quick and simple review
• FFmpeg for transcoding, playback, metadata viewing and extraction
• QCTools advanced quality control software for audio-visual files
• MediaInfo display and export technical metadata
• MediaConch for checking the conformance of a file

Being open source, they’re low cost (and free for the many commonly used flavors) allowing archives with limited resources to process the files encoded by RAWcooked on any computer, such as for cataloging or access purposes.

Funding plan

MediaArea.net has achieved implementation of RAWcooked’s basic module with thanks to AV Preservation reto.ch, who has guaranteed software development with funding of 10 000 €.

The archive community or interested industry users are invited to buy additional flavors required by their institution - such as components, bit depths, endianness, packing, etc. in order to help the long-term sustainability of the project and its further development. Required flavors will be fully tested by MediaArea.net to exact specification before being implemented into the software.

This model allows contributors the assurance that their files will work seamlessly with the software from the outset.

The price is 1000 € for basic support including bug fixes and one additional flavor, then 500 € for each following flavor which can be added later if you are not sure of how many you need immediately.

Status

RAWcooked had its first official release October 2018, with support for a limited number of DPX flavors. To use RAWcooked simply type rawcooked /RAWsequenceDPX/ which begins the encoding using FFmpeg, creating an MKV encoded file in the same folder, leaving the original DPX sequence intact. You can also use this command rawcooked yourdpx.dpx using the first DPX file in a sequence within your folder. To return the MKV back to its RAW source format use rawcooked yourdpx.mkv which uses RAWcooked software to unpack your RAW sequence bit-by-bit, including all original metadata and sidecar files.
Developments are still ongoing for a future version that will implement checksum hash embedding and reversibility check functions. Development snapshots are available and you are welcome to contribute to this process by testing the code.

Supported input flavors

D = Supported by default / K = Supported with a key, please contact us / N = Need file & development if interest, please contact us

Another format/flavor you want us to support (e.g. little endian…)? Contact us.

Supported output formats

Another format you want us to support (e.g. MOV…)? Contact us.

License:

It is Open-Source software which means that end users and developers have the freedom to study, improve and redistribute the program (BSD 2-Clause license for the code developed by MediaArea).