RAWcooked

Data is never RAW.
Data is always cooked in a way or another.

Jérôme Martinez

No Time To Wait, Vienna, November 2017

FFV1

Lossless video compression format

Open source, patent free

Adopted by several archives

Being standardized (IETF)

Frames are divided by slices, with checksums

Compression

Example with 1 second at 24 fps 10-bit HD film on a 6-core (12-thread) Skylake-X CPU:

• 24 DPX files (or in ZIP/TAR uncompressed): 189 MB
• 1 compressed ZIP file: 175 MB in 10 seconds
• 1 compressed LZMA2 file: 154 MB in 30 seconds
• 1 compressed LZMA2 solid (don't do that!) file: 144 MB
• 1 FFV1/MKV Intra 16-slice file: 105 MB in 1.5 seconds

Advantages

• Open source and free (FFmpeg)
• Package is 1.5-3x smaller than DPX/TIFF
• Cheksum by "Cluster" (usually 1 second) at container level
• Cheksum by "Slice" (you choose how many per frame) at video level
• Files are natively playable by lot of players (VLC, FFplay...)

Disadvantages

• You lose some metadata
  (DPX/TIFF header: scan software, some colorimetry info, film type, DPX time code, shutter angle, gamma...)
• Still not all in one package
  (sidecar files e.g. framemd5, metadata in XML)
• But...

RAWcooked

• Store TIFF/DPX header in Matroska elements
• Store other sidecar files as Matroksa attachments
• Conversion is reversible (bit-by-bit to original files)

Usages

• Storage
  Save HDD/LTO space (less €...)
• Transport
  Encode, transport, decode; you save bandwidth (€... and transfer speed) without changing something else in your workflow (same files after revert to DPX/TIFF)

Drawbacks

• More CPU intensive than uncompressed ZIP
  (future project: use GPU and/or SSE/AVX)
• You lose a complete slice content if there is a single corrupted bit
  (Mitigated with several slices per frame, MediaConch can fix a bitflip error and could be extended to a byte error)

Development

Few TIFF/DPX flavors first

Alpha release soon

Beta release in February 2018

First release in April 2018

We need you!

Developed by MediaArea
https://mediaarea.net

Main sponsorship by AV Preservation by reto.ch
https://avpres.net/RAWcooked/

Additional sponsorship by you?

Stay in touch

MediaArea: https://mediaarea.net, @MediaArea_net

RAWcooked / Reto: https://avpres.net/RAWcooked/

Jérôme Martinez: jerome@mediaarea.net

Slides: https://mediaarea.net/Events

License (except images): CC BY