RAWcooked

Data is never RAW.
Data is always cooked in a way or another.

Jérôme Martinez

iPRES, September 2019

MediaArea

Open source software company focused on digital media analysis. We work (different levels of involvement) on:

MediaInfo
Convenient unified display of the most relevant technical and tag data for video and audio files
MediaConch
Implementation checker, policy checker, & reporter
QCTools
Helps users analyze and understand their digitized video files through use of audiovisual analytics and filtering
BWF MetaEdit, AVI MetaEdit, MOV MetaEdit
Embedding, validating, and exporting of metadata
DV Analyzer
Checking presence of technical errors in DV captures

Raw A/V files

Huge size
(4K+ is there, can be 100 MB/frame, several TB per hour)

1 file per video frame (thousands of files in a directory)

Not playable as is by several players (VLC...)

So many DPX or TIFF format flavors
(interoperability issues)

Lossless video compression format

Open source, patent free

Adopted by several archives

Being standardized (IETF)

Frames are divided by slices, with checksums

Example with 1 second at 24 fps 10-bit HD film on a 6-core (12-thread) Skylake-X CPU:

Results vary depending on the content of files
From 0.4x to 2.4x (average 0.7x) real time/thread (encoding/decoding)
From 0.7x to 16x (average 3.5x) the speed of JP2k (FFmpeg)
Better compression ratio by 2% compared to JP2k
(9% with additional FLAC audio compression)
https://viaa.be/files/attachments/.669/VIAA_Preservation_Reformatting.pdf

You lose some metadata
(DPX/TIFF header: scan software, some colorimetry info, film type, DPX time code, shutter angle, gamma...)
Complicated command line
But...

Easy: just a short command line
"rawcooked YourDirectoryName"
Store DPX/TIFF headers/footers in a specific Matroska attachment
Store other sidecar files as Matroksa attachments
Output is a single Matroska/FFV1/FLAC file
Encoding is reversible (bit-by-bit to original files)
"rawcooked YourMatroskaFileName.mkv"

Check if the file is healthy
"rawcooked --check YourMatroskaFileName.mkv"
Check if DPX headers are conform to specs
"rawcooked --conch YourMatroskaFileName.mkv"
Add error correction codes while encoding the file
e.g. with overhead of 1.5%, you can lose 4 blocks every 252 blocks without losing any content
"rawcooked --ecc YourDirectoryName"
Fix the corrupted file
"rawcooked --fix YourMatroskaFileName.mkv"

Archive asks a digitilization to their supplier
Classic workflow with the scanner
+ "rawcooked --all YourDirectoryName"
Transport... (2x less file sizes, less costly)
Archive receives content & checks the integrity
(file health, DPX conformance...)
"rawcooked --check YourMatroskaFileName.mkv"
Archive can visually check the content with
e.g. VLC Media Player
Storage (cost divided by 2 due to compression)
Revert to exact original DPX if someone needs it
"rawcooked YourMatroskaFileName.mkv"