Reversibly Normalise Film Scans & Optimize Storage
State of RAWcooked
Data is never RAW.
Data is always cooked in a way or another.
Jérôme Martinez
No Time to Wait 6, October 2022
What is RAWcooked
From
- 1 DPX/TIFF file per video frame
- Not playable as is by several players (VLC...)
- Huge size
- So many DPX or TIFF format flavors
(interoperability issues)
To
- 1 file with all (video, audio, sidecar files)
- Playable by lot of players (MKV+FFV1+FLAC)
- Size divided by 2
- Reversible to source format if needed
Technical chalenges
So many DPX & TIFF flavors.
"padding bit shall be zero" in spec... Not in reality.
Some scanners are not spec compliant.
Need to have the user confident that all is reversible.
Supported input formats
- DPX/Raw: 8/10/12/16 bit, RGB/RGBA/Y
- TIFF/Raw: 16 bit, RGB
- WAV/PCM: 16/24 bit, 1/2/6/8 channel, 44/48/96 kHz
- AIFF/PCM: 16/24 bit, 1/2/6/8 channel, 44/48/96 kHz
- Based on files from our sponsors
- More formats or format flavors on request
RAWcooked sponsors
(+ a couple of unlisted entities)
RAWcooked sponsors
- AV Preservation by reto.ch (initial sponsor)
- National Audiovisual Centre Luxembourg (CNA)
- National Library of Norway
- Irish Film Institute (IFI)
- Northwest University Library
- National Library of Wales
- Walter J. Brown Media Archives
- The MediaPreserve
- British Film Institute
- New York Public Library
- Národní filmový archiv | National Film Archive (NFA CZ)
- ELEMENTS.tv
- Stanford Libraries
- Friedrich Ebert Stiftung|Foundation
- Preserve South
- National Film Archive Hungary
- Memnon
- State Library of New South Wales
- Vlaamse Radio- en Televisieomroeporganisatie (VRT)
- Jerusalem Cinematheque’s Israel Film Archive
- MWA Services
- And a couple of others
Performance
FFV1 is good, but slow like lot of lossless formats.
We are working on an GPU accelerated encoder & decoder.
The most requested improvement, but not quick to do.
Performance depends also a lot of storage I/O.
BFI uses GNU Parallel for optimizing speed, see github.com/bfidatadigipres/dpx_encoding
(also contains a lot more about RAWcooked automation)
Impact of corrupted block
With DPX/TIFF, 1 storage block failure means a couple of pixels lost.
With FFV1, 1 storage block failure means a slice lost.
By default we set an high count of slices per frame for limiting impact.
We are working on erasure code addition for error correction.
PoC ready.
Source files access
Wish to be even more like a ZIP.
Currently extraction of all or nothing.
Plan to have options for extracting only 1 file, a time span, only audio...
Long term.
File size
Avoiding thousands of files is great.
But 4K content is big, potentially 2 TB files.
Plan to have options for splitting by time, max size...
Long term.
Graphical interface
Command line is great but difficult for beginners.
Graphical interface would be nice.
Long term.
Financial sustainability
- Open source code provided without lock to sponsors
- Deliveries on our website are with a lock
- DPX 8/10 bit RGB & WAV 2ch 48kHz flavors are usable by default
- We provide a key for other format flavors and features (temporary key possible)
- 1000 € for first flavor/feature
+ 500 € per additional flavor/feature - (planned) 500 €/year for maintenance
- To be compared with storage cost saving
(storage cost divided by 2)
Financial sustainability
- Current sponsors filled ~80% of the spending.
(based on average senior developer rate) - We keep getting new sponsors.
- We plan to sell GPU/AVX accelerated versions.
- As well as other options previously indicated.
Stay in touch
MediaArea: https://mediaarea.net, @MediaArea_net
RAWcooked: https://MediaArea.net/RAWcooked
Jérôme Martinez: jerome@mediaarea.net
Slides: https://MediaArea.net/Events
License (except images): CC BY