Reversibly Normalise Film Scans & Optimize Storage

RAWcooked, FFV1
& GPU acceleration

 

Data is never RAW.
Data is always cooked in a way or another.

 

Jérôme Martinez

No Time to Wait 9, October 2025

What is RAWcooked

From

  • 1 DPX/TIFF file per video frame
  • Not playable as is by several players (VLC...)
  • Huge size
  • So many DPX or TIFF format flavors
    (interoperability issues)

To

  • 1 file with all (video, audio, sidecar files)
  • Playable by a lot of players (MKV+FFV1+FLAC)
  • Size divided by 2
  • Reversible to source format if needed

What we do

Compress data in open source compression formats

Matroska, FFV1 and FLAC

They are open and standards (IETF)

Store reversibility data in attachments

Rely on FFmpeg

Long term preservation

Technical chalenges

So many DPX & TIFF flavors.

"padding bit shall be zero" in spec... Not in reality.

Some scanners are not spec compliant.

Need to have the user confident that all is reversible.

Supported input formats

  • DPX/Raw: 8/10/12/16 bit, RGB/RGBA/Y
  • TIFF/Raw: 16 bit, RGB
  • WAV/PCM: 16/24 bit, 1/2/6/8 channel, 44/48/96 kHz
  • AIFF/PCM: 16/24 bit, 1/2/6/8 channel, 44/48/96 kHz
  • Based on files from our sponsors
  • More formats or format flavors on request

~35 RAWcooked sponsors


  • AV Preservation by reto.ch
  • Centre national de l’audiovisuel
  • Irish Film Institute
  • Nasjonalbiblioteket
  • Northwestern University Libraries
  • National Library of Wales
  • Walter J. Brown Media Archives
  • The MediaPreserve
  • British Film Institute
  • New York Public Library
  • New York Public Library
  • ELEMENTS
  • Stanford Libraries
  • Friedrich-Ebert-Stiftung
  • Preserve South
  • National-Film-Archive-Hungary
  • Memnon
  • State Library of New South Wales
  • VRT
  • IFA
  • MWA Services


 
 
 
and more

RAWcooked sponsors

  • AV Preservation by reto.ch (initial sponsor)
  • National Audiovisual Centre Luxembourg (CNA)
  • National Library of Norway
  • Irish Film Institute (IFI)
  • Northwest University Library
  • National Library of Wales
  • Walter J. Brown Media Archives
  • The MediaPreserve
  • British Film Institute
  • New York Public Library
  • Národní filmový archiv | National Film Archive (NFA CZ)
  • ELEMENTS.tv
  • Stanford Libraries
  • Friedrich Ebert Stiftung|Foundation
  • Preserve South
  • National Film Archive Hungary
  • Memnon
  • State Library of New South Wales
  • Vlaamse Radio- en Televisieomroeporganisatie (VRT)
  • Jerusalem Cinematheque’s Israel Film Archive
  • MWA Services
  • ...

Financial sustainability

  • Open source code provided without lock to sponsors
  • Deliveries on our website are with a lock
  • DPX 8/10 bit RGB & WAV 2ch 48kHz flavors are usable by default
  • We provide a key for other format flavors and features (temporary key possible)
  • 1000 € for first flavor/feature
    + 500 € per additional flavor/feature
  • Except for GPU support (1000 €)
  • To be compared with storage cost saving
    (storage cost divided by 2)

Cumulative budget in 2024

~30 sponsors

~45 k€ paid

~40 k€ spent

Thank you FFmpeg for the FFV1 encoder/decoder

Performance is a topic

FFV1 is good, but slow (like all lossless formats).

The biggest complain, especially with 4K+.

Performance depends also a lot on storage I/O.

BFI uses GNU Parallel for optimizing speed, see github.com/bfidatadigipres/dpx_encoding
(also contains a lot more about RAWcooked automation)

Issues

Skills: not everyone knows about optimizing a GPU

Motivation: not everyone is interested in lossless formats

Speed improvements uncertain

What is needed is not the core target of GPU vendors

So we can discover bugs in GPU drivers :)

What we did

Contact with FFmpeg main maintainer

Redirected to someone having the skill and motivation

We implement in FFmpeg

Go!

Technical choices

Choice of the development language

"Vulkan": open, multiple platforms, multiple vendors

But not supported by Apple Silicon GPUs :(

1024 slices by default

Cumulative budget in 2025

~35 sponsors

~50 k€ paid

~60 k€ spent

Encoding

Decoding

Analysis

The perf/price ratio is not stable

Running 3 instances is 3x faster!

Some bottlenecks in FFmpeg

Difficult to investigate

Could we have 6K5K16bit in real time with a RTX 4090?
(4 GB/s...)

Status

In FFmpeg 8.0 (released)

Integration in RAWcooked in progress

The future

Integration of GPU support in RAWcooked in progress

Trying to avoid FFmpeg bottlenecks

  • Focus on DPX files I/O in RAWcooked
  • FFV1 v4 with uncompressed lower bits
  • Parallelization of GPU instances in RAWcooked?
  • Avoiding the FFmpeg "pivot" picture format?
  • Using both the CPU and the GPU in parallel?
  • Reducing the count of slices for small resolutions?

Stay in touch

MediaArea: https://mediaarea.net, @MediaArea_net

RAWcooked: https://MediaArea.net/RAWcooked

Jérôme Martinez: jerome@mediaarea.net

Slides: https://MediaArea.net/Events

License (except images): CC BY