MediaConch

Implementation and policy checking
on FFV1, Matroska, LPCM, and more

Jérôme Martinez, MediaArea

Experience Workshop - November 2016

What is MediaConch?

MediaConch is a conformance checker

• Implementation checker
• Policy checker
• Reporter
• Fixer

What is MediaConch?

Implementation and Policy reporter

What is MediaConch?

Implementation report:

Policy report:

What is MediaConch?

General information about your files

What is MediaConch?

Inspect your files

What is MediaConch?

Policy editor

What is MediaConch?

Public policies

What is MediaConch?

Fixer

• Segment sizes in Matroska
• Matroska “bit flip” correction
• FFV1 “bit flip” correction
  

Integration

Archivematica is an integrated suite of open-source software tools that allows users to process digital objects from ingest to access in compliance with the ISO-OAIS functional model

MediaConch interfaces

• Graphical interface
• Web interface
• Command line
• Server (REST API)
• (Work in progress) a library (.dll/.so/.dylib)

MediaConch output formats

• XML (native format)
• Text
• HTML
• (Work in progress) PDF
• Tweakable! (with XSL)

Open source

• GPLv3+ and MPLv2+
• Relies on MediaInfo (metadata extraction tool)
• Use well-known open source libraries: Qt, sqlite, libevent, libxml2, libxslt, libexslt...

Supported formats

• Priorities for the implementation checker

  • Matroska
  • FFV1
  • PCM

• Can accept any format supported by MediaInfo for the policy checker

  • MXF + JP2k
  • QuickTime/MOV
  • Audio files (WAV, BWF, AIFF...)
  • ...

Supported formats

Can be expanded

• By plugins
  • Support of PDF checker: VeraPDF plugin
  • Support of TIFF checker: DPF Manager plugin
  • You use another checker? Let us know
• By internal development
  • More tests on your preferred format is possible
  • It depends on you!

Versatile

Several input formats are accepted

• FFV1 from MOV or AVI
• Matroska with other video formats
• (Work in progress) Extraction of a PDF or TIFF attachement from a Matroska container and analyze with a plugin (e.g. VeraPDF and DPF Manager)
• ...

Versatile

Input can be from:

• Files (local/network)
• FTP/FTPS/SFTP
• HTTP/HTTPS
• Amazon S3

Versatile

Binaries are provided for:

• Windows
• Mac
  Homebrew users:"brew install mediaconch", that's all!
• Linux (Ubuntu, Debian, Fedora, OpenSUSE...)
  Since Ubuntu 16.04 and Debian Testing/9 users:
  "apt-get install mediaconch" or in Ubuntu Store, that's all!
  (it is in the official distros repository)
• Embedded devices? Doable
  (we tested it on a Raspberry Pi )
• Can be ported on other distros (BSD...)

Standardization

• Matroska is widely used but not (yet) standardized
• FFV1 is gaining increasing usage in preservation contexts but is not (yet) standardized

CELLAR: IETF workgroup

• Open standards group
• Goal to IETF-standardize Matroska/FFV1/FLAC
• A lot of progress, especially with Matroska/EBML specs
• https://datatracker.ietf.org/wg/cellar/charter/

FFV1 performance

• NOA tested on SD 8-bit content:
  • i7-2600 (4 cores+HT, 3.4-3.8 GHz)
  • 3-4x real time
  • 4-5x decoding speed increase compared to JP2k
• VIAA is testing on SD 10-bit content (FFmpeg 3.2):
  • E5-2698V3 (16 cores+HT, 2.3-3.6 GHz)
  • 0.7x real time/thread, 11-12x real time/all cores+HT
  • 3-4x decoding speed increase compared to JP2k
  • Better compression ratio by 8-10% compared to JP2k

FFV1 performance

• This is an average, results varies depending on the content of files
• From 0.4x to 2.4x (average 0.7x) real time/thread (encoding/decoding)
• From 0.7x to 16x (average 3.5x) the speed of JP2k (FFmpeg)
• Not convinced?
  • Test on your own files
  • MediaArea will provide test scripts
  • We can perform tests for you

Worldwide

• 2 project leaders

  • Jerôme Martinez (Digital Media Analysis Specialist, France)
  • Dave Rice (Archivist, USA)

• Presentations worldwide

  • IASA, France
  • FIAT/IFTA, Austria
  • FOSDEM, Belgium
  • AMIA, USA
  • Code4Lib, USA
  • JTS, Singapore
  • (3-6 October 2016) IPRES, Switzerland
  • (25-29 September 2016) IASA, USA

Matroska research corpus

• We analyze all Matroska files from archive.org
• Interface with some statistics of Matroska elements usage (e.g. files with CRC-32 elements...) https://mediaarea.net/MediaConchCorpus/

What's next?

• Continue to improve handling of huge collections
• Continue to improve user interface
• Support of embedded attachments
• Statistics
• Finish standardization of Matroska and FFV1
• More conformance tests
• More fixing cases

And after PREFORMA sponsorship?

It depends on you!

• This is open source
• Driven by user requests
• Everyone can develop or sponsor a development
• Potential features:
  • Support of tests for your prefered format
    (MOV? MXF? JP2k? WAV?)
  • Support of other checkers
    (BWF MetaEdit? QCTools?)
  • Integration in your workflow
  • ...

Example (Plugins)

Example (Plugins)

Example (Plugins)

Stay in touch

MediaArea: https://mediaarea.net, @MediaArea_net

MediaConch: https://mediaarea.net/MediaConch, @MediaConch

Jérôme Martinez: jerome@mediaarea.net

Slides: https://mediaarea.net/Events

License: CC BY