Open source tools
Some examples of open source projects
used (present, future) by archivists
Jérôme Martinez
MediaArea
FIAF, Prague, April 2018
The four freedoms
- The freedom to run the program as you wish, for any purpose
- The freedom to study how the program works, and change it so it does your computing as you wish
- The freedom to redistribute copies so you can help your neighbor
- The freedom to distribute copies of your modified versions to others
FFmpeg
Tool for manipulating A/V files
Very versatile
Lot of formats supported
Lot of possibilities to manipulate A/V files
Command line tool
ffmprovisr
FFmpeg has a steep learning curve
ffmprovisr helps users through the command generation process so that more people can reap the benefits of FFmpeg
Maintained by archivists like you
ffmprovisr
QCTools
Helps users analyze and understand their digitilzed video files through use of audiovisual analytics and filtering
Based on FFmpeg
Graphical interface or command line
https://www.bavc.org/preserve-media/preservation-tools/qctools
(Development snapshots on
https://MediaArea.net/QCTools)
QCTools
QCTools
Colors of digitilzed video are weird? Let's check...
MediaInfo
Convenient unified display of the most relevant technical and tag data for video and audio files.
Graphical interface or command line or software library
https://MediaArea.net/MediaInfo
MediaInfo
MediaInfoOnline
MediaConch
Implementation and policy checking on FFV1, Matroska, LPCM (and more)
Based on MediaInfo
Graphical interface or command line
https://MediaArea.net/MediaConch
MediaConch
Implementation and Policy reporter
MediaConch
Implementation report: |
Policy report: |
MediaConch
General information about your files
MediaConch
Inspect your files
MediaConch
Policy editor
MediaConch
Public policies
BWF MetaEdit
Embedding, validating, and exporting of metadata in Broadcast WAVE Format (BWF) files
Supports the FADGI Broadcast WAVE Metadata Embedding Guidelines
Graphical interface or command line
BWF MetaEdit
BWF MetaEdit
AVI MetaEdit
Embedding, validating, and exporting of metadata in AVI (Standard and OpenDML) files
Supports the U.S. National Archives Guidelines recommandations
Graphical interface or command line
Is similar to BWF MetaEdit? Right, code from BWF MetaEdit was reused, limiting development cost
AVI MetaEdit
MOV MetaEdit
Embedding and editing of metadata in MOV (Apple QuickTime) or MP4 (ISO/IEC 14496-14 a.k.a. MPEG-4 Part 14) files.
It is currently focused on Universal Ad ID metadata and Pixel Aspect Ratio edition
Sponsors were not interested in more. Could be expanded on request
Graphical interface or command line
MOV MetaEdit
vrecord
Tool to digitise analogue video
Make videotape digitization or transfer easier.
Blackmagic Design capture cards currently supported
vrecord
VirtualDub FFV1
VirtualDub is a Windows transcoding tool with a GUI
No FFV1 supported by current version
Developers abandonned the tool (no more development)
But it is open source
"Nobody wants to update the tool? Let's add FFV1 support ourselves"
VirtualDub FFV1
RAWcooked
Encodes RAW audio-visual data (DPX/TIFF) into a losslessly compressed file (no more unplayable TAR!)
Metadata accompanying the RAW data are preserved (reversiblility)
Sidecar files, like MD5, LUT or XML, are in container attachments
RAWcooked
- Final package is 1.5-3x (usually 2x) smaller than DPX/TIFF
- Checksum by "Cluster" (usually 1 second) at container level
- Checksum by "Slice" (you choose how many per frame) at video level
- Files are natively playable by lot of tools (FFmpeg, VLC...)
RAWcooked
- Storage
Save HDD/LTO space: either ~2x less cost for same redundacy or 2x more redundancy for ~ same cost - Transport
Encode, transport, decode; you save bandwidth (€... and transfer speed) without changing something else in your workflow (same files after revert to DPX/TIFF)
RAWcooked
In development
Stable release planned next month
~20 DPX flavors (RGB/RGBA * bitdepth * filled/packed * Big/Little endian) supported
Uses FFmpeg FFV1 encoder (internal encoder planned), uses internal FFV1 decoder
Transparent development, all at https://github.com/MediaArea/RAWcooked
RAWcooked
Developed by MediaArea
https://MediaArea.net
Main sponsorship by AV Preservation by reto.ch
https://reto.ch
With additional financial support from some other archives:
Projects have similar patterns
This is open source
One can peek code from another project
Lot of libraries are shared
A communuity (e.g. archives from different countries) can build an ecosystem
Funding
Driven by user requests
Most of tools were funded after a need is detected by users
Why paying for something you don't control? (is your current choice future proof?)
Everyone (you included) can develop or sponsor a development
You can fork if you think you have a better idea than others
Users have full control
Actors
Funding: European Commission, MoMA, National Endowment for the Humanities, the Knight Foundation, Library of Congress and FADGI, NARA (National Archives and Records Administration), CNA (National Audiovisual Centre of Luxembourg), Nasjonalbiblioteket (National Library of Norway), IFI (Irish Film Institute), Northwestern University Libraries and lot of small sponsoring from tens of other small to large entities
Actors
Management: MediaArea, AVP, Bay Area Video Coalition...
Development: MediaArea, lot of individual developers, and lot of archivists!
Funding, management, development: not always same people. You decide.
Stay in touch
MediaArea: https://MediaArea.net, @MediaArea_net
Jérôme Martinez: jerome@MediaArea.net
Slides: https://MediaArea.net/Events
License: CC BY