access

Sep

2016

Digital Video in Mixed-Content Archives

On a recent trip to one of Britain’s most significant community archives, I was lucky enough to watch a rare piece of digitised video footage from the late 1970s.

As the footage played it raised many questions in my mind: who shot it originally? What format was it originally created on? How was it edited? How was it distributed? What was the ‘life’ of the artefact after it ceased to actively circulate within communities of interest/ use? How and who digitised it?

As someone familiar with the grain of video images, I could make an educated guess about the format. I also made other assumptions about the video. I imagined there was a limited amount of tape available to capture the live events, for example, because a number of still images were used to sustain the rolling audio footage. This was unlikely to be an aesthetic decision given that the aim of the video was to document a historic event. I could be wrong about this, of course.

When I asked the archivist the questions flitting through my mind she had no answers. She knew who the donor of the digital copy was, but nothing about the file’s significant properties. Nor was this important information included in the artefact’s record.

This struck me as a hugely significant problem with the status of digitised material – and especially perhaps video – in mixed-content archives where the specificities of AV content are not accounted for.

Due to the haphazard and hand-to-mouth way mixed-content archives have acquired digital items, it seems more than likely this situation is the rule rather than the exception: acquired bit by bit (no pun intended), maintaining access is often privileged over preserving the context and context of the digitised video artefact.

As a researcher I was able to access the video footage, and this of course is better than nothing.

Yet I was viewing the item in an ahistoric black hole. It was profoundly decontextualised; an artefact extracted to its most barest of essences.

Standard instabilities

This is not in any way a criticism of the archive in question. In fact, this situation is wholly understandable given that digital video are examples of ‘media formats that exist in crisis.’

Video digitisation remains a complex and unstable area of digital preservation. It is, as we have written elsewhere on this blog, the final frontier of audiovisual archiving. This seems particularly true within the UK context where there is currently no systematic plan to digitise video collections, unlike film and audio.

The challenge with digital video preservation remains the bewildering number of potential codec/ wrapper combinations that can be used to preserve video content.

There are signs, however, that file-format stabilities are emerging. The No Time to Wait: Standardizing FFV1 & Matroska for Preservation symposium (Berlin, July 2016) brought together software developers and archivists who want to make the shared dream of an open source lossless video standard, fit for archival purpose, a reality.

It seems like the very best minds are working together to solve this problem, so Greatbear are quietly optimistic that a workable, open source standard for video digital preservation is in reach in the not too distant future.

Metadata

Yet as my experience in the archive makes clear, the challenge of video digitisation is not about file format alone.

There is a pressing need to think very carefully about the kind of metadata and other contextual material that need to be preserved within and alongside the digitised file.

Due to limited funding and dwindling technical capacity, there is likely to be only one opportunity to transfer material currently recorded on magnetic tape. This means that in 2016 there really can be no dress rehearsal for your video digitisation plans.

As Joshua Ranger strongly emphasises:

‘Digitization is preservation…For audiovisual materials. And this bears repeating over and over because the anti-digitization voice is much stronger and generally doesn’t include any nuance in regards to media type because the assumption is towards paper. When we speak about digitization for audio and video, we now are not speaking about simple online access. We are speaking about the continued viability, about the persistence and the existence of the media content.’

What information will future generations need to understand the digitised archive materials we produce?

An important point to reckon with here is that not all media are the same. The affordances of particular technologies, within specific historical contexts, have enabled new forms of community and communicative practice to emerge. Media are also disruptive (if not deterministic) – they influence how we see the world and what we can do.

On this blog, for example, Peter Sachs Collopy discussed how porta-pak technology enabled video artists and activists in the late 1960s/ early 1970s to document and re-play events quickly.

Such use of video is also evident in the 1975 documentary Les prostituées de Lyon parlent (The prostitutes of Lyon speak).

Les prostituées documents a wave of church occupations by feminist activists in France.

The film demonstrates how women used emergent videotape technology to transmit footage recorded within the church onto TV screens positioned outside. Here videotape technology, and in particular its capacity to broadcast uni-directional messages, was used to protect and project the integrity of the group’s political statements. Video, in this sense, was an important tool that enabled the women – many of whom were prostitutes and therefore without a voice in French society – to ‘speak’.

Peter’s interview and Les prostituées de Lyon parlent are specific examples of how AV formats are concretely embedded within a social-historical and technical context. The signal captured – when reduced to bit stream alone – is simply not an adequate archival source. Without sufficient context too much historical substance is shed.

In this respect I disagree with Ranger’s claim that ‘all that really may be needed moving ahead [for videotape digitisation] is a note in the record for the new digital preservation master that documents the source.’ To really preserve the material, the metadata record needs to be rich enough for a future researcher to understand how a format was used, and what it enabled users to do.

‘Rich enough’ will always be down to subjective judgement, but such judgements can be usefully informed by understanding what makes AV archive material unique, especially within the context of mixed-content archives.

Moving Forward

So, to think about this practically. How could the archive item I discuss at the beginning of the article be contextualised in a way that was useful to me, as a researcher?

At the most basic level the description would need to include:

The format it was recorded on, including details of tape stock and machine used to record material
When it was digitised
Who digitised it (an individual, an institution)

In an ideal world the metadata would include:

Images of the original artefact – particularly important if digital version is now the only remaining copy
Storage history (of original and copy)
Accompanying information (e.g., production sheets, distribution history – anything that can illuminate the ‘life’ of artefact, how it was used)

This information could be embedded in the container file or be stored in associated metadata records.

These suggestions may seem obvious, but it is surprising the extent to which they are overlooked, especially when the most pressing concern during digitisation is access alone.

In every other area of archival life, preserving the context of item is deemed important. The difference with AV material is that the context of use is often complex, and in the case of video, is always changing.

As stressed earlier: in 2016 and beyond you will probably only get one chance to transfer collections stored on magnetic tape, so it is important to integrate rich descriptions as part of the transfer.

Capturing the content alone is not sufficient to preserve the integrity of the video artefact. Creating a richer metadata record will take more planning and time, but it will definitely be worth it, especially if we try to imagine how future researchers might want to view and understand the material.

Jul

2015

British Stand Up Comedy Archive’s audiovisual collections

Greatbear have recently worked with the British Stand Up Comedy Archive (BSUCA) to reformat a number of Digital Audio Tapes (DATs) and U-Matic video tapes from their collection.

Established in 2013 and based at the University of Kent’s Special Collections, the BSUCA aims ‘to celebrate, preserve, and provide access to the archives and records of British stand-up comedy and stand-up comedians.’

In 2014 the BSUCA became one of the University of Kent’s 50th anniversary ‘Beacon Projects‘.

Beacon Project funding will support work to ‘catalogue, preserve, digitise, and make accessible the existing collections, and identify new relevant collections.’

They will also hold a number of events which engage comedians ‘in conversation’ about their archives, excerpts from these events are documented here.

We are honoured that project archivist Elspeth Millar took time out of her busy archiving schedule to tell us a bit more about the BSUCA.

She told us:

‘I’m really enjoying the variety of material that I get to work on, including printed material (posters, flyers, letters, notebooks), audio-visual material on a range of formats (audio cassettes, VHS, DAT, MiniDisc, U-matic), and also born-digital records held on obsolete formats (such as 3.5” floppy disks).

In addition the content of the material is, of course, really interesting, and I feel that I am learning a lot from our collections, including about the history of stand-up comedy (from the alternative cabaret movement, to alternative comedy, to the comedy ‘industry’ today) but also political and social topics (for example Mark Thomas’ collection includes a lot of material on the arms trade and large corporations). We are also holding events with some fantastic comedians (Richard Herring, Stewart Lee, Mark Thomas, and at the Edinburgh Festival Fringe, Jo Brand, Alexei Sayle, Susan Calman) so it is wonderful to hear comedians themselves reflecting on their work and on material that they have deposited with the archive.’

You can keep up to date with the latest news from the BSUCA archive on twitter and view images from their collections on flickr.

Read on for more from Elspeth. Her answers cover issues such as selection and appraisal decisions, metadata and dissemination plans for the BSUCA.

They also provide useful insight into the digital preservation tools BSUCA use to manage their digitised and born-digital assets.

Once again, massive thanks to her for responding to our questions and best of luck to BSUCA in the future.

BSCUA Responses to Greatbear Questions

1. What motivated you to get the tapes you sent to us re-formatted now? i.e., what kinds of selection and appraisal processes were behind the decision?

The British Stand-Up Comedy Archive has AV material on a number of audio and moving image formats, magnetic and optical, including audio compact cassettes, MiniDiscs, DATs (Digital Audio Tapes), VHS, DVCams, Audio CD and U-matic tapes. None of these formats are suitable for archival storage and all material will need to be digitised or transferred from their original carrier to digital files. We can carry out the digitisation (or digital transfer) of some audio in-house and we have started our project by transferring material originally captured or stored on MiniDiscs, Audio CDs, and audio compact cassettes¹. After assessing all the formats we currently have it was decided to outsource the digitisation of DATs and U-matic tapes. Both of these are priority formats for transfer from a preservation perspective² and after some research I learnt that DATs can be problematic to transfer due to ‘DAT compatibility’ and dropout problems³. In addition, we have neither a DAT machine or U-matic machine already within Special Collections or within the University, and with the number of recordings on these formats currently limited, it was felt that it would not make sense to purchase already obsolete equipment, which would then need to be professionally maintained.

The other important reason for transferring the tapes of course was for accessibility, so that we can make the recordings accessible to researchers. In addition, our funding is currently only for one year⁴, so it is vital to ensure that audio-visual material on obsolete formats are transferred during this first phase of the project.

2. Can you tell us how metadata helps you to describe, preserve and aid discovery of the Stand Up Comedy archive.

Providing information about our audiovisual items (and resulting digital items) is incredibly important from both an access and preservation perspective. Metadata about analogue items (and subsequent digital files) and born-digital files will be included in the cataloguing collections management system used by the British Stand-Up Comedy Archive (which is part of the University of Kent’s Special Collections & Archives). The catalogue records will include descriptive metadata and administrative metadata. Metadata which comes under the ‘descriptive metadata’ heading describes the item/file and includes a summary of the contents of the recording, all of which helps to make recordings discoverable for researchers. This metadata is also vital from a preservation perspective as it allows archivists to retrieve and identify files. Metadata which comes under the ‘administrative metadata’ heading provides information to help manage the file(s)/recordings, and includes information related to Intellectual Property Rights (including copyright) and preservation information such as the file format and the digitisation/digital transfer. Researchers will be interested in some of these issues (e.g. copyright, as this determines how archived recordings can be used) but from a digital preservation perspective this metadata is extremely important as it records information about the format of the digital file, information about the original carrier, as well as fixity information, to measure whether the file has changed over time.

This metadata will be recorded in our catalogue and will be searchable via the University of Kent’s website and in the future some archive aggregators. However, we are also experimenting with different processes and tools for embedding metadata in files, and researching different metadata standards for this. The benefits of embedding some metadata within the file include the removal of the risk of losing the link between the metadata and the digital file that it is describing. In addition, metadata embedded in born-digital master and digitised master files can also be transferred to ‘access’ copies (generated at a lower specification/resolution) which will also assist in user accessibility. Embedded metadata has its limitations and it is not that flexible, which is why we are using a dual approach of embedding some metadata, but also keeping this information externally in our catalogue.

3. How do you manage, and plan to manage digital audio and audio visual materials in the Stand Up Archive? What digital preservation tools do you use?

The first process in managing digital AV materials in the BSUCA is to think about the file formats that we will use for long-term preservation and access. For audio material we are digitising as LPCM (Linear Pulse Code Modulation) in a Wave format (.wav) wrapper. The addition of embedding metadata into these wave files extends the file to become BWF .wav files, the standard recommended by the International Association of Audiovisual Archives (IASA).⁵

Deciding upon a file format for digitising moving image has been trickier, as the Greatbear team have already written about on this blog; we hope to get underway with digitisation of VHS in September and we are looking at using the FFv1 codec (an open-source lossless compressed codec) wrapped as either AVI or Matroska (MKV).

We are also experimenting with a number of digital preservation tools; one directory that has proved great for discovering such tools is the COPTR wiki (Community Owned digital Preservation Tool Registry), a really useful collated list of various digital preservation tools . One aspect of our digital preservation planning is the creation of checksums as early in the lifecycle of the digital file as possible. We are using a tool called Blackbush, which is a checksum tool⁶ which generates MD5 hash files which was developed for the British Library’s Sound Archive. To embed metadata into .wav files we are using the BWF MetaEdit tool, a free open-source tool developed by AV Preserve and the Federal Agencies Digitization Guidelines Initiative. When our archival master is a compressed format (such as an mp3 on a data or audio CD which has been deposited), we are using tools such as Adobe Bridge to embed metadata in the ID3 format (or Adobe Audition’s metadata tools as we transfer audio). The advantage of BWF MetaEdit for wav files is that it is a free open-source tool, which also has other functions such as batch editing (we can edit multiple wav files at once) and batch import and export functions, which will be useful for when we catalogue this material to item level.

Other tools that we have found useful include DROID (Digital Record Object Identification), developed by The National Archives, and, for other digital material we are using forensic imaging tools such as FTK Imager and ImDisk to mount virtual images of disk images.

4. How do you think the material will be used by future researchers? As a Stand Up Archive I imagine you get a lot of requests for material from broadcasters. How do you manage requests and expectations from different user communities?

The British Stand-Up Comedy Archive is still in its infancy; although we have had material since 2013, it has only been since the beginning of this year that we have been funded to digitise and preserve the material already deposited, start to catalogue it, make it accessible, and publicise what we have and what we are aiming to do.

But two of our core purposes are to ensure access (that these archives are universally discoverable and accessible), and to ensure that the archives are used, and used in a variety of ways (popular culture, academic research, teaching, journalism, general enjoyment). Our main user group at the moment is actually students studying stand-up and popular performance at the University of Kent (at BA and MA level) who have used AV material as part of their course, and we also have a number volunteering with the project, doing summaries of recorded interviews and stand-up performances.

Notes

[1] We have purchased an audio cassette deck (Denon DN-790R) and are using a MiniDisc deck on loan from colleagues within the University, and have also purchased an external audio capture card/A-D converter.

[2] https://psap.library.illinois.edu/format-id-guide/audiotape#dat and https://psap.library.illinois.edu/format-id-guide/videotape#umatic.

[3] https://siarchives.si.edu/sites/default/files/pdfs/digitalAudioTapesPreservation2010_0.pdf (page 5-8) and http://thegreatbear.co.uk/audio-tape/transferring-dats-to-digital-files/.

[4] The British Stand-Up Comedy Archive is part of the University of Kent’s Special Collections and Archives, but it currently has specific funding for one year (as a Beacon Project) to digitise and make accessible its current holdings; more about the Beacon projects can be found at http://www.kent.ac.uk/beacon/about.html.

[5] Guidelines on the Production and Preservation of Digital Audio Objects, IASA-TC 04, 2.8.2

[6] A checksum is ‘an algorithmically-computed numeric value for a file or a set of files used to validate the state and content of the file for the purpose of detecting accidental errors that may have been introduced during its transmission or storage. The integrity of the data can be checked at any later time by recomputing the checksum and comparing it with the stored one. If the checksums match, the data was almost certainly not altered’. National Digital Stewardship Alliance Glossary, http://www.digitalpreservation.gov/ndsa/ndsa-glossary.html.

Feb

2015

Digitising small audiovisual collections: making decisions and taking action

Deciding when to digitise your magnetic tape collections can be daunting.

The Presto Centre, an advocacy organisation working to help ‘keep audiovisual content alive,’ have a graphic on their website which asks: ‘how digital are our members?’

They chart the different stages of ‘uncertainty,’ ‘awakening’, ‘enlightenment’, ‘wisdom’ and ‘certainty’ that organisations move through as they appraise their collections and decide when to re-format to digital files.

Similarly, the folks at AV Preserve offer their opinion on the ‘Cost of Inaction‘ (COI), arguing that ‘incorporating the COI model and analyses into the decision making process around digitization of legacy physical audiovisual media helps organizations understand the implications and make well-informed decisions.’

They have even developed a COI calculator tool that organisations can use to analyse their collections. Their message is clear: ‘the cost of digitization may be great, but the cost of inaction may be greater.’

Digitising small-medium audiovisual collections

For small to medium size archives, digitising collections may provoke worries about a lack of specialist support or technical infrastructure. It may be felt that resources could be better used elsewhere in the organisation. Yet as we, and many other people working with audiovisual archives often stress, the decision to transfer material stored on magnetic tape has to be made sooner or later. With smaller archives, where funding is limited, the question of ‘later’ is not really a practical option.

Furthermore, the financial cost of re-formatting audiovisual archives is likely to increase significantly in the next five-ten years; machine obsolescence will become an aggravated problem and it is likely to take longer to restore tapes prior to transfer if the condition of carriers has dramatically deteriorated. The question has to be asked: can you afford not to take action now?

If this describes your situation, you might want to hear about other small to medium sized archives facing similar problems. We asked one of our customers who recently sent in a comparatively small collection of magnetic tapes to share their experience of deciding to take the digital plunge.

We are extremely grateful for Annaig from the Medical Mission Sisters for answering the questions below. We hope that it will be useful for other archives with similar issues.

1. First off, please tell us a little bit about the Medical Mission Sisters Archive, what kind of materials are in the collection?

The Medical Mission Sisters General Archives include the central archives of the congregation. They gather all the documents relating to the foundation and history of the congregation and also documents relating to the life of the foundress, Anna Dengel. The documents are mainly paper but there is a good collection of photographs, slides, films and audio documents. Some born digital documents are starting to enter the archives but they are still few.

2. As an archive with a modest collection of magnetic tapes, why did you decide to get the materials digitised now? Was it a question of resources, preservation concerns, access request (or a mixture of all these things!)

The main reason was accessibility. The documents on video tapes or audio tapes were the only usable ones because we still had machines to read them but all the older ones, or those with specific formats, where lost to the archives as there was no way to read them and know what was really on the tapes. Plus the Medical Mission Sisters is a congregation where Sisters are spread out on 5 continents and most of the time readers don’t come to the archives but send me queries by emails where I have to respond with scanned documents or digital files. Plus it was obvious that some of the tapes were degrading as that we’d better have the digitisation sooner than later if we wanted to still be able to read what was on them. Space and preservation was another issue. With a small collection but varied in formats, I had no resources to properly preserve every tape and some of the older formats had huge boxes and were consuming a lot of space on the shelves. Now, we have a reasonably sized collection of CDs and DVDs, which is easy to store in good conditions and is accessible everywhere as we can read them on computer here and I can send them to readers via email.

3. Digital preservation is a notoriously complex, and rapidly evolving field. As a small archive, how do you plan to manage your digital assets in the long term? What kinds of support, services and systems are your drawing on to design a system which is robust and resilient?

At the moment the digital collection is so small that it cannot justify any support service or system. So I have to build up my own home made system. I am using the archives management software (CALM) to enter data relating to the conservation of the CDs or DVDs, dates of creation, dates to check them and I plan to have regular checks on them and migrations or copies made when it will prove necessary.

4. Aside from the preservation issue, what are your plans to use the digitised material that Greatbear recently transferred?

It all depends on the content of the tapes. But I’ve already spotted a few documents of interest, and I haven’t been through everything yet. My main concern now is to make the documents known and used for their content. I was already able to deliver a file to one of the Sisters who was working on a person related to the foundation of the congregation, the most important document on her was an audio file that I had just received from Greatbear, I was able to send it to her. The document would have been unusable a few weeks before. I’ve come across small treasures, like a film, probably made by the foundress herself, which nobody was aware of. The Sisters are celebrating this year the 90th anniversary of their foundation. I plan to use as many audio or video documents as I can to support the events the archives are going to be involved into.

***

What is illuminating about Annaig’s answers is that her archive has no high tech plan in place to manage the collection – her solutions for managing the material very much draw on non-digital information management practices.

The main issues driving the decision to migrate the materials are fairly common to all archives: limited storage space and accessibility for the user-community.

What lesson can be learnt from this? Largely, that if you are trained as an archivist, you are likely to already have the skills you need to manage your digital collection.

So don’t let the more bewildering aspects of digital preservation put you off. But do take note of the changing conditions for playing back and accessing material stored on magnetic tape. There will come a time when it will be too costly to preserve recordings on a wide variety of formats – many of such formats we can help you with today.

If you want to discuss how Greatbear can help you re-format your audiovisual collections, get in touch and we can explore the options.

If you are a small-medium size archive and want to share your experiences of deciding to digitise, please do so in the comment box below.

Feb

2014

Open Source Solutions for Digital Preservation

In a technological world that is rapidly changing how can digital information remain accessible?

One answer to this question lies in the use of open source technologies. As a digital preservation strategy it makes little sense to use codecs owned by Mac or Windows to save data in the long term. Propriety software essentially operate like closed systems and risk compromising access to data in years to come.

It is vital, therefore, that the digitisation work we do at Great Bear is done within the wider context of digital preservation. This means making informed decisions about the hardware and software we use to migrate your tape-based media into digital formats. We use a mixture of propriety and open source software, simply because it makes our a bit life easier. Customers also ask us to deliver their files in propriety formats. For example, Apple pro res is a really popular codec that doesn’t take up a lot of data space so our customers often request this, and of course we are happy to provide it.

Using open systems definitely has benefits. The flexibility of Linux, for example, enables us to customise our digitisation system according to what we need to do. As with the rest of our work, we are keen to find ways to keep using old technologies if they work well, rather than simply throwing things away when shiny new devices come on the market. There is the misconception that to ingest vast amounts of audio data you need the latest hardware. All you need in fact is a big hard drive, flexible, yet reliable, software and an operating system that doesn’t crash so it can be left to ingest for 8 hours or more. Simple! Examples of open source software we use is the sound processing programme SoX. This saves us a lot of time because we are able to write scripts for the programme that can be used to batch process audio data according to project specifications.

Openness in the digital preservation world

Within the wider digital preservation world open source technologies are also used widely. From digital preservation tools developed by projects such as SCAPE and the Open Planets Foundation, there are plenty of software resources available for individuals and organisations who need to manage their digital assets. It would be naïve, however, to assume that the practice of openness here, and in other realms of the information economy, are born from the same techno-utopian impulse that propelled the open software movement from the 1970s onwards. The SCAPE website makes it clear that the development of open source information preservation tools are ‘the best approach given the substantial public investment made at the European and national levels, and because it is the most effective way to encourage commercial growth.’

What would make projects like SCAPE and Open Planets even better is if they thought about ways to engage non-specialist users who may be curious about digital preservation tools but have little experience of navigating complex software. The tools may well be open, but the knowledge of how to use them are not.

Openness, as a means of widening access to technical skills and knowledge, is the impulse behind the AV Artifact Atlas (AVAA), an initiative developed in conjunction with the community media archive project Bay Area Video Coalition. In a recent interview on the Library of Congress’ Digital Preservation Blog, Hannah Frost, Digital Library Services Manager at Stanford Libraries and Manager, Stanford Media Preservation Lab explains the idea behind the AVAA.

‘The problem is most archivists, curators and conservators involved in media reformatting are ill-equipped to detect artifacts, or further still to understand their cause and ensure a high quality job. They typically don’t have deep training or practical experience working with legacy media. After all, why should we? This knowledge is by and large the expertise of video and audio engineers and is increasingly rare as the analogue generation ages, retires and passes on. Over the years, engineers sometimes have used different words or imprecise language to describe the same thing, making the technical terminology even more intimidating or inaccessible to the uninitiated. We need a way capture and codify this information into something broadly useful. Preserving archival audiovisual media is a major challenge facing libraries, archives and museums today and it will challenge us for some time. We need all the legs up we can get.’

The promise of openness can be a fraught terrain. In some respects we are caught between a hyper-networked reality, where ideas, information and tools are shared openly at a lightning pace. There is the expectation that we can have whatever we want, when we want it, which is usually now. On the other side of openness are questions of ownership and regulation – who controls information, and to what ends?

Perhaps the emphasis placed on the value of information within this context will ultimately benefit digital archives, because there will be significant investment, as there already has been, in the development of open resources that will help to take care of digital information in the long term.

Jan

2014

Digital Preservation – Establishing Standards and Challenges for 2014

2014 will no doubt present a year of new challenges for those involved in digital preservation. A key issue remains the sustainability of digitisation practices within a world yet to establish firm standards and guidelines. Creating lasting procedures capable of working across varied and international institutions would bring some much needed stability to a profession often characterized by permanent change and innovation.

In 1969 The EIAJ-1 video tape was developed by the Electronic Industries Association of Japan. It was the first standardized format for industrial/non-broadcast video tape recording. Once implemented it enabled video tapes to be played on machines made by different manufacturers and it helped to make video use cheaper and more widespread, particularly within a domestic context.

The introduction of standards in the digitisation world would of course have very little impact on the widespread use of digital technologies which are, in the west, largely ubiquitous. It would however make the business of digital preservation economically more efficient, simply because organisations would not be constantly adapting to change. For example, think of the costs involved in keeping up with rapid waves of technological transformation: updating equipment, migrating data and ensuring file integrity and operability are maintained are a few costly and time consuming examples of what this would entail.

Although increasingly sophisticated digital forensic technology can help to manage some of these processes, highly trained (real life!) people will still be needed to oversee any large-scale preservation project. Within such a context resource allocation will always have to account for these processes of adaptation. It has to be asked then: could this money, time and energy be practically harnessed in other, more efficient ways? The costs of non-standardisation becomes ever more pressing when we consider the amount of the digital data preserved by large institutions such as the British Library, whose digital collection is estimated to amass up to 5 petabytes (5000 terabytes) by 2020. This is not a simple case of updating your iphone to the next model, but an extremely complex and risky venture where the stakes are high. Do we really want to jeopardise rich forms cultural heritage in the name of technological progress?

The US-based National Digital Stewardship Alliance (NDSA) National Agenda for Digital Stewardship 2014 echoes such a sentiment. They argue that ‘the need for integration, interoperability, portability, and related standards and protocols stands out as a theme across all of these areas of infrastructure development’ (3). The executive summary also stresses the negative impact rapid technological change can create, and the need to ‘coordinate to develop comprehensive coverage on critical standards bodies, and promote systematic community monitoring of technology changes relevant to digital preservation.’ (2)

File Format Action Plans

One step on the way to more secure standards is the establishment of File Format Action Plans, a practice which is being increasingly recommended by US institutions. The idea behind developing a file format action plan is to create a directory of file types that are in regular use by people in their day to day lives and by institutions. Getting it all down on paper can help us track what may be described as the implicit user-standards of digital culture. This is the basic idea behind Parsimonious Preservation, discussed on the blog last year: that through observing trends in file use we may come to the conclusion that the best preservation policy is to leave data well alone since in practice files don’t seem to change that much, rather than risk the integrity of information via constant intervention.

As Lee Nilsson, who is currently working as a National Digital Stewardship Resident at the US Library of Congress writes, ‘specific file format action plans are not very common’, and when created are often subject to constant revision. Nevertheless he argues that devising action plans can ‘be more than just an “analysis of risk.” It could contain actionable information about software and formats which could be a major resource for the busy data manager.’

Other Preservation Challenges

What are the other main challenges facing ‘digital stewards’ in 2014? In a world of exponential information growth, making decisions about what we keep and what we don’t becomes ever more pressing. When whole collections cannot be preserved digital curators are increasingly called upon to select material deemed representative and relevant. How is it possible to know now what material needs to be preserve for posterity? What values inform our decision making?

To take an example from our work at Great Bear: we often receive tapes from artists who have achieved little or no commercial success in their life times, but whose work is often of great quality and can tell us volumes about a particular community or musical style. How does such work stand up against commercially successful recordings? Which one is more valuable? The music that millions of people bought and enjoyed or the music that no one has ever heard?

Ultimately these questions will come to occupy a central concern for digital stewards of audio data, particularly with the explosion of born-digital music cultures which have enabled communities of informal and often non-commercial music makers to proliferate. How is it possible to know in advance what material will be valuable for people 20, 50 or 100 years from now? These are very difficult, if not impossible questions for large institutions to grapple with, and take responsibility for. Which is why, as members of a digital information management society, it is necessary to empower ourselves with relevant information so we can make considered decisions about our own personal archives.

A final point to stress is that among the ‘areas of concern’ for digital preservation cited by the NDSA, moving image and recorded sound figure highly, alongside other born-digital content such as electronic records, web and social media. Magnetic tape collections remain high risk and it is highly recommended that you migrate this content to a digital format as soon as possible. While digitisation certainly creates many problems as detailed above, magnetic tape is also threatened by physical deterioration and its own obsolescence challenges, in particular finding working machines to play back tape on. The simple truth is, if you want to access material in your tape collections it needs now to be stored in a resilient digital format. We can help, and offer other advice relating to digital information management, so don’t hesitate to get in touch.

Oct

2013

Digital Preservation and Copyright

Most customers who send us tape to digitise own the copyright of their recording: it is material they have created themselves, be it music, spoken word or film.

Occasionally customers are not so sure if they own the full copyright to their recordings. This is because a single piece of work can have multiple copyright holders.

For example, films and songs can have many different contributors, such as the person who made the recording, the songwriter and performers. There are performing rights royalties which are paid to a songwriter, composer or publisher whenever their music is played or performed in any public space or place; mechanical rights royalties which are paid to the songwriter, composer or publisher when music is reproduced as a physical product or for broadcast or online, and performers rights royalties which are paid to the people performing on the record. It can seem like a bit of a minefield, and you do have to be really careful, particularly if want to re-publish the works in a commercial context.

A collection of tapes that include original recordings made by the customer

The simple truth is, if you do not have full permission of all copyright holders, you would break the law if you digitised a tape and re-published it commercially.

Copyright, Intellectual Property and Digital Preservation is a tricky area to negotiate. Currently ‘there is still no exception in UK law for preservation copying. For materials which are still in copyright, permissions should be sought from copyright holders prior to any copying being done. This area is under consideration though with museums, libraries and archives lobbying for change’ (Jisc Digital Media).

What this means basically is that archives, libraries and museums are effectively restricted in how much material they can legally preserve in digital form. Andrew Charlesworth explains in a very useful report for the Digital Preservation Coalition on ‘Intellectual Property Rights for Digital Preservation’ (2012)

‘In “Chapter III: Acts permitted in relation to copyright works”, the C opyright D esigns and P atents A ct 1988 provides for a series of permissible activities that would otherwise be barred for breach of a rights holder’s exclusive rights. These include the “fair dealing provisions” which, for example, state that making transient copies is an integral and essential part of certain technological processes (s.28), and using all or part of a copyright work for non-commercial research or private study (s.29), criticism or review, or reporting current events (s.30), do not constitute infringements’ (11).

Clearly copyright law as it stands places immense restrictions in a digital environment where copying and sharing all kinds of things is pretty much the norm. What are the arguments then for changing copyright laws? In Imagine there is no copyright and cultural conglomerates too by Joost Smiers and Marieke Van Schinjdel, published by the Institute of Network Cultures‘ Theory on Demand series, they argue that removing copyright from cultural products will ensure that ‘our past and present heritage of cultural expression, our public domain of artistic creativity and knowledge will no longer be privatised’ (6).

Making cultural heritage publicly available is an argument for transforming current copyright laws across the range of political positions. While Smiers and Van Schinjdel interpret privatisation embedded in copyright law as linked to commercial power, the implicit argument in the DPC report is that opening up current restrictions can only be good for business. In this particular domain we see how the value of archival information has shifted in the digital landscape, so that it is increasingly seen as a resource through which money can be made.

A transformation of copyright laws would not necessarily lead to a weakening of commercial interests as Smiers and Van Schinjdel speculate, but would most probably enable the re-use of information across a range of profit and profit-making initiatives. Charlesworth insists we are ‘clinging to copyright practices that reflect outdated business models rather than attempting to establish new practices to address the prevailing mixed analogue/digital environment’ (7).

The digital information revolution has required all sectors of society to change how they relate to, use, record, save and consume information. While we have all become, to a lesser or more degree, record keepers, this brief survey of copyright law may help us appreciate the challenges professional archivists face in negotiating this complex area. After all, ‘life would be much simpler for archivists if the law relating to the preservation of copyright works in general, and digital works in particular, was both clarified and, where necessary, extended to permit more robust strategies for collection, preservation and reuse of copyright works’ (5).

Oct

2013

Parsimonious Preservation – (another) different approach to digital information management

We have been featuring various theories about digital information management on this blog in order to highlight some of the debates involved in this complex and evolving field.

To offer a different perspective to those that we have focused on so far, take a moment to consider the principles of Parsimonious Preservation that has been developed by the National Archives, and in particular advocated by Tim Gollins who is Head of Preservation at the Institution.

In some senses the National Archives seem to be bucking the trend of panic, hysteria and (sometimes) confusion that can be found in other literature relating to digital information management. The advice given in the report, ‘Putting Parsimonious Preservation into Practice‘, is very much advocating a hands-off, rather than hands-on approach, which many other institutions, including the British Library, recommend.

The principle that digital information requires continual interference and management during its life cycle is rejected wholesale by the principles of parsimonious preservation, which instead argues that minimal intervention is preferable because this entails ‘minimal alteration, which brings the benefits of maximum integrity and authenticity’ of the digital data object.

As detailed in our previous posts, cycles of coding and encoding pose a very real threat to digital data. This is because it can change the structure of the files, and risk in the long run compromising the quality of the data object.

Minimal intervention in practice seems here like a good idea – if you leave something alone in a safe place, rather than continually move it from pillar to post, it is less likely to suffer from everyday wear and tear. With digital data however, the problem of obsolescence is the main factor that prevents a hands-off approach. This too is downplayed by the National Archives report, which suggests that obsolescence is something that, although undeniably a threat to digital information, it is not as a big a worry as it is often presented.

Gollins uses over ten years of experience at the National Archives, as well as the research conducted by David Rosenthal, to offer a different approach to obsolescence that takes note of the ‘common formats’ that have been used worldwide (such as PDF, .xls and .doc). The report therefore concludes ‘that without any action from even a national institution the data in these formats will be accessible for another 10 years at least.’

10 years may seem like a short period of time, but this is the timescale cited as practical and realistic for the management of digital data. Gollins writes:

‘While the overall aim may be (or in our case must be) for ―permanent preservation […] the best we can do in our (or any) generation is to take a stewardship role. This role focuses on ensuring the survival of material for the next generation – in the digital context the next generation of systems. We should also remember that in the digital context the next generation may only be 5 to10 years away!’

It is worth mentioning here that the Parsimonious Preservation report only includes references to file extensions that relate to image files, rather than sound or moving images, so it would be a mistake to assume that the principle of minimal intervention can be equally applied to these kinds of digital data objects. Furthermore, .doc files used in Microsoft Office are not always consistent over time – have you ever tried to open a word file from 1998 on an Office package from 2008? You might have a few problems….this is not to say that Gollins doesn’t know his stuff, he clearly must do to be Head of Preservation at the National Archives! It is just this ‘hands-off, don’t worry about it’ approach seems odd in relation to the other literature about digital information management available from reputable sources like The British Library and the Digital Preservation Coalition. Perhaps there is a middle ground to be struck between active intervention and leaving things alone, but it isn’t suggested here!

For Gollins, ‘the failure to capture digital material is the biggest single risk to its preservation,’ far greater than obsolescence. He goes on to state that ‘this is so much a matter of common sense that it can be overlooked; we can only preserve and process what is captured!’ Another issue here is the quality of the capture – it is far easier to preserve good quality files if they are captured at appropriate bit rates and resolution. In other words, there is no point making low resolution copies because they are less likely to survive the rapid successions of digital generations. As Gollins writes in a different article exploring the same theme, ‘some will argue that there is little point in preservation without access; I would argue that there is little point in access without preservation.’

This has been bit of a whirlwind tour through a very interesting and thought provoking report that explains how a large memory institution has put into practice a very different kind of digital preservation strategy. As Gollins concludes:

‘In all of the above discussion readers familiar with digital preservation literature will perhaps be surprised not to see any mention or discussion of “Migration” vs. “Emulation” or indeed of ―“Significant Properties”. This is perhaps one of the greatest benefits we have derived from adopting our parsimonious approach – no such capability is needed! We do not expect that any data we have or will receive in the foreseeable future (5 to 10 years) will require either action during the life of the system we are building.’

Whether or not such an approach is naïve, neglectful or very wise, only time will tell.

Jun

2013

Digital Preservation – Planning for the Long Term

There are plenty of reflections on the Great Bear tape blog about the fragility of digital data, and the need to think about digitisation as part of a wider process of data migration your information will need to make in its lifetime.

We have also explored how fast moving technological change can sometimes compromise our capacity to construct long term strategies for the survival of digital data.

This why it is so important that organisations such as the Digital Preservation Coalition, founded in February 2002, articulate a vision that aims to make ‘digital memory accessible tomorrow.’ Their website goes on to say:

Our generation has invested as never before in digital resources and we’ve done so because of the opportunity they bring. They have grown in volume, complexity and importance to the point that our children are baffled by the inefficiencies of the analogue age. Pervasive, fluid and fragile: digital data is a defining feature of our age. Industry, commerce, government, law, research, health, social care, education, the creative industries, the heritage sector and private life depend on digital materials to satisfy ubiquitous information needs and expectations. Digital preservation is an issue which all organisations, particularly in the knowledge sector, will need to address sooner or later.

As providers of a digitisation service it is important for us to understand digitisation in line with the ideas articulated above. This means creating high quality, uncompressed files that will make it as easy as possible for data migrations to happen in the future should they need to.

Organisations such as the Digital Preservation Coalition are providing sensible advice and creating forums for learning and debate about the problems and possibilities of digital preservation.

These are two things that are needed as we come to navigate an information environment heavily populated by ‘pervasive, fluid and fragile’ digital data.

Nov

2011

Convert, Join, re encode AVCHD .MTS files in Ubuntu Linux

One of our audio and video archive customers has a large collection of AVCHD video files that are stored in 1.9GB ‘chunks’ as xxxxx.MTS files. All these files are of 60 minute and longer duration and must be joined, deinterlaced, re encoded to a suitable size and bitrate then uploaded for online access.

This is quite a task in computer time and file handling. These small domestic cameras produce good HD movies for a low cost but the compression to achieve this is very high and does not give you a file that is easily edited. The .MTS files are MPEG transport stream containers for H264 encoded video.

There are some proprietary solutions for MacOS X and Windows that will repackage the .MTS files into .MOV Quicktime containers that can be accessed by MacOS X or re-encoded to a less compressed format for editing with Final Cut Pro or Premiere. We didn’t need this though, just a reliable and quick open source workflow.

The first and most important issue is to rejoin the camera split files.
These cameras use FAT32 file systems which cannot handle individual files larger than 2GB so they split the .MTS video file into chunks. As each chunk in a continuous sequence references the other chunks these must be joined in the correct order. This is easily achieved with the cat command.
The rejoined .MTS files can now be reencoded to a more manageable size using open source software such as Handbrake. We also needed to deinterlace our footage as it was shot interlaced and it would be accessed on progressive displays. This will increase the encoding time but without it any movement will look odd with visible artifacts.
Finding the ‘sweet spot’ for encoding can be time consuming but in this case was important as projected text needed to be legible but the file sizes kept manageable for reasonable upload times!

access