Optimising Production Storage using Archiware and Limecraft : Limecraft

Webinar: ‘Optimising storage for high-end documentary production using Archiware and Limecraft.’

Capitalising on their relentless focus to eradicate manual work, and using a smart configuration of cold storage managed by Archiware and Scale Logic near-line storage, Hotel Hungaria managed to bring a layered production storage architecture to life which is at the same time accessible, very secure and cost-effective.

In thit webinar, Olivier Struyven (CTO and head of postproduction at Hotel Hungaria), Maarten Verwaest (CEO at Limecraft) and David Fox (Archiware) shared first-hand best practices on how to implement a hybrid storage architecture that scales at marginal cost and which is therefore suitable for high volume and high quality production .

Using Limecraft Cloud Connector

00:02:33 – Limecraft is a Cloud-native Media Asset Management solution used by producers of all types of content. It is particularly suited for at high volume/high quality productions, coping with challenges of scalability, security, environmental footprint.

The key to the solution lies in setting up a hybrid storage model by integrating local storage using Limecraft ‘Cloud Connector’. Cloud Connector allows to swiftly archive or stage collections from LTO tape (managed by Archiware) to online storage (Scale Logic), which is a perfect balance of cost and performance allowing the producer to scale to larger volumes at marginal cost. Under the water surface, Limecraft uses Automatic Speech Recognition (ASR) and Image Recognition for computer assisted logging, making sure content remains accessible when the inventory scales to 100.000’s of media files.

00:12:18 – Doing so, the business case and optimisation for a lower environmental impact go hand in hand. By storing the bulk of the content on data tape (LTO-8), thereby avoiding having to store 1,5PB on online storage, Hotel Hungaria realised 100.000€ in direct savings. Because data tape is “cold storage” (not using electric power unless accessed), Hotel Hungaria saves an estimate of 10kWh per day, i.e. the equivalent of a average household.

About Archiware and Archiware P5

00:15:31 – Archiware PR is a software product installed next to or directly on storage servers. The modules for Backup and Archive may use LTO data tape. Archiware also has a replication product. Backup is a disaster recovery (DR) tool. It runs daily against all the production storage, and creates an updated copy of everything on tape. Most customers use 2 or 3 backups. Archive is a different proposition, it is where content is moved for preservation purposes. In the case of Hotel Hungaria, this is at the very beginning of the pipeline. Right after high-risk footage is shot. Or it may be used to take completed work offline at the end of the project.

The key difference between Archive and Backup

Compared to backup, Archive is different. Backup is on a file level and is everything – an exact and complete copy. Archive is more selective. You need a properly structured archive to be useful. For the Archive, Hotel Hungaria uses a folder structure which is similar to the project structure in Limecraft, which makes the Archive more usable and accessible.

Storage Hardware for Archive and Backup

Smaller companies often use a pile of USB drives. This is not reliable over a period of 10 years. It is not cheap, nor fast. It is hard to create redundancy. Archiware also supports 3rd party cloud services from Amazon, Microsoft, Blackblaze, Wasabi and more, but usually the cost of these is much higher compared to the use of LTO data tape. Considering data tape, Archiware supports for all generations of tape and all common vendors of tape hardware.

About Hotel Hungaria end the Wildlife Project ‘Our Nature’

00:21:58 – Hotel Hungaria is a team of 800 professionals, free-lance included, together working on 100 new projects per year. 850 shooting days output 500TB of new content per year. Different departments work on different types and stages of projects, including marketing, short form, scripted, non-scripted and post-production. The ability for them to collaborate online is an essential ingredient for overall efficiency.

Storage Architecture, Security, Accessibility and Environmental Impace

00:23:31 – Original footage is ingested using Limecraft Edge, making copies and transcodes, and transferred to the various storage locations including checksums. Next, Limecraft ‘Cloud Connector’ creates a disaster recovery copy. Cloud Connector is a dedicated piece of software component that interacts with storage nodes on the one hand side, and reports back to Limecraft Flow running in the cloud.

Hotel Hungaria maintains a total of 2.3PB on cold storage. Collections can be restored from archive on the production server (Scale Logic) for online editing, but the majority of the material remains on cold storage all the time.

One of these projects is ‘Our Nature’, including 80.000 clips. Shot on Sony in X-OCN format at 650Mbps for a total of 1,5PB, and using 1800 days of editing and 120 days of colour grading.

Tip1 – Make sure the storage is ‘Human readable’

00:26:38 – To maximise business continuity, Hotel Hungaria took great care the storage, both the online production storage as well as the archive storage on data tape, remain ‘human readable’: the storage uses a structure of project folders similar to how material is organised by the MAM system.

From the perspective of the MAM system user, collections can be archived or restored from archive as needed, without the intervention of technical staff.

Tip 2 – Security Considerations

00:29:40 – Hotel Hugaria is safe because media are stored on LTO twice. Furthermore, security comprises several other important aspects.

800 named access the system using Multi-Factor Authentication. When a person leaves the company, access controls are removed with a single click of the button.
When a project is finished , it is closed. Members can still access the content by using the Limecraft search engine and using the proxies, albeit read-only. Changes are not possible, unless the collection is restored from archive first.
Then, there is the aspect of business continuity. The production server is read-only as well, only Limecraft ‘Cloud Connector’ is capable of writing onto it. Users can access the production server, but can’t change or delete anything by accident.
Finally, by nicely storing all projects on cold storage, you are much less susceptible to ransomware and malware in general.

Tip 3 – Accessibility

00:32:47 – Given the size of the project, accessibility is key. Edit producers need to efficiently retrieve shots for editing. To do so, Hotel Hungaria applied a variety of methods. The cameramen typically deposit any metadata in a spreadsheet, which is ingested and processed by Limecraft. Subsequently, Limecraft uses various AI services including Automatic Speech Recognition, Face Detection and Image Recognition (detection of animals, optical character recognition) to index the content.

Furthermore, Limecraft can also be used for securely sharing content with 3rd parties for verification. This eliminates file transfers or direct access to production servers. In case the user needs to download a copy of the files, Limecraft Cloud Connector temporarily stages the content thereby creating a secure gateway.

Committed to Sustainability

00:35:14 – The company currently uses 450TB of online storage. As the company grow, they mainly need to add additional tapes at a cost of 10€ per TB, i.e. at a marginal operational cost and without significantly increasing their power consumption. By using the online accessibility of the content, Hotel Hugaria drastically reduced transportation costs. Finally, the remote editing and online accessibility allow people to collaborate regardless of their physical location, and without the obligation to commute.

More info about this specific case: https://www.limecraft.com/cases/hotel-hungaria-onze-natuur/