10 Tips for a more efficient Video Ingest - Limecraft Blog : Limecraft

Automate repetitive Tasks of Video Ingest and save Time for Storytelling

A good ingest process is the key to efficient post-production and enabling remote post-production. An efficient ingest process automates repeated manual tasks like audio sync, uses proper file and folder names, and combines LUTs and watermarks as part of the encoding proxies. A good ingest process does quality checks and reduces errors to a minimum. The best ingest pipelines reduce the processing time by 50% and, the remainder of the work is automated creating availability for other work.

In case of a poorly set up ingest process, the data handler or the edit assistant help themselves using a hodgepodge of different tools for offloading, manipulating the content, encoding proxies, and manually setting file names. Copying and pasting data between the different applications is frustrating, it causes delays, and it is considered the most common source of errors.

Applications like Hedge, Woody Technologies, Film Databox, Telestream ContentAgent, or Limecraft Edge make it easy for data handlers and edit assistants to automate parts or all of the ingest process. Paired with Limecraft Flow, you can bring the level of collaboration and the quality of your work to the next level. Using automated ingest orchestration, you don’t need to worry anymore about encoding types and file formats, copying hundreds of files into different storage locations, and adding right the metadata to the Media Asset Management system.

💡 Note: You’ll need a Limecraft account to use the workflow in this piece. If you don’t have an account yet, it’s free to get started.

Table of contents

💡 Note: To get you started ingesting content, we make available Limecraft Edge on Windows and Apple free of charge. You can read more about setting up your ingest workflow here.

Offload Media and Checksum Calculation

Offloading Media means writing all original source material to one or more destinations for security purposes before you execute any other processes, including folder structures and metadata. It creates a carbon copy of the source onto the destination, within a folder hierarchy as set using the ‘Destination Path’. The estimated time to completion of the backup job is typically shown as well as the progress for each clip.

Using Windows explorer or Apple Finder to manually copy files and comparing the size of the folders before and after, is not reliable, as the source may contain all sorts of hidden folders and files that are transferred. To make sure you are not losing any valuable content, you should use a proper tool for offloading, such as Hedge, Shotput Pro, Silverstack or Limecraft.

A good backup or offload solution allows you to pre-configure and manage the folder structure and file names according to the specifications of the editor, so as to avoid manual correction of the file names at a later point in time.

After the writing of each file, a good offload process performs a content similarity verification by reading the file of the destination volume and comparing it with the original source file. An ‘MD5’ or similar checksum is used to verify the copy process is successful.

💡 Pro Tip: a good offload solution allows you to define source drives, system drives and backup drives and prevents you from writing on system or source drives.

Audio Synchronisation and Audio Layout Configuration

Audio handling often gets complicated. While you might be recording a certain number of discreet audio channels, edit proxies will typically use 2, 4 or 8-channel audio and most cloud-based collaboration platforms such as Filestage, Vimeo or Frame.io don’t support multi-channel audio at all. Limecraft Flow is currently the only online collaboration platform supporting multi-track audio, allowing you to switch between audio channels and audio layouts during the play out.

So, a typical high-end shoot will result in batches of video files offloaded from the camera media and a set of audio files from the audio recorder. Before uploading the raw material to the edit suite or the collaboration platform, we recommend to synchronise audio and video. The most reliable method to equalise the clocks of the cameras and the audio. It takes a bit of discipline and dedication, but it pays off as saves you one to two hours of manual work per shooting day. Alternatively, when using prosumer gear without reliable clocks, you can opt for audio synchronisation using audio synchronisation using Linear Timecode (LTC), which is offered by Limecraft since the 2021 winter release.

Coming back to the point of audio layout configuration, the best is to make sure your production is properly set up before you start ingesting. Using Limecraft, you can centrally manage the production settings including the setup of audio layouts.

Metadata extraction

While all video ingest software can encode and transfer proxies, only a few including Woody Technologies and Limecraft Edge properly extract and handle metadata created by cameras and audio devices. Limecraft extracts over 100 technical metadata, the camera card manifesto, and allows you to add additional descriptions and modify the selection of circled shots.

Encoding of Proxies

Limecraft Edge can be configured to drive third-party encoders such as Telestream Vantage, CODEX or Kolorfront, but the built-in encoding pipeline process all common consumer and professional file formats, thereby creating low-resolution copies for collaboration purposes and edit proxies for the editing process. Such low-resolution copies for collaboration are typically 2Mbps (1GB per hour), but in the case of very high-resolution sources or complex imagery, you might need to increase the bitrate in order to avoid artefacts. Edit proxies are used for cutting the video and are typically delivered in a DNx or ProRes flavour for cutting on Avid Media Composer, Adobe Premiere or similar. Considering these requirements, you should definitely aim for some flexibility when looking for an encoding solution.

One of the key advantages of using Limecraft for your video encoding requirements is to centrally manage all the encoding parameters centrally, regardless of which machines or instances of the software you are using to create proxies. Whether you are using Limecraft Edge, Cloud Connector or Watch Folder, as soon as you touch a particular project, the software will pull its settings according to the most up to date list of allowed encoding profiles.

Doing so, as an edit assistant or post-production supervisor, you can be confident at all times that all encoding services are using the agreed specifications. For operations that run at scale, this approach structurally solves the number one cause of quality problems in the edit suite.

💡 Pro Tip 1: high-end cameras use lossless compression of very high-resolution images, at a bitrate of 450Mbps for SONY X-OCN up to 2400Mbps for ProRes4:4:4 or ARRIRAW High-Density Encoding (HDE) formats. That is roughly up to 1TB per hour of content. In such cases, you should not use a cloud-based encoding. You will always need local storage and a local instance of software that runs the encoding processes.

💡 Pro Tip 2: encoding high-resolution formats can take a long time and may be demanding for your hardware. In order to ensure you can use your machine for other tasks while the encoding process ongoing, you should look for a solution that lets you control the hardware resources the encoder is allowed to access. If not, your machine will not be usable for other tasks as long as the process is running.

Lookup Tables or LUTs

High-end cameras typically produce a raw file format created to contain as much visual information as possible, but the images need colour correction to give them a representative look. Lookup Tables or LUTs are simple ‘one size fits all’ transformations of the colours of an image. They are typically provided by the Digital Image Technician (DIT). Where applicable, LUTs are typically rendered as part of the encoding process.

It is essential to use LUTs when creating low-resolution proxies for collaboration, if not the images are not suitable for quality control or pre-cutting. LUTs are typically not rendered on the high-resolution footage, as the online editors will want full freedom during the colour grading process. Lastly, whether or not LUTs should be applied for edit proxies, depends on the volumes and the specific requirements imposed by the post-production supervisor. Bear in mind that different batches of content, for example interior or exterior shots, may require different LUTs.

💡 Pro Tip 1: when you are looking for a solution to render LUTs as part of the ingest process, look for a solution that allows you to specify which specific versions need and don’t need it

💡 Pro Tip 2: look for a solution that allows you to use multiple LUTs, and configure which batches of content should use which LUT

Watermarking

Proxies use watermarks for several reasons.

sometimes the watermark contains the card names to make sure the files can be easily identified;
sometimes it makes sense to put the timecode as a watermark, allowing faster logging or pre-cutting;
watermarks may contain a company logo or indication of origin;
for security purposes, some producers opt to earmark the images with the name of the person that downloads the content; or
any combination of the above.

Using Limecraft as the solution for your video ingest process, adding visual or textual watermarks is at the tip of your fingers without the need for integrating third-party software. Rendering watermarks is executed as part of the encoding process, saving you the burden of transferring the file to a third-party service and reducing the turnaround time.

File transfers

Arguably the most important and most critical step of the ingest process is the actual file transfer. In a typical use case, the original media are stored on a device optimised for preservation and managing large volumes (such as LTO data tape) in one or two copies, the edit proxies are transferred to shared storage accessible to the edit bays, and the web proxies are sent to your collaboration platform of choice.

It is very important to look for software that runs reliably and can cope with unstable network conditions, as transferring gigabytes (if not terabytes) of content may take a while. It is highly recommended to recalculate the MD5 verification (see above) after each transfer, so as to confirm the integrity of the data.

If you use Limecraft Edge to orchestrate the file transfers, you can pre-configure the transfer settings, thereby indicating exactly which encoding profiles are supposed to be transferred to what destinations. These can include temporary drives to store the high-resolution footage for physical transport to the post facility where the media can be stored at their final destination.

💡 Pro Tip: if you use Limecraft Edge to manage the transfers in combination with Limecraft Flow as your workspace or Media Asset Management system, Limecraft will keep track of all versions. This means the items in your online workspace are linked to media that may be stored locally or elsewhere. Doing so, upon export to edit, distribution or archive, Limecraft can send instructions to that local storage and execute a partial retrieve as if the storage would be directly accessible through the cloud.

Naming conventions

When transferring the media to disk or the online workspace, you might want to set an agreed nomenclature of file names and folder structures according to the specification post-production supervisor or edit assistant. These are important because they affect how the media are handled downstream to create the edit bins.

Updating the MAM system

So the proxies have been encoded, and the files have been stored on tape or disk. The (almost) final step is to update your workspace or the Media Asset Management (MAM) system with thumbnails, metadata and any other ancillary files. As of this point, if the file is not in your records, it will exist.

As suggested above, after creating the records, all available metadata and side care files, as well as the references to all known copies of the files must be registered in the MAM system. These include de MD5 hashes of the media, as they are the proof of integrity. If you plan to ingest cards, it is important to look for a system that properly supports card structures and understands the card manifest. Later, upon export to edit, it will prevent you from having to transfer entire cards or shooting days, allowing you to realise significant savings on online storage and minimise transfer delays.

Artificial Intelligence cuts shots into Scenes

The hat-trick of using Limecraft Edge for ingest in combination with Limecraft Flow as your workspace of preference, is the use of Artificial Intelligence (AI) to automatically cut shots into scenes. Limecraft understands the similarity between the takes of a shot and links these to the right scene. As a result, the proxies can be easily visualised in the order of the script, instead of per shooting days or otherwise. This is a massive timesaver for anyone on the team that has to follow up on the continuity, apart from the director and the editors also including set script supervisors, dressers, Make-up Artists, etc.

Summary

In summary, these are the three most important key takeaways you should keep in mind when looking for an efficient video ingest solution

While there are several great specialised applications to solve parts of the ingest process (e.g. Hedge for offload, Da Vinci Resolve for audio sync and grading), reducing the number of point solutions also reduces the number of transcodes, processing time, file transfers, and eventually the chances to make errors due copying and pasting data from one system to another.
Limecraft Edge is the only application available on the market that combines a full functionality for offloading, audio sync, LUTs processing, encoding of proxies, and secure file transfer using proper naming conventions
A workspace based on the combination of Limecraft Flow and Limecraft Edge gives you a raft of additional perks including a smooth solution for remote post-production using a proxy-based workflow, central configuration management to avoid configuration inconsistencies and thereby avoiding errors, and automated pre-cutting.

New to Limecraft? Limecraft provides workspaces for video teams of all sizes, and supports all types of content. It integrates with your existing tools without complicated code. Sign up for free to start using Limecraft.