Clear communication is critical to the content creation process. And while today’s productions somehow manage to compensate for inefficient communication mechanisms, there is a growing and urgent need to streamline the way we communicate and exchange information as we continue to scale up to meet the increasing demand for content. In our blog “Cloud. Work. Flows”, we identified some missing components that are required to enable software defined workflows. We highlighted that a more efficient messaging system will be critical to improve communication between participants (which could be people or machines) in a complex workflow system. We’ve addressed communication elements in the Ontology for Media Creation which covers some aspects on what needs to be communicated. Recently we’ve been turning our attention to the how to express that communication in the most efficient manner. For example, in the 2030 Vision our first principle states that content goes straight to the cloud and does not need to be moved. Once ingestion to the cloud has completed, the first participants in the chain will need to be notified that the content is now in the cloud and ready to be worked on and ideally including a location for that content. A similar workflow notification message is required when a task has been finished and the work is ready for review by another team member. In this post we’ll discuss the benefits of a common approach to communicating these repeating types of workflow messages. In a subsequent post we’ll get into the technicalities of how we think such a system could be built including considerations to enable it to span cloud infrastructures and tools.
The Art of the Message
We need to deal with both simple messages, in near real-time, between two participants, like this:
Synchronous Real-Time Messages between two known participants.
And more complex messages, especially as we move to more automated systems, where the participants may not know who will pick up the messages and may not receive replies for hours or even days. Take for an example:
Asynchronous Messages between one sender and multiple potential recipients.
In this example the Render Manager (the message sender) doesn’t know which nodes may respond or when they may respond. There are thousands of such nuanced examples in production workflows that we need to consider when thinking about the sorts of messages that could be sent between systems. We need a messaging approach that can accommodate all of these message types, and also the complexity of multi-cloud infrastructure when messages may be flowing between systems that are not all owned/leased or operated by the same organization and on the same infrastructure.
Software Messaging Systems
At MovieLabs we’ve been thinking about approaches to these messaging problems. One approach is using point-to-point API calls between all these disparate systems and while appropriate for many use cases we don’t believe this will scale to whole productions or studios – there would simply be too many custom integrations to get all the possible components of a workflow to work together[1]. We see the best way to manage the highly asynchronous delivery of information to multiple (potentially unknown to the sender) destinations is to decouple the mechanics of the communication – the what from the how. In software systems this can be managed in a more automated way using Message Queues[2]. A message queue allows a message to be sent blindly (the sender does not need to specifically know who will read it). Specific queues are typically associated with a particular topic, any other participant with an interest in that topic can then subscribe to the queue and receive its messages whenever they’re ready.
Message queues – or, more broadly, message systems – are a natural fit for software-defined workflows: their raison d’être is to provide a communication mechanism where senders and receivers can operate without knowing anything about each other beyond how to communicate (the message queue) and some expectations about the contents of the communication (the messages.) This separation allows applications to run independently and, just as importantly, be developed independently.
As long as the sending and receiving applications can both access the message queue, it doesn’t matter where the applications are running; they can be in the same cloud, in different clouds, on a workstation in a cloud, or even in two organizations. Agreeing on commonality of some aspects of message headers and message contents can enable interoperability especially in that cross-organizational use case. For example, if a message from an editorial department to a VFX house includes a commonly agreed upon place to put shot and sequence identifiers, a workflow management system at the VFX house can route that message to the appropriate recipients internally.
The use of messaging systems rather than point-to-point integrations also makes it easier to gather together operational data for logging, dashboards, and detecting/acting on exceptions and errors.
Benefits
As we look to the 10 Principles of the 2030 Vision, we can see that messaging is key to enabling the “publish” function (where access to files is pushed through a workflow, as tasks are created) and enabling participants to “subscribe” to those files, tasks, or changes. Principles 1 and 2 of the 2030 Vision state that assets go to the cloud and do not need to move, which means sharing the location of those files becomes a key message that will also need to be shared between systems. By enabling a robust multi-cloud[3] ecosystem with broadly distributed and understandable messages, we hope to unlock the true flexibility of software-defined workflows But we do not need to wait for the entirety of the 2030 Vision ecosystem to be built before we can take advantage of messaging systems – there are many use cases that can be deployed now to enable a more interoperable and flexible workflow in 2022 and beyond.
Next Up…
In our next post we’ll discuss the software elements of a messaging system, types of messages and the use cases we hope to enable with one. Make sure you stay tuned, so you get the message…
[1] Not to say that API calls don’t have their place in interoperability. We’re very supportive of applications exposing APIs for system-to-system communication with small amounts of data or messages that need a guaranteed quickish response.
[2] Message queues are a familiar element of software engineering, suited to a wide a variety of inter-process communication problems. Operating systems use them internally, and business and process automation software make heavy use of them.
[3] We define “cloud” in the 2030 Vision as private, public and hybrid infrastructures connected to the internet and therefore envision productions that need to span across all of those. See the multi-cloud blog here for more details.
