By Tom Coughlin, President, Coughlin Associates
Flash memory is finding increased use in client and enterprise applications because of its higher performance compared to hard disk drives. As the $/GB price of flash memory declines these applications are increasing. For professional media and entertainment projects dealing with higher resolution and higher dynamic range workflows, performance matters. We will look at developments in flash memory in digital media content capture and creation, post production, content distribution and even in archiving.
Flash in Content Capture
Professional video cameras are undergoing rapid evolution, driven by higher resolution content, faster frame rates, and higher dynamic range as well as stereoscopic content capture. Many of these cameras are used in the field. For applications requiring rugged field use or fast playback response, flash memory either as cards or solid state drives (SSDs) is becoming more popular.
The physical storage media for professional cameras is undergoing rapid evolution as film and magnetic digital tape (used for many years in professional video cameras) declines. This is due to the rapid file access convenience of hard disk drives and the ruggedness and speed of flash-based, solid-state storage.
Figure 1 shows the percentage of various recording media used by the 2016 Digital Storage in Media and Entertainment survey participants in professional video cameras. Note that the 2017 survey is still open (until May 15, 2017).
Flash memory is the clear leader in professional video camera media, increasing from 19% in 2009 to 54% in 2016 while magnetic tape shows a consistent decline over the same period, in particular, magnetic tape declines from 34% to 2%. Optical discs use between 2009 and 2016 bounced around between 7% and 17%.
Figure 1. Percentage of Various Recording Media in Professional Video Cameras
Professional cameras using flash memory include those from ARRI, Canon, Grass Valley/Thompson, Ikegami, Panasonic,
Before the 2017 NAB show, Micron’s Lexar unit introduced its latest microSD UHS-II (U3) cards with capacities up to 256GB. This product (shown in Figure 2) provides a premium memory solution for sports camcorders. It quickly captures, plays back, and transfers media files, including 4K, 3D, and 1080p full-HD video. It provides high-speed performance leveraging UHS-II Technology for a read transfer speed up to 150MB/s (1000x)1, a write speed of up to 90MB/s, and includes a microSD™ UHS-II USB 3.0 reader for high-speed file transfer. The product is verified to achieve maximum compatibility with GoPro cameras.
Other professional media formats include the CFast compact flash format introduced in 2013 by SanDisk (shown in Figure 3), the flash memory modules for Panasonic P3 cameras and the Sony camcorder modules such as the one shown in Figure 4.
Figure 2. 256GB Professional 1000x microSD UHS-II (U3) Card
Figure 3. SanDisk CFast Compact Flash Card
Figure 4. Sony SR Memory Flash Memory Camcorder Module
Third party display and recording devices for the field often use flash memory. As an example, the Convergent Design Odyssey7Q supports a wide range of recording formats. 2.5" Premium SSD media supports read/write bandwidths in excess of 500 Mbytes/sec per drive, enabling Canon 4K Raw at 60 fps on a single recorder.
At the 2016 NAB and in the upcoming 2017 NAB show, 360-degree video capture is a hot topic. 360-degree video requires a great many more cameras and can require significant computational resources to stitch together the video images without artifacts.
In order to provide a better viewing experience in the 360-degree field of view (FOV), either the resolution of the entire view must be better (e.g. the Jaunt One high-resolution version has 8K per eye and thus 16K total displayed resolution) or there must be a way to increase the resolution in the most significant FOV in a video, so at least in that FOV, the resolution leads to a greater feeling of reality.
New tools and stitching software, much of this cloud-based, will enable these workflows for most folks who want to participate in this revolution in content. The role of a director is as importance as ever as new methods are needed to tell stories and guide the viewer to engage in this story.
Figure 5 shows the Jaunt ONE 360-degree camera. The Jaunt ONE high-resolution version (24R) has a 1” 20 MP rolling shutter sensor with frame-rates up to 60 fps with 3200 ISO Max, 29 dB SNR at ISO800. It has 10 stops per camera modules with customer 130-degree diagonal FOV, 4/2.9 optics and with up to 16K resolution (8K per eye).
Jaunt ONE at 60 fps provides 200 GB/minute uncompressed. This can fill a 1TB SSD in five minutes. They are forced to use compression to be able to use currently affordable storage devices. This compression creates 11 GB per minute, which can fill a 1 TB SSD in 90 minutes.
Figure 5. Jaunt ONE 360-Degree Video Camera
As shown in the survey, flash memory use in professional cameras is increasing and all the professional video camera companies now offer flash memory-based digital cameras. However, because of the higher cost of flash memory compared to other storage devices, it is very common to quickly empty flash memory cartridges to another storage device. Often this is an external hard disk drive. As the price of flash memory drops we expect to see dramatic growth in the use of flash memory for professional content capture.
External hard disk drives are also often used in the field for aggregating content captured originally on solid-state storage modules, so the solid state modules can be erased and reused. It is common to unload content captured on flash memory modules onto external hard disk drives in order reuse these relatively expensive storage modules (i.e. a 64 GB P2 storage modules costs at least $315).
Flash in Post Production
Figure 6 is a schematic of a non-linear editing station showing local storage and DRAM for basic caching and storage of edited content as well as optional connections to shared on-line (or real time) storage via a SAN or NAS using some sort of network connection (GbE or HBA). We also show cloud-based storage in a remote data center, which can enable collaborative workflows of space and time.
Figure 6. Modern Post Production Storage
Although many folks use hard disk drives for local storage on their editing station there are media workstations that are built around the use of solid state storage. Some of these are shown in Figure 7. These include the Apple Mac Pro, often used by media and entertainment professionals a well as notebook computers such as the HP Zbook Studio Workstation. There are also external USB or Thunderbolt storage devices containing solid state drives that are used by some professionals, often for transport of content from field recording.
Figure 7. Flash-Based Video Workstations
Note that DRAM is often used as a buffer in various parts of the NLE system to reduce the impact of system latencies. Buffering and caching in NLE and other post-productions systems are moving from DRAM only to use NAND flash as well as DRAM for buffering and caching. There are many suppliers offering NAND flash enhanced post-production equipment today. Some examples of flash memory storage for video production and post-production are described below.
GB Labs introduced an SSD storage system targeted for 8K video-production. Their Space SSD Ultimate is a networked storage system for 8K grading, editing, compositing and motion graphics. Space SSD Ultimate is a shared storage system that allows online editing of Super Hi-Vision files. It is tuned and configured to provide IO/s for simultaneous read and write operations without degrading performance. Its 9,000MB/s can handle of up to 66 streams of uncompressed HD, 15 streams of 4K DPX or 6 streams of 8K video, with multiple workstations, able to access up to 1.4 PB of storage over 40GbE.
At the 2015 NAB show, HP announced a Z-Turbo PCIe SSD drive on their video workstations. This was a Gen2 NVMe flash memory product from Samsung with sequential data rates of 2.15 GB/s read and 1.55 GB/s writes. Capacities of these MLC products are 256 GB and 512 GB.
NetApp has been showing their EF All-Flash Array for media metadata, transaction and analytic processes at NAB shows. They have been promoting it for 4K and high-frame-rate production workgroups who need to edit, render and transcode concurrently. The NetApp E-Series is focused on large broadcast production workgroups, large content repositories, and some 4K facilities. Bitcentral was demonstrating its
Other companies have flash memory based solutions for media workflows, including EditShare’s XStream Field 2 for a high capacity mobile storage array for work in the field. EMC, NetApp,
At the 2016 NAB show,
It should be noted that
Play-out of content over a distribution system is an application where performance can have real value. This had led to an expectation that flash memory based storage systems may find a role in this market niche. Toshiba, as well as other companies, have a number of content delivery products using flash memory. The Toshiba On-Air Max Flash, Flash Memory Playout Server is shown in Figure 8.
Figure 8. Toshiba On-Air Max Flash
The flash memory in this system consists of chips on a blade rather than having an SSD based system. Toshiba has a whole family of flash-based content delivery servers for content delivery networks (CDNs) such as it’s ExaEdge solid-state HTTP streaming and cache server.
Cinergy is a software-based service that can provide cloud-based service for playout of content. They have 3K playout capability from Amazon Cloud and they provide an all IP-based TV service. They provide end-to-end workflow MAM for a major broadcaster in Turkey with 400 TB of online assets and 4 PB in a disk-based archive. They also provide ingest services for BBC where they were used on Planet Earth. They have been using flash memory for the cache but with the decrease in the costs of flash memory, they may start using this higher performance storage for more applications.
In our 2016 survey, we saw that about 31% of the participants working in content distribution used flash memory on their edge servers in 2016.
Note that edge servers to support VOD delivery using flash memory-based solid state drives (SSDs) were introduced by several vendors such as SeaChange and Oracle, Toshiba and others over the last ten years. Such systems were seen to provide more reliable remote edge operation with good read performance rates, but our survey shows their percentage use is fairly stagnant. We show an IBM flash-based content server in Figure 9. High-speed enterprise solid-state drives (SSDs) and other solid-state storage technology for VOD edge content delivery should become more popular as their price goes down and particularly to provide VoD services and network PVR edge support.
Figure 9. IBM Flash-based Content-Delivery Servers
Flash in Archiving
The lower costs of magnetic tape make it attractive for applications where longer latency for initial data access is not a problem. Note that once mounted, magnetic tapes can have very high data rates. In fact, there are several companies that include flash memory in archive/backup write and sometimes read cache for tape-based storage libraries.
One interesting example is the Spectra Logic Black Pearl Family. This initially started out as a write-cache for writing to tape. The company is now selling this as an Object Storage Gateway allowing content to be archived on disk drives,
Figure 10. Flash Memory in a Spectra Logic Black Pearl Object Storage System
Flash memory will play a role in what are called active archives, where content in the archive is regularly accessed and serves as a content library. Flash memory offers a tier of storage that can be used to access frequently access content and the disaggregation of flash (as well as HDD storage) from the servers in hyper-scale data centers is becoming much more common as a means to more efficiently use this high performance, but more expensive, storage asset.
Conclusion and Summary
Flash Memory is finding its way into more and more professional media and entertainment applications from content capture and creation, through post-production, distribution and even some archive applications.
In 2015 we estimated that 62.7% of the total storage media capacity shipped for all the digital entertainment content segments was in HDDs with digital tape at 27.0%, 6.5% optical discs and flash at 3.9% as shown in Figure 11.
Figure 11. 2015 Storage Type Market Share in Media and Entertainment
By 2021 we expect that tape capacity shipment has been reduced to 18.6%, HDDs shipped capacity is 69.2%, optical disc capacity is down to about 2.3% and the flash capacity percentage is at 9.9%.
In addition to the increase in flash capacity, we expect total flash revenues to increase by an additional 13% of total storage revenues in the M&E industry between 2015 and 2021.
The speed and ruggedness of flash memory give this storage technology a growing future in the media and entertainment business as the cost goes down and as endurance and other issues are mitigated by the move to 3D flash memory and better flash life cycle management.
2017 Creative Storage Conference
At the 2017 Creative Storage Conference (May 24, 2017 in Culver City, CA) you can learn more about the use of flash memory in M&E as well as the growth in VR content in professional video and how this will drive new digital storage demand and technologies to support the high data rates needed for captured content and cloud-based VR services. This is the 11th year of the conference and we look forward to having you join us.
About the Author
Thomas M. Coughlin, President, Coughlin Associates is a widely respected storage analyst and consultant. He has over 30 years in the data storage industry. Dr. Coughlin has many publications and six patents to his credit. Tom is also the author of Digital Storage in Consumer Electronics: The Essential Guide, published by Newnes Press. Tom publishes the Digital Storage Technology Newsletter, the Digital Storage in Media and Entertainment Report, and other reports.
Tom is active with SNIA, SMPTE, IEEE, and other professional organizations. He is Education Chair for the SNIA Solid State Storage Initiative. He is Chair of Future Directions for the IEEE Consumer Electronics Society as well as past Director for IEEE Region 6. He is a long-standing member of the CE Society BoG and was Vice President of Operations for three years. Tom is the founder and organizer of the Annual Storage Visions Conference as well as the Creative Storage Conference. He is the general chairman of the annual Flash Memory Summit. More on Tom at www.tomcoughlin.com.