For those who have not taken a look at virtual sets recently, it's probably a good time for an update. Virtual sets are being used in the studios of major networks as well as at local call letter stations and even small production houses. Applications range from local news production and talk shows to music videos and sporting events. Prices continue to come down, and quality continues to go up - dramatically. Those of us trained in the television industry can usually tell when a virtual set is being used, but often the average consumer doesn't notice until it is pointed out to them.
Given the great inroads that virtual set technology has made in just the past eighteen months, there is still the sense that it is "new," and I often find that broadcasters are not even sure what questions to ask when first looking at virtual sets. If you are considering the purchase of a virtual set, what do you look for? This article will examine the major issues that should be considered. For example, how do you evaluate the "real cost" of the set when you take into account space savings and construction costs of a real studio? Can you use your existing hardware - cameras, lenses, chroma keyer, lighting, etc - or will you have to build a complete studio from the ground up? Finally, should you even consider the purchase of a virtual set today given that everything may go to HD in a short time? Not all of these questions have straightforward answers, but by examining some of the issues, you should be able to determine whether or not a virtual set is right for your operation and then be in a position to evaluate the various solutions that are available.
Definitions
Let's start by defining what we mean by a virtual set. In basic terms, a virtual set or virtual studio system is a tool which allows you to place live actors in graphically generated 3D environments for live productions. In a typical virtual studio production, the talent will perform in front of a blue screen background. The actual background that will appear in the final output is a graphic 3-D image that resides in the computer. The foreground and background images are digitally composited using a chroma-keyer. The concept is much the same as the familiar bluescreen and keying technology used for television weather reports. The difference with virtual set technology is that the image keyed onto the blue screen is a three dimensional graphical set that the actors interacts with. More importantly, with virtual sets, you have the ability to move the camera freely, and synchronizing the camera with the 3D graphical set convinces the eye the set is more realistic. It provides a believable and visually interesting show for the viewer.
The computer graphics are produced with modeling software packages and then imported into the virtual studio software. To build a realistic set or at least one that is believable, the software needs controls for proper texturing, lighting and model parameters.
During production, actors move about the virtual set, the camera operators follow the action, and the set is synchronized with a true perspective.Both foreground and background objects may be moved and manipulated in real-time; therefore, an actor can not only go in front or behind virtual objects, but walk or move inside them.
How It Works
A virtual studio system consists of four basic components. The camera tracking system which will electronically or mechanically extract and calculate the camera position parameters, a computer workstation, an off-the-shelf 3-D modeling software package for rendering a 3D virtual set, and a chroma keyer which combines the foreground and background for the seamless picture.
Integrating real-time TV images with a three dimensional computer generated environment is achieved through software which functions as a virtual camera -- it maps the images of the virtual scene (including virtual characters) onto the real camera images.
In order for a seamless blending of the virtual scene with the real actors without jittering or jumping, the virtual camera has to be given a complete set of accurate position data from the real cameras and the actors. The virtual set software analyzes the co-ordinate spaces of the real world and the virtual world and then matches them to each other exactly. To do this, the computer graphics system obviously has to have information about the real world coordinates -- x, y, and z, pan, tilt and roll axes as well as distance units. This information gives position, orientation and scale data needed to place real world objects properly in the virtual environment. Today there are three different techniques that are employed to obtain the necessary position data: opto-mechanical, pattern recognition, and infrared detection. Virtual set systems may use one technique exclusively or use various techniques in combination.
The first technique utilizing opto-mechanical sensors is a technology which has been around for a number of years. Mechanical sensors are mounted and tightly coupled onto a pan/tilt head to measure the pan and tilt motion as the camera is handled. Further, the lens is coupled with mechanical gears which measure the zoom position. This information is usually combined and sent to the rendering computer to position the 3D model. The advantages of this system are accuracy and cost-effectiveness in single camera systems. Costs go up, however, with each additional camera simply because of the high cost of the sensor-configured pan/tilt heads. Other disadvantages are the inability to move the camera for x, y and z and the system calibration requires the pan/tilt head to be measured very accurately.
In the pattern recognition method, a vinyl carpet is painted in two Ultimatte colors (blue and super-blue) in a line formation, the "grid," on which every point is unique and identifiable by the computer as the image is received from the camera. The concept is not dissimilar from the way in which bar-codes work. The vinyl carpet can be any size and is hung flat on a wall. The foreground video is fed to a fast, parallel processing computer which recognizes the grid and by analyzing the grid angles and applying pre-determined mathematical algorithms the computer can calculate the x, y, z, pan, tilt, zoom and roll camera location. About ten percent of the grid needs to be in the frame of the camera shot for the computer to have enough information to calculate camera position. The advantage of this method is that cameras can be moved freely -- and this is true for handhelds Steadicams and dollies. It supports multiple cameras and no calibration is required on an on-going basis. The disadvantage is that the grid must be in the frame of the shot, which limits shooting angle to about 180 degrees.
For the infrared detection method, IR beacons and directional receivers are used for automatic location of moving objects in the studio -- such as the cameras and actors. IR detection modules are used for automatic depth-keying and other functions which require precise locations.
Beyond the basics, there are a number of items to note and consider when looking at virtual sets. For instance, major consideration will have to be given to the design of the virtual set. There are a wide array of 3D software programs that can be used to create virtual sets, but your real decision will be whether to have in-house designers create the sets or outsource the task. There are a number of independent companies which now specialize in creating virtual sets and even offer off-the-shelf sets from their set libraries.
Preview capability is another factor to consider, as directors will want to view camera outputs before switching cameras. Shadow capabilities are now more widely available, so that real shadows from live talent can be seen or shadows can be created in the virtual set software. Lip syn must be also be accounted for - some systems have a delay of one or more frames which will vary depending on which tracking system is used.
Weighing the cost and making a selection
You may be surprised to learn that entry level systems are now available as low as $100K. And when you consider this along with some of the other savings discussed above, you can no longer dismiss virtual sets on a cost argument.
The first step in evaluating a virtual studio is to assess your specific requirements. You will want to determine the number and range of productions which may be executed. This is important because some virtual studio architectures are better equipped to support multiple productions from one blue screen area. If you are planning only one virtual studio production, then positioning of the cameras can be fixed. However, if you are going to support several productions from a single virtual studio, you will have to take camera recalibrations into effect.
When evaluating virtual studios, it is best to be aware of your existing facility and what equipment on can be utilized in a potential virtual studio set up. Some virtual sets will require specific approved equipment -- this is not necessarily a disadvantage if you are building a studio from scratch, but you should perform a cost analysis. General requirements for all virtual studios include a chroma keyer, blue screen area, computer (SGI or Windows NT), and lighting. Check with the virtual set vendors as to what existing equipment can be used with their system. Finally, given the developments of in our industry related to DTV and the fast pace at which we are moving toward HDTV production, any virtual set you invest in today should have an existing upgrade path to HD.
Enhancing Reality
Techniques and computer processing power have improved to the point where now virtual sets are no longer criticized for their lack of realism. Today, virtual set techniques allow for closer interaction of the actors with the virtual set. Actors can walk in front of or behind virtual objects, or even inside them. On one news channel, the anchor walks to the center of the set, a virtual railing rises from the floor to surround her, and then she is transported to the upper level of the virtual set.
The technology has advanced so much that users no longer simply attempt to create sets that are valid imitations of real life. Their aspirations are greater now. In fact, many virtual set users have it as there goal to create totally unique visual experiences -- experiences that are better than "real."
