Friday, December 30, 2016

Building a 1950's Cinerama experience in VR



"This is Cinerama." What was Cinerama? Cribbing from Wikipedia: "Cinerama is a widescreen process that originally projected images simultaneously from three synchronized 35 mm projectors onto a huge, deeply curved screen, subtending 146° of arc." I had to look up subtending - basically a line joining the endpoints of the screen's arc. We'll keep that in mind as we develop our virtual reality recreation of a Cinerama screen. I recommend reading the wiki article, though in summary the process involved filming with three overlapping cinema cameras to project a panoramic image, an effect that is very similar to Imagineering's Soarin' projection technology and very like the multiple camera 360 pano capture devices that are flooding into public hands.



When we ask "can Hollywood make a VR movie" the simple answer is that they sorta' already did in 1962, and it worked quite well. As the documentary Cinerama Adventure demonstrates, some directors were frustrated by the limitations the format imposed on their work, while others embraced the creative possibilities within those boundaries. Both facets are demonstrated in the epic MGM movie How The West Was Won. As one subject put it, the scene is often front lit, side lit, and back lit. That sounds very like a 360° VR shoot. Since Turner has released a Blu-ray version for which their technicians have done a brilliant computer-stitched version of the three camera's frames and even included border material that most theaters lost in projection, I think it is a perfect opportunity to attempt to recreate the Cinerama experience in VR.


The first step involved creating a screen geometry for projection. Since the Blu-ray material extends the HTWWW screen dimensions to nearly 180°, I used Autodesk Maya to build a partial sphere that covered that angle. Interviews with Cinerama filmmakers have made reference to a "Cinerama zone" where I imagine the sweet spot for viewing takes place, much like any current theater. I found that this was not in the center of the sphere, as this gave a feeling of being too close to the screen. In addition, I found that a cylindrical screen - which is what would have been found in Cinerama theaters - led to an effect of stretching at the top and bottom of the image, probably because these areas are at a greater dimensional distance from the viewer, but was likely a necessary concession to audience members who were not seated in the "zone" and also would have been much less costly to build than a spherical dome/screen. We don't have that limitation in VR, although the VR scene we're using can display on any surface form.

After some testing, the screen was scaled up 15 times. One reason viewers are uncomfortable setting close to a screen might be that their eyes are focused at twenty feet when their mind is seeing objects that are supposed to be at or near infinity. It doesn't seem to work the other way, it is not uncomfortable to focus far away on a subject that seems close to the camera, so I scaled our screen geometry to allow the VR viewer to focus at a distance, making the massive screen seem more comfortable.

Because we're using the Unity engine to display the scene in an HTC Vive, there is no projector, instead the pixels are mapped in the GPU via UV position to points on our screen geometry using the GPU's warping capability. It's just texture-mapping with a movie instead of an image, and Unity is up to the task on my MSI laptop, even after changing the texture import settings to 100% quality. This takes a long time to import, and was frustrating when tweaks needed to be made.

Unity won't play a Blu-ray directly, so I needed to convert the movie to MP4 files, and keep the file size under Unity's two gigabyte texture limitation, so in order to keep the quality pegged I exported into several 1GB chunks and wrote a script to swap the movie textures as needed. This made it easier to jump around within the film for testing as well, after I bought the movie here.



Here are the controls and movie swapping code attached to the 3D screen0 model object in Unity:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;


public class PlayMovieInSpace : MonoBehaviour
{
    public Texture[] textures;
    public int currentSegment = -1;
    private float timeLeft;
    private int nextSegment;
    private bool pauseButton = false;
    public MovieTexture movie;
    private bool firstRun = true;
    public GameObject audiencePosition;
    public AudioSource audioPlayer { get; private set; }
    public float seatAdjustmentPosition;

    void Start()
    {
    }

    void Update()
    {
        if (firstRun)
        {
            firstRun = false;
            audiencePosition = GameObject.Find("screen0");
        }

        if (Input.GetButtonDown("Jump"))//equals spacebar
        {
            pauseButton = !pauseButton;
            if (pauseButton)
            {
                movie.Pause();
            }
            else
            {
                movie.Play();
            }
        }

       
        //skip to next chapter
        if (Input.GetButtonDown("Fire1"))//equals catarel key
        {
            timeLeft = 0;
            movie.Stop();
        }
       
        //adjust seat position in theater
        if (Input.GetKey(KeyCode.UpArrow))
        {
            seatAdjustmentPosition = seatAdjustmentPosition - 0.1f;
            audiencePosition.transform.position = new Vector3(transform.position.x, transform.position.y, eatAdjustmentPosition);
        }
        if (Input.GetKey(KeyCode.DownArrow))
        {
            //Debug.Log(audiencePosition.name);
            seatAdjustmentPosition = seatAdjustmentPosition + 0.1f;
            audiencePosition.transform.position = new Vector3(transform.position.x, transform.position.y, seatAdjustmentPosition);
        }


        timeLeft = timeLeft - Time.deltaTime;
        //Debug.Log(timeLeft);
        if (timeLeft < 0)
        {
            //Debug.Log("0 timer loop");
            AudioSource audioPlayer = GetComponent<AudioSource>();
            Renderer r = GetComponent<Renderer>();
            r.material.mainTexture = textures[currentSegment];
            movie = (MovieTexture)r.material.mainTexture;
            audioPlayer.clip = movie.audioClip;
            while (!movie.isReadyToPlay) { } //do something}

            nextSegment++;
            currentSegment++;

            movie.Play();
            audioPlayer.Play();

            timeLeft = movie.duration;
        }
    }
}


There's an obvious and lazy bug in that it will try to play a non-existent movie segment after the final one, though by then you have already left the theater and they're sweeping up popcorn so it's not top on my list of things to fix. I am, of course, not going to upload the movie, though I will upload the Unity scene without the movie files to GitHub if anyone takes notice of this post.



Now the review: we've succeeded in recreating a Cinerama dome in VR, although as Leonard Maltin points out in the doco, much of the sensory-overload of the Cinerama experience is due to one's peripheral vision being filled by the panorama, so it should feel as though cowboys and Indians are flying around you and past you. Vive has limited peripheral vision; although we can turn and look at the action in seemingly any direction (except behind the viewer), we cannot feel the peripheral effect. I tried the same setup in Oculus with very little difference in the outcome. These are not limitations of this program, just the current state of the art of VR in general. To paraphrase Richard Widmark from the movie, "That's just new life going on." Soon, I hope, VR display's will overcome this limitation.

An unfortunate addendum: Debbie Reynolds passed away. On that same day I finally finished importing the movie segments into Unity and watched the full movie in the Vive headset. Although I never worked with Miss Reynolds, she did make an appearance at the occasional Paula Kent Meehan Christmas party and was a delight. Seeing her performance in How The West Was Won singing the film's rendition of Greensleaves is quite moving.



Tuesday, October 18, 2016

Virtual steality - take your own Disney Treasures - Me-Gifting - AKA out of word play

It's not that I don't appreciate the fine merchandise in the souvenir store at Disneyland Park. It's just that Duffy, Itchy and the Gang are not for me. So if I am to leave Disneyland Paris with a tchotchke that I can appreciate, I am going to have to be creative. Enter Agisoft.


Ignoring the mournful gaze of the Phantom Manor cast member (or was it mere pity?) I began photographing the plaque outside the queue entrance. By taking a couple of dozen shots with a Sony A7S2, I was able to rebuild my would-be souvenir as a 3D model in Agisoft using the magic of photogrammetry. Unfortunately this was not good enough for my pointlessly high standards. The spiky teeth of the freaky head guy were missing from this reconstruction.

According to the Agisoft manual, by taking around one hundred pictures of this checkerboard would enable the software to compute a more precise model of my camera's lens distortions, and make a better model.

Fortunately one kindly Imagineer took pity upon me and pointed out that the checkerboard cannot have the same pattern if it is rotated 180°, so I had to 'shop out the last row of squares from each and every photo. Urf. Something to do with OpenCV Brownian Distortion model algorithms are at play here.

Upon feeding the calibration into Agisoft...
He's still gamming at me! Next I tried bumping up the Clarity in Photoshop in an attempt to create more localized contrast for Agisoft to key on when the focus is not perfect. This also sometimes helps with photos that don't align properly. Then, I tried super-sampling the images before doing any tweaks.

I was quickly coming to the conclusion that my efforts to make my souvenir had failed. Perhaps the Sony did not have gift-shop quality resolution, or perhaps the photos were taken at too wide of an angle (35mm) to be useful.

 

With only one option left, I bit the bullet and purchased a Sony 90mm macro lens, a linear polarizer to block reflections, and airplane tickets back to Paris. This was now the most expensive souvenir of 2016, so far.



155 photos later, we have pointy teeth! Now it is just a simple hopelessly complicated matter to place the data in a 3D printer and push the "Should have just bought the Duffy" command. Would that it were so easy. The Agisoft export has holes and no thickness, so some further processing is needed. I turned to DesignX to do some more robust manipulations than Agisoft currently provides.

Note to self: don't be afraid to brush away spider-webs, they had a visible impact on the quality of the Crypt Keeper's right side.

The penultimate step is to bring the design into a program called Geomagic Sculpt, which is the cleanest method of voxelizing 3d printer data to ensure water-tight meshes and manifold geo. It is also a good place to smooth out any rough patches and refine the model in an intuitive sculptural manner. This is because one can feel the 3D model using Sculpt's haptic feedback stylus. Here's another heisted souvenir in Sculpt:



Ready to print using Photoshop CC, our go-to support generation software for extruded filament printers.



NEXT HEIST: WE STEAL THE ENTIRE PHANTOM MANOR

Monday, November 18, 2013

Finding an apartment that was sealed off 75 years ago

It’s 1938 on the rooftop of a downtown Los Angeles building, the owners of a penthouse apartment with a chewing gum view of the snowcapped San Gabriel mountains (very Wrigley,) closed the door and walked away. For seventy five years, the apartment sat empty. With downtime during a photo shoot setup, playfully examining the rooftop, I began turning door knobs. One opened….

The first indication that a bit of mystery was involved, the household appliances sat where they had been the last time they were used, the GE Refrigerating Machine keeping the leftovers cool, as our penthouse apartment dwellers looked down at the neon marquees announcing the premieres of Snow White, Gone With The Wind, The Wizard of Oz. The maid’s quarters offered easy access to the then exceedingly luxurious spoils of a machine that actually washed your clothes for you!
Looking at the bathroom, I realized that life back in time wouldn’t have seemed much different from today. Everybody poops.

Even the circuit breakers in a closet off of the maid’s room exuded retro-coolosity:

It seems there might have been a spitting problem, as this sign on the sun deck evinces:
From the 1938 patent date, the air conditioning unit on the roof must have been the latest and greatest. Why would they leave?
Imagine this view at night, with the neon lit.
The Eastern Columbia department store would have made for convenient shopping, though today it stands as the finest example of Art Deco era architecture surviving in Los Angeles.
If our mysterious tenants needed an excuse to stay, this million-dollar view would have been worth, in those days, over seven hundred dollars!.
Although world war would effect the City of Angeles a few years later, requiring roof light black-outs during air-raid (tests), I think the pano view of the penthouse best explains the possible reasons for deserting this magical loft: the pipes that travel through the apartment walls reflect laws that arose in the depression era that required high-rise residences and hotels to retrofit fire sprinkler systems, and unfortunately the only way to make that happen was to seal off the rooftop apartment, and leave it alone for us to discover.