Live DV combining with CG

[robmilliken]

I've shot some video with my daughter in an attempt to master mixing live DV video with AM.

 

My technique: I am having my daughter use a painted popcan the same color as the main color of my CG popcan as a prop. Using BINK/SMACKER tools, I am converting the avi video uncompressed single frame images using the BLEND method of deinterlacing. I then use PHOTOSHOP to fill in the purple popcan the rest of the way (pretty easy with the paintcan and adjusting tolerance along the way since it is the same color). Then, using EYEBATCH, I am changing this purple color to an ALPHA channel. All this is plastered onto a flat grid in AM in COOKIE CUTTER mode. I then have the CGI can modeled in AM behind the single shot of my daughter. Using locked rendering as a godsend of a helper tool in AM, I can get it placed fairly close to the prop cutout. I have the single shot on the grid lit with a single KLIEG light and a couple lights for the can (blue sky outside shining through nearby kitchen window and was shot indoor with one bulb); this was done by separate light lists for the popcan and the flat grid with the single photo shot. Comments are very welcome. I used photoshop for some touchup, but only a little smudging, and not much -- most of the output is directly from AM. I only have 300 frames to go! lol!

 

 

PROBLEM: I have a question for the group though -- the can is TOO perfect for the slightly out of focus live shot. To compensate so far, I make them both a little more blurry by altering the far focus aspect of the camera. This has the undesirable affect though of blurring the live video as well, something I don't really want. Is there a way to blur the CGI can slightly out of focus and not the flat plane photo? Something similar to the light lists which separate the light?

 

Comment comment comment!!!

[Rodney]

Rob,

 

You could render the can out separately to a sequence of TGAs or even AVI...blurring or softening it.

Then composite the DV footage over the top of the blurred A:M can.

 

Multiple Rotoscope might be best to do the trick.

 

Kill all the lighting before you render though. you don't want the lighting to shine on the image rotoscopes.... unless you do.... in which case..... nevermind that last one.

[heyvern]

Hmm...

 

Have you tried the dof blur using a camera rotoscope instead of the cookie cutter model method?

 

You know you can have the rotoscope ON TOP in the camera view, the alpha on the image would act the same as a cookie cutter. Whatever is "masked" in the image will be transparent in the rotoscope.

 

I think using a camera rotoscope the dof would not effect the image. This also might make animating the can to the image easier since you would not have to worry about any other models in the chor. The image is locked to the camera.

 

I beleive the camera rotoscope in a choreography was intended for this purpose.

 

Just an idea.

[heyvern]

Dagnabbit!

 

Just had another idea!

 

This is a ton of more work but it would be cool to try sometime...

 

Have 2 rotoscopes! One for the background and one for the hands or anything in front of the can. You would have to mask out only the rotoscope with the hands. As a matter of fact this would be no extra work since you are doing that already! It would be a hit on memory with AM since you would have 2 sets of animated rotoscopes of course.

 

Keep a set of totally untouched frames for the background rotoscope (behind the can) create a mask on the second set of rotoscopes for the for the hands in front of the can.

 

This would allow you a little more flexibility. Have 2 camera rotoscopes, one set behind one set in front. Another bonus this would add is that you could have a hand "stand in" model to cast shadows on the can, since this model would be covered by the rotoscope it should work pretty well.

 

If you use the clone tool to paint over a little of what is behind the can on the first rotoscope images this would cover any slight mistakes in compositing.

 

Like I said, this might be a real pain in the arse.

 

Vernon "I keep forgetting my sig!" Zehr

[heyvern]

Me again!

 

the idea intrigued me so....

 

This is just a quick little gif that shows the concept I was talking about in my previous post.

[robmilliken]

Thank you Rodney and Vern for the help!!

 

Vern, on a Monday morning before I went to work, that example image had me in stitches! Made my Monday much more bearable, thank you!

 

I tried your techniques. In the past I steered away from cam rotos because for some reason I had troubles getting the alpha out... it never seemed to work. Trying your technique though just in the last hour, I had no troubles at all -- it just makes me pick a key color instead, and I didn't have any trouble with this -- fact now I can forgo the Eyebatch program making a key color alpha for all the frames at once. Another reason was because the popcan has reflection and the rotos don't reflect anything off the model. Just a nitpick though, and fixable, I think I will plaster a decal of the DV on the inside 'dome' of a sphere and have it just behind the camera and lit right should reflect back on the can just fine. I really like your idea too of stand in model hands that put in a realistic shadow. It doesn't even have to be perfect since they are never seen!

 

Sure your technique might be more memory consuming, but I think it would work better for my needs as I don't have to make the alpha channel of the prop can just perfect and do all the finessing that filling in the prop entails. I can make the cg can slightly larger than the prop to cover it up. It also solves a problem for a later scene where the can is going to warp out in her hands (using the cookie cutter, the warps that bulge out would have gone beyond the prop and it would have gone behind the dv image). All this too renders FAST!

 

I am SOOO glad I consulted here before going ahead with this, as this REALLY helps!!! THANK YOU!

 

BTW: this project was inspired by a scene from television's Enterprise, a series that I think is underrated for it's CG work. There was a scene where a character holds a cup by the handle and is about to take a sip, then lets go and the cup wanders off weightless away from him. I had always wanted to try to get something like that to work with it's problems on how the fingers cover the cup, lighting it right, etc., in AM.

 

I hope at some time to have the finished project posted in the 'completed' forum. A ways to go, but I have a method now!!!

 

Thanks again!!

 

Rob Milliken

[heyvern]

Hey!

 

Great!

 

So glad you liked my image. I was trying to think of what to replace the can with and there was my character Bob, with his arms out like some kind of action figure...

 

I was wondering about the problems with alpha on a rotoscope so I did a "real" test in AM just to make sure I am not insane.....I HEARD THAT!

 

I would still recommend using an alpha (it does work!) on the rotos only because you can blur the edges to match the image. Using a key color the edges are aliased, hard, and don't match the scene.

 

In the attached image you can see that the edges of the hands are blurred slightly to match the slight out of focusness of the whole image. You wouldn't get that using a key color.

 

If you need help figuring out how to save an alpha let us know here. For some reason people seem to have problems with this. When saving the file out of Photoshop or any other application that supports alpha just make sure the alpha is black for invisible white for opaque, and save as a 32 but TARGA

 

Vernon "I have real work now!" Zehr

[heyvern]

...32 BIT targa... not but targa... There is no such thing as a 32 but targa...

 

For reflections on the can you could use an environment map using the rotoscope images, or as you said, put a panel in front of the can that stays aligned with the camera using the same images.

 

 

Good luck fine sir!

 

Vernon "!" Zehr

[robmilliken]

I think my problem with Alpha rotoscopes in the past was the 32 bit part. I think I was saving them as 24. There is a pretty good tutorial for those following this thread on how to do what your examples show above in photoshop:

 

http://www.lindaworks.com/tutorials/photos...oshop_alpha.htm

 

As far as cutting them out I used the magnetic lasso and used the wand to touch up what it did and did not grab for my little tests on how to use this technique.

 

Again, very helpful. I am not sure what you meant about the key color being a hard edge and the alpha technique you can blur? This is probably something very photoshop specific. The alpha technique I have used so far from what I can see does a hard edge too. In what way can you blur the alpha channel? Below is a zoomed in view of her hand and a brighter popcan put 'under it' and out of place to illustrate.

 

Thanx again!

 

Rob.

[heyvern]

The alpha technique I have used so far from what I can see does a hard edge too. In what way can you blur the alpha channel?

According to the technique you posted there is one thing left out.

 

You can go in to that channel and modify it after it is created. When you save the alpha channel click on it in the Photoshop channels pallete and you can then paint on it, blur it what ever you want or need to do to clean up the edges.

 

A couple of tricks would be to use the lasso or magic wand step. Then after saving this selection as an alpha use the noise/dust and scratches filter in PS to clean up the edges. Then you can blur this slightly and possibly use levels to adjust the width of the alphas edges. or you can just use the paint brush to clean up the edges.

 

You can also view the mask as you adjust it by turning on the eyeball icon on the RGB channel in the channels pallette in photoshop.

 

This will display both the rgb image AND the extra alpha channel at the same time. Double click the alpha channel and you can change how it is displayed by setting the color and opacity of the channel. This will not change the alpha only how it is displayed in photoshop.

 

I hope this helps. Please don't hesitate to ask for more info if this is unclear.

 

Vernon Zehr

[heyvern]

One thing I forgot to mention,

 

The Magic wand tool will almost always give you some aliasing on the edges. It is not my first choice for creating a good mask. Actually it has been many years since I have ever used the magic wand too to be honest.

 

Using the select/color range command in photoshop might give you better results. Or even duplicating one of the rgb channels and modifying it with levels or curves is also a technique I use.

 

Vernon Zehr

[Bill_Y]

Strangely I have an affinity for using paths to make my selections... ;P

[robmilliken]

Below is the result of fixes to the original made by suggestions here. I think you can see a clear difference and it looks much better. I do think I am beginning to obsess with this one frame, but if I get the technique right I figure the other frames will obtain more continuity.

 

I was able to blur the alpha (even with my bad wand selection, I think it looks fine) per your suggestion. I wanted to impliment your suggestion with shadows on the can. I really tried. I even tore off the hands of some models on the CD (can you imagine if some federal agency is going through the Internet looking for key phrases of terror right now?) but they didn't cast shadows at all. I think this may be because of the light list to the can. I even tried including the model's hands in the light list to no avail. Finally, I decided to take some black cubes make em 90 transparent and set density to 1. Since the hands are already on an alpha on top, placement was pretty easy -- they didn't affect the hands at all.

 

Thanks again for all your suggestions! On to frame TWO!! Woo hoo!

 

Rob Milliken

[heyvern]

Strangely I have an affinity for using paths to make my selections... ;P

Me too!

 

I was trying to think of ways to create the alpha that could be done with a photoshop action. Using the paint brush and path tools is kind of hands on.

 

If the colors are similar on each frame and the scene doesn't change much you can use the color range selection gizmo, or duplicate a channel then save the selection and fiddle with other filters and levels and they can be recorded in an action. If you get pretty close it is a breeze then to run that action on all of the frames.

 

I have done this before but it requires that all the images be very similar, and also a fairly unique key color for selection purposes in photoshop.

 

I love photoshop actions...

 

p.s. hey Rob! If you send me a few frames from the original vid (one from the beginning middle and end) I might be able to create a photoshop action that would process all of the frames and save out targas with one click. let me know. You could look at the action and learn from it as well. Much more fun to have a cup of coffee while the computer does all the work.

 

Vernon "Man of action" Zehr

[robmilliken]

Vern,

 

Here is a copy of the last frame. You said on your previous post that you were able to do this trick of selecting a color range if the frames were similar. Well, posted here is frame 37 (yes, I said hundreds of frames earlier -- this is only one clip in a few) and the last frame of the clip. As you can see, the clip hasn't changed much since frame 0 -- she is exclaiming 'mmmm...!' at this can which later will be animated in her hand. I thought this clip would be best to start with. This is also in better focus. Are you talking about some kind of 'select color range' within a selection in the frame? That would be cool! But would also be cool if it could be automated! I'd like to learn the method.

 

The selection method with lasso, blur, etc. with an alpha, so far seems to work well on individual frames. I hope to get something basic maybe done by SUN (I work an 8 hour job that has nothing to do with animation... something I hope to change some day) to at least see if the edging varies too much to be noticable... my biggest fear at this point. There is a martial arts film that comes to mind in doing this technique where this actor who was supposed to combat himself and you could see the actual cut lines were made whenever the actor crossed over with his other self -- it was very noticable. I don't want something like that if I can help it. I figure this is something very basic that needs to be mastered if DV and CG are going to be mastered as a mix. I appreciate your help.

 

 

Rob Milliken

[heyvern]

I just did a quick test.

 

Man! If only that can was bright green.....

 

Tip for the future. If you ever try this again, anything you want to "mask" should not be in the same color range as stuff you want to keep.... like skin for instance! The can has the same color values as some of the areas of the hands. In order for an action to do its magic it has to be "hands off", no painting in a specific spot if you follow. If the can were green you could easily grab a mask from the magenta red hues, manipulate the levels, do a little blur, reduce the noise, and bingo.

 

Now if you used a Sprite can it would have only taken a minute to do....

 

I have to go out for a while this afternoon unfortunately. I will try to play with it a little more and see if I can "find" the mask in the image. It is there somewhere you just have to "tweak" it out.

 

The magic wand works great but the action records it as specific pixel locations. I thought it would use colors. So if I find a similar "trick" to find the colors it might work.

 

I hope I can post an action file for you later.

 

Vernon Zehr

[heyvern]

Well, I think I got it.

 

First off, this is a Photoshop 7 action file. I am not positive but it may work with 5 or 5.5.

 

To import just get the Action pallette up in PS, click the triangle, right top. It says Load Action. Select this and select the action you downloaded from here.

 

You should get a new folder in your actions pallete called robs_can_movie.atn. Open this folder and inside is the action (robs_can).

 

You should test it on a few frames just to be sure. I did a bunch of "seemingly" wierd commands in the action to get the needed mask.

 

The big thing I did was convert to LAB color (don't worry I switched it back to RGB at the end). I did this because I was able to get a better selection from the color range command in LAB.

 

When you import the action there are 3 commands at the bottom not "checked" (they won't do anything if they don't have a check).

 

The first unchecked command is "invert". This can be ignored unless you have unexpected results. Let me know if the alpha isn't being created properly. There are a couple of spots that might cause trouble depending on how channels are set to display. I am pretty sure it will work fine.

 

The second command that is unchecked is the "Save" command. This is the save as targa step of the action and will allow you to batch save images in one folder to another folder as 32 bit targas with an alpha.

 

The third is the close command. It closes the document after it is saved as targa. You need this and the save command to be checked (on) ONLY when batch processing. (you don't want 300 files open in photoshop at the same time right?) If you are just testing the action keep these items unchecked.

 

To run a batch make sure "robs_can" action is selected in the action pallete then under file/automate/batch... you can choose a source folder and a destination folder. When you choose the destination folder (create a new folder for this) there is a check box to Overide action "save as" command. Check this box, you will get a warning but just hit okay.

 

When you are ready and hit start or whatever, it will open each image in the source folder, run the action and save it to the destination folder. This may take a few minutes... go get coffee, watch some tv whatever.

 

Phew! I hope this works. Don't worry I spent more time writing these instructions than creating the action.

 

Vernon Zehr

robs_action.zip

[zacktaich]

wow, great results on that last one. I've forgotten how to make selections with paths, it always ends up just filling my paths in w/ some color. I don't really like when it does this. I'm a bit of a photoshop n00b.

[robmilliken]

The primary reason I didn't use a standard chromakey color on the can is because with my initial tests, yellow and greenish colors especially the neon ones bled onto the hands. On purpose, I selected the same color as the primary color of the can. At the time, I thought I was going to use cookie cutter method, but that Alpha blur technique rocks! I don't think project would look the way it now does without that suggestion. I think the 'cut out the hand' part, though done with mag lasso and wand is about as perfect as it can get. I think if it was green, the green would bleed through part of the blur. Though I understand for keying out a color for a mask it is a pain. I took the time to do it though on Saturday, before I saw the lastest post on actions. I do thank you for the time. I want to spend time when I am wide awake (as opposed to right now) to really study what you did, because I am sure it has good application.

 

Now that I am on to getting the can to match the hand movements I am finding a new pain in the arse. I really DO wish I had marked a bullseye on the middle uppermost part of the can or something so I can track it for placement easier. I have seem to spend as much or more getting placement right than selecting the hands. Would be cool if there were a mini version of Icarus for AM, just so you can monitor the live action pixels and constrain the X and Y (I can handle Z) of a CG model with it. Kinda a constrain to live action feature.

 

Here the attached is the latest with ALL 38 frames! Compressed into DIVX format. Work to be done: can placement (?! - I have seen this 1 second footage for sooo long, I'm really not sure how much more I can do to change), definately shadow work under the hands - perspective isn't right though I haven't spent much time on it yet, getting focus/lighting right -- I think it's about there - maybe just minor adjustments.

 

Enjoy! My daughter has 14.59 minutes of fame left!

 

Rob Milliken

finaldivx.zip

[johnl3d]

she's a good actor...I love the refrigerator with the magnets..reminds me of ours..looks pretty good to me so far....

[heyvern]

The primary reason I didn't use a standard chromakey color on the can is because with my initial tests, yellow and greenish colors especially the neon ones bled onto the hands....

Yes,

 

but on the other hand with the techniques I use adjusting the channels I probably could have eliminated the bleeding of the colors.

 

It is a matter of extremes and similar pixel colors. The purple can color had almost the same color values as the skin tones. Even so, I was still able to get a mask. The main difficulty was the shadow areas. The shadow color between the fingers and below the hand on the can were identical.

 

I hope to see if my action works someday!

 

I was not sure if it would be effective on the whole series.

 

For positioning, you could use a point in the video frames as a reference, then constrain the can to a null with an offset and use the null to line up with the reference point.

 

Vernon Zehr

[robmilliken]

More!

 

Here is another 1 second fragment of what I am working on - here the 'legs' are coming out of the can. I used a lot of the techniques I used that Vernon helped me with on the last clip. Again, this is more mixing CG with live action. I think this looks a little bit better than the last clip -- it certainly went faster -- due to more experience?

 

Shadows jumping all over the place and cg can placement across a moving live hand are a constant concern of mine. Though only a second, I keep looking at it over and over and see glaring problems, though a casual viewer may not catch it since a second goes by pretty fast. I am VERY glad I used a purple prop same as the CG can for this clip, as you can see the reflection matching nicely on some of her fingers. Green or yellow would not have worked as well. It's the little things.

 

In regards to the last post. Pick a point, make it null then constrain the prop to that point? Wouldn't you have to make the null point do then the same thing the can would do? I am absolutely sure I am missing something. For the first clip I did earlier, I did pick a point on the lettering on the can and it helped to keep the tip of her finger on it.

 

Anyone know of a method to make the shadows that clearly flicker here blurred or something so they don't have the obvious look of being cut out -- I did a blur from frame to frame, but that didn't seem to help.

 

Thanks for helping so far!! Hope some other people are learning from these tips too!!!

 

Rob Milliken

divxfinal.zip

[robmilliken]

This project is no longer a WIP, it is finished. See the entry, "Laugh Pack Attack" in AM:FILMS for the final results that could not have happened without the people in this forum who offered their assistance.

 

Rob Milliken.