Steve & Chris Rear Window

[mouseman]

Steve posted a teaser render in the main thread. Here's a WIP shaded render of what we have so far for animation. No bells and whistles (i.e. no hair), and a lot of the walking at the beginning will be edited out, but this sets us up for the rest of the animation.

 

2 weeks to go! And a lot of animation to go! I'm glad Steve picked up a bunch of render nodes!

Blocking1_Shaded7.mov

[fae_alba]

that is looking good!

[TheSpleen]

nice!

[largento]

Looking good! I felt the weight when he climbed the stairs.

[Gerry]

I want to know what happens next!

[Shelton]

Walk sequence. Still working with cape

 

Steve

walk.mov

[Walter Baker]

Oh ya beautiful job, up the steps and the walk, nice

[kwhitaker]

Just saw this, even without any bells and whistles it looks great. makes me wonder-whats in the package.

[mouseman]

Just saw this, even without any bells and whistles it looks great. makes me wonder-whats in the package.

The important thing is what isn't in the package!

[Wildsided]

The way the guy turns off the TV before he answers the door would indicate he's watching something he doesn't want others to see and whatever the delivery guy is dropping off is in a plain brown package. I'm thinking he's getting a bit of x-rated entertainment.

[TheSpleen]

The belly on him is priceless!

[Kamikaze]

Well, that first tease was leaving me wanting more .......Hope my simple but skit won't cause the project any let down...smile...

[Vertexspline]

Chris ....thanks for the sneek peek......... I think it is coming along very very nicely . Its got great storytelling for 15 seconds long or so ----that's awesome ---and yes like others ----cannot wait to see what is going to happen.......

[Shelton]

Well netrender is killing me.

 

I sat there and watch it endlessly. It is addictive!

 

Here are a couple of shots of the progress. Robert and Paul I may be a couple of days late. It states I have another 36 hours to render with 32 cores. I still have to add 4 more machines to netrender

 

Steve

[Rodney]

You'll be the ultimate render wrangler when this is finished.

[Shelton]

I have a total of 84 cores available. All the machines are decent machines to full blown gaming machines to dell precision workstations. I have learned a lot and will pass on some issues with netrender that I have to work around while the renderings are going on,

 

But it has been fun. I have made a good friend in this process in Chris with our collaboration of the Rear Window. I think we have both learned a great deal in the process and it has been well worth the effort. We are ready to begin our next project together

 

Steve

[robcat2075]

Well netrender is killing me.

 

I sat there and watch it endlessly. It is addictive!

 

Yup, I've don't that too.

 

 

 

Here are a couple of shots of the progress.

 

That looks great.. Can you get an actual screencapture of that screen? That would be cool to post just for people to salivate at.

 

 

Robert and Paul I may be a couple of days late. It states I have another 36 hours to render with 32 cores. I still have to add 4 more machines to netrender

 

Fine by me!

[Shelton]

Robert I will get all the machines into the netrender que and then get a screen shot

 

Steve

[fae_alba]

I have a total of 84 cores available. All the machines are decent machines to full blown gaming machines to dell precision workstations. I have learned a lot and will pass on some issues with netrender that I have to work around while the renderings are going on,

 

I really want to hear your comments about setting up your render farm.

 

 

But it has been fun. I have made a good friend in this process in Chris with our collaboration of the Rear Window. I think we have both learned a great deal in the process and it has been well worth the effort. We are ready to begin our next project together

 

Steve

 

For me, this is the primary reason for these projects!

[Shelton]

I have a Dell 24 port managed switch and have found that it is not as critical in a rendering situation where each frame takes a great deal of time to have a gig switch. Render server is not as taxed in this scenerio. But for those projects and rendering where the frames are around 2 minutes or less to render, it is very critical to have the fastest switch you can get! When you have more than 20 cores render server can really get bogged down sending the file information to the slaves when they are rendering each frame quickly. Escpecially if you have render slaves running on the machine that is hosting render server, which I do. That machine is a 6 core amd 1090t with 32 gig of ram and it has trouble in that situation. What would be nice under the license if render server could be place on a machine designed to host render server alone.

 

Anyway my thoughts on that.

 

What I have found in this render is that the AMD machines are being killed by the Intel machines. In fact, one machine I decided to throw into the pool is an older Core 2Duo E 8400 3.0 ghz with 8 gig of ram and it out performs any of my other machines i7, fx 8150 bulldozer (8 cores). It is rendering 1.75 frames per every 1 frame of a 3.6 8150 amdfx machine. The i7 is closer. I may be way off but a machine with more than four cores is struggles to keep up with a dual core machine. I wonder if render slave is not optimized for multiple cores such as the amd 1090t (6 cores) or amd 8150 (8 cores). Also some of the four core machines that have 16 gig of ram will have two cores running well and 2 cores that are struggling.

 

Just somethings I noticed. Will keep you updated.

[Shelton]

First half completed. Only one machine gave me difficulties during the render, 8 core. Anyway will begin moving data off the render network and start tomorrow on the second half

[Shelton]

I have found on microsoft's tech site that the AMD bulldozer technology broke somethings in Win7. There are several patches available off the amd website or microsofts site. It helps with the multiple cores and timing out issues. I hope this was what was causing issues with render slaves as it seemed the machine would just simply quit working.

 

Steve

[Shelton]

Hi Paul,

 

Will have the renderings shortly. We ran into a little snag. So bear with me a couple of days.

 

This project has generated renewed interest and passion for future projects. I started pulling out old stuff and thinking if I tweak a little hear and there.... Anyway thanks for letting us participate in this project. Looking forward to seeing what everyone has put together.

 

Steve

[Shelton]

Some finally thoughts on netrender.

 

It is an awesome piece of software. I can not imagine how long it would have taken on one machine to render this project.

 

Some things that really bothered me were the rendering times of machines. My workstation that I use to model and animate is an I7 3.2 4 core machine. When it finally got to rendering files it was slower than Core2 Duo Extreme processor QX9650 running at 3. In fact it was only the third fastest machine in the render pool. The core 2 duo E 8400 beat the I7 machine in times by 3-5 secs per render.

 

Also all of the Core2 Duo machines outperformed all of the AMD processors by almost 2 to 1. I put a Pentium D Dual core 3.0 with 6 gigs of ram running Vista 64 and it out ran all of the AMD machines except for one. The only AMD processor that kept up with the Core 2 Duo machines was the AMD 1090T 6 core processor. It was in 4 place by only 8 seconds per frame. In some frames it was the fastest computer in the pool. But overall time it was fourth.

 

What have I learned?

 

Patience

 

Timing of inserting the machine into the pool is critical. If a machine is placed in the pool while a great deal of traffic from netrender is going on then that machine may take a great deal of time to load the project or crash all together.

 

When the pool has more than 25 cores net render server seems to have issues when new machines are introduced.

 

Fastest switch you can buy is needed. Keeps the collisions of the netrender traffic.

 

Patience

 

Some machines no matter what I did in preparing and setting up the network would not load in netrender. I still had 5 machines that I did not use in the rendering process.

 

Take a little extra time to review settings before starting. Even though you can modify the project in the pool, its better to make sure you got it right before you enter machines.

 

Review the out come of frames while the rendering is going on.

 

Patience

 

I caught myself just watching the machines render, staring at the screen of netrender.

 

With all of that some what negative comments I want to say, I love the program. I believe it is essential for AM's future.

 

Steve

[robcat2075]

Some finally thoughts on netrender....

 

If you ever have time to summarize/organize this into an article or tutorial that would be a fine thing to do.

[mouseman]

Some finally thoughts on netrender....

If you ever have time to summarize/organize this into an article or tutorial that would be a fine thing to do.

Or maybe it could be added to MMZ_Timelord's write-ups.

[Rodney]

Fastest switch you can buy is needed. Keeps the collisions of the netrender traffic.

 

Can you elaborate on this a little?

What do you mean by this, fastest network switch?

And what exactly is that?

[Shelton]

I have several different switches i tried. Ended with a gigabyte network switch. I tried a "green" switch that seemed to go to sleep to save energy and this caused some issues. Even tried economical switch but had some issues. The best switch i used was netgear gig switch, business class. I will try a cisco switch that touts better collision resolution. I will keep you posted.

[Shelton]

My partner in crime, caught a huge error. Netrender appears to have not rendered several frames in one of th chor. So back into netrender.

 

Steve

[mouseman]

These are some of the general things we learned on this project. We thought we would present them here in hopes that others might find it useful.

 

 

Lessons Learned

 

GENERAL, PLANNING

 

Less is more. Don't add too much material. Longer video length means longer render times, and more to fix if there is a problem to be fixed.

 

Just do it. Get in and do it. If you mess up, it can be fixed or re-done.

 

Don't be afraid to show your work early and often. Have your work reviewed, don't be intimidated. (And on the reviewer side, focus on what could be added or plussed.)

 

Script - get specific ideas laid in as soon as possible. Having specific plot points is great, but you need the in-between stuff figured out, too, otherwise the characters are just wandering.

 

COLLABORATION

 

"One man, one machine, one movie" has its limits. You can do a lot more together.

 

Working with another person is a gold mine. Different point of view, different ideas for ways of doing something.

 

Video conferencing is invaluable. Skype has the best quality, but no longer has free screen sharing. GTalk/google hangouts is acceptable and has screen sharing, but is not as good for live movement (i.e. playing an animation). We may tinker with Microsoft Lync in the future.

 

Having a set schedule of times to talk is important. Although some flexibility is necessary (family obligations, etc.), regular interaction helps. We met twice per week.

 

REVISION CONTROL / SUBVERSION

 

Need to have someone that knows Subversion well, including how to deal with problems such as conflicts. There is a little bit of a learning curve. Need a host. The repository acts a location for backing up the project; if you lose your machine, you can re-get everything from the repository. Everyone needs to be up to date on how to do it; maybe have training sessions though GTalk.

 

Commit often so the others have access to your latest work. Commits are, to an extent, a measure of progress. Always add a comment.

 

It is important to create non-changing file names and not change them, not even for alternate versions. (One exception might be having a pre-exported version of a rigged character and a post-exported version of the rigged character.) References to assets from other files can get easily messed up and point to the wrong version. If you want to make a local backup that you don't want to commit, use explorer and hit CTRL-C to copy and CTRL-V to paste. Otherwise, just commit the current version of the file. You can always get access to older versions through SVN.

 

Multiple people cannot work effectively on a single choreography at the same time. There must be a hand-off procedure.

 

Versioning systems are good for creating versions of assets. However, it is not good for a general file sharing or transfer system. Consider using dropbox or an FTP site or something similar, although these often have storage or transfer limitations. The versioning system is also not a place to store or transfer copies of generated files (such as the final rendered animation sequence).

 

Always be mindful of what you are checking in. Animation:Master currently adds nonsense changes to files, especially choreographies. These have no real meaning, and can cause merge conflicts in the future that will need to be resolved. This is especially frustrating if one person is making significant changes to that file and the other person is not. Revert any changes you can't account for.

 

Non-changes that A:M keeps adding or changing:

A:M changes names of chor actions - adds a different number, e.g. FacialAnim3 becomes FacialAnim2 (when we never added a number ourselves)

Lots of empty ROTOSCOPE entries in choreography files.

CHANNELDRIVER

 

ANIMATION

 

Cloth:

Combine cloth to model, pre-roll. But you don't have to do it. No sudden movements; start in a neutral pose, let cloth relax, then do movements.

 

Rigging:

Don't be afraid to do your own sub-set of rig. We might be heading towards creating own face rig, but we'll try them all first. Shoulder and hips are still hard to weight correctly. Weighting is always hard. Remember to weight only to geometry bones (and fan bones).

 

Reuse models wherever possible, or rework them (e.g. we made a chair into a sofa and a single bed into a double bed). Don't model if it's not visible. (e.g. we had a door that wasn't in the set, but didn't need to be visible; we didn't model it, just made the characters act as if it were there.)

 

Animating helps us learn about how to model and rig better. For example, when you move some bones in a reasonable fashion and the mesh deforms in an undesirable way, it may be due to a bad choice for which direction the splines go. We will always model with a spline that goes from the side of the mouth to where the hinge of the jaw would be.

 

RENDERING

 

Create presets to do animations of the entire scene but with a high step value. The presets include start and end times and the step value, so you may need at least one for each choreography. Make sure enough options are turned on to be able to judge things like hair, particles, and lighting (or even custom to one feature). Avoid committing to a big render until you are happy with the quick animations.

 

For NetRender, Multiple machines and cores - understand the basics of networking itself. How to do a shared file/folder. The slave machines must have the exact same paths to the data files on all machines. If there is a problem, look for permission issues; make sure the client can create files in the data folders.

[Rodney]

Now THAT is an awesome Lessons Learned write up. Thanks for that.

 

A couple questions:

 

Non-changes that A:M keeps adding or changing:

A:M changes names of chor actions - adds a different number, e.g. FacialAnim3 becomes FacialAnim2 (when we never added a number ourselves)

Lots of empty ROTOSCOPE entries in choreography files.

CHANNELDRIVER

 

Is it possible that an effective workaround might be to avoid naming actions with a number at the end? Or perhaps spelling that number out (i.e. FacialAnimThree)? I'm curious if A:M (or SVN) sees that number as an increment and gets confused in the saving process. Avoiding the number on the end might resolve this. Of course, It'd be good to know more about this problem so that it can be reported to A:M Reports and squashed entirely.

 

 

Versioning systems are good for creating versions of assets. However, it is not good for a general file sharing or transfer system.

 

I'd like to know more about why you believe this is the case. Besides people's general aversion to using versioning systems I thought SVN was pretty good for maintaining collections of assets. I do understand that SVN is a centralized and not a distributed model like Git and Mercurial and that is a little problematic. But it shouldn't be too difficult to use SVN for file sharing with small teams. If one of the issues is that it's hard to keep track of what is new/old/updated at a glance by just looking at the directory structure then I think I know what you mean.

 

Thanks Chris! I'm glad to see successful collaborations like this.

[Shelton]

Rodney

 

The biggest problem was on my part in the SVN adventure. I would rename a chor or file by using a number such as BillFace2. Now I have broken the revisioning process. Plus I would forget to add this to the list to committ to our repository. Now not only does Chris not have the updated model I have not saved the revised version. Most of the issues of SVN was a lack of knowledge and procedures on my part.

 

Steve

[Rodney]

Rodney

 

The biggest problem was on my part in the SVN adventure. I would rename a chor or file by using a number such as BillFace2. Now I have broken the revisioning process. Plus I would forget to add this to the list to committ to our repository. Now not only does Chris not have the updated model I have not saved the revised version. Most of the issues of SVN was a lack of knowledge and procedures on my part.

 

Steve

 

Thanks Steve,

I recall a similar learning curve with TWO that made working on the film difficult at first because not only was everyone learning how to animate a feature film... tough enough(!)... now they had to get in sync with this revisioning system that did not seem to be easy to understand or use. Just diving in and using it would have been advantageous but I think several were distracted by this to the point of thinking, "I don't care about this stuff, I just want to animate." Learning at least the basics of how files will move through the pipeline is one of those necessary evils in collaborative works. It's different than a one man production where none of the resources have to be shared.

 

I do think SVN is pretty forgiving and as you've seen, the difficulty can be overcome.

I've stared at other revisioning programs such as Git and Mercurial in an effort to see if they work better for more straightforward file sharing, which is what is important to the end user/animator. The versioning aspect is more important to the person who has a big picture perspective on all of the assets and where they are in relationship to the final product.

 

I think the current versioning software is primarily text based and in time we'll see a more graphical approach. I think this graphical element is the essential part of the process as it aides in the communication of the state of each resource and what changes are required to get where the project needs to go. Having this understood at a mere glance or during a quick review is the difficult part. Programs such as SVN are more concerned with the process itself than with the communication that is gained or frustrated by the user interface.

 

At any rate, I'm impressed that you've used SVN for this project. Bravo!

[mouseman]

Sorry for my delay in responding.

 

Non-changes that A:M keeps adding or changing:

A:M changes names of chor actions - adds a different number, e.g. FacialAnim3 becomes FacialAnim2 (when we never added a number ourselves)

Is it possible that an effective workaround might be to avoid naming actions with a number at the end? Or perhaps spelling that number out (i.e. FacialAnimThree)?

The thing I was referring to was not the names of choreographies themselves (though we had a different issue with that which I also mentioned later). The issue I'm mentioning is that choreography ACTIONS that were given a name without any number on them (such as "FacialAnim") would have a number appended to it at some point by A:M. Then it would later be changed in future saves. I realize I should probably put some effort into finding out what sequence of steps causes it and create a report. I believe it is in no part related to Subversion.

 

Versioning systems are good for creating versions of assets. However, it is not good for a general file sharing or transfer system.

I'd like to know more about why you believe this is the case. Besides people's general aversion to using versioning systems I thought SVN was pretty good for maintaining collections of assets. I do understand that SVN is a centralized and not a distributed model like Git and Mercurial and that is a little problematic. But it shouldn't be too difficult to use SVN for file sharing with small teams. If one of the issues is that it's hard to keep track of what is new/old/updated at a glance by just looking at the directory structure then I think I know what you mean.

A versioning system is good for things that need to have a history tracked. This is important in things like source code, where you need to be able to track down when a bug was introduced into the code (and who did it, what were they trying to accomplish when the bug was introduced, etc.).

 

A repository/versioning system technically could be used as a file transfer system, where one person adds and commits a file to the repository, and when the other person gets the file when they update their own working copy. But it's not an ideal use of the repository.

 

One disadvantage of a versioning system is that you have a copy of every version. This means every version uses disk space. While disk space is getting cheaper and cheaper all the time, the reality is that the disk that the central repository is on is of a finite space. Assuming there are backups of that server, the backup time takes longer as there are more files and takes more space. Also, if you check in a huge file, then others will be downloading it the next time they do an update, and might be quite annoyed with the wait. There is a "cost" of sorts for checking something in.

 

Obviously, you should check in your PRJ files, your CHO files, your MDL files, your texture files/images, and other assets directly needed to re-render your work. No question to that. Reference drawings or videos? Probably that's fine.

 

Now, if I render a test render with weird options to a sequence of PNG or JPG files and when it was done used them to generate an MP4 video, would those PNG or JPG files need to be checked in? Well, assuming everything needed to create them was checked into the repository, they are things that can be regenerated by exporting the old revision of the tree and re-rendering from that. You would probably not need to go back to reference that file after it was used to generate an animation. Also, there is little likelihood that you would need to differentiate frame 739 from one test render to another. There is no need to have a history of the generated file and there is no need to ever refer back to it, so it doesn't make sense to put it into the versioning system. Similarly, the video files of the test renders usually don't need to be kept for posterity, so putting them in the repository so the other person can see what it looks like is not an optimal use of the repository.

 

In short, any item whose value is known to be a very short time is not a good candidate for putting into the repository, and would be better handled by some other transfer means.