Netrender Event Commands (Running additional scripts/programs automatically)

[Rodney]

I want to add some information related to Netrenders Event Commands that allow us to run batch files and programs after a job completion (or frame completion).

This process can allow use of other programs to post process output from Animation:Master's Netrender and help us understand how we can do the same basic process without Netrender as well.

 

There are three basic components of this process:

1.  Set up the Event Command in Netrender

2. (Optionally) structure a (textual) batch file to take arguments from Netrender and pass them on to other programs

3. Run the desired programs and processes

This does assume a basic understanding of running Hash Inc's Netrender and how to use a text editor (to create batch files)

 

Setting up the Event Command in Netrender

 

Here we've selected Job Completed rendering as our trigger event.

Commands added here will be executed upon successful completion of a rendering job.

Note the arguments/variables that Netrender can pass to other programs; pool name, job name, time of completion, number of frames rendered, elapsed time of rendering and the output folder files were rendered into.

In this example our command will pass all of these arguments to a Windows batch file where we can then use those as needed.

Note the specific formatting of the path where the program, in this case a batch file is identified in quotes with extra back slashes to account for one slash being an escape character.  If or location was deeper in the directory structure we might need to repeat that pattern thusly:  "F:\\deep\\deeper\\stilldeeper\\runme.bat"

The arguments we will use as variables later are then added after that command:  %p %j etc.

 

The (Optional) Batch File

A Windows batch file is simply a text file with the .bat extension that can be used to run useful commands at the command line.

Note that a batch file can be ran independently of Netrender so in many cases we might simply run the batch file rather than wait for Netrender to run it.  Here we do want to pass information from Netrender to other programs so we want to take advantage of that capability in batch files.

So we open our favorite text editor and create a text file named "runme.bat". (as that is referenced by Netrender via the Command Event.

In our runme.bat batch file we might create something like the following:

 

I won't explain everything here but the important part is that we are allowing Netrender to run a script that in turn can now run other programs.  Here we take the passed arguments from Netrender (the %1, %2, %3 and other arguments) and use them to set up variables we can use elsewhere in our batch file.  In this way we can refer to something recognizable like "UsethisVery SpecificVariableName" rather than %8 which we may forget what it references.  Note that %0 references the current program being run which in this case is our batch file.

So next we use 'set' to store the arguments from Netrender as recognizable variables.

Here I've used the echo command just to display information to the screen.

In the case of our story this might be the program we run where we use those variables.  Each time the batch script encounters a variable it uses that variable's value in its place.

I've added the pause command at the end to make sure the user has a chance to see everything and acknowledge the information before the program closes.  Pause can be given specific messages after the command but by default simply asks for a key to be pressed in order to close the program.

 

Running the Program (Program Output)

Here we see what our automated batch script has produced:

 

 

Rather than just output text or information we might prefer to convert an image sequence from PNG images into a GIF animation, or an MP4 video... scale images up/down... run backups... feed the cats... or any crazy little thing we can dream up.

 

Running some of these useful options, such as using FFMPEG to modify, convert and merge images, video and audio is what we will try to explore next.

[Rodney]

For anyone that wants to lean forward a little here is a batch script (to be named runme.bat) that targets all PNGs in the target directory to create an MP4 video using FFMPEG.  If successful in creating the MP4 file it then zips up the PNG files (using Windows Tar.exe utility).  If the zip file is created successfully it then deletes the PNG files.

Disclaimer:  This batch file is more complex than it really needs to be but I started to dive into error checking which allows for stripping of special characters from the variables passed by Netrender and checks to ensure files are properly created before continuing and especially removing/deleting files.  The 'setlocal enabledelayed expansion' is new to me as well and presumably allows for the most current value of variables at time of command execution.  I need to research more to determine how to strip all special characters out of a variable but it might be better to simply... simplify. 

TODO:  I would like to master timestamping of files to better facilitate archiving and backups.  Gotta deal with those special characters...

@echo off
setlocal enabledelayedexpansion
cls
echo This is a batch file running with the following variables:
echo %0 %1 %2 %3 %4 %5 %6
echo.

set program=%0
set pool=%1
set job=%2
set time=%3
set frames=%4
set elapsedtime=%5
set outputfolder=%6

rem Strip quotes from variables
for %%I in (program pool job time frames elapsedtime outputfolder) do (
    call :stripQuotes %%I !%%I!
)

echo program %program% (batch file)
echo pool %pool%
echo job %job%
echo time %time%
echo frames %frames%
echo elapsedtime %elapsedtime%
echo outputfolder %outputfolder%

REM Run FFMPEG with the dot 4 digit wildcard pattern (image.0000.png)
ffmpeg -framerate 30 -i "%outputfolder%\image.%%04d.png" -c:v libx264 -pix_fmt yuv420p "%outputfolder%\output.mp4"

REM Simplified error checking
if exist "%outputfolder%\output.mp4" (
    echo MP4 file created successfully.

    REM Create a zip archive of all PNG images
    echo Creating zip archive of PNG images
    tar -a -cvf "%outputfolder%\images%job%.zip" -C "%outputfolder%" *.png

    REM Check if the zip file was created successfully
    if exist "%outputfolder%\images%job%.zip" (
        echo Zip archive created successfully: %outputfolder%\images%job%.zip

        REM Optionally delete the original PNG files
        echo Deleting original PNG files
        del "%outputfolder%\*.png"
        
        echo PNG images archived and original files deleted
    ) else (
        echo Failed to create zip archive: %outputfolder%\images%job%.zip
    )
) else (
    echo Failed to create MP4 file.
)

pause
exit /b

:stripQuotes
set "%1=%~2"
exit /b

This script does expect to find FFMPEG so if not in the Windows Environmental path it should probably be placed in the target directory or other location where it can be found by the script.  We could also hard code the exact path to FFMPEG.

 

[Rodney]

Here is what we might expect to see in the console window if our project file is set to render 24 frames:

This is a batch file running with the following variables:
"F:\runme.bat" "Pool1.rpl" "TheJob" "06/24/2024 12:56 AM" 25 " 0:01:26" "F:\am\renderfolder"

program F:\runme.bat (batch file)
pool Pool1.rpl
job TheJob
time 06/24/2024 12:56 AM
frames 25
elapsedtime  0:01:26
outputfolder F:\am\renderfolder
ffmpeg version N-112991-g081d69b78d-20231215 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13.2.0 (crosstool-NG 1.25.0.232_c175b21)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-gpl --enable-version3 --disable-debug --enable-shared --disable-static --disable-w32threads --enable-pthreads --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libharfbuzz --enable-libvorbis --enable-opencl --disable-libpulse --enable-libvmaf --disable-libxcb --disable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-chromaprint --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libaribcaption --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libvpl --enable-openal --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags= --extra-libs=-lgomp --extra-version=20231215
  libavutil      58. 33.100 / 58. 33.100
  libavcodec     60. 35.100 / 60. 35.100
  libavformat    60. 18.100 / 60. 18.100
  libavdevice    60.  4.100 / 60.  4.100
  libavfilter     9. 14.100 /  9. 14.100
  libswscale      7.  6.100 /  7.  6.100
  libswresample   4. 13.100 /  4. 13.100
  libpostproc    57.  4.100 / 57.  4.100
Input #0, image2, from 'F:\am\renderfolder\image.%04d.png':
  Duration: 00:00:00.83, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: png, rgb48be(pc, gbr/unknown/unknown), 200x200, 30 fps, 30 tbr, 30 tbn
File 'F:\am\renderfolder\output.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0000023141102c80] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0000023141102c80] profile High, level 1.2, 4:2:0, 8-bit
[libx264 @ 0000023141102c80] 264 - core 164 - H.264/MPEG-4 AVC codec - Copyleft 2003-2023 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'F:\am\renderfolder\output.mp4':
  Metadata:
    encoder         : Lavf60.18.100
  Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p(tv, progressive), 200x200, q=2-31, 30 fps, 15360 tbn
    Metadata:
      encoder         : Lavc60.35.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
[out#0/mp4 @ 000002313eec1500] video:28kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 3.984505%
frame=   25 fps=0.0 q=-1.0 Lsize=      29kB time=00:00:00.76 bitrate= 313.7kbits/s speed=16.6x
[libx264 @ 0000023141102c80] frame I:1     Avg QP:23.13  size:  2367
[libx264 @ 0000023141102c80] frame P:13    Avg QP:27.04  size:  1478
[libx264 @ 0000023141102c80] frame B:11    Avg QP:27.13  size:   606
[libx264 @ 0000023141102c80] consecutive B-frames: 28.0% 40.0%  0.0% 32.0%
[libx264 @ 0000023141102c80] mb I  I16..4: 34.3% 42.0% 23.7%
[libx264 @ 0000023141102c80] mb P  I16..4:  0.6%  7.0%  1.4%  P16..4: 22.9% 26.1% 15.0%  0.0%  0.0%    skip:26.9%
[libx264 @ 0000023141102c80] mb B  I16..4:  0.2%  1.8%  0.2%  B16..8: 37.1% 17.4%  5.5%  direct: 1.9%  skip:35.8%  L0:40.8% L1:43.2% BI:16.0%
[libx264 @ 0000023141102c80] 8x8 transform intra:63.2% inter:61.5%
[libx264 @ 0000023141102c80] coded y,uvDC,uvAC intra: 62.8% 63.9% 34.4% inter: 24.8% 18.5% 3.2%
[libx264 @ 0000023141102c80] i16 v,h,dc,p: 72%  1% 18%  8%
[libx264 @ 0000023141102c80] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 31%  9% 21% 10%  4%  7%  3%  6%  9%
[libx264 @ 0000023141102c80] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24% 16% 23%  6%  7% 10%  5%  3%  7%
[libx264 @ 0000023141102c80] i8c dc,h,v,p: 59% 13% 20%  7%
[libx264 @ 0000023141102c80] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0000023141102c80] ref P L0: 80.0% 15.9%  2.9%  1.1%
[libx264 @ 0000023141102c80] ref B L0: 95.0%  5.0%
[libx264 @ 0000023141102c80] ref B L1: 99.5%  0.5%
[libx264 @ 0000023141102c80] kb/s:271.08
MP4 file created successfully.
Creating zip archive of PNG images
a image.0000.png
a image.0001.png
a image.0002.png
a image.0003.png
a image.0004.png
a image.0005.png
a image.0006.png
a image.0007.png
a image.0008.png
a image.0009.png
a image.0010.png
a image.0011.png
a image.0012.png
a image.0013.png
a image.0014.png
a image.0015.png
a image.0016.png
a image.0017.png
a image.0018.png
a image.0019.png
a image.0020.png
a image.0021.png
a image.0022.png
a image.0023.png
a image.0024.png
Zip archive created successfully: F:\am\renderfolder\imagesTheJob.zip
Deleting original PNG files
PNG images archived and original files deleted