Ive been busy lately with many projects. The v002 application which has been hinted at is coming along very nicely thanks to help from ‘bangnoise’, aka Tom Butterworth. Tom wrote the lovely ‘real’ datamosh plugin for Quartz Composer, and has been amazingly helpful in many regards. I had the pleasure of meeting up with Tom in London. Lovely guy, and I am really happy to have his help.

I also have some newer plugins for QC as well as doing some graphics pipeline improvements to Open Emu. During this ive been posting small updates via twitter, if you care, you can find me @_vade on twitter, or email me at vade [ at ] vade [ dot ] info if you are interested in beta testing things.

Here is a taste of an Open Emu upcoming feature, and stay tuned for some updates to the v002 Movie Player which has some nice improvements and bug fixes thanks to Tom as well. Heres some retarded fun with Open Emus new Quartz Composer filtering pipeline…

Open Emu - New Filters/Scaler system from vade on Vimeo.

Ongoing OpenEmu development has seen a lot of changes recently. One of the exciting new features is leveraging Quartz Composer for post processing frames from the various console emulators using OpenGL. These filters allow high quality scaling using GLSL and whatnot. But a lot more fun can be had with the Quartz Composer backend…

This filter aims to emulate what gaming is like on an older CRT monitor.. literally. We borrow from Stellas new CRT emulation code, and build on it by adding iSight powered reflections in the CRT, and distortion to match the curvature of the tube.

This is just a relatively simply example of what can be done using Quartz Composer as a filter system, there are a lot more opportunities to have fun and adjust the output image to suite hard core tweakers and experimenters

Fun.

Just a sneek peak at an currently un-optimized filter/functionality.

Some images and video from LPM 2009. In short, awesome.

Outpt and myself will be performing at Live Performers Meeting 2009, and I will be dropping in on the Quartz Composer workshop, to talk about some advanced topics, (programming plugins, including Quartz Composer into a custom application), and potentially with some surprises.

If you will be at LPM, let me know.

More Urban Imagery:

A user suggested I add audio analysis to the movie player plugin. Turns out Apple has some pretty easy APIs to add FFT analysis to your QT movie playback code, so I implemented it tonight. Its a bit rough, but this is nice because it grabs only audio from the currently playing movie (meaning you could, in theory have 2 of these going and get different FFTs out), and runs based off of volume in the patch, so you can fade out your audio and the FFT respects that. Kind of neat!

Update: I’ve re-tested Resolume with the latest available update - 3.0.1 at the time of this writing, and some results have changed:

So there was a question on the VDMX forum regarding output quality of various VJ softwares, and questioning which is best. I decided to do a little test of Resolume Avenue, Modul8, VDMX and v002 (why not..?) compared to Quicktime player for quality, just out of curiosity. What I found was rather interesting for a variety of reasons. I wont pretend to be an expert at any of these softwares, so bear with me. If I make obvious mistakes, point them out!

Testing:

My test clip is a rendered out from Final Cut Pro 6.0.5 clip of bars (and tone..), with time code and my logo (because im like that). It may not be the best real world clip to use, but the choice mistakenly lead to an interesting revelation. Almost all commercial VJ software fucks up on handling non 1:1 pixel aspect ratios correctly, and none get the color right. DVCProHD is actually, if you are counting, 960×720 pixels, but the image is displayed at 1280. This apparently is enough to through off most Vj apps.

Quicktime Player

This is our reference. Lets jot some quantitative color values down. (I grabbed these from the Digital Color Meter in Application/Utilities)

• Light Grey: 186,186,186
• Yellow: 181, 193, 0
• Cyan: 0,192, 198
• Green: 0 198, 0
• Magenta: 219, 0, 198
• Red: 215, 0, 0
• Blue: 34, 0, 199
• Purple: 71, 0, 120
• White: 255, 255, 255
• Black: 0, 0, 0
• Pluge (dont ask), 7, 8, 8

Now, those values (if you are a video nerd…) seem whacky. Bear with me (or jump to the conclusion…). Otherwise aspect ratio is correct, and everything looks and plays fine as expected.

VDMX

The first thing I did was make sure my output settings were correct for the DVCProHD, and set them to 1280 x 720. Made a new project, a new media bin, a new layer, and loaded in my clip. The aspect ratio was definitely off, and I could not really get VDMX to display it at a proper 16×9 without going to ’stretch’ mode, which seems, to me, like a bug (reported).

Color wise, we are not to far off, but not on the ball:

• Light Grey: 179, 179, 179
• Yellow:179, 179, 0
• Cyan: 0, 179, 179
• Green: 0, 179, 0
• Magenta: 179, 0, 180
• Red: 179,0, 0
• Blue: 0, 0, 179
• Purple: 47, 0, 101
• White: 255, 255, 255
• Black: 0, 0, 0
• Pluge: 5, 5, 5

Hrm. Interesting. Thats not quite right (according to our reference). Video quality seemed fine, even though the aspect was off a bit. I did not notice any weird edges or odd artifacts in the image, but its hard to say without it being displayed with the right aspect ratio. David Lublin suggested I try various blend modes because VDMX can leverage raw OpenGL or Core Image pipelines internally. Switching between OpenGL Over and / Source Atop blend modes made no difference (which is good if you think about it). Setting the layer position mode to stretch looked just like QT.

Modul8

Now, getting Modul8 to actual display my clip properly was a real pain in the ass, I had to manually adjust the scale of the clip in the transform controls, as I could not get it to display correctly at scale 1.0 (or.. maybe thats 2.0 x in Modul8 lingo? I have no idea, the UI says 2 which is what it started with launching the app. How that makes sense I have no idea) . To actually fully display (it was being clipped on output initially, see the screenshot), I had to adjust the transform scale to somewhere between 1 and 2. However, once I adjusted the scale to see the full extent of the image, the aspect ratio was indeed correct. And the color, dead on with VDMX’s ‘interpretation’ of our quicktime movie.

• Light Grey: 179, 179, 179
• Yellow:179, 179, 0
• Cyan: 0, 179, 179
• Green: 0, 179, 0
• Magenta: 179, 0, 180
• Red: 179,0, 0
• Blue: 0, 0, 179
• Purple: 47, 0, 101
• White: 255, 255, 255
• Black: 0, 0, 0
• Pluge: 5, 5, 5

Qualitatively, the image looks softer, with some blurring/issues on the edges of diagonal lines, especially the text/timecode readout and my v002 logo. It looked almost as if the image had been re-sampled. You have to look pretty hard to see it, but it makes me think video was being scaled or processed at a set resolution. Thats speculation, I have no idea about the internals of Modul8. Its not huge, but it looks like its there.

Resolume Avenue

Edit: Pixel aspect ratio handling is fixed in Resolume 3.0.1., and the color results have changed: see the addendum at the end of the Resolume Avenue section

It took me some time to figure out how to set the current ‘Composition’ to 1280×720, and when I did, I was presented with something far stranger than I had anticipated: It not only messed up the pixel aspect ratio, but decided to fill in the rest of the missing space with the right hand side of the bars blue:

Color wise…

• Light Grey: 161, 161, 161
• Yellow: 161, 161, 0
• Cyan: 0, 161, 161
• Green: 0, 161, 0
• Magenta: 161, 0, 162
• Red: 161, 0
• Blue: 0, 0, 161
• Purple: 29, 0, 83 (looks more like dark blue here..)
• White: 255, 255, 255
• Black: 0, 0, 0
• Pluge: 0, 0, 0

The whole image was much darker, and seemed shifted. The lack of pluge means you loose definition / detail in your blacks. I dont know if this is indicative of using DVCProHD, but its not necessarily a good sign… the shift in the blue also means something else weird is going on. Perhaps they are using a custom YUV ->ARGB conversion path with different constants than DVCProHD expects (ie, HD Rec 709 if I recall).

Update to Resolume 3.0.1

Resolume 3.0.1 fixes the pixel aspect ratio issue, so thats moot. Good news is also either I was incorrect in my testing or 3.0.1 also changes some color output slightly. Either way the new values for the same DVCProHD file are :

• Light Grey: 179, 179, 179
• Yellow: 179, 178, 0
• Cyan: 0, 179 ,178
• Green: 0, 179, 0
• Magenta: 179, 0, 180
• Red: 179, 0
• Blue: 0, 0, 179
• Purple: 48, 0, 100
• White: 255, 255, 255
• Black: 0, 0, 0
• Pluge: 5, 5, 5

This is much better than previous versions, and the pluge results indicate that we dont miss out on details in our blacks, which some may find important. Good news, and puts Resolume Avenue at almost exactly the same values as VDMX and Modul8. Thats kind of amusing, but I guess this tells us the Quicktime API makes sure of this and no one is doing anything too crazy behind the scenes pipeline wise that would effect the color output.

v002

Ok, so, im jumping the gun a bit but this is going to be my open source VJ app, and the UI is not finished so no pictures. Anyway, it handles the aspect ratio correctly because I happened to load a DVCProHD movie a few weeks ago and noticed it doing what VDMX did, seemingly playing it back at 960×720 rather than 1280. In Quicktime 7, this is called your frame aperture mode, if you set it to presentation or production your issues tend go away, presentation being the preferred for a variety of reasons. Read here for more info.

Color wise, it shows exactly the same colors as Quicktime Pro does:

• Light Grey: 186,186,186,
• Yellow: 181, 193, 0
• Cyan: 0,192, 198
• Green: 0 198, 0
• Magenta: 219, 0, 198
• Red: 215, 0, 0
• Blue: 34, 0, 199
• Purple: 71, 0, 120
• White: 255, 255, 255
• Black: 0, 0, 0
• Pluge, 7, 8, 8

And the image seems identical to Quicktimes as well. Yay for me and shit.

Trust No One (except…)

Wait. How do we know Quicktime Player is the “right” reference? Those colors its displaying are the definition of weird. Those Bars are motherfucking COLOR BARS. THEY ARE REFERENCE. THEY ARE LIKE THE WORD OF GOD. NO ENGINEER WORTH HIS SALT WOULD PICK 181, 193, 0 FOR YELLOW. What the shit is going on?

Well, of course, when in doubt, blame Apple. Quicktime uses Colorsync to display and output its video, and not the proper video gamma of 1.8 with proper video color conversion matrices. If you’ve installed Final Cut Studio and open Quicktime Player preferences, it adds a checkbox to allow for Final Cut Pro Color Compatibility, which ’skips’ over Colorsync and uses Final Cut Pros color correction. Apple giveth. The faith is restored, for $1500.

QT Color with proper 1.8 Gamma and FCP compatibility enabled:

• Light Grey: 191, 191, 191
• Yellow: 191, 191, 0
• Cyan: 0, 191, 191
• Green: 0, 191, 0
• Magenta: 191, 0, 192
• Red: 191, 0
• Blue: 0, 0, 191
• Purple: 65, 0, 120
• White: 255, 255, 255
• Black: 0, 0, 0
• Pluge: 11, 11, 11

Goddamn it. It looks like no one correctly handles color properly. I just wasted your time didn’t I?

Final Thoughts

Something I noticed when testing these apps is, at one point, I had all of them open and playing video (VDMX, Modul8, Avenue, v002 and Quicktime), and VDMX consistently had higher frame-rate playback than any other app, including Quicktime Player. I could tell because I had incrementing time code. Thats huge.

Regarding the color accuracy and image quality, this is not something everyone is going to see, or notice, because firstly NO ONE USES DVCPROHD as a playback codec. I just happened to choose it since I had some bars on my drive, and then noticed the aspect ratio issue in some apps, so it piqued my curiosity. PJPEG, MJPEG, AIC, DXV etc all use a pixel aspect ratio of 1.0 so this is why no one has noticed. The only issue here is the component YUV to ARGB conversion, and the differences noted in VDMX and Modul8 are indicative of a lossy YUV->ARGB. Ask any seasoned After Effects user, they have seen this for some time.

Secondly, almost everyone is playing out to non-color calibrated monitors, or even worse, projectors, which are fucking impossible to get any real calibrated color out of unless you have a huge budget and lots of time on your hands. I think the lesson here is this is basically of a non-issue, and I only stumbled on some bugs in some apps handling of odd pixel aspect ratios. Try not to care about all this, and make with the pretty… I feel like I just wasted a lot of time.

Andrew Benson of Cycling 74 released earlier today a really badass GPU/GLSL based Optical Flow Implementation. “Oh wow that sounds nerdy” you say. Fuck you, it kicks ass, and lets you do things like this:

I ported it to QC with his permission, and will be releasing it soon on the v002 site. Thanks Andrew for letting me port, and for sharing it with the world. Badass shit brother your kung fu is strong! Here is a quick video demonstrating in Quartz Composer:

There was a recent topic on the QC list regarding antialiased rendering in QC. Due to the way Open GL pixel formats work, and how Quartz Composer (editor) creates its pixel format you dont typically have much room to maneuver. If you are rendering geometry, you are basically out of luck. However, if you are outputting an image, or are using 3D geometry in a render in image patch you could theoretically use the relatively newish OpenGL extension for Antialising for Frame Buffer Objects. The setup is ugly, and its a multistep process, but the results are nice. Here is a sample plugin (also posted to the Apple Quartz Composer Dev list) which shows one way of rendering MSAA + FBO. Because you are using an FBO target the lack of a multisample pixel format being specified the plugins host is not a killer. Just check for the required GL extensions (gl_ext_framebuffer_multisample & gl_ext_framebuffer_blit) , if they exist, go for it. You get antialiased output for very cheap.

In theory, with Kinemes unofficial API you could create an Antialised Render to Image macro patch. Someone needs to get on that ASAP. It would be hot.

Heres the plugin: Download: QCPluginFBOMSAA.zip

Here are before and after images

Interpreting Adobe Photoshop CS3 executable binary as imagery via the upcoming v002 Glitch File Reader QC plugin. Some interesting structures.