Frameworks, plugins, FBOs, MSAA, v002

Ive had my head down working hard on some new things, specifically for my v002 VJ application, refactoring code, optimizing, modularizing and so forth. Things not terribly interesting to post about, so ive been spending my time just banging away on problems when I can. Ive worked though some modularization issues, made a pretty decent plugin framework for the application to allow very easy development of new source plugins. So now doing weird and interesting things with the app is that much more straightforward.

One of the things other I have not been super happy about with most VJ apps (including older versions of v002 and v001) is the quality of the rendering, especially concerning antialiasing when drawing 3D geometry. Ive taken some time to learn about modern rendering pipelines, and have worked them into my re-factored v002 code base.

In Quartz Composer Editor, when you render geometry, there is no antialiasing happening anywhere. Just instantiate a cube renderer and pay attention – lots of jaggies on the edges. Its pretty horrible especially considering its almost 2009… But to make matters worse, VJ applications dont typically render directly to a window, most render to some sort or texture or buffer, so post processing effects can be applied, like blur, color correction and the like. Getting antialiasing working in scenarios like this is tricky, especially because out of the box API for OS X dont really work with AA on the card. For example, it you can’t render to a Core Video OpenGL Pixel Buffer and get antialiasing. It just doesnt work. This means you have to roll your own solutions to the problem and get a bit deeper into GL land. Some folks beat this by using super sampling (just rendering to a larger buffer, but this kills performance, and is a general bad idea – especially if you do it in Quartz Composer using Render in Image, which is ridiculously slow).

In OS X, it turns out there are two ways to approach the problem, one OS X specific and one using straight OpenGL. The first way is to use a hidden window, with a shared OpenGL context, and ‘screen scrape’ the surface window texture to your main context using [NSOpenGL createTexture: fromView: internalFormat] method call. It took me a while to figure this one out, and it works. Now to get antialiasing, you have to make sure to enable MSAA on your main and shared contexts pixel formats. You can now “render to texture” with antialiased context, and it will work on basically every card OS X supports. The code setup is its fairly complex, it requires a copy stage using who knows what internal mechanism (its internal mechnism is private), shared contexts and hidden views, limited to OS X only, and is really not the proper future GL path.

The other solution is to render to a frame buffer object, which has a multi-sampled render storage buffer attached. You then render you scene into this frame buffer, and blit out its context to a second FBO which has a texture color attachment. This sounds complex, and is only supported on newer hardware, but its really the ‘right’ way to handle it. Added bonuses are you do not need a hidden window, a shared context and any slower copying mechanisms. Its fast, cross platform, and has will only get better as newer cards support multi-sampled textures, you can remove the whole blit stage for added speed. Either way, FBO/MSAA is fast, much faster that using CVOpenGLPBuffers, or Quartz’ Render in Image patch. Another nice thing is you can easily fall back to a regular FBO pipeline without antialiasing very easily, should the card not support MSAA.

Attacking problems like these are important (if somewhat boring and not particularly glamourous) but is adds polish, and I think it makes a subtle but drastic difference for those interested in quality of their output. So, whats it look like?

Here is a shot straight from Quartz Composer, showing the teapot and a cube as reference:

QC output

Note the ugly jaggies on the edges. Really not pleasing.

Now from v002 with render to FBO with multisample antialiasing:

v002 output

Pretty huge difference, with the same FPS. How much does this cost us GPU wise? Lets check out GL profiler:

Here is Quartz Composers OpenGLMonitor profile as a reference:

QC GL profiler

Note the huge spikes in the “CPU wait for GPU”. Im not sure what that is all about, but its there… And v002s:

v002 GL profiler

Yup, this is basically for free folks :) Awesome. So this is just a taste of the type of nerditry I’ve been up to lately…

5 Responses to “Frameworks, plugins, FBOs, MSAA, v002”

  1. toby*spark Says:

    thou rockest mostest.

    this takes me back to the era of softvns… it simply cared about the output, and even processing at 320×240 apps that used it for their rendering would trump others doing things at twice the resolution.

  2. vade Says:

    Thanks toby :)

    While I never really used SoftVNS that much when I was in Max/MSP land, I’ve heard only amazing things, and have seen its performance and capabilities first hand. I really cant wait to get v002 to a point where I can release it in somewhat workable/usable state. It wont try to have nearly as many features as other apps, but it can grow in to it. I want quality, speed, agility, fast adaptation for improvisational performance and light-weightiness to be paramount. A usable, interesting and understandable UI for performers is also very important to me.


  3. George Toledo Says:

    Interesting observations to say the least.

  4. fsk Says:

    cant wait to see this app :).

  5. ellimac Says:


    happy new year !!!

    i ‘m also developing a vj software that integrate quartz compositon , quicktime movie , shockwave flash animation and sur the quality of rendering is primordial(je sais pas comment on dit en anglais).i’ve got my answer to get anti-aliasing on quartz now i try to setup FBO in my app (arrrgss!!!)


Leave a Reply