Experiments in inter app video sharing. In other words, glorified screen capture, staying on the GPU.

Update: Ive released the plugin.

QT7 required.

What you are seeing here, is Processing, and Max/MSP Jitter being captured in realtime, at full resolution and framerate into v002, and having their output be effected as a source input into v002.

Edit: Fixed some issues so it works within VDMX. VDMX + P5 example.

p.s. in ur face girl!1!36336

13 Responses to “Source-ry”

  1. Klangfreund Says:

    This looks awesome! Can you talk a little bit more about it? Is the whole GUI of OSX OpenGL? Like every window on its own plane? Does this even matter for this capturing you do here? The whole video-stream is handled on the GPU. Is the keyword pixelshader or is there another essential “thing” you’ve used for capturing a region of the screen? I guess the area of interrest must be visible on the screen and can’t be covered by another window.
    Again, great work!

  2. dan winckler Says:


    You know how long I have been waiting for this. At least three years, I think.

    Will you marry me?

  3. dan winckler Says:


    In a nutshell, yes, the whole GUI is sent to the graphics card as textures by the Window Server. The basic technique is demonstrated in the SonOfGrab developer example ( Apple added a new, very fast screen grabbing technique in Leopard, the CGWindow API.

  4. vade Says:

    Actually, thats somewhat backward. The plugin is using the old method, basic screen scraping: Its pulling a sub texture off of the front GL Buffer, which means just stuff that is on screen. So windows have to be completely visible, but, since things are staying on the GPU, it is *very* fast.

    The new OS X 10.5 function, CGWindowListCreateImage() returns a Bitmap CGimage from a window id, in other words, the window server transparently reads a texture off of the GPU, and creates a CPU image for you. This is *slow*. However, it has some advantages:

    The target window can be FULLY occluded, meaning, it can be behind another window, and you still get ONLY your target window – not the window above it.
    The target window can be mostly off screen, which saves TONS of screen real estate.
    You can restrict the image from getting stuff you dont want, so you dont have to mess with alignment.

    However, its really fucking slow :) This is basically iShowU, but without the saving to a QT movie, it just keeps it on the GPU.

  5. Klangfreund Says:

    Dan Winckler and vade, thank you for your explanations!

  6. Jonas Jongejan Says:

    Hey, this looks awesom! Are you going to release the source code? Im very interested in porting it for other opengl applications, like OpenFrameworks. Would be very cool.

  7. Create Digital Motion » Inter-App Video: A Mac GPU Hack, More Ideas? Says:

    […] Source-ry [] […]

  8. vade Says:

    Yup. Hopefully ill release everything, source included, VERY VERY soon now :)

  9. amon robe Says:

    wow great! if i see correctly it´ll work with maxmsp too? if so yes!!!!

  10. Jonas Jongejan Says:

    Thank you! I got one (major) bug. If i got more then one monitor attached to the computer, and a open the qtz file in VDM, or another host app, it crashes! But if i have one monitor on, turn on the quartz plugin, and then attach another monitor it does work (but it cant grab from the secondary monitor)

  11. vade Says:

    Jonas: Odd. Ive tested this with multiple monitors on my system. Are you using two graphics cards, or one card with dual output? I suspect the former will crash, but the latter should work.

    I *may* have a fix for this, but this has to do with ‘virtual screens’ and how OS X and OpenGL interface with more than one ‘renderer’. If the OS has to maintain more than one virtual screen, it wont work. this is the case with more than one graphics card. The reason for this is because the two different monitors are on two different pieces of hardware, and resource sharing across them, the way I am doing it, (for speed), wont work.

    Let me know, im curious if this is the case. Thanks for the report.

  12. Jonas Jongejan Says:

    Ahh ofcourse. Yes my setup was multiple graphiccards. Makes good sense. Is it possible to make it more advanced, so that that i wont crash if i got multiple graphiccards active, but ofcourse only being able to output to the same graphicscard? And maybe giving me the ability to have multiple instances of the plugin on different cards doing each their own work. (i mean, GPU 1 ca grab form GPU1, and GPU 2 can grab from GPU 2 etc).

  13. vade Says:

    That sounds reasonable. I may have a rig I can test on too. I have some idea on how I can handle this. Thanks for confirming.

Leave a Reply