Pixel Bender tutorial

With the advent of Flash Player 10 at the end of 2008, Adobe opened up a new generation of image processing techniques using the Pixel Bender system. In its simplest form, Pixel Bender allows the Flash or Flex developer to run mathematical algorithms used for image processing (and in fact other tasks) at a greatly increased rate. Flash Player does this by offloading those calculations to the computer's graphics processing unit (GPU) which is often faster than the main CPU.

This tutorial demonstrates the use of the Pixel Bender system in Flex Builder 3.

Targeting Flash Player 10
The first thing we need to do is create a Flex project targeting Flash Player 10. This is pretty straightforward. Firstly ensure that you have the FP10 version of playerglobal.swc - this will be in sdks\3.4.0\frameworks\libs\player\10 of your Adobe Flex Builder installation directory.

Now, create your new Flex in Flex Builder. Under Project->Properties->Flex Build Path, choose the Library path tab, open up "Flex 3.4" and remove the existing playerglobal.swc entry. Then click "Add SWC" and select the playerglobal.swc in your sdks\3.4.0\frameworks\libs\player\10 directory above.

fp10lib.jpg

This will now allow you to use the FP10 library. You'll also need to make sure you create the HTML code to ensure the user is using FP10. Under project properties again, go to Flex Compiler and make sure the Required Flash Player version is set to at least 10.0.0.

fp10html.jpg

Pixel Bender Toolkit

Our Flex project is now ready to code. However, we will first need to create our Pixel Bender filter. This filter is not written in AS3, but a proprietary format to Adobe. To compile the filter code to something that Flex can use, we need to use Adobe's Pixel Bender Toolkit. You can download this from Adobe labs at http://labs.adobe.com/technologies/pixelbender/. Once installed, launch the toolkit and copy and paste this code into the bottom window:

<languageVersion : 1.0;>
 
kernel FilterTest
<   namespace : "Preinvent";
    vendor : "Preinvent";
    version : 1;
    description : "Passthrough filter - does nothing"; >
{
   input image4 sourceImage;
   output pixel4 dst;
 
   void evaluatePixel()
   {
      dst = sampleNearest(sourceImage, outCoord());
   }
}

Let's take a moment to review this code. The first line simply names the filter, in this case "FilterTest". The next block contains some meta data about this filter including namespace, vendor version and a description. The interesting bit is the code within the braces:

   input image4 sourceImage;
   output pixel4 dst;
 
   void evaluatePixel()
   {
      dst = sampleNearest(sourceImage, outCoord());
   }

Firstly we define our input and output params. In the case of Flex, we always have at least an input parameter of type "image4" and an output parameter of type "pixel4". These types are native to Pixel Bender. "image4" is an image containing "pixel4" types. "pixel4" i simply a pixel with r, g, b and alpha components.

Our single function, evaluatePixel() is where it all happens. Pixel Bender calls this function repeatedly or every pixel in the output image. Here it performs a nearest neighbor resample of sourceImage at pixel location given by outCoord (which is a built-in function) and assigns the value to dst. You can test this filter in the toolkit by loading in a test image (File->Load Image 1) and clicking Run. As you will see, the image does not change. This is because our filter simply returns the pixel value sent in.

So let's make it do something more interesting. Change the function to look like this:

   void evaluatePixel()
   {
      dst = sampleNearest(sourceImage, outCoord())*pixel4(1,0,0,1);
   }

You'll notice that we've added "*pixel4(1,0,0,1)" to the end of our calculation. This will take the output of the resample and multiply each pixel element by the value within our newly defined pixel4. For example, assume that a particular pixel as r,g,b,alpha values of "144,89,215,255". Multiplying this by pixel4(1,0,0,1) will multiply 144*1, then 89*0, then 215*0, then 255*1. This will return "144,0,0,255". The astute amongst us will notice that this effectively removes the green and blue channels. If you run this now, you'll see exactly this happening.

Now, let's get more interesting. Say we want to alter the level at which we modify the red channel. Firstly we need to define a new parameter to the script and define some boundaries. We will also modify our calculation to include that parameter:

<languageVersion : 1.0;>
 
kernel FilterTest
<   namespace : "Preinvent";
    vendor : "Preinvent";
    version : 1;
    description : "Passthrough filter - does nothing"; >
{
    parameter float redMultiplier
    <minValue: 0.0;
     maxValue: 2.0;
     defaultValue: 1.0;>;
 
   input image4 sourceImage;
   output pixel4 dst;
 
   void evaluatePixel()
   {
      dst = sampleNearest(sourceImage, outCoord())*pixel4(redMultiplier,0,0,1);
   }
}

You can see that we have defined the parameter redMultiplier to have a default value of 1.0 and min/max limits of 0.0 and 2.0. if you now run this script, on the right hand side of the toolkit window you'll see a slider that will let you change the redMultiplier value and instantly see the effect.

toolkitwindow.jpg

Let's do the same for the green and blue channels:

<languageVersion : 1.0;>
 
kernel FilterTest
<   namespace : "Preinvent";
    vendor : "Preinvent";
    version : 1;
    description : "Passthrough filter - does nothing"; >
{
    parameter float redMultiplier
    <minValue: 0.0;
     maxValue: 2.0;
     defaultValue: 1.0;>;
 
    parameter float greenMultiplier
    <minValue: 0.0;
     maxValue: 2.0;
     defaultValue: 1.0;>;
 
    parameter float blueMultiplier
    <minValue: 0.0;
     maxValue: 2.0;
     defaultValue: 1.0;>;
 
   input image4 sourceImage;
   output pixel4 dst;
 
   void evaluatePixel()
   {
      pixel4 multiplier = pixel4(redMultiplier,greenMultiplier,blueMultiplier,1);
      dst = sampleNearest(sourceImage, outCoord())*multiplier;
   }
}

Note that I have defined a new pixel4 on the previous line, to make the code more readable. Test this and you'll see that we can now modify the R, G and B channels.

Implementing in Flex

OK, so we can now write some basic Pixel Bender code. The next job is to get this doing something in Flex. Firstly, you'll need to save your source code as a PBK file. I've saved mine in the assets/pixelbender directory in my Flex project and called it "rgbchannels.pbk". What's important to note here is that Flex requires compiled code as a PBJ file - it can not compile it itself. In Pixel Bender Toolkit, choose File->Export Filter for Flash Player and save this in the same assets/pixelbender directory as "rgbchannels.pbj".

Now for the easy bit - Flex. Add an Image component to your MXML file and embed an image file. I've copied one of the Pixel Bender images as an example into assets/images:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute">
	<mx:Image x="10" y="10" source="@Embed(source='../assets/images/Canyonlands.png')"/>
</mx:Application>

Now we need to embed our PBJ file and create an instance of it:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute">
	<mx:Image x="10" y="10" source="@Embed(source='../assets/images/Canyonlands.png')" />
 
	<mx:Script>
		<![CDATA[
			[Embed(source="../assets/pixelbender/rgbchannels.pbj", mimeType="application/octet-stream")]
			private var RGBChannelsClass:Class;
		]]>
	</mx:Script>
</mx:Application>

Now we need to apply this filter to the image. We do this by adding a ShaderFilter as a filter to the Image component. I do this when the image component has been created. As you can see in the following code, we take the bytes of the PBJ file, pass them into the constructor of the Shader object, then create a ShaderFilter and add it to the list of filters to the Image component:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute">
	<mx:Image id="img" x="16" y="88" source="@Embed(source='../assets/images/Canyonlands.png')" creationComplete="imageLoaded()"/>
 
	<mx:Script>
		<![CDATA[
			[Embed(source="../assets/pixelbender/rgbchannels.pbj", mimeType="application/octet-stream")]
			private var RGBChannelsClass:Class;
 
			private var shader:Shader;
 
			private function imageLoaded():void
			{
				shader = new Shader(new RGBChannelsClass() as ByteArray);
				img.filters = [new ShaderFilter(shader)];
			}
 
		]]>
	</mx:Script>
</mx:Application>

If you run this, you'll see the unchanged image. This is because we're using the default values in our Pixel Bender code which returns the same pixel as the source image. We need to have the ability to modify the RGB multiplier parameters. We'll do this by adding some sliders. When the slider values change, the method "refreshFilter" is called which sets the slider values to our three parameters and forces a render of the filter by re-adding it to the Image filters:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute">
	<mx:Image id="img" x="16" y="88" source="@Embed(source='../assets/images/Canyonlands.png')" creationComplete="imageLoaded()"/>
 
	<mx:Label x="16" y="10" text="Red:"/>
	<mx:Label x="16" y="36" text="Green:"/>
	<mx:Label x="16" y="62" text="Blue:"/>
	<mx:HSlider x="60" y="11" id="redSlider" value="1" minimum="0" maximum="2" snapInterval="0.1" liveDragging="true" change="refreshFilter()"/>
	<mx:HSlider x="60" y="36" id="greenSlider" value="1" minimum="0" maximum="2" snapInterval="0.1" liveDragging="true" change="refreshFilter()"/>
	<mx:HSlider x="60" y="61" id="blueSlider" value="1" minimum="0" maximum="2" snapInterval="0.1" liveDragging="true" change="refreshFilter()"/>
 
	<mx:Script>
		<![CDATA[
			[Embed(source="../assets/pixelbender/rgbchannels.pbj", mimeType="application/octet-stream")]
			private var RGBChannelsClass:Class;
 
			private var shader:Shader;
 
			private function imageLoaded():void
			{
				shader = new Shader(new RGBChannelsClass() as ByteArray);
				img.filters = [new ShaderFilter(shader)];
			}
 
			private function refreshFilter():void
			{
				shader.data.redMultiplier.value[0] = redSlider.value;
				shader.data.greenMultiplier.value[0] = greenSlider.value;
				shader.data.blueMultiplier.value[0] = blueSlider.value;
 
				img.filters = [new ShaderFilter(shader)];
			}
		]]>
	</mx:Script>
</mx:Application>

If you now run this, you'll be able to change the R, G and B channels via the three sliders we've added. Result! The demo can be seen at http://blog.preinvent.com/demos/PixelBenderDemo/index.html and the full project download can be accessed at the bottom of this post.

Conclusion

The Pixel Bender Toolkit gives us a real-time test environment for creating Pixel Bender scripts. There are several example scripts included with the toolkit that you can experiment with. The full language specification is available for download at http://download.macromedia.com/pub/labs/pixelbender/pixelbender_referenc....

Implementing these scripts in Flex is as easy as adding a Shader created from the compiled PBJ file to any component (not just an Image as used in this tutorial).

Note that Pixel Bender isn't limited to image processing. Essentially all it is is a way of performing complex mathematical functions very quickly without taxing the CPU, so there's no reason why it couldn't also be used for accounting calculations, complex trigonometry etc.