I
ost games nowadays have multiplayer capabilities; however, the only interaction that goes on among online gamers is the occasional text message. Imagine hav-ing the ability to see the expression on your opponent's face when you just pass them before reaching the finish line, or when they get fragged by your perfectly placed rocket. Web cams allow you that functionality, and with high-speed Internet slowly becoming standard, it's becoming feasible to send more data to more clients.This gem demonstrates a straightforward approach to implementing Web cam methodologies into a game. We'll be using Video for Windows to capture the Web cam data, so Windows is required for the Web cam initialization function. We will cover numerous approaches for fast image culling, motion detection, and a couple of image manipulation routines. By die end, we will have a fully functional Web cam application tliat can be run and interacted widi at reasonable frame rates.
Initializing the Web Cam Capture Window
The following code demonstrates how to use Video for Windows to set up a Web C^^l>3 camera window in an application. Note to die reader: when dealing with video drivers ONTHICD from hardware vendors: You can never have too much error checking and handling
code (review source code on CD for a more thorough implementation).
// Globals
HWND hWndCam = NULL;
BOOL cam_driver_on = FALSE;
int wco_cam_width = 160, wco_cam_height = 120;
int wco_cam_updates = 400, wco_cam_threshold = 120;
// WEBCAM_INIT
void webcam_init(HWND hWnd) {
// Set the window to be a pixel by a pixel large hWndCam = capCreateCaptureWindow(appname,
WS_CHILD | WS_VISIBLE | WS_CLIPCHILDREN | WS_CLIPSIBLINGS,
153
0,0,
1,1,
hwnd, 0);
if(hwndCam) {
// Connect the cam to the driver
cam_driver_on = capDriverConnect(hWndCam, 1);
// Get the capabilities of the capture driver if(cam_driver_on)
{capDriverGetCaps(hWndCam, &caps, sizeof(caps));
// Set the video stream callback function
capSetCallbackOnFrame(hWndCam, webcam_callback);
// Set the preview rate in milliseconds capPreviewRate(hWndCam, wco_cam_updates);
// Disable preview mode capPreview(hWndCam, FALSE);
// Initialize the bitmap info to the way we want capwnd.bmiHeader.biSize = sizeof(BITMAPINFOHEADER);
{ // Assign memory and variables webcam_set_vars();
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T,
GL_REPEAT);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER,
GLJ.INEAR);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glGenTextures(1, &webcam_tex.gl_grey);
glBindTexture(GL_TEXTURE_2D, webcam_tex.gl_grey);
glTex!mage2D(GL_TEXTURE_2D, 0, 1, webcam_tex.size, webcam_tex.size, 0, GLJ.UMINANCE, GL_UNSIGNED_BYTE, webcam_tex.greyscale);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T,
GL_REPEAT);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER,
GL_LINEAR);
else {
cam_driver_on = FALSE;
The above function retrieves the handle to the Web cam window we're capturing from through the function capCreateCaptureWindow(). We then initialize it with win-dows properties such as its size, and whether it should be visible. In our case, we do want it to be visible; however, we're only going to set the window to a 1x1 pixel, so it's basically invisible. This is required because we don't actually want to display the image subwindow, but we want to receive the data updates from Windows through the call-back function.
We then retrieve driver information, set the callback function (more on this later), the number of times per second we want to refresh the Web cam, and then reset all our variables. The driver is then tested to see if it can handle returning the stan-dard bitmap information in which we are interested. Upon success, we initialize all the memory for all our movement buffers, as well as the OpenGL texture. We pull a little trick when deciding how big to make this texture, which will come in handy later on. Based on whatever height we set up our Web cam window to be, we find and allocate our memory to the next highest power of 2. Even though we are allocating a bigger buffer than the Web cam image, we save ourselves an expensive texture resize operation, by just doing a memcpyQ right into the larger buffer — at the cost of some small precision loss in the Web cam image.
Retrieving Data
Once we have our video window initialized, we need a way to retrieve the data from the Web cam every frame. To let Windows know which callback function it should send the data to, we must call capSetCallbackOnFrameQ with the address of the call-back function. When Windows decides it's time to update the Web cam, it will pass us the bitmap information inside the VIDEOHDR structure.
In our case, we'll make the callback function process all the Web cam data to decide if we want to create a texture out of it. We can pass all of that data to the web-cam_calc_movement () function for further processing, which will determine if enough data has changed since die last frame, after which, we can update the texture.
// WEBCAM_CALLBACK
// Process video callbacks here
LRESULT WINAPI webcam_callback(HWND hwnd, LPVIDEOHDR videojidr) {
// Calculate movement based off of threshold if(webcam_calc_movement(video_hd r,
webcam_tex.delta_buffer, wco_cam_width,
wco_cam_height, webcam^tex.size, wco_cam_threshold)) {
webcam_make_texture(videojidr, wco_cam_rendering);
}
return TRUE;
}
Windows defines the LPVIDEOHDR structure as:
typedef struct videohdr_tag {
LPBYTE IpData; // pointer to locked data buffer DWORD dwBufferLength; // Length of data buffer
DWORD dwBytesllsed; // Bytes actually used
DWORD dwTimeCaptured; // Milliseconds from start of stream DWORD dwUser; // for client's use
DWORD dwFlags; // assorted flags (see defines) DWORD dwReserved[4]; // reserved for driver
} VIDEOHDR, NEAR *PVIDEOHDR, FAR * LPVIDEOHDR;
Windows saves the Web cam data in the buffer called If Data. This is the primary variable we are interested in, but dwTimeCaptured and some of the flags may prove useful as well. Now that we've captured the data from the Web cam, let's test it to see if it's useful.
Motion Detection
We now want to weed out any unnecessary frames which have barely changed so we can avoid unnecessary updates to our texture. Updating textures is a notoriously slow operation in a 3D API such as OpenGL.
The following source code compares delta buffers, and returns true or false if the given threshold has been breached. Note that returning early when the threshold has been exceeded could optimize this function further; however, that would hamper us from using the delta buffer later on. Ectosaver [FiretoadOO] uses these unsigned bytes of delta movement to calculate the amplitude of the waves it causes, and to determine when there is no one moving around.
// GLOBALS
unsigned char wco_cam_threshold=128; // This is a good amount (0-255)
// WEBCAM_CALC_MOVEMENT
// This is a simple motion detection routine that determines if // you've moved further than the set threshold
BOOL webcam_calc_movement(LPVIDEOHDR video_hdr, unsigned char *delta_buff,
int webcam_width, int webcam_height, int gl_size, unsigned char thresh) {
unsigned char max_delta=0;
int i=0, j=0;
int length;
unsigned char *temp_delta = (unsigned char *)malloc(
sizeof(unsigned char)* webcam_width * webcam_height);
length = webcam_width * webcam_height;
webcam_tex.which_buffer = webcam_tex.which_buffer 7 0 : 1 ; if(!video_hdr->lpData)
return FS_TRUE;
for(i=0; i<length; i++) {
// Save the current frames data for comparison on the next frame // NOTE: Were only comparing the red channel (IpData is BGR), so / / i n theory if the user was in a solid red room, coated in red // paint, we wouldn't detect any movement....chances are this //isn't the case :) For our purposes, it this test works fine webcam_tex.back_buffer[webcam_tex.which_buffer][i]
= video_hdr->lpData[i*3];
// Compute the delta buffer from the last frame
// If it's the first frame, it shouldn't blow up given that we // cleared it to zero upon initialization
temp_delta[i] =
abs(webcam_tex.back_buffer[webcam_tex.which_buffer][i] -webcam_tex.back_buffer[!webcam_tex.which_buffer][i]);
/ / I s the difference here greater than our threshold?
if (temp_delta[i] > max_delta) max_delta = temp_delta[i] ;
// Fit to be inside a power of 2 texture for(i=0; i<webcam_height ;
memcpy(&delta_buff [i*(gl_size)] ,
&temp_delta[i*(webcam_width)] , sizeof (unsigned char)*webcam_width) ;
f ree(temp_delta) ; if(max_delta > thresh)
return TRUE;
else
return FALSE;
Manipulating Web Cam Data Get the BGR Pixels
Once we've performed all our testing and culling, we are ready to manipulate the data we were sent from Windows. For this, we will simply copy the pixels from die VIDEOHDR data struct (the native format Windows returns is BGR) into a buffer that we've allocated to have a power of 2. Note that this technique avoids resizing the texture data's pixels, as it simply copies the pixels straight over, preserving the pixel aspect ratio. The only drawback to this technique is that it will leave some empty space in our texture, so we're left with a bar of black pixels at the top of the image. We can eliminate that bar by manipulating texture coordinates (once mapped onto 3D geometry) or resizing the texture.
// WEBCAM_MAKE_BGR
void webcam_make_bgr(unsigned char *bgr_tex, unsigned char *vid_data, int webcam_width, int webcam_height, int glsize)
{ int i;
for(i=0; i<webcam_height; i++) {
memcpy(&bgr_tex[i*(glsize*3)],
&vid_data[i*(webcam_widtn*3)],
sizeof(unsigned char)*webcam_width*3);
Convert to Grayscale
Once we've captured the BGR data, we could convert it to grayscale. This would result in an image that is one-third the size of our regular textures, which would be practical for users who have slow Internet connections, but still want to transmit Web cam data.
Here is a function that multiplies each RGB component in our color buffer by a scalar amount, effectively reducing all three color channels to one:
// WEBCAM_MAKE_GREYSCALE
void webcam_make_greyscale( unsigned char *grey,
unsigned char *color, int dim) {
int i, j;
// Greyscale = RED * 0.3f + GREEN * 0.4f + BLUE * 0.3f for(i=0, j=0; j<dim*dim; i+=3,
grey[j] = (unsigned char)float_to_int(0.30f * color[i] + 0.40f * color [i+1] + O.SOf * color[i+2]);
Real-Life Cartoons
Once we've successfully converted all our data to grayscale, we can manipulate the data to draw the picture in a cartoon-like fashion. This method splits the image into five different levels and six different colors, coloring different ranges of pixel values with solid values. All we have to do is perform some simple comparisons and evaluate each pixel based on our heat intensity constants.
The final result is compared against a lookup from either the grayscale buffer or our delta buffer. If we want to see the image every frame (single buffer), we will need to compare against the grayscale. To give different results, we'll assign random color intensities for each pixel based on our heat intensity constants.
// WEBCAM_INIT_CARTOON
void webcam_init_cantoon(cartoon_s *cartoon_tex) {
char i;
for(i=0; i<3; i++) {
// Pick random colors in our range
cartoon_tex->bot_toll_col[i] = rand()%255;
cartoon_tex->min_toll_col[i] = rand()%255;
cartoon_tex->low_toll_col[i] = rand()%255;
cartoon_tex->med_toll_col[i] = rand()%255;
cartoon_tex->high_toll_col[i] = rand()%255;
cartoon_tex->max_toll_col[i] = rand()%255;
tfdefine MIN CAM HEAT 50
tfdefine LOW_CAM_HEAT 75
#define MED_CAM_HEAT 100
#define HIGH_CAM_HEAT 125
#define MAX_CAM_HEAT 150
// WEBCAM_MAKE_CARTOON
void webcam_itiake_cartoon( unsigned char *cartoon, cartoon_s cartoon_tex,
unsigned char *data, int dim) {
int i, j, n;
for(i=0, j=0; j<dim*dim; i+=3, {
if(data[j] < MIN_CAM_HEAT)
for(n=0; n<3;
cartoon[i+n] = cartoon_tex.bot_toll_col[n] ; }
if(data[j] > MIN_CAM_HEAT && data[j] < LOW_CAM_HEAT)
for(n=0; n<3;
cartoon [i+n] = cartoon_tex.min_toll col[n];
}
if(data[j] > LOW_CAM_HEAT && data[j] < MED_CAM_HEAT) for(n=0; n<3;
cartoon[i+n] = cartoon_tex.low_toll_col[n] ; }
if(data[j] > MED_CAM_HEAT && data[j] < HIGH_CAM_HEAT)
for(n=0; n<3;
cartoon[i+n] = cartoon_tex.med_toll_col[n] ; }
if (data[ j] > HIGH_CAM_HEAT && data[j] < MAX_CAM_HEAT)
for(n=0; n<3;
cartoon [i+n] = cartoon_tex.high_toll_col[n] ; }
if(data[j] > MAX_CAM_HEAT) for(n=0; n<3;
cartoon[i+n] = cartoon_tex.max_toll_col[n] ;
Uploading the New Texture
Now, all that's left is uploading the texture to OpenGL. The first step is to get the color values from Video for Windows. Once the new color values are calculated, we can go on to converting it to grayscale, and then go on to our cartoon Tenderer.
Once all the image manipulation is finished, we call glTexSubImage2D() to get it into the appropriate texture. It is then ready for use in a 3D application as a texture.
// WEBCAM_MAKE_TEXTURE
void webcam_make_texture(LPVIDEOHDR video, webcam_draw_mode mode) {
// Build the color first
webcam_make_bgr(webcam_tex.bgr, video->lpData,
wco_cam_width, wco_cam_height , webcam_tex.size) ;
if (mode == GREYSCALE || mode == CARTOON) webcam_make_greyscale (webcam_tex . greyscale ,
webcam_tex.bgr, webcam_tex.size) ; // Note: Could also pass in the delta buffer instead of // the greyscale
if (mode == CARTOON)
webcam_make_cartoon (webcam_tex . bgr , webcam_tex . cartoon , webcam_tex. greyscale, webcam_tex.size) ;
// Upload the greyscale version to OpenGL if (mode == GREYSCALE)
{
glBindTexture(GL_TEXTURE_2D, webcam_tex.gl_grey) ; glTexSub!mage2D(GL_TEXTURE_2D, 0,0,0,
webcam_tex . size , webcam_tex . size , GL_LUMINANCE,
GL_UNSIGNED_BYTE, webcam_tex. greyscale) ; }
// Upload the color version to OpenGL else
{
glBindTexture(GL_TEXTURE_2D, webcam_tex.gl_bgr) ; glTexSub!mage2D(GL_TEXTURE_2D, 0,0,0,
webcam_tex . size , webcam_tex . size ,
GL_BGR_EXT, GL_UNSIGNED_BYTE, webcam_tex.bgr) ;
Destroy the Web Cam Window
After we're done using the Web cam, we need to destroy the window and set our call-back function to NULL, so Windows knows to stop sending messages to it. In addi-tion, we must free up all the memory we previously allocated to our color, grayscale, and delta buffers.
// WEBCAM_DESTROY
void webcam_destroy(void) {
if (cam_driver_on)
capSetCallbackOnFrame(hWndCam, NULL);
DestroyWindow(hWndCam) ; hWndCam = NULL;
if (webcam_tex . bgr) f ree(webcam_tex.bgr) ; if (webcam_tex . grayscale )
free(webcam_tex. grayscale) ; if (webcam_tex . delta_buf f er)
f ree(webcam_tex.delta_buffer) ; if (webcam_tex.back_buffer[0] )
f ree(webcam_tex.back_buffer[0]) ; if (webcam_tex.back_buffer[1 ] )
f ree(webcam_tex.back_buffer[1 ] ) ;
Conclusion
Web cams have a lot of untapped potential that game developers may not realize.
They have the ability to be used as input devices, as in the way a mouse is used, by tracking color objects and translating their rotations from 2D to 3D [Wu99] . It's even possible to replace your standard mouse using a Web cam, by performing data smoothing and color tracking algorithms on the input frames.
References
Microsoft Developer Network Library http://msdn.microsoft.com/library/devprods/
vs6/visualc/vcsample/vcsmpcaptest.htm.
[FiretoadOO] Firetoad Software, Inc., Ectosaver, 2000 www.firetoads.com.
[Wu99] Wu, Andrew, "Computer Vision REU 99" www.cs.ucf.edu/-vision/reu99/
profile-awu.html.