What: Image Analysis App for iPhone (iOS)
How: X-Tools, iPhone SDK, objective-C
Type of Project: Academic
For: Prof. Betke's CS585 class titled "Image and Video Computing" (CS585)
When: April, 2009
The following is the Final Project Report.
BU CS585:Image and Video Computing
Final Project(04/30/09): Image Enhancing Applications for the iPhone
Overview:
Cell phones are evolving into general purpose hand-held computers with many of the features and capabilities of lap-top computers. The Apple iPhone is a popular and prime example. The camera and image display capabilities of the iPhone are of particular interest. The goal of this project is to develop an iPhone “App” for image manipulation and editing (that is, the basis for a “mini-Photoshop”) that runs directly on the iPhone and works on images produced by the iPhone camera and on the images stored in the “Photo Library” on the iPhone. The application will also work on images stored on the Apple iPod Touch.
Benefits:
The image editing application will extend the camera capabilities and forms the framework for a commercial platform. Apple manages a marketing program for iPhone apps through the iTunes store as well on the Apple website.
Specific Goals:
- Learn to develop applications for the iPhone using the Apple iPhone SDK.
- Develop a general image editing and image display iPhone application foundation.
- Develop an image pixel transformation for removing or minimizing “shadows” in the images produced by the camera. This extends the “dynamic range” of the camera.
- Demonstrate the image editing and display capabilities of the iPhone and the image application foundation on a simple binary image (and mask) generator.
Figure 1: Image adjustment application using method 2 running on the iPhone. The top image is the original image. The bottom image is the modified image. The slider bar adjusts the level of the gain applied, with 0 being no change to the original. Note that it is easy to see the person walking down the road in the modified images but not the original. One may scroll the images in each window using a finger touch (or mouse drag in the simulator).
Development Tools Used:
- Intel-based Mac running OS X 10.5 (Leopard)
- X-Tools
- iPhone SDK, including the iPhone Simulator
Fortunately, X-Tools and the iPhone SDK are free.
The iPhone OS and Mac OS are written in Objective C (yet another object-oriented extension of C and quite a bit different from C++ and Java. Main differences are in messaging and memory management). The iPhone is not set up to do video editing. There is also no direct access to the pixels in an image. Each bitmap for an image must be taken apart to access the raw data.
Method:
Dynamic Range Adjustment (DRA) extends the 'dynamic range' of an image similarly to the well-know HDR concept where dynamic range is maximized by taking indentical shots at different focal lengths and combining into one image. However unlike HDR, this method only requires one image, thus allowing for future use on a video application.
The methods used to minimize shadows in an image, effectively increasing the dynamic range of the camera, involves computing, pixel by pixel, a gain. The new pixel value is computed from the old pixel value by multiplying the old value by the gain.
In the first method, the gain is based on the “intensity” of the pixel, and the same gain is applied to all 3 color components (RGB) in the pixel. In the second method, a gain is computed for each color channel or component, based on the magnitude of the color component.
Dynamic Range Adjustment (DRA) Method 1 (used in ImageDRA):
p = “intensity” of pixel = (R + B + G)/3
p’ = new pixel intensity, with R’ = gR, B’ = gB, G’ = gG where g is the gain.
g = 5 a (1 – p/255)4 + f(p,a) + 1
where a = the slider value (range of 0 to 1) and
f(p,a) = -0.75 sin2(PI*p/128) a if p < 64
and
f(p,a) = 0.75 [sin(PI(p-64)/384) – 1] a otherwise
Method 2 (used in ExampleDRA):
p = pixel color component (R, G, or B)
p’ = g p (each color component is multiplied by a different gain)
where
g = 5 a (1 – p/255)n + 1
and n depends on a and p:
n(p,a) = a/2 + ½ p < 11
= 3a/2 + ½ 11< p < 30
= 5a/2 + ½ 30 < p < 60
= 2a + 1 p > 60
Binary Images and Masks (used in binaryGEN and binaryMask):
The binary threshold is set by the pixel intensity:
p = 0.3 R + 0.59 G + 0.11 B.
The slider bar adjusts the threshold value (0 to 255).
Results:
The shadow reducing methods and the binary image and mask generators iPhone apps were run on the iPhone simulator. An example result for method 1 is shown below. The method allows one to see detail on the side walls and steps that are not easy to see in the original.
Figure 2: Method 1 applied to a sample image from the iPhone photo catalog, This is the iPhone display while the ImageDRA app is running. The image on the top is the original. The shadows in the image on the bottom have been lightened. The slider bar adjusts the level of adjustment in real-time, with 0 equal to no change and 100 equal to max change.
Method 2 was applied to a dusk scene and worked quite well. The person walking gown the street is easy to see in the modified image, but difficult to impossible to recognize in the original. The application running on the iPhone is displayed in the top image above.
The binary mask application produced the expected results with the benefit of adjusting the threshold by eye using the slider bar. A typical result is given on the right in the image below.
Figure 3: Basic navigation of the binaryMask application. On the left is the iPhone main display with the application icons. The binaryMask application allows the user to select an image from the Photo Library on the iPhone (middle). On the right, the result of apply a binary mask to an image is displayed. The original image (from Hawaii album) is displayed on the top, and the binary masked image on the bottom. The slider bar allows one t instantly adjust the threshold for the binary mask. The threshold is based on the pixel intensity.
Discussion:
The iPhone proves to be a versatile platform for image display and real-time visual editing, once the image raw data management foundation is developed. Methods can be developed beyond what is now available to extend the capabilities of the built-in camera as well as other images added in the field. One does not have to wait until downloading to a laptop to see if some images are useable when working in the field.
The shadow removal methods worked fairly well at improving the dynamic range in the images, but the particular gains we developed clearly need improvement at the brighter image levels. There are also other methods (removing the highlights, for example) that should be added for more capability in image editing.