Photometric Stereo

1. Experiment Purpose

The goal of this experiment is to implement the Photometric Stereo algorithm, which uses multiple images under different lighting directions to estimate the surface normal vectors and albedo, and then re-render the image under a specified lighting direction.

• Normal Vector Calculation: According to the Lambertian model formula , where is the albedo, is the normal vector, and is the lighting direction. The albedo and lighting direction can be uniquely determined when at least three images with known lighting directions are provided.
• Shadow and Highlight Processing: Shadows and highlights break the linear Lambertian model. A simple solution is to sort all the observations for each pixel and discard a certain percentage of the brightest and darkest pixels to remove shadows and highlights.

2. Experiment Principle

According to the Lambertian model (a purely diffuse reflection model), for each point on the surface, there is a fixed albedo , and the intensity of the reflected light depends only on the albedo and the angle of the incident light (the angle between the light direction and the normal vector). The formula is as follows:

When we have images under different lighting directions, we can use the Lambertian model to analyze each pixel individually. With three lighting angles, we can express the Lambertian model equation in matrix form as: Since the reflected light intensity and the lighting directions are known, we can uniquely determine the albedo and normal vector through matrix inversion when three different lighting directions are available.

With the albedo and normal vectors, re-rendering the image under a specified lighting direction is easy; we just apply the Lambertian model to calculate the light intensity at each pixel.

3. Experiment Content

In the provided code framework, the primary task is to implement the myPMS.m file.

First, I modified the original function declarations because the final output requires three images: the normal map, albedo map, and re-rendered image, while the original framework only output the normal map. So, I added the albedo map and re-rendered image to the function's return. The input now also includes a new parameter shadow_removal_percentage, which specifies the percentage of the brightest and darkest pixels to be discarded when handling shadows and highlights (e.g., if the value is 20, then the darkest 20% and the brightest 20% pixels are discarded).

In the data preprocessing phase, N, albedo, and re_rendered_img represent the normal map, albedo map, and re-rendered image, respectively. They are all three-channel images. I is used to temporarily store the light intensities for each channel.

Through simple loops, the original images were first processed with a mask to extract the subject and avoid the influence of the background. Then, for each RGB channel, the given light_intensity was used to divide the original image, retrieving the true light intensity I for each channel.

Then, for each pixel, the light intensities of all the images were sorted, and the brightest and darkest pixels of the given percentage were discarded, storing the results in I_col_filtered. Based on the formula , we can first compute by using the least squares method on the filtered light directions s_filtered and light intensities I_col_filtered. The norm of the result gives the albedo value, and dividing the result by the albedo gives the unit normal vector.

With this information, re-rendering becomes straightforward. Simply traverse the pixels and apply the Lambertian model formula for each pixel. Since the assignment did not require the RGB components of the incident light, I assumed them to be 1 by default.

Thus, the main function for Photometric Stereo is complete. Below are some small modifications to the Baseline:

The function call now includes the modified inputs and outputs. The shadow and highlight removal percentage is set to 20, meaning the darkest 20% and the brightest 20% pixels are discarded. To make the final re-rendered image more visually appealing, it is also normalized.

Here is the full code for Photometric Stereo (expand to view):

(On my GitHub repository ComputerVision/Homework 1 you can also find the code)

Folding Click to view more

function [N, albedo, re_rendered_img] = L2_PMS(data, m, shadow_removal_percentage)

    num_images = size(data.s, 1);
    [height, width, ~] = size(data.imgs{1});
    
    N = zeros(height, width, 3);
    albedo = zeros(height, width, 3);
    re_rendered_img = zeros(height, width, 3);
    I = zeros(num_images, length(m), 3);
    
    % Extract pixel intensities for each image
    for c = 1:3
        for i = 1:num_images
            img = double(data.imgs{i});
            img = img(m);
            img = img / data.L(i, c);
            I(i, :, c) = img;
        end
    end
    
    for c = 1:3
    % Remove shadows and highlights, compute normals and albedo
        for i = 1:length(m)
            I_col = I(:, i, c);
            
            % Sort intensities and remove shadows/highlights
            [sorted_I, idx] = sort(I_col);
            num_to_remove = round(length(sorted_I) * shadow_removal_percentage / 100);
            valid_idx = idx(num_to_remove+1:end-num_to_remove);
            
            % Filter light source directions and intensities
            s_filtered = data.s(valid_idx, :);
            I_col_filtered = sorted_I(num_to_remove+1:end-num_to_remove);
            
            % Solve for normal and albedo
            A = s_filtered \ I_col_filtered;
            albedo_val = norm(A);

            norm_A = A / albedo_val;
            [row, col] = ind2sub([height, width], m(i));
            N(row, col, :) = norm_A';
            albedo(row, col, c) = albedo_val;
        end
    end
    % Re-render the image using recovered normals and albedo
    viewing_direction = [0, 0, 1];
    for c = 1:3
        for i = 1:height
            for j = 1:width
                normal = squeeze(N(i, j, :));
                if norm(normal) > 0
                    re_rendered_img(i, j, c) = max(0, dot(normal, viewing_direction)) * albedo(i, j, c);
                end
            end
        end
    end
    end
    

4. Experiment Results

	Normal Map	Albedo Map	Re-rendered Picture
bear
buddha
cat
pot