Image sharpening and its effects on perceived image quality Ari Partinen

98 Pages·2009·9.48 MB·English

Image sharpening and its effects on
perceived image quality

Ari Partinen

Diplomityö, joka on jätetty opinnäytetyönä tarkastettavaksi diplomi-insinöörin
tutkintoa varten Espoossa 22.1.2009.

Työn valvoja Professori Pirkko Oittinen

Työn ohjaaja DR Mazin Musa
TEKNILLINEN KORKEAKOULU Diplomityön tiivistelmä

Tekijä: Ari Partinen
Työn nimi: Image sharpening and its effects on perceived image quality

Päivämäärä: 25.1.2009 Kieli: Englanti Sivumäärä: 82

Tutkinto-ohjelma: Tietoliikennetekniikka

Vastuuopettaja: Professori Pirkko Oittinen
Ohjaaja: DR Mazin Musa

Diplomityössä tutkittiin terävöityksen vaikutusta koettuun kuvanlaatuun. Aluksi tehtiin
selvitys mitkä ovat tavanomaisimmat syyt epäterävyyteen käytettäessä kamerakännyköitä
ja kuinka näitä voitaisiin välttää sekä mitä tulisi ottaa huomioon kameramoduulien
suunnittelussa. Erityisesti tutkittiin kännykkäkameroiden linssien ja sensorin aiheuttamaa
epäterävyyttä. Lisäksi tarkasteltiin kuinka kontrasti, kohina ja kuvan kirkkaus vaikuttavat
koettuun terävyyteen. Havaittiin, että kaikilla edellä mainituilla kuvanlaatuattribuuteilla
voidaan vaikuttaa kuvan koettuun terävyyteen.

Lisäksi tutkittiin kuinka paljon terävöitystä kamerakännykällä otettuun kuvaan tulisi lisätä
kuvan jälkikäsittely vaiheessa ja millaisella terävöityksellä saavutetaan optimaalinen
kuvanlaatu. Tätä tutkittiin käyttämällä kuutta luonnollisissa oloissa, eri
valaistusolosuhteissa otettua kuvaa. Vastaavat valaistusolosuhteet luotiin myös
studioympäristössä ja näin voitiin kuvata erityinen testi kenttä josta voitiin mitata kuvan
alkuperäinen terävyys sekä kohina. Testikuvat otettiin viiden megapikselin SMIA85-
standardin mukaisella CMOS-kameramoduulilla. Luonnolliset testikuvat sekä
laboratoriokuvat käsiteltiin käyttäen simulointi ohjelmaa joka simuloi kännykkäkameran
kuvaprosessointia. Testikuvista luotiin 11 eri versiota joissa terävöityksen määrää
vaihdeltiin ja terävöityksen määrä oli myös mitattavissa testikentästä otetuista kuvista.
terävöityksen suuruuden mittariksi valittiin resoluutiovaste joka on yleisesti käytetty mittari
tutkittaessa yliterävöityksen määrää.

Terävöityksen vaikutusta koettuun kuvanlaatuun testattiin käyttäen kymmentä
testihenkilöä. Testistä kävi selvästi ilmi, että mobiilikameralla otettu kuva kaipaa
terävöitystä, mutta liiallinen terävöitys huonontaa kuvanlaatua. Lisaksi havaittiin, että
kirkkaassa valaistuksessa otettuja kuvia pitää terävöittää enemmän kuin huonossa
valaistuksessa otettuja, jotta saavutetaan optimaalinen kuvanlaatu. Päädyttiin esittämään,
että kirkkaassa valaistuksessa, yli 1000 luxia, optimaalinen kuvanlaatu saavutetaan
käyttämällä terävöitysalgoritmia joka tekee kuvaan 2.1 resoluutiovasteen antavan
terävöityksen. Valaistuksen ollessa 1000-300 luxia paras kuvanlaatu saavutettiin 1.8
resoluutio vasteella sekä alle 300:n sadan luxin valaistuksessa resoluutiovasteeksi saatiin
samoin 1.8. Näillä arvoilla saavutettiin hyväksyttävä kuvanlaatu 70-60%:n varmuudella.

Avainsanat: kuvan terävyys, terävöitys, terävöitys algoritmi, resoluutiovaste, MTF
Helsinki University of Technology Abstract of the Master’s Thesis
Author: Ari Partinen
Name of thesis: Image sharpening and its effects on perceived image quality

Date:25.1.2009 Language: English Pages: 82

Study program: Telecommunications

Supervisor: Professor Pirkko Oittinen
Instructor: DR Mazin Musa

The objective of this master’s thesis was to examine how much sharpening a mobile
imaging system should introduce to images to achieve optimal image quality. Also the
concept of perceived sharpness was examined. First I examine what are the major
reasons contributing to the unsharpness in mobile imaging devices. Especially I focus on
reasons deriving from the lens and the sensor of digital imaging system. I also investigate
how contrast, noise, and brightness affect the perceived sharpness of final image.

In the experimental part I studied how much sharpening imaging system should add to the
raw image from SMIA85-standard CMOS imaging sensor, fairly commonly used in mobile
imaging devices. The affect of sharpening to overall image quality was studied by usage
of image signaling pipe simulator, emulating a real mobile phone image processing.
Several natural content images were captured in varying lighting conditions. Six I3A
compatible images were chosen for subjective evaluation. The lighting conditions where
natural content images were captured were replicated in studio and both set of images
were run through image signaling pipe simulator with varying amount of additive
sharpening. The amount of sharpening added was measured as resolution response,
which is commonly used measure for over sharpening.

The affects of sharpening to perceived image quality were tested with ten naïve test
persons. The test revealed that to obtain optimum image quality sharpening is clearly
needed, but excessive sharpening will deteriorate the image quality. It was also
discovered that images taken in bright lighting conditions needed more sharpening to
achieve optimum image quality that their low light counterparts. The result of the test was,
that to obtain optimal image quality in Bright ambient lighting, over 1000lux, a sharpening
equal to 2.1 resolution response should be used. This yields 70% acceptability. For
Medium and low ambient lighting, under 1000lux, the optimum resolution response is 1.8
yielding 60% acceptability.

Keywords
Image quality, Sharpening, image sharpness, perceived sharpness, sharpening
algorithms
ACKNOWLEDGEMENTS

This Master of Science Thesis was written for Nokia Corporation, Southwood UK as a
Fulfillment of Master of Science degree in Helsinki University of Technology. Several
people have contributed to my work but especially I would like to thank my instructor Dr.
Mazin Musa for the support he has given me during my work, Philip Trevelyan for helping
me in finalizing the work and my team in Nokia for creating a great work atmosphere.
Also I would like to thank my professor Pirkko Oittinen for getting me into the imaging
world and of course for the guidance she has given me during my work with the thesis.
The work hasn’t been always easy and my family and especially my godson Miska has
helped me to get my mind of the work when needed. For this I’m grateful for them.
By far the most important person during this time has anyway been Laura, and she
deserves a great big hug, for just being there.

I guess that it’s time to turn of the computer and go and check what’s happening in London
to night. Over and Out.

There’s more to the picture than meets the eye
-Neil Young

I
TABLE OF CONTENTS
1 Introduction..................................................................................................................1
1.1 Background............................................................................................................1
1.2 Scope of the study.................................................................................................2
1.3 Justification and the structure of the thesis............................................................3
1.4 The test process....................................................................................................3
2 Image quality................................................................................................................4
2.1 General..................................................................................................................4
2.2 Image quality evaluation........................................................................................6
2.3 Working definition of image quality........................................................................9
3 Fundamentals in imaging technology.........................................................................10
3.1 Camera system....................................................................................................10
3.2 Mobile phone camera module.............................................................................12
3.3 Basics of image sensors......................................................................................12
3.4 Sensor efficiency.................................................................................................13
3.5 Charge Coupled Device.......................................................................................14
3.6 Complementary Metal-Oxide-Semiconductor (CMOS)........................................17
3.7 Fundamentals in optics........................................................................................19
3.8 Optical chain........................................................................................................19
3.8.1 Lens system.................................................................................................19
3.8.2 Aperture and shutter.....................................................................................23
3.8.3 Infrared filter.................................................................................................24
3.8.4 Anti-aliasing filter..........................................................................................24
3.9 Field of view.........................................................................................................25
3.10 Depth of field....................................................................................................26
3.11 Conclusions of the Field of View and the Depth of Field..................................30
4 Image sensor enhancing technologies.......................................................................31
4.1 General................................................................................................................31
4.2 Anti-blooming.......................................................................................................31
4.2.1 Microlenticular arrays...................................................................................32
5 Limited optical system performance...........................................................................33
5.1 General................................................................................................................33
5.2 Monochromatic aberrations.................................................................................35
5.2.1 General.........................................................................................................35
5.2.2 Spherical aberration.....................................................................................35
5.2.3 Comatic aberration.......................................................................................36
5.2.4 Astigmatism..................................................................................................37
5.2.5 Curvature of field..........................................................................................40
5.3 Polychromatic aberrations...................................................................................42
5.3.1 General.........................................................................................................42
5.3.2 Chromatic aberration....................................................................................42
5.3.3 Correction of chromatic aberration...............................................................43
6 Image sharpness........................................................................................................45
6.1 Interpretation of sharpness..................................................................................45
6.2 Resolution and acutance.....................................................................................47
6.3 Interactions between sharpness and other image quality parameters................48
6.3.1 Sharpness and contrast................................................................................48
6.3.2 Sharpness and noise....................................................................................50
6.3.3 Sharpness and brightness............................................................................51
6.4 Sharpness and viewing distance.........................................................................52
6.5 Definition and measurement of image sharpness...............................................52
6.6 Slanted edge MTF...............................................................................................53
II
6.7 Edge width as a tool for calculating perceived blur and ringing...........................55
7 Image sharpness enhancement.................................................................................56
7.1 General................................................................................................................56
7.2 Unsharp mask......................................................................................................57
7.3 Edge enhancement..............................................................................................58
8 Test arrangements.....................................................................................................60
8.1 General................................................................................................................60
8.2 Choosing, generation and categorization of the test images...............................60
8.3 Viewing conditions...............................................................................................65
8.4 The subjective test...............................................................................................65
9 Results.......................................................................................................................66
10 Conclusions.........................................................................................................78
References.........................................................................................................................80

III
ABBREVIATIONS
- listing all abbreviations used in the thesis doc; alphabetical order
Abbreviation Explanation
MTF Modulation Transfer Function
JND Just Noticeable difference
ISP Image Processing Pipe
CCD Charge Coupled Device
CMOS Complementary Metal Oxide Semiconductor
MOS Metal Oxide Semiconductor
CFA Color Filter Array
IR Infra Red
DSC Digital Still Camera
SLR Single Lens Reflex
DOF Depth Of Field
EDoF Extended Depth of Field
OTF Optical Transfer Function
SFR Spatial Frequency Response
OECF Opto-Electronic Conversion Function
LMF Luminance Masking Function

IV
1 INTRODUCTION
1.1 Background
The digital imaging industry has experienced a massive upswing in past few years. In 2005
ABI research /22/ made a study and concluded that in two years camera phones are going
to replace the markets from low end digital cameras. Although, it’s hard to say if this has
actually happened, but what we can say is, that Nokia is the biggest camera manufacturer
in the world /26/.2005 ABI also announced that within two years the mega pixel race in
mobile imaging industry will hit the 5 megapixel mark. That has also happened and first
mobile 8 Mega pixel imaging product is on the markets. /31/
The massively increased usage of digital imaging products and cameras has raised an
increasing need to develop quality measurement techniques that could predict or model
perceived image quality automatically and in an objective manners. It has been proved in
several studies that subjective evaluation, and quality metrics give the best assessment, but
they are extremely costly and time consuming to arrange and this is why they are not
suitable for everyday system integration. Instead of using subjective evaluation, increasing
effort is aimed to develop objective metrics that would predict the subjective opinion of the
image quality. But because of the complex nature of human vision system and ambiguous
nature of image quality in general, this has proven to be quite a challenging task. /19/

The increasing amount of network usage has also produced several metrics that are usable
for evaluate the image quality deteriorations but the biggest default in majority of these
metrics is that they are based to usage of reference image or full reference evaluations
where one supposes that reference, or perfect image can be used to evaluate the image
quality deteriorations and this is often not the case when evaluation image quality from
digital cameras. /46/
When thinking about the image quality in TV for example the problem is to preserve the
quality while transferring the broadcast through the TV-network. So, in TV case the image
quality is measured basically as the amount of deteriorating happened during the
transmission of broadcast. This is quite not the case when digital cameras and especially
the image processing pipe (from now on called also with abbreviation ISP) of the digital
imaging devices is in question. Yes, we can compare the ISP to television networks on
some level. They both convey image data and introduce some changes to it. But when
television network or internet tends to deteriorate the image quality, the scope of the ISP is
to improve it. This is quite a fundamental difference, and because of this difference the
much studied methods of full reference evaluations are not usable. Because all this, a
whole new set of no reference methods have to be developed.
1
Camera phone sets really challenging requirements for the imaging system. The imaging
system needs to have low cost and simple optical structure because of the high volumes. In
addition to that the system needs to be small in size and very robust because of the way
these devices are used every day. Because of the demands of small size and low cost, the
image quality of these devices, without any image processing is relatively poor and
substantial amount of post processing is needed to achieve acceptable image quality.
However, the processing capacity in a cellular phone is finite because of the requirements
of low power consumption and cost. All these requirements and limitations culminate in
the need to have simple and powerful post processing algorithms that achieve best possible
outcome, from relatively poor starting points, from a relatively small amount of processing
power. This is quite a technical handful.
To design an imperfect imaging system, such as a mobile phone camera with aberration
limited optics and image sensors, it’s very important to find a quality metrics that is related
to human observer way to judge the image quality. Several different quality measures have
been used to estimate the subjective image quality of the imaging system, meaning the
image quality that a human observer perceives.
In many publications the image sharpness has been recognized as one of the key metrics
that human observers use when evaluating image quality. Unfortunately in mobile imaging
devices the sharpness of optical systems can be very limited and the perceptions of
sharpness have to be created in post processing. This is where the sharpening algorithms
and optimizing them come into the picture.
1.2 Scope of the study
The scope of the study is to examine the effects of sharpening to the image quality and
ultimately try to find the limit values for right amount of sharpening. The amount of
sharpening is examined by usage of maximum resolution response and ringing metric. This
is done by modifying the amount of sharpening in one image signaling pipe (ISP) without
changing the other ISP parameters. The usage of ISP simulator provides possibility to
simulate mobile phone ISP in the way that mobile phone post processes the raw sensor
images between the caption of an image and displaying or saving it.
The subjective sharpness was chosen to be the image quality parameter under the scope
because it has been proven to be one of the most important image quality parameters when
subjective evaluations are carried out. /28//35/ It’s also fairly easy to change the amount of
sharpening with out affecting greatly to other image quality attributes. This makes the
study especially interesting because it enables the result to be applied to several imaging
systems.
However, it must be made clear that the exact amount of optimum sharpening is always, to
varying degree, system dependent. But never the less, the results work as a good guide line
for future ISP and sharpening algorithm design.
2
1.3 Justification and the structure of the thesis
The image unsharpness as a parameter is rather interesting one, because it can be identified
as a artifactual attribute as a nature, because whenever it’s present it has a tendency to
lower the perceived quality of the image. But when looking sharpening as an attribute, it
can be seen as a preferential attribute, because it also has tendency to improve image
quality to some extend and after that it starts to deteriorate it. /28/ So with this in mind, it
should be possible to recognize the optimum amount of sharpening for every imaging
system and finding the optimum amount of sharpening is the question I’m about to tackle.
In the thesis theory section I shall introduce the imaging system in general, because
unsharpness in image can derive from several sources in imaging pipe of camera systems. I
particularly focus on the reasons for unsharpness caused by the sensor and the optics in
imaging system. I shall also introduce other image quality attributes and discuss about their
affects on the final image quality, but because of the complicated nature of the image
quality only the optimization of the sharpening was chosen to be the scope of the
experimental part. In the theory part, some time is also spent examining the sharpening
algorithms and interactions between perceived image sharpness and other image quality
attributes. In results chapter, I naturally present the results of the objective and subjective
measurements and spend some time finding the connections between the objective
sharpening metrics and the subjective evaluations.
1.4 The test process
The goal of this thesis is to determine optimal amount of sharpening the image processing
pipe can introduce without starting to deteriorate the image quality. This is done by taking
multiple I3A compatible images with 5 Mega pixel mobile imaging device in various
lighting conditions, both indoors and outdoors. The idea is to cover bright-, medium- and
low light conditions. After this I’m going to replicate the real life conditions in laboratory
conditions and capture images of special test target in these replicated lighting conditions.
This way we can introduce the same image artifacts and parameters like analog gain and
exposure time to both natural images and the test target images. This allows us to use the
specific test target images to compute the image artifacts and processing also introduced to
the natural images.
This is possible, because sensor introduced several artifacts like noise that are depended on
the prevailing lighting conditions, not the actual photographic scenes. Also, the processing
of the images uses only information dependent of the lighting conditions. For example ISP
can utilize different amount of sharpening, or noise filtering, to images taken in different
lighting level conditions, but the actual photographed scene doesn’t affect to any of these
artifacts or processing methods.
In practice the test has three phases. First natural images and studio images are taken in
similar lighting conditions, making sure that for every natural image, we have
corresponding objective test image. With corresponding, meaning that the Analog gain,
exposure time and aperture are the same, hence the image quality attributes are very
3