Mock article for Studia Informatica

STUDIA INFORMATICA
Volume 37
2016
Number 3B (126)
Adam POPOWICZ
Silesian University of Technology, Institute of Automatic Control
Aleksander R. KUREK
Jagiellonian University Astronomical Observatory
AN ALGORITHM FOR JOINT AND BONE LOCALIZATION IN USG
IMAGES OF RHEUMATOID ARTHRITIS
Summary. The assessment of degree of joint inflammation using USG imaging is a
challenging task, and thus the results may differ between experts. Therefore, the evaluation of
images by a computer program may be a vital solution for objective assessment of disease
progression. In this paper an original pipeline, developed for precise estimation of bone line
and joint location in USG images, is proposed. The presented set of methods provides
valuable input for further classification tools, which will aim at identification and rating of
the degree of joint inflammation.
Keywords: medical image processing, segmentation
ALGORYTM WYZNACZANIA POŁOŻENIA STAWU ORAZ KOŚCI
W OBRAZACH USG PALCÓW DŁONI
Streszczenie. Ocena stopnia zapalenia stawów palców dłoni z wykorzystaniem
obrazów USG jest zwykle trudnym zadaniem i może być różna w zależności od eksperta
analizującego obrazy. Z tego względu analiza wykorzystująca program komputerowy
stanowić może obiektywny wskaźnik progresu choroby. W artykule zaprezentowano zestaw
procedur przetwarzania obrazów, umożliwiający wykrycie położenia kości oraz stawu
w obrazach USG. Wyniki zwracane przez algorytm stanowią istotną podstawę do dalszych
kroków, mających na celu wykrycie oraz gradację stopnia zapalenia.
Słowa kluczowe: przetwarzanie obrazów medycznych, segmentacja
8
A. Popowicz, A.R. Kurek
1. Introduction
Ultrasound imaging, which enables detailed insight into internal structures of human
body, has also proved its usefulness for precise estimation of a degree of joint inflammation
in Rheumatoid Arthritis [1]. By a visual assessment of a USG image, a physician scores the
progression of disease from 0 to 3, using the OMERACT definitions [2]. Unfortunately, the
rating is usually somehow subjective since it depends on the experience of an examiner and
may be affected by the standardized atlases currently utilized [3-4]. Therefore, it was
frequently observed that the diagnosis of experts differ one from another. That is why it was
decided to initiate a research (MEDUSA project, [5]) on possible automatic detection and
rating of joint inflammation using image processing and classification tools.
The location and size of synovitis region depends on severity of the disease. Initially, it
develops from a joint, extending gradually along a bone toward a skin layer. The pathological
region is usually more or less dimmer than the normal tissue (however, it is not a strict rule).
Three examples of different degree of joint synovitis are presented in Fig. 1. The marked
characteristic parts include inflammation region (green), bone line (red), skin line (violet) and
joint (turquoise) as indicated by an experience physician.
Fig. 1. Examples of USG images of joint of various degree (from the left: 1, 2 and 3). Synovitis,
bone and joint are marked by green, red and turquoise, respectively
Rys. 1. Przykłady obrazów USG stawu o różnym stopniu zapalenia. Obszary zapalenia kości oraz
stawu zostały zaznaczone kolorem odpowiednio zielonym, czerwonym i fioletowym
In this paper, a pipeline for automatic detection of joint and bone line is proposed. It was
indicated by physicians that the synovitis develops along the bone and starts from the joint, it
was concluded, that the proper localization of these two characteristic regions is of the
highest importance. In presented algorithm a novel simple method of filtering of USG images
is employed. The filter enhances the bones which are further processed by simple
morphological operations (dilation, thresholding, calculation of region areas) leading to the
estimation of a bone line. A joint position is subsequently obtained by analyzing the profile of
bone, looking for its local minima.
It should be noted, that there have been already some attempts to localize bones and joint
in such USG images [6-8]. However, in contrast to those methods, the pipeline presented in
this paper is significantly simpler and requires much less computational effort. It does not
An algorithm for joint and bone localization in USG images of rheumatoid arthritis
9
incorporate sophisticated classifiers which require and, unfortunately, are dependent on
selection of a training set. As its operations can be easily understood, the implementation of
proposed pipeline is straightforward. Its outcomes can be also utilized in parallel with the
results provided by much more sophisticated algorithms, to increase the overall efficiency of
such a multi-output detection system.
2. Pipeline
2.1. Detection of bone and joint position
The first part of the pipeline aims at the enhancement of regions of finger bones. It is
accomplished with a use of a novel filter developed especially for such USG images. The
filter is based on the physics of assessment of an acoustic signal on obstacles, e.g. in form of
bone. The basic absorption Lambert’s law indicates that the amplitude 𝐽 of acoustic wave
moving in a body decreases as:
𝑑
𝐽(𝑑) = 𝐽(0) 𝑒 −𝜎 ,
(1)
where d is the distance, 𝜎 is the parameter of the tissues (the denser the tissue, the lower
the 𝜎). The properties of encountered tissues result in various reflections and absorption of
incoming sound wave . The sound is reflected strongly by the bones, thus the intensity of this
type of objects in USG images is distinctively higher. Due to the reflection, the wave passing
through bone is strongly attenuated and therefore the intensity of pixels below bone are much
dimmer. Based on this observation, the assumption was made that the bone pixels are the
ones which are not proceeded (along the image column) by any brighter pixel. To reveal such
regions, the pixels of original image are filtered by reducing its intensity by a maximum
intensity of pixels located in a rectangular region below:
𝐼(𝑥0 , 𝑦0 ) = 𝐼(𝑥0 , 𝑦0 ) −
min
𝑦 > 𝑦0 +𝑅,
𝑥 𝜖 {𝑥0 −𝑀 ∶ 𝑥0 + 𝑀}
𝐼(𝑥, 𝑦) ,
(2)
where 𝐼(𝑥, 𝑦) is the intensity of pixel at (𝑥, 𝑦), R and M are parameters. The R parameter
can be adjusted to allow for wider (higher R) or shorter (lower R) bone regions. The second
parameter M is the width of filtering window and it provides for a possible scattering of
acoustic waves in x-direction of an image. To suppress the speckle noise present in USG
images, a simple median filter (K×K mask) is employed before the filtration process. The
optimization of the values of filter parameters (R, M and K), as well as other tunable
parameters included in the pipeline, will be discussed in the next sections of this paper.
Exemplary outcomes of the proposed routine are presented in Fig. 2.
10
A. Popowicz, A.R. Kurek
(a)
(b)
(c)
Fig. 2. Exemplary USG image processed by the proposed filter: (a) original image, (b) median
filtered image (11×11 pixels mask), (c) filtered outcome
Rys. 2. Przykłady obrazów USG przetworzonych za pomocą proponowanego filtru: (a) obraz
oryginalny, (b) wynik filtracji medianowej, (c) wynik filtracji proponowaną techniką
The computational burden of such a filtering schema can be significantly reduced by (1)
calculating the intensity maxima progressively in each line and then (2) by considering only
the maxima located with a specified shift (R, M) relatively to currently processed pixel. The
MATLAB code that allows for such efficient calculation of filter output is given below:
function [filtered] = filter_image(image,M,R)
[X,Y] = size(image);
image = medfilt2(image,[11,11]);
local_max = 0*image;
for y = 1:Y
column_max = 0;
for x = X:-1:1
column_max = max(column_max,image(x,y));
local_max(x,y) = column_max;
end
end
filtered = -inf*image;
M_2 = round(M/2-1);
for x = 1:X-R
for y = M:Y-M
filtered(x,y) = max(local_max(x+R,y-M_2:y+M_2));
end
end
filtered = image – filtered;
filtered (filtered <0) = 0;
end
In the next step of pipeline, the filtered image is thresholded with an optimized value T,
(the optimization is explained further in Section 3). The detected regions are labeled, so that
each one has its area calculated. The largest one is selected as final bone mask and all the
pixels below and above this region are discarded from further analysis. Then, the areas of
remaining regions are recalculated and the largest one is added to the final mask only if its
area is larger than 10% of the first, initially selected region. This process is repeated 3 times,
which appeared to be just enough to reveal the largest bones even after the thresholding they
were not marked as fully consistent regions (i.e. split into several smaller ones). Examples of
adding bone regions are presented in Fig. 3.
An algorithm for joint and bone localization in USG images of rheumatoid arthritis
(a)
(b)
(c)
11
(d)
Fig. 3. Steps leading to the creation of bone mask: (a) filtered image, (b) thresholded image, (c) first
largest bone detected, (d) second largest bone detected. In third iteration there was no region
which satisfied minimal area criterion
Rys. 3. Kolejne kroki prowadzące to uzyskania maski kości: (a) obraz przefiltrowany, b) progowanie,
(c) pierwsza wykryta kość, (d) druga wykryta kość. W trzeciej iteracji algorytmu nie wykryto
obszarów spełniających kryteria detekcji
After the bones are detected, their upper line is determined. It is done by simple
subtracting a bone mask from its dilated version. This gives the contour of bones which is
further processed by picking only the first marked pixel in each image column. Exemplary
result of estimation of a bone line is given in Fig. 4.
(a)
(b)
Fig. 4. Determination of a bone line: (a) largest bones detected, (b) final bone line
Rys. 4. Określanie linii kości: (a) największe wykryte kości, (b) ostateczna linia kości
Such a definition of a bone line allows converting it to a signal (BS=f(x)). It is known,
that the bone line should be smooth and therefore this signal should to be filtered using
smoothing filter to reduce any unwanted irregularities. In this processing step, y coordinate of
each point of a bone line is replaced by the median of y coordinates of bone line points within
a local window ( ±10 pixels in x axis). Example of filtering of the bone line from Fig. 4 is
depicted in Fig. 5. This routine not only smooths the line, but also rejects possible outliers
appearing e.g. due to the false detections of bones within the interference-related artifacts.
12
A. Popowicz, A.R. Kurek
Fig. 5. Smoothing the bone line from Fig. 4: blue and red dots represent bone line before and after
median filtering, respectively
Rys. 5. Wygładzanie linii kości z rysunku 4. Kolorami niebieskim oraz czerwonym zaprezentowano
linię kości, odpowiednio, przed oraz po wygładzaniu
2.2. Detection of joint position
The next part of the algorithm aims at detecting joint position. It was observed that the
joint is located usually in a local minimum of a bone line, surrounded by the elevation of a
line from both, left and right side. To find such a place, bone signal (BS) is conditionally
filtered by subtracting maxima of BS signal on the left and right side of currently processed
pixel:
max
𝐵𝑆𝑓 (𝑥) = 𝑥𝜖{𝑥− 𝑑∶𝑥−1}
𝐵𝑆(𝑥)+
2
max
𝑥𝜖{𝑥+1:𝑥+𝑑}
𝐵𝑆𝑓 (𝑥) = 0 if 𝐵𝑆(𝑥) >
𝐵𝑆(𝑥)
max
− 𝐵𝑆(𝑥)
{𝐵𝑆(𝑥)}
𝑥𝜖{𝑥− 𝑑∶𝑥−1}
(3)
𝐵𝑆𝑓 (𝑥) = 0 if 𝐵𝑆(𝑥) >
max {𝐵𝑆(𝑥)}
{
𝑥𝜖{𝑥+1∶𝑥+𝑑}
where BSf is a filtered bone signal, d is a tunable parameter, which controls the width in x
axis within which a local minimum is calculated. The conditional statements in (3) ensure
that the average height (i.e. y position) of a bone line on the left and right side is larger than
the height of a joint. To further reduce the rate of false detections, only the pixels within a
1
7
limited x range (4 𝑥: 8 𝑥) are considered. Such asymmetrical selection of a region of interest
is justified by the fact that joint is usually located slightly on the right side of an image center.
It is done to expose the left-side bone which is frequently covered by the inflammation.
Eventually, a joint position is determined from the maximum of BSf signal. Exemplary
BSf signal calculated for a sample USG image is depicted in Fig. 6. Note the consecutive
steps of processing of bone signal: limitation of bone line (green points), filtering (red points)
and, finally, finding the position of joint (black circle).
An algorithm for joint and bone localization in USG images of rheumatoid arthritis
13
Fig. 6. Estimation of a joint position in a sample USG image. Above: USG image with indicated
bone line (red) and joint position (green cross). Below: bone signal (blue) with its filtered
version (BSf, green) with a joint position indicated (black circle)
Rys. 6. Estymacja położenia stawu w przykładowym obrazie USG. Powyżej: obraz USG
z zaznaczonym na czerwono położeniem kości oraz zielonym wskaźnikiem stawu. Poniżej:
linia kości (niebieska) wraz z jej odfiltrowaną wersją (BSf, kolor czerwony) oraz wskaźnik
położenia stawu (czarny okrąg)
3.
Optimization
3.1. Reference data set
The reference set of 276 USG images, depicting various degree of joint synovitis, was
collected during the realization MEDUSA project, in a public health institution, Helse Forde,
Norway [9]. The set consists of USG images with regions of bone line, joint and area of
synovitis indicated by an experienced physician. All images depicted only finger crosssections, two types of joints: metacarpophalangeal joint (MCP) and proximal interphalangeal
joint (PIP). Additionally, each image was assigned with a given degree of inflammation (0-3,
where 0 stands for no visible synovitis). Several examples of images from our dataset are
presented in Fig. 1 and Fig. 8.
3.2. Quality measures
To assess the proximity between regions indicated by the presented algorithm and the
ones marked by an expert, a special method was developed. Since the detection is not a single
14
A. Popowicz, A.R. Kurek
pixel, but rather a set of pixels, the distance transformation of binary images was applied.
First, the bone line indicated by an expert is processed by the distance transformation so that
the map of distances to detected line is created. Then, the mean bone distance (Qb) between
reference and determined bone lines can be calculated by averaging the distances assigned to
the pixels marked as a bone line by the presented pipeline. The distance (Qj) between
reference and detected joint position can be obtained similarly, using distance transformation.
The distance transformation is a well-known image processing procedure, which allows for
estimation of Euclidian distance between each pixel and its nearest indicated object. The
algorithm uses the double-scan algorithm and is described in details in [10].
In the case of our images, the joint estimation error is simply the value of distance from
the joint indicated by an expert. An example of distances estimated in such a quality
assessment are presented in Fig. 7. The distances were calculated for the red, reference
points, while the mean distance was calculated be averaging distances assigned to green
pixels.
3.3. Parameters optimization
The reference set of 276 USG images was processed with the pipeline employing various
values of parameters. The optimized parameters were: R, M – parameters of bone filter, K –
mask of median filter, T – threshold used for bone selection, and d – width of a window
utilized during the search of local minima in filtered bone signal. The range of possible
values of the parameters is presented in Table 1. The smallest values of quality indicators
(Qf = 8.6 pix. and Qb = 33.9 pix) was achieved for the parameters well within a specified
range and are given in Table 2. Several examples depicting various types of USG images
with indicated bone and joint by both, expert and by the optimized pipeline are exposed in
Fig. 8. The mean distance of automatic and reference detections, as calculated using Q
measures, is highlighted with green and yellow, respectively for bone and joint.
4.
Summary
The USG imaging allows for detailed view of joint inflammation in Rheumatoid Arthritis.
Unfortunately, the image complexity is so high that the grading of synovitis can differ even
between the experts. Therefore, for the purpose of objective and reproducible assessment of
such images, a dedicated computer program is being developed in MEDUSA project.
An algorithm for joint and bone localization in USG images of rheumatoid arthritis
15
Fig. 7. Assessment of the error of pipeline results (Qb=3.5 pix., Qj=10 pix.). Green markers show the
output of the proposed pipeline, while the red ones depict the markings indicated by an expert.
Pixels intensities correspond to the distances from the reference points
Rys. 7. Ocena błędu wyznaczania obszarów (Qb=3.5 pix., Qj=10 pix.). Zielonym markerem
zaznaczono wyniki działania algorytmu. Czerwony kolor przedstawia wskazania lekarza.
Jasności pikseli odpowiadają dystansowi od wskazań referencyjnych
Table 1
Range of parameters values utilized in the optimization
Parameter
Range
Step
R
M
T
K
D
10:40
2:10
2:10
11:21
60:120
10
2
2
2
20
Table 2
Optimal values of parameters for joint and bone line estimation
Bone estimation
Joint estimation
Parameter
(Qb = 8.6 pix)
(Qj = 33.9 pix)
.
R
M
T
K
D
20
4
8
11
N/A
20
2
2
15
100
16
A. Popowicz, A.R. Kurek
Fig. 8. Examples of automatic and manual annotation of bone line and joint position in USG images.
Bone and joint indicated by an expert are marked by red and blue points respectively. Green
bold line and yellow circle show automatically detected bone line and joint. Green and yellow
highlighted regions expose the areas within mean distance as calculated using Q measures
Rys. 8. Przykłady manualnej oraz automatycznej detekcji kości oraz stawu w obrazach USG. Kość
i staw wskazane przez lekarza zostały zaznaczone kolorem czerwonym oraz niebieskim.
Zielona pogrubiona linia oraz żółte kółko przedstawiają wykrytą kość oraz staw. Na zielono
oraz żółto podświetlono obszary średniego błędu Qb oraz Qj
An algorithm for joint and bone localization in USG images of rheumatoid arthritis
17
The first part of such a system includes the precise estimation of bone line and joint
position. It is a challenging and crucial task, which directly affects an overall efficiency of
automated grading system. For this purpose, a dedicated pipeline was presented in this paper.
In a first step of the algorithm, a novel filter, incorporating simple procedures of image
processing, was utilized to highlight the regions associated with bones. A computationally
efficient implementation of the filter was also provided in this work. For the purpose of
localization of joint position, the pipeline performs the search for local minima in estimated
bone line. Several modifiable parameters allows for fine-tuning of the algorithm, so that its
outputs are consistent with the reference indications provided by an expert.
The pipeline, in contrast to the competitive solutions which employ machine-learning
methods and complex classification procedures, does not require much computational effort.
It is straightforward and may be easily implemented in any programing language.
Additionally, this simplicity allows for future modifications and improvements in particular
stages of the pipeline. The results of the method may also support the outcomes of much
more sophisticated approaches, improving the final efficiency of detection system.
BIBLIOGRAPHY
1. Węgrzyn S.: Podstawy automatyki. PWN, Warszawa 1978.
2. Østergaard M., Ejbjerg B., Szkudlarek M.: Imaging in early rheumatoid arthritis: roles of
magnetic resonance imaging, ultrasonography, conventional radiography and computed
tomography. Best Practice & Research Clinical Rheumatology, 2005, vol. 19(1),
p. 91÷116.
3. Østergaard M., Edmonds J., McQueen F., Peterfy C., Lassere5 M., Ejbjerg B., Bird P.,
Emery P., Genant H., Conaghan P.: An introduction to the EULAR–OMERACT
rheumatoid arthritis MRI reference image atlas. Annals of the Rheumatic Diseases, 2005,
vol. 64, p. i3÷i7.
4. Terslev L., Torp-Pedersen S., Savnik A., von der Recke P., Qvistgaard E., DanneskioldSamsøe B., Bliddal H.: Doppler ultrasound and magnetic resonance imaging of synovial
inflammation of the hand in rheumatoid arthritis: a comparative study. Arthritis &
Rheumatism, 2003, vol. 48(9), p. 2434÷2441.
5. Kulbacki M., Segen J., Habela P., Janiak M., Knieć W., Fojcik M., Mielnik P,
Wojciechowski K.: Collaborative Tool for Annotation of Synovitis and Assessment in
Ultrasound Images. International Conference on Computer Vision and Graphics ICCVG
2014.
18
A. Popowicz, A.R. Kurek
6. Mielnik P.: Brief information on research work implemented under the MEDUSA
project. Norsk Rheumabulletin, 2013, vol 4, p. 3.
7. Segen J., Kulbacki M., Wereszczyński K.: Registration of Ultrasound Images for
Automated Assessment of Synovitis Activity. Intelligent Information and Database
Systems, 2015, vol. 9012, p. 307÷316.
8. Segen J., Kulbacki M., Wereszczyński K., Wojciechowski K.: Optimization of Joint
Detector for Ultrasound Images Using Mixtures of Image Feature Descriptors. Intelligent
Information and Database Systems, Intelligent Information and Database Systems, 2015,
vol. 9012, p. 277÷286.
9. Wereszczynski K., Segen J., Kulbacki M., Mielnik P., Fojcik M., Wojciechowski K.:
Identifying a joint in medical ultrasound images using trained classifiers. International
Conference on Computer Vision and Graphics ICCVG 2014.
10. Kulbacki M., Segen J., Habela P., Janiak M., Knieć W., Fojcik M., Mielnik P.,
Wojciechowski K.: Collaborative Tool for Annotation of Synovitis and Assessment in
Ultrasound Images. International Conference on Computer Vision and Graphics ICCVG
2014.
11. Borgefors G.: Distance transformations in digital images. Computer Vision, Graphics,
and Image Processing, 1986, vol. 34, no. 3, p. 344÷371.
Acknowledgment
The research leading to these results has received funding from the Norwegian Financial
Mechanism 2009-2014 under Project Contract No. Pol-Nor/204256/16/2013 and was
performed using the infrastructure supported by POIG.02.03.01-24-099/13 grant: GCONiI Upper-Silesian Center for Scientific Computation.
Omówienie
Ocena stopnia zapalenia stawów palców dłoni z wykorzystaniem obrazów USG jest
niezwykle trudnym zadaniem. Gradacja stopnia zapalenia może być różna w zależności od
eksperta oceniającego obraz. Z tego względu analiza wykorzystująca program komputerowy
stanowić może obiektywny wskaźnik progresu choroby. Program taki jest głównym celem
projektu MEDUSA. W artykule zaprezentowano zestaw procedur przetwarzania obrazów,
umożliwiający wykrycie położenia kości oraz stawu w obrazach USG. Główna część
An algorithm for joint and bone localization in USG images of rheumatoid arthritis
19
algorytmu wykorzystuje nową technikę filtracji obrazów, dzięki której uwidaczniane
i następnie ekstrahowane są obszary obrazu związane z obecnością kości. Poszukiwanie
położenia stawu odbywa się dzięki ocenie lokalnych minimów w linii tworzonej przez
przekrój kości. Zaprezentowane metody zostały zoptymalizowane dzięki wykorzystaniu kilku
modyfikowalnych parametrów, tak aby zmaksymalizować podobieństwo wyników algorytmu
oraz referencyjnych wskazań dokonanych przez eksperta. Zestaw algorytmów dostarcza
istotnych informacji wejściowych wymaganych do dalszych kroków związanych z detekcją
i gradacją stopnia zapalenia stawu.
Addresses
Adam POPOWICZ: Silesian University of Technology, Institute of Automatic Control,
ul. Akademicka 16, 44-100 Gliwice, Poland, [email protected]
Aleksander R. KUREK: Jagiellonian University Astronomical Observatory, ul. Orla 171
30-244 Kraków, Poland, [email protected]