amos urjc 2

14 Pages·2015·0.21 MB·English

The Amos Project: An Approach
to Reusing Open Source Code
Manuel Carro
[email protected]
Technical University of Madrid (Spain)
Germán Puebla Carlo Daffara
[email protected] [email protected]
Technical University of Conecta s.r.l (Italy)
Madrid (Spain)
http://www.amosproject.org
Presente y futuro de la ingeniería del software libre
URJC, Madrid, May 21, 2003
URJC-May21,2003 http://www.amosproject.org/ –p.1/27
Software: Ubiquity and State
• Software is nowadays pervasive
• The preferred way to add intelligence to objects
• Basis for most of the current world economic infrastructure,
improvements in health care, logistics, manufacturing...
• Of 24 technologies of the future, 18 are software intensive
• Despite this: still unable to cope with software complexity
• Basically the same “software crisis” first described in 1968!
URJC-May21,2003 http://www.amosproject.org/ –p.2/27
Software Companies
• Standish Group report 1998: for US companies, regarding
software/IT projects:
• 26% succeeded (it was 27% in 1996)
• 28% failed
• 46% are “challenged” (“challenged” preferred to failures)
• Worldwide loss for bad software around 550.000 million euros
per year (extrapolated from reported numbers for U.S. companies)
URJC-May21,2003 http://www.amosproject.org/ –p.3/27
Reusing Software
• One of the solutions is software reuse
• Many different techniques:
• Software components
• Runtime binary package discovery / integration
• Component infrastructures & architecture
• ...
• In addition, of course, to friendly and helpful software
development tools
URJC-May21,2003 http://www.amosproject.org/ –p.4/27
Source Code and Reuse
• Experience in software engineering has demonstrated that
access to source code can reduce the effort needed for reuse
• Free/Open source gives a natural way to explore code reuse
URJC-May21,2003 http://www.amosproject.org/ –p.5/27
Source Code: Where?
• The problem:
finding suitable components
(or at least, a small set of candidate packages)
• Some companies make a living from finding (source code)
components for its customers building final software systems
• It is difficult:
• To search through many different packages
• To extract the best ones for the purpose at hand
• To estimate integration effort
URJC-May21,2003 http://www.amosproject.org/ –p.6/27
Tried Approaches
Formal methods:
• They give perfect matches
• Only small code fragments are handled
• Formal description is required
URJC-May21,2003 http://www.amosproject.org/ –p.7/27
Tried Approaches (Cont.)
Artificial intelligence:
• Nearly automated, use comments, function names
• Sometimes unable to extract sensible terms
• Multi-language projects difficult to handle
• Hit rate and precision not very high
URJC-May21,2003 http://www.amosproject.org/ –p.8/27
Tried Approaches (Cont.)
Brute force:
• Simple, fast
• Cannot handle knowledge on the package
• Confused by synonyms, too many false hits
• Lack of structure in the description of packages
URJC-May21,2003 http://www.amosproject.org/ –p.9/27
The AMOS approach
• Ontology for source package description (adds context for
searches, allows exchange of semantic information)
• Dictionary with descriptive vocabulary (including semantic part,
synonyms, generalizations)
• Database describing packagess by adding dictionary terms to
the ontology
• Search using a specialized engine to answer the queries
URJC-May21,2003 http://www.amosproject.org/ –p.10/27
Ontology
• In the realm of Amos: a classification of the properties and needs
of source code packages
• I.e., which language it is written in, which license does it have,
which capabilities are provided by it, which capabilities are
needed.
• Source code packages: not necessarily .rpm, .deb, .mdk —
even more: usually none of these
• Rather, real source code, which can be modified, (re)compiled,
chopped down to pieces and rebuilt
URJC-May21,2003 http://www.amosproject.org/ –p.11/27
Package Descriptions
• Pieces of source code placed in the onto-
logy tree by describing their (relevant) at-
tributes
• Vocabulary: it grows as new items are added to the database (a
dictionary ensures completeness and avoids dangling terms)
• The dictionary adds intelligence to the
EdgeDetection:imageprocessing,graphicfilter,
gradientdetection,...
system by including synonyms and gene-
ImageSmoothing:imageprocessing,graphicfilter,
coloraverage,...
ralizations GraphicFormatConversion:imageprocessing,
graphicformatlibraries,...
• Finally, packages are described
GhostScript ImageMagick The Gimp
using dictionary terms
• A single application may contain many (source code) packages
URJC-May21,2003 http://www.amosproject.org/ –p.12/27
An Illustratory Example
Dictionary of terms: edge detection, histogram, equalization, GIF,
PNG, JPG, video_input, video_output, video_compression, ...
Descriptions of source code packages:
• motion:
Needed: video_interface, ...
Provided: edge_detection, video_capture, out(JPG)
• gnomemeeting:
Needed: video_interface, audio_interface...
Provided: video_capture, video_output, video_compression,
video_decompression, audio_compression,
audio_decompression, audio_capture, audio_output
URJC-May21,2003 http://www.amosproject.org/ –p.13/27
Storage and Retrieval
• Descriptions are (conceptually) stored in a database
• Users state which capabilities they need
• A matching engine retrieves packages:
• Based on their description
AMOS
• Trying to fulfill as much as
engine
possible the user’s desires
• Probably using some heuristic,
in order to reduce search spa-
GhostScript
ce
ImageMagick
URJC-May21,2003 http://www.amosproject.org/ –p.14/27
A Real Retrieval Sample
• Based on a (test) database made from the RPMs installed on a
(Linux) laptop — 733 packages and 7484 search terms
• Not the final aim, but database easy to build and realistic
• WWW interface and search algorithm written in Ciao Prolog
• Search Terms: emacs latex gcc gnat latex2html
• Using no heuristics
• Search time: 50 ms. (in a 2.0 GHz Xeon)
• Results:
• Packages to consider: tetex-latex, bash-2.05a, emacs,
info, mktemp, perl-XML-Grove-0.46alpha
• Plus a list of unfulfilled capabilities which do not depend on
other packages (i.e., gnat in this case)
• And the list of all installed files (some 2600 files)
URJC-May21,2003 http://www.amosproject.org/ –p.15/27
More on Package Description
• Each package Ap is described using (among others):
q
• A set of required capabilities p: what the package needs
• A set of returned capabilities q: what the package offers
• Real example (actual syntax, but no commitment):
motion :=
[video_interface]
==>
[edge_detection, video_capture, out(JPG)]
• Syntax may change when data moved to external database
• User requirements launch a search against the package
description database
• Outcome: set of packages + set of unfulfilled capabilities
URJC-May21,2003 http://www.amosproject.org/ –p.16/27
Performing the Assembly
Input: R, a set of requirements
Output:(P,R)–sets of needed packages and unfulfilled requirements
Algorithm:
F := ∅ -- What has been fulfilled so far
P := ∅ -- Packages used so far
=
do -- Invariant: R ∩ F ∅
select Ap ∈ {Ap | q ∩ R 6= ∅}
q q
P := P ∪ {A}
F := F ∪ q
R := (R - q) ∪ (p - F)
until <no Ap can change R>
q
return (P, R)
1. Select (and remove) an element q from R
2. Look for a package Ap which satisfies it
q
3. If found, add package requirements to R; otherwise add q to the
set of non-satisfiable requirements
4. If R is nonempty, go to 1
URJC-May21,2003 http://www.amosproject.org/ –p.17/27
Efficiency of Matching Algorithm
• Efficient, but only to some extent
• Sample search on a (structured) Prolog database of some 7000
terms & 700 packages: 170 ms
• How does using a flat/external D.B. affect performance?
D.B. implementation Time (ms.)
Original Prolog implementation 171
Plain fact-based Prolog database 2533
MySQL database 29193
• Some more intelligence is needed!
URJC-May21,2003 http://www.amosproject.org/ –p.18/27
Matching Algorithm II
• Essentially: precompute some search into inverted indices
• Packages are not scanned looking for those offering some
capability
• Instead a list of packages is associated to each capability
• Very good speedup achieved!
D.B. implementation Time (ms.)
Original Prolog implementation 6
Plain fact-based Prolog database 7
MySQL database 76
• Enough to fulfill speed requirements in realistic DB
• Code still easy to understand
• Other optimizations (e.g., caching) can still be applied
URJC-May21,2003 http://www.amosproject.org/ –p.19/27
The Whole Enchilada
• WWW-based interface
• Interaction with toolset based on CGIs
WWW WWW
• CGIs relatively light
Browser Daemon
• DBServer: part of Ciao CGI
(On demand)
Prolog libraries
• It provides an interface Add Review
Search
packages Packages
to persistent data
• Also makes CGIs to be
Daemon
smaller (Permanent)
DB Server
• Same technology can
be applied to search in-
terface, if necessary Permanent
database
Non permanent
database
URJC-May21,2003 http://www.amosproject.org/ –p.20/27