Author: Gustavo Niemeyer
Date: 2004-12-05 14:29:24 -0200 (Sun, 05 Dec 2004)

# Smart Package Manager

Author: | Gustavo Niemeyer
Contact: | niemeyer@conectiva.com
Revision: | $Rev$
Date: | $Date$

Contents

- Overview
- Project Status
- Features
  - Modular
  - Smart Transactions
  - Multiple Interfaces
  - Channels
  - Priority Handling
  - Autobalancing Mirror System
  - Downloading Mechanism
  - Removable Media Support
- Running Smart
- Building Smart
  - Dependencies
- Case Studies
  - Case 1 - APT
  - Case 2 - APT & YUM
  - Case 3 - APT & YUM
  - Case 4 - APT
- Credits

# Overview

The Smart Package Manager project has the ambitious objective of creating smart and portable algorithms for solving adequately the problem of managing software upgrading and installation. This tool works in all major distributions, and will bring notable advantages over native tools currently in use (APT, APT-RPM, YUM, URPMI, etc).

From The Free On-line Dictionary of Computing:

    smart

        1. <programming> Said of a program that does the {Right Thing}
        in a wide variety of complicated circumstances. (...)

# Project Status

The development of Smart Package Manager started on May 4th, 2004, and version 1.0 was released on Aug 14th, 2008, after extended beta testing.

# Features

## Modular

Smart has been developed with modularity and flexibility in mind. It's completely backend-based, and package-manager-agnostic. Support is currently implemented for RPM, DPKG, and Slackware package management systems, and porting it to new systems should be very easy.

## Smart Transactions

That's one of the most interesting aspects of Smart Package Manager, and the one who has motivated calling it smart. Computing transactions respecting the relations involved in the package management world may become an unpleasant task when thousands of packages and relations are being considered, or even when just a few complex relations turn the most obvious choice into the unwanted one.

While other applications try to find a possible solution to satisfy the relations involved in some user-requested operation, and sometimes even fail to do so [1], Smart goes beyond it. In the kernel of Smart Package Manager lives an algorithm that will not only find a solution, if one is available, but will find the best solution. This is done by quickly weighting every possible solution with a pluggable policy, which redefines the term "best" depending on the operation goal (install, remove, upgrade, etc).

This behavior has many interesting consequences. In upgrades, for instance, while precedence is given to newer versions, intermediate versions may get selected if they bring a better global result for the system. Packages may even be reinstalled, if different packages with the same name-version pair have different relations, and the one not installed is considered a better option.

Another important goal achieved with the transaction algorithm is that, even though it is able to check and fix relations in the whole system, it will work even when there are broken relations in installed packages. Only relations related to the operation being made are checked for correctness.

[1] | Check Case Studies for real cases where the algorithm works better than what is implemented in other applications.

## Multiple Interfaces

Smart has multiple native and completely integrated interfaces:

- Command line interface, with several useful subcommands: update, install, reinstall, upgrade, remove, check, fix, download, search, and more.
- Shell interface, with command and argument completion, making it easy to perform multiple operations quickly using a local or remote terminal.
- Graphic interface, offering the friendliness of visual user interaction.
- Command line interface with graphic feedback, allowing one to integrate the power of command line with graphic environments.

Besides these interfaces, ksmarttray is also included in the Smart package. It notifies users about available updates using a KDE tray icon.

## Channels

Channels are the way Smart becomes aware about external repositories of information. Many different channel types are supported, depending on the backend and kind of information desired:

- APT-DEB Repository
- APT-RPM Repository
- DPKG Installed Packages
- Mirror Information
- Red Carpet Channel
- RPM Directory
- RPM Header List
- RPM MetaData (YUM)
- RPM Installed Packages
- Slackware Repository
- Slackware Installed Packages
- URPMI Repository

## Priority Handling

Priorities are a powerful way to easily handle integration of multiple channels and explicit user setups regarding preferred package versions.

Basically, packages with higher priorities are considered a better option to be installed in the system, even when package versions state otherwise. Priorities may be individually assigned to all packages in given channels, to all packages with given names, and to packages with given names inside given channels.

With custom priority setups, it becomes possible to avoid unwanted upgrades, force downgrades, select packages in given channels as preferential, and other kinds of interesting setups.

## Autobalancing Mirror System

Smart offers a very flexible mirror support. Mirrors are URLs that supposedly provide the same contents as are available in other URLs, named origins. There is no internal restriction on the kind of information which is mirrored. Once an origin URL is provided, and one or more mirror URLs are provided, these mirrors will be considered for any file which is going to be fetched from an URL starting with the origin URL.

Mirror precedence is dynamically computed based on the history of downloads of all mirrors available for a given origin URL (including the origin site itself). The fastest mirrors and with less errors are chosen. When errors occur, the next mirror in the queue is tried.

For instance, if a mirror http://mirror.url/path/ is provided for the origin ftp://origin.url/other/path/, and a file in ftp://origin.url/other/path/subpath/somefile is going to be fetched, the mirror will be considered for being used, and the URL http://mirror.url/path/subpath/somefile will be used if the mirror is chosen. Notice that strings are compared and replaced without any pre-processing, so that it's possible to use different schemes (ftp, http, etc) in mirror entries, and even URLs ending in prefixes of directory entries.

## Downloading Mechanism

Smart has a fast parallel downloading mechanism, allowing multiple connections to be used for one or more sites. The mechanism supports:

- Resuming
- Timestamp checking
- Parallel uncompression
- Autodetection of FTP user limit
- Cached file validation

and more.

At the moment, the following schemes are nativelly supported:

- file
- ftp
- http
- https
- scp

Additionally, the following schemes are supported when pycurl is available:

- ftps
- telnet
- dict
- ldap

## Removable Media Support

Smart Package Manager implements builtin support for removable media (CDROMs, DVDs, etc) in most of the supported channel types. The following features are currently implemented:

- Mountpoint autodetection
- Support for multiple simultaneous media drives
- Medias may be inserted in any order
- Installed system is guaranteed to maintain correct relations between media changes
- Remote removable media support using any of the supported schemes (ftp, http, scp, etc)

# Running Smart

Smart Package Manager may be run in many different ways, depending on the interface in use and on the intended goal.

The following command would install the foobar package, for instance:

    smart install foobar

While the following command would install the foobar package, but with graphic output:

    smart --gui install foobar

To open the graphic interface in interactive mode, one may simply run:

    smart --gui

Similarly, the following command would open the shell interface:

    smart --shell

Extensive help is available for all commands, by using the --help switch:

    smart --help
    smart install --help
    smart channel --help
    ...

# Building Smart

## Dependencies

Core: | Smart is written in Python, with some core modules rewritten as C extensions for memory savings and performance gains. With that in mind, the core system of Smart depends on Python 2.3 or higher, and a C compiler to build the extensions.
Graphic Interface:
| The "gtk" graphic interface depends on pygtk 2.4 or higher. The "qt" graphic interface depends on pyqt 3.3 (not 4.x).
RPM backend: | The RPM backend depends on the Python rpm module of RPM 4.4 or higher, due to a limitation which was present in previous versions of the ts.dbMatch() method, and the availability of the readHeaderFromFD() function. In the contrib/patches/ subdirectory there are patches for previous RPM versions including the missing functionality. There are also pre-packaged binary versions which include the patched module without requiring changes in other tools.
DPKG backend: | There are no extra dependencies besides DPKG itself.
Slackware backend:
| There are no extra dependencies besides the packaging scripts installpkg, upgradepkg and removepkg.

# Case Studies

In this section will be described real cases showing Smart behavior in comparison with other tools, or handling unusual situations.

Notice that Smart was not tuned to work in any of these cases, and the reason it works is because handling unusual situations was the initial project goal.

## Case 1 - APT

This case happened in a real world environment where a weakness in the algorithm used by APT (which is the same used in APT-RPM) turned a simple operation into a problem of obscure results. Smart Package Manager was used in the same environment to show its results.

The problem starts when an installation of xscreensaver is tried:

    [root@damien:/root] apt-get install xscreensaver
    Reading Package Lists... Done
    Building Dependency Tree... Done
    Some packages could not be installed. This may mean that you have
    requested an impossible situation or if you are using the unstable
    distribution that some required packages have not yet been created
    or been moved out of Incoming.

    Since you only requested a single operation it is extremely likely that
    the package is simply not installable and a bug report against
    that package should be filed.
    The following information may help to resolve the situation:

    The following packages have unmet dependencies:
      xscreensaver: Depends: libglade-2.0.so.0
                    Depends: libxml2.so.2
    E: Broken packages

The error shown makes the user believe that libglade-2.0.so.0 and libxml2.so.2 are not available. That's not the case:

    [root@damien:/root] apt-get install libxml2
    Reading Package Lists... Done
    Building Dependency Tree... Done
    Some packages could not be installed. This may mean that you have
    requested an impossible situation or if you are using the unstable
    distribution that some required packages have not yet been created
    or been moved out of Incoming.

    Since you only requested a single operation it is extremely likely that
    the package is simply not installable and a bug report against
    that package should be filed.
    The following information may help to resolve the situation:

    The following packages have unmet dependencies:
      libxml2: Depends: glibc-iconv but it is not going to be installed
    E: Broken packages

Another misguiding error message. Let's go further:

    [root@damien:/root] apt-get install glibc-iconv
    Reading Package Lists... Done
    Building Dependency Tree... Done
    Some packages could not be installed. This may mean that you have
    requested an impossible situation or if you are using the unstable
    distribution that some required packages have not yet been created
    or been moved out of Incoming.

    Since you only requested a single operation it is extremely likely that
    the package is simply not installable and a bug report against
    that package should be filed.
    The following information may help to resolve the situation:

    The following packages have unmet dependencies:
      glibc-iconv: Depends: glibc-gconvdata (= 2.3.3) but 1:2.3.2-586_1cl is to be installed
    E: Broken packages

Version 2.3.3 is needed, but 1:2.3.2-586_1cl is to be installed. This message is mostly correct. The only problem is, "1:2.3.2-586_1cl" is already installed:

    [root@damien:/root] apt-cache policy glibc-gconvdata
    glibc-gconvdata:
      Installed: 1:2.3.2-586_1cl
      Candidate: 1:2.3.2-586_1cl
      Version Table:
     *** 1:2.3.2-586_1cl 0
            100 RPM Database
         0:2.3.3-69473cl 0
            500 file: conectiva/all pkglist

The problem was found. A package from another repository (586_1cl shows it's not native, in that specific case) has a higher epoch than the one available in the usual repository. This clearly shows that the APT algorithm marks a single version as candidate, and when this is not the wanted version for some operation, the whole operation is compromised.

When testing Smart Package Manager in the same environment, the expected result is obtained:

    [root@damien:/root] smart install xscreensaver
    Updating cache...              ######################################## [100%]

    Computing transaction...

    Downgrading packages (1):
      glibc-gconvdata-0:2.3.3-69473cl.i386

    Installing packages (4):
      glibc-iconv-0:2.3.3-69473cl.i386
      libglade2-2.4.0-68154cl.i386
      libxml2-2:2.6.13-67598cl.i386
      xscreensaver-4.15-69825cl.i386

    Confirm changes (Y/n)?

Smart correctly selected glibc-gconvdata for downgrading as the only possibility of performing the user requested operation.

## Case 2 - APT & YUM

This is another real case, and is being reproduced in a controlled environment for tests with YUM, APT-RPM, and Smart.

The issue is, a package named A requires package BCD explicitly, and RPM detects implicit dependencies between A and libB, libC, and libD. Package BCD provides libB, libC, and libD, but additionally there is a package B providing libB, a package C providing libC, and a package D providing libD.

In other words, there's a package A which requires four different symbols, and one of these symbols is provided by a single package BCD, which happens to provide all symbols needed by A. There are also packages B, C, and D, that provide some of the symbols required by A, but can't satisfy all dependencies without BCD.

The expected behavior for an operation asking to install A is obviously selecting BCD to satisfy A's dependencies, on the other hand, YUM and APT fail to deliver that as a guaranteed operation, as is shown below.

First, let's see how YUM deals with the problem:

    [root@burma ~]% yum install A
    Setting up Install Process
    Setting up Repo:  localpub
    repomd.xml                100% |=========================|  951 B    00:00
    Reading repository metadata in from local files
    localpub  : ################################################## 5/5
    Resolving Dependencies
    --> Populating transaction set with selected packages. Please wait.
    ---> Downloading header for A to pack into transaction set.
    A-1.0-1cl.i386.rpm        100% |=========================| 1.0 kB    00:00
    ---> Package A.i386 0:1.0-1cl set to be installed
    --> Running transaction check
    --> Processing Dependency: libD for package: A
    --> Processing Dependency: libC for package: A
    --> Processing Dependency: libB for package: A
    --> Processing Dependency: BCD for package: A
    --> Restarting Dependency Resolution with new changes.
    --> Populating transaction set with selected packages. Please wait.
    ---> Downloading header for D to pack into transaction set.
    D-1.0-1cl.i386.rpm        100% |=========================| 1.0 kB    00:00
    ---> Package D.i386 0:1.0-1cl set to be installed
    ---> Downloading header for C to pack into transaction set.
    C-1.0-1cl.i386.rpm        100% |=========================| 1.0 kB    00:00
    ---> Package C.i386 0:1.0-1cl set to be installed
    ---> Downloading header for B to pack into transaction set.
    B-1.0-1cl.i386.rpm        100% |=========================| 1.0 kB    00:00
    ---> Package B.i386 0:1.0-1cl set to be installed
    ---> Downloading header for BCD to pack into transaction set.
    BCD-1.0-1cl.i386.rpm      100% |=========================| 1.0 kB    00:00
    ---> Package BCD.i386 0:1.0-1cl set to be installed
    --> Running transaction check

    Dependencies Resolved
    Transaction Listing:
      Install: A.i386 0:1.0-1cl

    Performing the following to resolve dependencies:
      Install: B.i386 0:1.0-1cl
      Install: BCD.i386 0:1.0-1cl
      Install: C.i386 0:1.0-1cl
      Install: D.i386 0:1.0-1cl
    Is this ok [y/N]:

YUM selected all packages for installation, even though BCD alone would satisfy A's dependencies.

Let's see how APT deals with that:

    [root@burma ~]% apt-get install A
    Reading Package Lists... Done
    Building Dependency Tree... Done
    The following extra packages will be installed:
      B BCD
    The following NEW packages will be installed:
      A B BCD
    0 upgraded, 3 newly installed, 0 removed and 0 not upgraded.
    Need to get 0B/4055B of archives.
    After unpacking 0B of additional disk space will be used.
    Do you want to continue? [Y/n] n

As a coincidence, APT did a better job, and selected only B and BCD to satisfy A's dependency, which is still not right.

Now, let's see how Smart would solve the problem:

    [root@burma ~]% smart install A
    Updating cache...               ######################################## [100%]

    Computing transaction...

    Installing packages (2):
      A-1.0-1cl@i386     BCD-1.0-1cl@i386

    2.7kb of package files are needed.

    Confirm changes (Y/n)?

Smart correctly selected only BCD, since it's necessary anyway, and solves all dependencies.

## Case 3 - APT & YUM

That's another interesting case which was tested with APT-RPM and YUM.

In this case, there's a package A version 1.0 installed in the system, and there are two versions available for upgrading: 1.5 and 2.0. Version 1.5 may be installed without problems, but version 2.0 has a dependency on B, which is not available anywhere.

In this case, the best possibility is upgrading to 1.5, since upgrading to 2.0 is not an option.

Let's see how APT reacts to this situation:

    [root@burma ~]% apt-get upgrade A
    Reading Package Lists... Done
    Building Dependency Tree... Done
    The following packages have been kept back
      A
    0 upgraded, 0 newly installed, 0 removed and 1 not upgraded.

APT seems to refuse to upgrade A, even though version 1.5 might be installed without problems.

What happens when forcing APT to install A:

    [root@burma ~]% apt-get install A
    Reading Package Lists... Done
    Building Dependency Tree... Done
    Some packages could not be installed. This may mean that you have
    requested an impossible situation or if you are using the unstable
    distribution that some required packages have not yet been created
    or been moved out of Incoming.

    Since you only requested a single operation it is extremely likely that
    the package is simply not installable and a bug report against
    that package should be filed.
    The following information may help to resolve the situation:

    The following packages have unmet dependencies:
      A: Depends: B but it is not installable
    E: Broken packages

It really refuses to install the newest version, and doesn't consider the possibility of using version 1.5.

Now, let's see how YUM would handle it.

Update: | This test case was showing the wrong results for YUM, since it was using a cached version of the package header that didn't present the missing dependency. I apologise for showing invalid results.

    [root@burma ~]% yum update
    Setting up Update Process
    Setting up Repo:  localpub
    repomd.xml                100% |=========================|  951 B    00:00
    Reading repository metadata in from local files
    primary.xml.gz            100% |=========================|  809 B    00:00
    MD Read   : ################################################## 3/3
    localpub  : ################################################## 3/3
    Resolving Dependencies
    --> Populating transaction set with selected packages. Please wait.
    ---> Downloading header for A to pack into transaction set.
    A-2.0-1cl.i386.rpm        100% |=========================| 1.3 kB    00:00
    ---> Package A.i386 0:2.0-1cl set to be updated
    --> Running transaction check
    --> Processing Dependency: B for package: A
    --> Finished Dependency Resolution
    Error: missing dep: B for pkg A

Just like APT, YUM selected version 2.0 and didn't consider the availability of an intermediate version.

Now, let's see how Smart would behave in the same situation:

    [root@burma ~]% smart upgrade
    Loading cache...
    Updating cache...               ######################################## [100%]

    Computing transaction...

    Upgrading packages (1):
      A-1.5-1cl@i386

    1.3kb of package files are needed.

    Confirm changes (Y/n)?

Smart correctly selects the intermediate version 1.5, which is the only viable possibility given the current options.

## Case 4 - APT

This case presents the following situation: there's a package A, installed in the system, which depends on libfoo, currently being provided by B 1.0. What happens if B is upgraded to version 2.0, but libfoo is moved to be provided by package C?

The expected behavior would be to upgrade B to version 2.0, and install C to satisfy A's dependency.

That's not what happens with APT:

    [root@burma ~]% apt-get dist-upgrade
    Reading Package Lists... Done
    Building Dependency Tree... Done
    Calculating Upgrade... Done
    The following packages will be upgraded
      B
    The following packages will be REMOVED:
      A
    1 upgraded, 0 newly installed, 1 removed and 0 not upgraded.
    Need to get 0B/1321B of archives.
    After unpacking 0B of additional disk space will be used.
    Do you want to continue? [Y/n]

Let's see Smart in the same situation:

    [root@burma ~]% smart upgrade
    Loading cache...
    Updating cache...               ######################################## [100%]

    Computing transaction...

    Upgrading packages (1):
      B-2.0-1cl@i386

    Installing packages (1):
      C-2.0-1cl@i386

    2.6kB of package files are needed.

    Confirm changes (Y/n)?

Smart correctly selected package C for installation as a viable possibility of leaving A installed in the system while upgrading B.

# Credits

This is the credit section, where people and institutions that have somehow contributed to the project are mentioned.

Conectiva, Inc.:
| Funded the creation of Smart, and its development up to August of 2005.
Canonical Ltd: | Is funding Smart development since September of 2005.
Wanderlei Cavassin:
| Conectiva's research & development coordinator, who believed the project was viable and encouraged the author to work on it.
Ednilson Miura & Herton Ronaldo Krzesinski:
| Conectiva employees, helped setting up many distributions for tests whenever necessary.
Andreas Hasenack:
| Conectiva employee, helped as being the first brave pre-alpha tester, and contributed with many ideas, discussions, etc.
Arnaldo Carvalho de Melo:
| Conectiva board member, helped with the "channel of mirrors" idea and by encouraging the author to build a generic channel information method.
Others @ Conectiva:
| Many other people in Conectiva helped with ideas and alpha-testing in general during the pre-release period of Smart development.
Guilerme Manika & Ruda Moura:
| Ancient Conectiva employees, now board members of the Haxent company, helped by testing Smart extensively in Fedora, reporting many bugs and suggesting changes. They have also created the Smart FAQ.
APT-RPM & Debian:
| Experience on packaging and ideas for a better framework were developed while the author of Smart worked as the APT-RPM maintainer.
Jeff Johnson: | Contributed as being the RPM maintainer itself, and in many discussions regarding packaging theory in general.
Seth Vidal: | YUM author, and member of the Duke University, contributed to Smart with the development of the XML MetaData repository format and discussions about it.
Michael Vogt: | Currently the maintainer of the Synaptic, used to co-maintain it with the author of Smart. Many of his ideas ended up being adopted in Smart as a consequence.
Sebastian Heinlein:
| Author of the package icons for Synaptic, that were mercilessly stolen to be used in Smart's graphic interface.
TaQ/PiterPunk at #slackware-br:
| These guys helped Smart development by explaining details of Slackware practices regarding packaging.
Matt Zimmerman: | Debian/Ubuntu developer and co-maintainer of the APT software, helped by shining some light regarding details of the DPKG pre-depends ordering expectations.
Mauricio Teixeira:
| FAQ maintenance, YaST2 channel maintainer, "tracker cleaner", general suggestions and code contributions.
Jonathan Rocker:
| Documentation help.