<!-- vim: set tw=78 sts=2 ai expandtab: -->
# debian-reference (Version 2 series)

Fri, 09 Mar 2018 15:30:36 +0000

Osamu Aoki, <osamu at debian dot org>

This Debian Reference (v2) is intended to provide a broad overview of the
Debian system as a post-installation user's guide. It covers many aspects of
system administration through shell-command examples for non-developers.

## Package status

The package status of `debian-reference` can be checked at the following URLs:

 * https://bugs.debian.org/src:debian-reference    (BTS)
 * https://tracker.debian.org/pkg/debian-reference (Package)
 * https://www.debian.org/doc/manuals/debian-reference/ (Latest HTML page)
 * https://www.debian.org/doc/user-manuals#quick-reference (Links on debian.org)

## Bug report

For bug report, please use the `reportbug` command against the
`debian-reference` source package.

```
 $ reportbug --src debian-reference
```

For bug reports, it is strongly recommended to write what you want explicitly
as an alternative.  "It does not work...", "I can do this...", etc. aren't
very useful.  Bug report such as "The section 9.3.3 should be rewritten as
.... because .... " is desirable style.  (I do not wish to spend time to
second-guess or search where you see the issues.)

For simple grammar errors and spell errors, diff -u against the plain text
file or the source file are most appreciated.

See also, http://bugs.debian.org for how Debian BTS works.

## Source tree

Please check following web page and download the source tree using the `git`
command from salsa.debian.org.

 * https://salsa.debian.org/debian/debian-reference

For read-only access:
```
 $ git clone https://salsa.debian.org/debian/debian-reference.git
```

For read-write access with SSH:
```
 $ git clone git@salsa.debian.org:debian/debian-reference.git
```

The latest relesed source package can be downloaded by:
```
$ apt-get source debian-reference
```
Here, `deb-src` line in `/etc/apt/sources.list` file should be pointing to
unstable.

You are welcomed to join the maintenance team and update the source tree.

For the English portion and the build scripts, please contact us with your
intended action at the Debian documentation project
<debian-doc@lists.debian.org> or file bug fix patches to the Debian BTS before
committing the changes to minimize major breakage.  

For translation via PO files, it is nice if you check their correctness before
committing the changes.  In practice, I have been quite permissive for the
translation updates and have fixed them manually if I find problems later.

All Debian Developers (DD) have the write access with the "Master" status for
this debian-reference salsa repository.  If you are a non-Debian Developer
with `*-guest` account on salsa who needs write access to this
debian-reference salsa repository, you can inform your `*-guest` account to me
(or any DD) with clear intent and reasonable rationale and let me (or any DD)
invite you to the write access with the "Developer" status from [the main
debian-reference project
page](https://salsa.debian.org/debian/debian-reference) via Setting-Member
menu.

See https://wiki.debian.org/Salsa/Doc for how to use this site.

### Updating the repository

If you are updating the English content in asciidoc/*, please make sure to
update debian-reference.raw.xml and commit it.

If you are updating the translation only, please only commit the updated
`po/*.po` to the `master` branch.  I mean no commit of `po/templates.pot`,
`*.xml`, `*.ent`, and `asciidoc/*.txt` files.  In other word, please don't use
"`git commit -a`".  Please note that the merging of PO files are non-trivial
task which doesn't work easily with the `git` merging infrastructure.  This is
safely realized by creating a local branch and work in the branch.  Then
commit only a non-conflicting changes to the remote `master` branch.

If you are not familiar with git, it may be a good idea to submit your updated
`po/*.po` to the BTS.  Then the package maintainer can merge and update the
source.

If you wish to push your work-in-progress local branch to the remote server as
a private backup, please use the branch name such as `wip-foo-guest` if your
account name is `foo-guest`. Others are expected not to use such branches and
you are recommended to remove them later.

```
 $ git checkout master
 $ git checkout -b wip-foo-guest
 $ make wrap;make po        # check fuzzy.log
  ... hack-hack-hack
 $ git push origin wip-foo-guest # private backup
  ... hack-hack-hack
 $ git checkout master      # get the latest
 $ git pull origin master
 $ gitk -a po/<language>.po # verify no recent commit on po/<language>.po
 $ git checkout wip-foo-guest po/<language>.po
 $ make wrap;make po        # check fuzzy.log
 $ git add po/<language>.po
 $ git commit -m "Describe commit ..."
 $ git push origin :wip-foo-guest
```

You can avoid using local branch as above if you use `git stash` and `git
stash pop`.

Any missing PO strings of `zh-tw`/`zh-cn` are temporarily auto converted from
the existing `zh-cn`/`zh-tw` PO strings during the build time.  So you may
find non-optimal translation strings not-found in the source tree to show up
in the generated document.  I chose this build setting since non-optimal
strings are better than the original English ones for most Chinese readers.
You can have the correct translation strings by adding proper PO strings.

Due to bugs in the TeX tool chain for the PDF generation, generation of the
proper PDF files may be disabled and only the placeholder PDF files may be
generated.

### Translation approaches

Translator are encouraged to use the dedicated PO file editors such as
`poedit`, `gtranslator`, `gettext.el` (with `emacs`), and `lokalize` on the PO
files `po/*.po` after obtaining the source tree with the `git` command.  This
manual approach helps debugging of the PO file by locally test building the
source tree.

Recently, people also seems to be contributing translation via the weblate
service.

 * https://hosted.weblate.org/projects/debian-reference/

(I am not quite sure how and who merge patches from weblate to the source.)

### Translation tips

####  Sanity check of a PO file:
```
 $ msgcat some.po >/dev/null
```
 -> Look for unquoted " and accidental folding of lines.

#### Remove duplicate entries in a PO file
```
 $ mv some.po some.old.po
 $ msguniq some.old.po >some.po
````

#### Add a new PO

You can create a new template PO file for a new language and edit it as:
```
 $ cd po
 $ cp templates.pot ??.po
 $ poedit ??.po
```

#### Merge PO files

When 2 person update a PO file, you need to merge them using tools in the
`gettext` package.

You can check if 2 PO files are different using:
```
 $ msgcat --less-than=2 filename.po.mine filename.po.their
```
If you have no order of preference:
```
 $ msgmerge filename.po.mine filename.po.their >filename.po
```
Look for "`#-#-#-#-#`" where manual resolution is needed and resolve it
via editor.

If your order of preference puts yours over theirs:
```
 $ msgcat --use-first filename.po.mine filename.po.their >filename.po
```

### Test build tips

#### Full build

You can test build the whole tree in `testing`/`sid` environment as:

```
 $ sudo apt-get update
 $ sudo apt-get install devscripts grep-dctrl wget git gitk
 $ sudo apt-get build-dep debian-reference
 $ cd /path/to/debian-reference
 $ debuild
  ...
```

Please note this takes very long time.

#### Partial build after translation

If you just translated a PO file, e.g. `po/it.po`, please
reformat and test it first by generating HTML files only:

```
 $ make wrap; make po
 $ make test "LANGPO=it"
```

You have English and Italian HTML files under the `html/` directory.

Specifying "`LANGPO`" variable as above saves build time by skipping other
languages.

#### Partial build after minor English updates

If you just updated typos in the ASCIIDOC file, e.g. `asciidoc/foo.txt`, please
test it first by generating HTML files only:

```
 $ make debian-reference.raw.xml
 $ make test "LANGPO="
```

You have English HTML files under the `html/` directory.

If this test causes no error, clean the source tree and commit it with:
```
 $ make clean
 $ git add asciidoc/*.txt
 $ git add debian-reference.raw.xml
 $ git add bin/replace_package
 $ git commit -m "Updated ...."
 $ make wrap; make po; make clean
 $ git commit -a -m "make po"
```

Please note many PO files are changed in the way `git` doesn't merge cleanly.

#### Full build after major English updates

For major English updates in ASCIIDOC files with new URL references or new
packages, please `distclean` and `entity` first to update the package
database.
```
 $ make distclean
 $ make entity
```

The rest are the same as the minor English updates.

### Tricks used for ASCIIDOC and PO files

There are some non-conventional tricks used in ASCIIDOC to work around special
characters.

#### `@@@` in ASCIIDOC

You see quite a bit of `@@@` in asciidoc source.  That is placed there
when idiosyncrasies of asciidoc hit me.  The result of Asciidoc to raw.xml 
conversion is filtered with the sed script bin/replace to get right 
characters into the resulting XML file (debian-reference.en.raw) without 
worrying about asciidoc converting them into something funny.

For example: 

 * `@@@sq@@@` is converted to `'` 

These conversions occur right after asciidoc conversion but before
storing to raw.xml files.  So this is used as purely the workaround for 
the asciidoc idiosyncrasies.

#### `@-@` in ASCIIDOC and RAWXML

You see quite a bit of `@-@` in asciidoc and generated XML source.  That
is placed there when auto-generated data replaces them as entity in the
final form (`*.xml`, `*.po`, `*.html`, ...).

For example:

 * `@-@foo@-@`     is converted to the entity definition &foo;
 * `@-@popcon1@-@` is converted to popcon value and URL link of package in column 1.
 * `@-@psize1@-@`  is converted to size value and URL link of package in column 1.
 * `@-@popcon2@-@` is converted to popcon value and URL link of package in column 2.
 * `@-@psize2@-@`  is converted to size value and URL link of package in column 2.

These survive asciidoc conversion but are transformed into XML when conversion
from `raw.xml` to `en.xml` happens.

### URL in the English ASCIIDOC source and PO files

When selecting URL in the English ASCIIDOC source, please make sure to pick an
internationalized URL which understands the browser language preferences.

The top entries of PO files contain URLs.  If URL is internationalized, such
URLs can be kept as in in the PO files. I.e., (`msgid` == `msgstr`):
```
  msgid "https://www.debian.org/"
  msgstr "https://www.debian.org/"
```

If URL uses UTF-8 characters, use original ones instead of ones with "%".  (If
"%" exists in PO file data, it will break the build.) The proper PO file data
rule is simpler to explain by an example.  For example, French translation for
`https://en.wikipedia.org/wiki/Character_encoding` can be found at
`https://fr.wikipedia.org/wiki/Codage_des_caractères` but copying its URL box
in the browser yields percent encoded string
`https://fr.wikipedia.org/wiki/Codage_des_caract%C3%A8res`. 

#### Best PO file entry format:

```
  msgid "https://en.wikipedia.org/wiki/Character_encoding"
  msgstr "https://fr.wikipedia.org/wiki/Codage_des_caractères"
```

#### Acceptable PO file entry format:

```
  msgid "https://en.wikipedia.org/wiki/Character_encoding"
  msgstr "https://fr.wikipedia.org/wiki/Codage_de_caract&percnt;C3&percnt;A8res"
```

#### Illegal and unacceptable PO file entry format (I.e., Please don't copy-and-paste):

```
  msgid "https://en.wikipedia.org/wiki/Character_encoding"
  msgstr "https://fr.wikipedia.org/wiki/Codage_des_caract%C3%A8res"
```

Please note most wikipedia pages have link to the translated text at the left
of web page.

### Don't translate wrong part

Do not change "`&foo;`" in the `msgstr` of a PO file for translating URL
references.  Look for the top part of the PO file defining the "`foo`" entity.


### Don't translate the XML tag name itself

You may see XML tags such as `<literal>` and `<ulink url=\"&gnome;\">` in
`msgid`.  Don't change them in translation in `msgstr`.

### Flow chart for the building of this documentation

The English source is kept as ASCIIDOC(*) files under asciidoc/.
The translation source is kept as PO files under po/.

```
  >>> all the asciidoc/*.txt files 
                   :(make rawxml)                  
                   v
  >>> debian-reference.raw.xml ------>  (make entity) <=> remotely fetched
        |          |   |                              :    data files (+)
        |          |   +---------------------+        :
        |          |                         |        :              
        |          v                         v        v              
        |   debian-reference.en.xml   common.ent   url.ent and *.ent files
        |          : (**)        |           |        |      
        v          v             |           |        |
      debian-reference.en.xmlt   |           |        |
        |                        |           |        |
        v                        |           |        |
      templates.pot              |           |        |
        |                        |           |        |
        v (make po)              |           |        |
  >>> ja.po                      |           |        |
        |    fuzzy.log (++)      |           |        |
        v                        |           |        |
      debian-reference.ja.xml    |           |        |
        |                        |           |        |
        v                        v           v        v
        +----------+-------------+-----------+--------+
                   |(make all), (make test),
                   |(make html), (make txt), (make pdf), ...
                   v
      all documents (*.*.html *.*.txt *.*.pdf)
```

 * (`*`)  ASCIIDOC used in this package is `bin/asdciidoc` which is a
  customized script based on the older version.
 * (`+`)  When the English source changes URL link location or packages
  mentioned in the package list table, you need to update `*.ent` files in the
  source by running "`make entity`" with remote file access.
 * (`**`) `debian-reference.en.xmlt` effectively replaces untranslatable
  contents in `debian-reference.en.xml` with "DUMMY" to reduce clutter in PO
  files.
 * (`++`) `fuzzy.log` file contains number of fuzzy translation.  If not zero,
  please update PO file.

### Build failure due to po/*.add

If `po/foo.add` can't locate a matching string, build fails.   Since no
`debian-reference.foo.xml` for the problematic language is build, it's a bit
difficult to debug situation.  If you face such problem, please rename such
`po/foo.add` to something like `po/foo.add-keep`, and execute
"`make LANGPO=foo html`" to make `debian-reference.foo.xml`.  Then you can
look for the location of interest and make it the matching string.  Then
rename `po/foo.add-keep` back to `po/foo.add` and update the matching string.

It's tricky.

### List of make targets

```
 $ make all       # build all
 $ make test      # build html for testing (for Translator)
 $ make package   # build html and text from RAWXML/PO for DDP
 $ make publish   # build html text pdf from RAWXML/PO for DDP
 $ make clean     # clean files ready for tar 
 $ make distclean # clean files to reset RAWXML/ENT/POT
 $ make rawxml    # update RAWXML from ASCIIDOC
 $ make entity    # update ENT and local cache data from RAWXML/REMOTE
 $ make po        # update all PO from RAWXML
 $ make wrap      # wrap all PO
 $ make xml       # update all *.XML from RAWXML/ENT/PO/ADD
 $ make css       # update CSS and IMAGES in $(PUBLISHDIR)
 $ make html      # update all HTML in $(PUBLISHDIR)
 $ make txt       # update all Plain TEXT in $(PUBLISHDIR)
 $ make pdf       # update all PDF in $(PUBLISHDIR)
 $ make tex       # update all TeX source in $(PUBLISHDIR)
 $ make url       # check duplicate URL references
```

Wikipedia URLs:

`bin/interwiki` script can be used to check translated URLs.

## Publication to the www.debian.org servers

The contents of `debian-reference` are published to the www.debian.org servers
and they are linked from:

 * https://www.debian.org/doc/user-manuals#quick-reference

These are generated from the latest released packages by the `cron` script run
on the master www.debian.org server and they are mirrored to all the
www.debian.org servers.

 * https://anonscm.debian.org/git/debwww/cron.git (Maybe moving to salsa soon)
   * `parts/1ftpfiles`
   * `parts/7doc`

Osamu
