Internationalization (i18n)

SecureDrop is translated into a number of languages. We use a web-based collaborative translation platform called Weblate to make it easier. Under the hood, all translation is done using GNU gettext.

With gettext, text to be translated is specially marked in source code. A Python example:

if not (msg or fh):
    flash(gettext("You must enter a message or choose a file to submit."), "error")
    return redirect(url_for('main.lookup'))

In this code, the string You must enter a message or choose a file to submit. can be automatically extracted for translation. The gettext function to which it is passed is used as a marker by pybabel or similar tools to collect the strings to be translated and store them into a .pot file at securedrop/translations/messages.pot. For instance:

#: source_app/main.py:111
msgid "You must enter a message or choose a file to submit."
msgstr ""

The .pot file serves as a template for all the language-specific .po files, which are where Weblate stores the contributed translations. For each language to be translated, a directory is created, such as securedrop/translations/fr_FR, and populated with a .po file derived from the template. For instance, securedrop/translations/fr_FR/LC_MESSAGES/messages.po is almost identical to securedrop/translations/messages.pot except for the msgstr fields, which will contain the French translations, e.g.:

#: source_app/main.py:111
msgid "You must enter a message or choose a file to submit."
msgstr "Vous devez saisir un message ou sélectionner un fichier à envoyer."

There’s one last type of file in the gettext system, a machine-readable version of the .po translations called a .mo file. Applications use these to get translations at runtime. The .po files are compiled to .mo files when the SecureDrop package is built.

The desktop icons installed on SecureDrop workstations are also translated. The icon templates are in the install_files/ansible-base/roles/tails-config/templates directory. Their labels are collected in the desktop.pot file and translated in the corresponding .po files in the same directory (fr.po, de.po etc.). All translations are merged from the *.j2.in files into the corresponding *.j2 file and committed to the SecureDrop repository. They are then installed when configuring Tails with the tasks/create_desktop_shortcuts.yml tasks.

We don’t expect translators to deal with all these files directly. Translation happens on our Weblate server, which is configured to use a fork of the main SecureDrop repository.

At the start of the release process, the localization manager collects string changes on the develop branch in the main SecureDrop repository and merges them to the i18n branch of the securedrop-i18n repository. The changes will then appear in Weblate, and translation can begin. At the end of the translation period, the localization manager collects the changes to the PO files on securedrop-i18n/i18n in a pull request for securedrop/develop. Once that pull request is merged, the translations are backported to the release branch in the main SecureDrop repository.

i18n_tool.py

Most of the work in managing translations within the SecureDrop code base is supported by securedrop/i18n_tool.py. It provides convenient wrappers around pybabel and gettext , and is used to update strings to be translated; pull translations from Weblate; to compile translations before running tests and while packaging SecureDrop.

Development tasks

Add a new language

A user with weblate admin rights must visit the Weblate translation creation page and the Weblate desktop translation creation page to add a new language.

SecureDrop only supports a subset of all the languages being worked on in Weblate: some of them are partially translated or not fully reviewed. The list of fully supported languages is hard-coded in the i18n_tool.py file, in the SUPPORTED_LANGUAGES variable. When a new language is completely translated and reviewed, the i18n_tool.py file must be manually edited to add this new language to the SUPPORTED_LANGUAGES variable.

Update strings to be translated

Whenever strings are modified in the SecureDrop source, whether in Python code, HTML templates, or desktop icon labels, the translation files should also be updated by running make translate in the root of the SecureDrop working copy.

The translate target runs i18n_tool.py translate-messages and i18n_tool.py translate-desktop, which in turn use pybabel extract to gather source strings. These commands update the .pot files for the SecureDrop server code and the desktop icons, as well as the .po files for each language.

After running make translate, carefully review the output of git diff. Check securedrop/messages.pot first for updated strings, looking for problems like:

  • overly idiomatic English

  • fragmented text, such as pieces of a sentence intended to be concatenated together, which can be difficult to translate

  • messages that are marked with plain gettext and contain plurals based on numeric placeholder variables – these should generally be marked with ngettext so that they can be translated properly in languages with complex plural forms

Then review the messages.po of one existing translation, with a focus on new translations, which are often marked fuzzy. There is no need to review multiple languages’ .po files because they are processed in the same way.

Once you’ve reviewed the changes, submit them in a pull request for the develop branch in the main SecureDrop repository.

The new source strings will only be visible to translators in Weblate after they’ve been merged to securedrop/develop and that branch has been merged into securedrop-i18n/i18n. The localization manager does this at the beginning of our release cycle.

Merge develop into the Weblate fork

1) First make sure the translation files on the develop branch of the main SecureDrop repository contain the latest source strings. Follow the steps under Update strings to be translated.

2) Then, translation must be suspended in Weblate, and any uncommitted changes committed and pushed, to avoid conflicts:

Weblate commit Lock

  • Click Commit.

  • Click Push.

  • And finally, click Lock.

Weblate commit Locked

  1. The securedrop/develop branch can now be merged into securedrop-i18n/i18n:

$ git clone https://github.com/freedomofpress/securedrop
$ cd securedrop
$ git remote add i18n git@github.com:freedomofpress/securedrop-i18n.git
$ git fetch i18n
$ git checkout -b i18n i18n/i18n
$ git merge origin/develop
$ git commit --amend -m 'l10n: sync with upstream origin/develop'
$ git push i18n i18n

  1. Verify that Weblate has the latest changes, and unlock the repository.

  • Go to the Weblate commit page for SecureDrop and verify the commit hash matches the last commit of the i18n branch. This must happen instantly after the branch is pushed because Weblate is notified via a webhook. If it is different, ask for help.

  • Click Unlock.

Weblate commit Unlock

Translation can now begin. As translators make progress, Weblate pushes the translations done via the web interface in commits to the i18n branch of the securedrop-i18n repository (a fork of the main SecureDrop repository). When the translation period ends, these commits will be collected into a pull request for the main SecureDrop repository.

Weblate commit Unlocked

Merge translations back to develop

Weblate automatically pushes the translations done via the web interface as a series of commits to the i18n branch in the securedrop-i18n repository, which is a fork of the develop branch of the main SecureDrop repository. These translations need to be submitted back to the securedrop/develop branch via pull requests. When you create a branch for this, begin its name with i18n-, as that prefix triggers special CI tests for translations.

To fetch the latest translations from the securedrop-i18n/i18n branch into your working copy of the SecureDrop repository, run these commands in your repo root:

$ git checkout -b i18n-merge origin/develop
$ securedrop/bin/dev-shell ./i18n_tool.py --verbose update-from-weblate
$ securedrop/bin/dev-shell ./i18n_tool.py --verbose update-docs --docs-repo-dir /path/to/documentation

You now have the latest translations on your i18n-merge branch.

Note

It is very important to check that each translated string looks like a plausible translation, with no markup. Even if the reviewer does not understand the language, if a translated string looks strange, someone other than the reviewer must be consulted to verify it means something. It is extremely unlikely that a contributor will manipulate a translated string to introduce a vulnerability in SecureDrop, but any suspicious translation should be investigated.

To check the new translations, you’ll need to compile them and verify them by running our automated tests and, ideally, by checking them in the SecureDrop source and journalist interfaces.

Compile translations

At runtime, gettext needs a compiled file for each language (the .mo files). Before you can check the translations in the SecureDrop web interfaces, these need to be created:

$ securedrop/bin/dev-shell ./i18n_tool.py --verbose translate-messages --compile

For the desktop icons of the source and journalist interfaces, compilation updates their template files with all the translations collected from the .po files.

This can be done by running the following command:

$ securedrop/bin/dev-shell ./i18n_tool.py --verbose translate-desktop --compile

Verify translations

SecureDrop web interfaces

After a translation is compiled, the web page in which it appears can be verified visually by starting the SecureDrop development servers and navigating via http://localhost:8080 for the source interface or http://localhost:8081 for the journalist interface. You can start the development servers with:

$ make dev

The translations can be checked automatically by running the SecureDrop page layout tests:

$ export PAGE_LAYOUT_LOCALES="en_US,fr_FR"  # may be set to any supported languages
$ make test TESTFILES=tests/pageslayout
[...]
tests/pageslayout/test_journalist.py::TestJournalistLayout::test_account_edit_hotp_secret[en_US] PASSED
tests/pageslayout/test_journalist.py::TestJournalistLayout::test_account_edit_hotp_secret[fr_FR] PASSED
[...]

Note

if unset, PAGE_LAYOUT_LOCALES defaults to en_US (US English) and ar (Arabic).

After running the tests, screenshots for each locale are available in securedrop/tests/pageslayout/screenshots/<locale>, e.g. securedrop/tests/pageslayout/screenshots/fr_FR. Screenshot filenames can be found in the tests that created them, in securedrop/tests/pageslayout/test_journalist.py or securedrop/tests/pageslayout/test_source.py.

Desktop icons

The translated templates for the desktop icons are:

  • install_files/ansible-base/roles/tails-config/templates/desktop-journalist-icon.j2

  • install_files/ansible-base/roles/tails-config/templates/desktop-source-icon.j2

Check that each of them contains a Name line for each of SecureDrop’s supported locales.

Push your branch and create a pull request

After you’ve checked the translations, you’re ready to push your i18n-merge branch and create a pull request to get the translations merged to the SecureDrop develop branch.

Note

If there have been multiple commits per language, as can happen if source strings need to be translated again after being changed to correct critical errors, or to incorporate suggestions from the source string feedback period, they should be combined via an interactive rebase. Reorder the commits to group them by language, then squash the commits for each language into one. The goal is to end up with one commit per supported language on the merge branch.

When you’re happy with the state of language commits on your merge branch:

$ git commit -m "l10n: compile desktop icons' translations" # if needed
$ git push -u origin i18n-merge

Note

The CI job translation-tests will automatically run the above page layout tests in all supported languages on branches named with the prefix i18n-. If you’ve followed that naming convention, the translation tests should soon be run on your pull request.

If you have an abundance of time, you can run all the translation tests locally with:

$ make translation-test

And at long last, you’re done. Go to https://github.com/freedomofpress/securedrop and propose a pull request.

Note

Unlike the SecureDrop application translations, the desktop icon translations are compiled and merged into the repository. They need to be available in their translated form when securedrop-admin tailsconfig is run, because the development environment is not available.

Update Weblate screenshots

You can use the script securedrop/upload_screenshots.py to update UI screenshots that are used to illustrate strings in Weblate. The script depends on the existence of up-to-date layout test results, which you can generate using this command in the base directory:

$ LOCALES=en_US make translation-test

Inspect the screenshots in the directory securedrop/tests/pageslayout/screenshots/en_US and make sure that their content corresponds to the expected version of the codebase.

Obtain your API key in Weblate. Export the token to the environment variable WEBLATE_API_TOKEN. You can now run this command to perform an upload:

$ securedrop/upload-screenshots.py

If new screenshots were added as part of this run, make sure to associate them with relevant strings in Weblate, which you can do from the screenshots list.

Release Management

Two weeks before the release: string freeze

When features for a new SecureDrop release are frozen, the localization manager for the release will:

  • Merge develop into the Weblate fork.

  • Update Weblate screenshots so translators can see new or modified source strings in context.

  • Update the i18n timeline in the translation section of the forum.

  • Add a Weblate announcement for the securedrop/securedrop component with the translation timeline for the release.

    • Important: Make sure the Notify users box is checked, so that translators receive an email alert.

    • You can view a history of past announcements in Weblate’s Django admin panel, or use this template:

      Translation for the SecureDrop X.Y.Z release has begun. If you have suggestions for source strings, please get them to us by YYYY-MM-DD. Translation will end on YYYY-MM-DD.

    • Set the Expiry date to release day itself (the day after the translation deadline).

  • Remind all developers about the string freeze in Gitter, for example using this template:

    Hello! We’ve just opened translations for the upcoming SecureDrop 2.3.0 release. If you have suggestions for source strings, please get them to us by 2022-03-20. Translation will end on 2022-03-27.

    Translations are done using Weblate (https://weblate.securedrop.org/projects/securedrop/securedrop/). If you haven’t used it before, <https://docs.securedrop.org/en/stable/development/translations.html> has instructions on how to get started.

  • Update Localization Lab via the SecureDrop Coordination channel in the IFF Mattermost.

  • During the feedback period, monitor Weblate comments and suggestions, and open a pull request for every source string suggestion coming from translators.

Remember that supported languages are the priority during this period. That is, while translation contributions are welcome for all languages, the pre-release goal is to keep the current set of supported languages at 100% translation in Weblate. Localization Lab can marshal individual translators to help meet this goal.

Release day

Prior to cutting the final release, the localization manager must:

Then, post-release, either same day or day-after, the localization manager should:

  • Remove the Weblate announcement about this release’s translation timeline (if you set an end-date on the original announcement, this may happen automatically)

  • Update the i18n timeline in the forum.

Translator credits

Correct acknowledgment of translators’ contributions is important, so i18n_tool.py makes it easy to list the translators who have helped since the last merge of Weblate translations, with i18n_tool.py list-translators. A list of everyone who has ever contributed translations to SecureDrop can be obtained with i18n_tool.py list-translators --all. There are Makefile targets for these, list-translators and list-all-translators, e.g:

$ make list-all-translators
ar:
  A. Nonymous
  Ahmad Gharbeia
  Ahmed Essam
  Ali Boshanab
[...]

Weblate administration

Note

The privilege escalation workflow is different for code maintainers and translation maintainers.

A translation admin has special permissions on Weblate and the repositories. When someone is willing to become an admin, a thread is started in the translation section of the forum. If there is consensus after a week, the permissions of the new admin are elevated. If there is not yet consensus, a public vote is organized among the current admins.

The privileges of an admin who has not been active for six months or more are revoked, but they can apply again at any time.

The community of SecureDrop translators works very closely with the SecureDrop developers and some of them participate in both groups. However, the translator community has a different set of rules and permissions, and therefore independent policies from SecureDrop itself.

Admin permissions

The full set of admin permissions can be granted at:

Granting reviewer privileges in Weblate

  • Visit https://weblate.securedrop.org/admin/weblate_auth/user/.

  • Click on the user name.

  • In the Groups block:

    • Select Localizationlab in the Available groups list and click on the right arrow to move it to the Chosen groups list.

    • Select Users in the Chosen groups list and click on the left arrow to remove it.

Update the Weblate full text index

Weblate’s full-text index can occasionally get out of sync. When this happens, Weblate’s search may fail to find a word that you know exists in the source strings. You can rebuild the index with:

$ ssh debian@weblate.securedrop.org
$ cd /app/weblate
$ sudo docker-compose run weblate rebuild_index --all --clean

Note that the new index may not be used right away. Some workers may still have the old index open. If the index is holding up translators with a release looming, the server can be rebooted.