libtaskotron: unicode issues and dnf --disableplugin=noroot broken on F26
Open, HighPublic

Description

This has been reported in Bugzilla. Prioritizing, because not only sudo support seems not to be working in certain locales, but the whole dnf support seems to be broken in F26.

When I run taskotron locally (using tox from [1]), it shouts on me I need rpm-python:

Version-Release number of selected component (if applicable):
libtaskotron-0.4.20-1.fc26.noarch

[libtaskotron] 14:33:46 INFO    Execution started at: 2017-05-30 14:33:46 UTC
[libtaskotron] 14:33:46 DEBUG   Using libtaskotron 0.4.20
[libtaskotron] 14:33:46 DEBUG   Parsed arguments: Namespace(arch='noarch', debug=False, item='python-twine-1.8.1-3.fc26', jobid='-1', libvirt=False, local=False, no_destroy=False, override=[], patch=None, ssh=None, ssh_privkey=None, task='runtask.yml', type='koji_build', uuid='20170530_143346_886492')
[libtaskotron] 14:33:46 DEBUG   Using config file: /etc/taskotron/taskotron.yaml
[libtaskotron] 14:33:46 DEBUG   Using config profile: development
[libtaskotron] 14:33:46 INFO    Task artifacts will be saved in: /var/lib/taskotron/artifacts/20170530_143346_886492
[libtaskotron] 14:33:46 INFO    Checking installed state of 1 packages...
[libtaskotron] 14:33:47 WARNING No DNF cache available, metadata will be need to be downloaded. If your DNF cache doesn't persist across task executions (e.g. on a disposable VM), consider creating the cache in your base system image to speed up execution.
[libtaskotron] 14:33:47 CRITICAL Traceback (most recent call last):
  File "/usr/bin/runtask", line 11, in <module>
    load_entry_point('libtaskotron==0.4.20', 'console_scripts', 'runtask')()
  File "/usr/lib/python2.7/site-packages/libtaskotron/main.py", line 156, in main
    overlord.start()
  File "/usr/lib/python2.7/site-packages/libtaskotron/overlord.py", line 96, in start
    runner.execute()
  File "/usr/lib/python2.7/site-packages/libtaskotron/executor.py", line 60, in execute
    self._prepare_task()
  File "/usr/lib/python2.7/site-packages/libtaskotron/executor.py", line 77, in _prepare_task
    % " ".join([pipes.quote(rpm) for rpm in rpms]))
TaskotronError: Some packages are not installed. Please run "dnf install rpm-python" to install all required packages.

This is of course not true, as no such package exists on Fedora 26, but I have the thing that provides it:

$ LANG=C.utf8 sudo dnf install rpm-python
Package python2-rpm-4.13.0.1-4.fc26.x86_64 is already installed, skipping.
Dependencies resolved.
Nothing to do.
Complete!

The package is specified in the runtask section:

environment:
    rpm:
        - rpm-python

But when I change it to:

environment:
    rpm:
        - python2-rpm

It sill tracebacks:

TaskotronError: Some packages are not installed. Please run "dnf install python2-rpm" to install all required packages.
$ rpm -q python2-rpm
python2-rpm-4.13.0.1-4.fc26.x86_64

I believe rpm_utils.is_installed() is broken. Also I was curious that sudo dnf install rpm-python actually fixed the problem for a while and I realized it's the sudo that fixes it. I.e. when I run sudo echo and enter the password, the code in rpm_utils.is_installed() goes into the os_utils.has_sudo() condition and uses sudo and that fixes the problem.

I've examined the problem further. Running with --debug:

[libtaskotron] 14:49:31 INFO    Execution started at: 2017-05-30 14:49:31 UTC
[libtaskotron] 14:49:31 DEBUG   Using libtaskotron 0.4.20
[libtaskotron] 14:49:31 DEBUG   Parsed arguments: Namespace(arch='noarch', debug=True, item='python-twine-1.8.1-3.fc26', jobid='-1', libvirt=False, local=False, no_destroy=False, override=[], patch=None, ssh=None, ssh_privkey=None, task='runtask.yml', type='koji_build', uuid='20170530_144931_095987')
[libtaskotron] 14:49:31 DEBUG   Using config file: /etc/taskotron/taskotron.yaml
[libtaskotron] 14:49:31 DEBUG   Using config profile: development
[libtaskotron:logger.py:182] 2017-05-30 14:49:31 DEBUG   Stream logging enabled with level: DEBUG
[libtaskotron:overlord.py:38] 2017-05-30 14:49:31 DEBUG   Found task: dist.python-versions
[libtaskotron:overlord.py:90] 2017-05-30 14:49:31 INFO    Task artifacts will be saved in: /var/lib/taskotron/artifacts/20170530_144931_095987
[libtaskotron:overlord.py:74] 2017-05-30 14:49:31 DEBUG   Execution mode: local
[libtaskotron:rpm_utils.py:184] 2017-05-30 14:49:31 INFO    Checking installed state of 1 packages...
[libtaskotron:os_utils.py:37] 2017-05-30 14:49:31 DEBUG   Deciding whether we have a password-less sudo access. Running: sudo --validate --non-interactive
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 861, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 734, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 465, in format
    record.message = record.getMessage()
  File "/usr/lib64/python2.7/logging/__init__.py", line 329, in getMessage
    msg = msg % self.args
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 11: ordinal not in range(128)
Logged from file os_utils.py, line 43
[libtaskotron:rpm_utils.py:229] 2017-05-30 14:49:31 DEBUG   Deciding whether DNF cache is available. Running: dnf --cacheonly repolist
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 861, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 734, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 465, in format
    record.message = record.getMessage()
  File "/usr/lib64/python2.7/logging/__init__.py", line 329, in getMessage
    msg = msg % self.args
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 19: ordinal not in range(128)
Logged from file rpm_utils.py, line 234
[libtaskotron:rpm_utils.py:191] 2017-05-30 14:49:31 WARNING No DNF cache available, metadata will be need to be downloaded. If your DNF cache doesn't persist across task executions (e.g. on a disposable VM), consider creating the cache in your base system image to speed up execution.
[libtaskotron:os_utils.py:37] 2017-05-30 14:49:31 DEBUG   Deciding whether we have a password-less sudo access. Running: sudo --validate --non-interactive
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 861, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 734, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 465, in format
    record.message = record.getMessage()
  File "/usr/lib64/python2.7/logging/__init__.py", line 329, in getMessage
    msg = msg % self.args
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 11: ordinal not in range(128)
Logged from file os_utils.py, line 43
[libtaskotron:rpm_utils.py:198] 2017-05-30 14:49:31 DEBUG   Running: dnf --assumeno --disableplugin=noroot install rpm-python
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 861, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 734, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 465, in format
    record.message = record.getMessage()
  File "/usr/lib64/python2.7/logging/__init__.py", line 329, in getMessage
    msg = msg % self.args
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 14: ordinal not in range(128)
Logged from file rpm_utils.py, line 203
[libtaskotron:logger.py:88] 2017-05-30 14:49:32 CRITICAL Traceback (most recent call last):
  File "/usr/bin/runtask", line 11, in <module>
    load_entry_point('libtaskotron==0.4.20', 'console_scripts', 'runtask')()
  File "/usr/lib/python2.7/site-packages/libtaskotron/main.py", line 156, in main
    overlord.start()
  File "/usr/lib/python2.7/site-packages/libtaskotron/overlord.py", line 96, in start
    runner.execute()
  File "/usr/lib/python2.7/site-packages/libtaskotron/executor.py", line 60, in execute
    self._prepare_task()
  File "/usr/lib/python2.7/site-packages/libtaskotron/executor.py", line 77, in _prepare_task
    % " ".join([pipes.quote(rpm) for rpm in rpms]))
TaskotronError: Some packages are not installed. Please run "dnf install rpm-python" to install all required packages.

Now I have a pretty decent idea what's going on here.

$ sudo --validate --non-interactive
sudo: je vyžadováno heslo
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 11: ordinal not in range(128)

Char 11 is ž.

$ \dnf --assumeno --disableplugin=noroot install rpm-python
Chyba: Tento příkaz musí být spuštěn pod uživatelem root.
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 14: ordinal not in range(128)

Char 14 is ř.

I believe I've identified several problems here:

  1. You are using Python 2 and thus you handle unicode in "I don't care" mode - be careful or use Python 3 instead to fix that :)
  2. You are calling external commands and doing stuff with the output but you do not set locale to English
  3. dnf --disableplugin=noroot does not work on Fedora 26

    [1] https://github.com/fedora-python/taskotron-python-versions
kparal created this task.Jun 13 2017, 1:24 PM

So, I looked into the brave new world in T939. As it seems, we'll likely remove sudo support, since it requires to run tasks as root. So that's first part "resolved". The current WIP code in D1195 also removes all code that uses dnf install and uses ansible for that instead. If that turns out of be the way we handle things, that's the second half of this "resolved", we'll see.

I'm going to block this on T939. Once that is implemented, we can look at if we still need to fix something here, or if all of our relevant existing code had been thrown away.

kparal moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Jun 19 2017, 11:55 AM

This has been partially "fixed" in D1216 - figuring out whether packages are installed now works under a non-root account, as long as you use plain package names/file paths as requirements. That should cover the major issues until we replace this whole code.