#101 non-string variables don't work in libtaskotron runner
Closed: Fixed Opened by tflink.

This is an interesting subtle consequence of our mixing of python and yaml that isn't handled well in the current code base.

When non-string data is returned from directives, it is converted directly into a string which makes using that data much more difficult than it should be. For example, returning a simple list like ['foo-1.2-3.fc99.noarch.rpm', 'bar-2.3-4.fc99.noarch.rpm'] makes something very difficult to parse on later use.

The problem stems from how we're doing variable substitution. Reusing the variable looks like:

somedirective:
arg1: "{{ my_var }}"
Since we're forced to make my_var into a string so that the yaml parses correctly, ['foo-1.2-3.fc99.noarch.rpm', 'bar-2.3-4.fc99.noarch.rpm'] becomes "['foo-1.2-3.fc99.noarch.rpm', 'bar-2.3-4.fc99.noarch.rpm']". Any attempt to address my_var as an iterator/list will retrieve chars from the string instead of elements of the list.

Changing the jinja variable syntax to something that yaml doesn't try to parse as a dict isn't a solution here because we're still inserting my_var directly into yaml-ish before directive execution and yaml doesn't work with python list syntax.

The possible solutions I can think of are:
# Change directives to yaml-ify everything before returning it
# Return something more text-based like CSV and re-parse at directive initialization

This task includes:
* investigate possible solutions, not limited to the ones listed above
* propose a fix
* implement the proposed fix after discussion


This ticket had assigned some Differential requests:
D108

yaml doesn't work with python list syntax.

Are we sure of this? It should be possible, YAML is a superset of JSON:
http://en.wikipedia.org/wiki/YAML#Lists

Are we sure of this?

No, not really. Especially when you have quotes from the YAML spec like that :) I wonder if a tweak or two to the directive logic to reparse the yaml could fix this?

As a preview, this is my current fix: F2238

I got rid of Jinja, because it can only return strings and sometimes I need to return different types (lists, dicts) -- that's what this ticket requests. Instead I replace the variables manually. I changed the variable syntax to ${name}. The end result is that variable expansion should work without any quoting, and you can both use just a single variable for passing intelligent structures, as well as mix variables with strings (like cmdline: "--debug --arch ${arch}") to receive strings.

Now I'll remove the hard-coded approach to replace variables only in the first level of the passed dictionary, and make it work with arbitrary nesting. That is needed to fix #148 and is highly related to this. Once that is done, this should work as well:

      python:
          file: upgradepath.py
          callable: main
          custom_args: [--debug, ${tag}]

For the sake of discussion, here's some example code that makes jinja/yaml behave: P1

Executing this code outputs:

"name: run example_reporting\nyumrepoinfo: {arch: '${ arch }', kojitag: '${ tag }', listy: '${ listy }'}\n"
u"name: run example_reporting\nyumrepoinfo: {arch: x86_64, kojitag: fedora-19-updates-testing-pending, listy: ['frog', 'bear', 'kangaroos', 'heffalumps', 'woozles']}"

{'name': 'run example_reporting',
'yumrepoinfo': {'arch': 'x86_64',
'kojitag': 'fedora-19-updates-testing-pending',
'listy': ['frog', 'bear', 'kangaroos', 'heffalumps', 'woozles']}}

For the sake of discussion, here's some example code that makes jinja/yaml behave: P1

Turns out that approach causes as many problems as it solves since it treats everything as not-a-string instead of everything as a string.

I'm slowly coming to agree with @kparal that we need to get rid of jinja.

I ended up adding .strip() to all the var_name parsing and changing VARS_RE to re.compile('\$\{[\w ]+?\}') to deal with variables that have whitespace in them (eg ${ var } instead of ${var}). I realize that F2238 is a preview but figured I would mention it.

I'm slowly coming to agree with @kparal that we need to get rid of jinja.

In case that wasn't clear, I have no more objections to proceeding with the method outlined in #9

I ended up adding .strip() to all the var_name parsing and changing VARS_RE to re.compile('\$\{[\w ]+?\}') to deal with variables that have whitespace in them (eg ${ var } instead of ${var}). I realize that F2238 is a preview but figured I would mention it.

Is there any benefit in allowing whitespace inside variable identifiers? You can't do that in bash.

@kparal: I guess it is just less error-prone? /me can see no particular reason for not allowing it, but I might be missing something, of course.

Is there any benefit in allowing whitespace inside variable identifiers? You can't do that in bash.

I guess that I have a habit of using whitespace before and after the variable name when using {{ var }} or ${ var } because I think it often makes things more readable.

I hadn't been thinking about spaces in the middle of variable names, so maybe '\$\{ ?(\w+) ?\}' would be better? It gets rid of the need for strip() and the static-number substring since only the group is returned in re.findall().

Closed by commit rLTRN1f9cc58e1a88.

Metadata