This is an interesting subtle consequence of our mixing of python and yaml that isn't handled well in the current code base.
When non-string data is returned from directives, it is converted directly into a string which makes using that data much more difficult than it should be. For example, returning a simple list like ['foo-1.2-3.fc99.noarch.rpm', 'bar-2.3-4.fc99.noarch.rpm'] makes something very difficult to parse on later use.
['foo-1.2-3.fc99.noarch.rpm', 'bar-2.3-4.fc99.noarch.rpm']
The problem stems from how we're doing variable substitution. Reusing the variable looks like:
somedirective: arg1: "{{ my_var }}" Since we're forced to make my_var into a string so that the yaml parses correctly, ['foo-1.2-3.fc99.noarch.rpm', 'bar-2.3-4.fc99.noarch.rpm'] becomes "['foo-1.2-3.fc99.noarch.rpm', 'bar-2.3-4.fc99.noarch.rpm']". Any attempt to address my_var as an iterator/list will retrieve chars from the string instead of elements of the list.
my_var
"['foo-1.2-3.fc99.noarch.rpm', 'bar-2.3-4.fc99.noarch.rpm']"
Changing the jinja variable syntax to something that yaml doesn't try to parse as a dict isn't a solution here because we're still inserting my_var directly into yaml-ish before directive execution and yaml doesn't work with python list syntax.
The possible solutions I can think of are: # Change directives to yaml-ify everything before returning it # Return something more text-based like CSV and re-parse at directive initialization
This task includes: * investigate possible solutions, not limited to the ones listed above * propose a fix * implement the proposed fix after discussion
This ticket had assigned some Differential requests: D108
yaml doesn't work with python list syntax.
Are we sure of this? It should be possible, YAML is a superset of JSON: http://en.wikipedia.org/wiki/YAML#Lists
Are we sure of this?
No, not really. Especially when you have quotes from the YAML spec like that :) I wonder if a tweak or two to the directive logic to reparse the yaml could fix this?
As a preview, this is my current fix: F2238
I got rid of Jinja, because it can only return strings and sometimes I need to return different types (lists, dicts) -- that's what this ticket requests. Instead I replace the variables manually. I changed the variable syntax to ${name}. The end result is that variable expansion should work without any quoting, and you can both use just a single variable for passing intelligent structures, as well as mix variables with strings (like cmdline: "--debug --arch ${arch}") to receive strings.
${name}
cmdline: "--debug --arch ${arch}"
Now I'll remove the hard-coded approach to replace variables only in the first level of the passed dictionary, and make it work with arbitrary nesting. That is needed to fix #148 and is highly related to this. Once that is done, this should work as well:
python: file: upgradepath.py callable: main custom_args: [--debug, ${tag}]
For the sake of discussion, here's some example code that makes jinja/yaml behave: P1
Executing this code outputs:
"name: run example_reporting\nyumrepoinfo: {arch: '${ arch }', kojitag: '${ tag }', listy: '${ listy }'}\n" u"name: run example_reporting\nyumrepoinfo: {arch: x86_64, kojitag: fedora-19-updates-testing-pending, listy: ['frog', 'bear', 'kangaroos', 'heffalumps', 'woozles']}"
{'name': 'run example_reporting', 'yumrepoinfo': {'arch': 'x86_64', 'kojitag': 'fedora-19-updates-testing-pending', 'listy': ['frog', 'bear', 'kangaroos', 'heffalumps', 'woozles']}}
Turns out that approach causes as many problems as it solves since it treats everything as not-a-string instead of everything as a string.
I'm slowly coming to agree with @kparal that we need to get rid of jinja.
I ended up adding .strip() to all the var_name parsing and changing VARS_RE to re.compile('\$\{[\w ]+?\}') to deal with variables that have whitespace in them (eg ${ var } instead of ${var}). I realize that F2238 is a preview but figured I would mention it.
.strip()
VARS_RE
re.compile('\$\{[\w ]+?\}')
${ var }
${var}
In case that wasn't clear, I have no more objections to proceeding with the method outlined in #9
Is there any benefit in allowing whitespace inside variable identifiers? You can't do that in bash.
@kparal: I guess it is just less error-prone? /me can see no particular reason for not allowing it, but I might be missing something, of course.
I guess that I have a habit of using whitespace before and after the variable name when using {{ var }} or ${ var } because I think it often makes things more readable.
{{ var }}
I hadn't been thinking about spaces in the middle of variable names, so maybe '\$\{ ?(\w+) ?\}' would be better? It gets rid of the need for strip() and the static-number substring since only the group is returned in re.findall().
'\$\{ ?(\w+) ?\}'
strip()
re.findall()
Closed by commit rLTRN1f9cc58e1a88.