Workflow expression evaluator

[markuswondrak]

Hi,

I stumbled across the workflow expressions and had a closer look at workflows/expressions.py. The evaluator is its own parser, around 1.5k lines including the condition diagnostics. The syntax is basically Jinja2 ({{ }}, pipe filters, and/or/not), and ARCHITECTURE.md calls it "Jinja2-like".

I couldn't find anything on why an existing library (e.g. a sandboxed Jinja2) wasn't used. Was that a conscious decision, or did it just grow that way?

I thought replacing the custom implementation with a library could take some complexity out of the codebase. What do you think? I'd be happy to try a prototype if there's interest.

[mnriem]

Thanks for raising this. Evaluating a library-backed replacement seems worthwhile, but the current syntax is Jinja2-like rather than fully Jinja-compatible. We would need to account for typed single-expression results, missing-path handling, numeric-string comparisons, custom filter behavior, and condition diagnostics.

Alongside Jinja2, these alternatives seem worth considering:

Library	Potential approach	Main consideration
Jinja2	Replace parsing and evaluation with a restricted, sandboxed environment	Closest syntax, but existing semantics are not a drop-in match
Lark	Define the existing language with a formal grammar	Could replace handwritten parsing while retaining our evaluator and compatibility
pyparsing	Define the grammar directly in Python	Similar preservation strategy to Lark; evaluation remains our responsibility
python-liquid	Adopt a restricted template engine with familiar delimiters and filters	Different filter syntax, condition semantics, and rendering behavior
simpleeval	Use a configurable Python-AST expression evaluator	Typed results, but no native support for our interpolation or filter syntax
cel-python	Adopt Common Expression Language	An intentional language migration rather than a compatible replacement

I would initially shortlist Jinja2 and Lark, with pyparsing as another parser-focused option. They address different goals: Jinja2 could outsource more of the evaluator, while Lark could remove fragile parsing machinery without changing the workflow language.

A useful prototype would compare against the existing expression and condition cases, identify behavioral differences, and show whether the compatibility layer actually leaves us with less code and maintenance. That would give us evidence for choosing a library rather than assuming similar syntax means equivalent behavior.