Global data object

Think of the case where your workflow contains tasks with the following functionality:

1. Read indata and perform intermediate calculations.

2. Perform main calculation based on results from previous step.

3. Report results.

How would you make the results from step 1 available for step 2? One solution could be to write the results to a file in step 1, and later use this file as input in step 2. This is a good solution if step 2 is a program call that actually relies on reading input files, but for a more data-driven workflow with pure Python functions, this is not optimal.

A better solution would be to have a common space that different functions can access for storing and reading data as pure Python objects. This is where the global data object g comes into play. g is part of the context that is set up when caliber is imported in your Python scripts or modules.

Import it in your modules

If you have a data-driven workflow based on functions collected in a separate module: use g to access data across the different functions. Import caliber in your module and access g from the context caliber.context.g or import the context directly from caliber import context and access it from the context as context.g.

Under the hood of the global data object

The global data object is a namespace object, i.e. a container for symbolic names along with objects that the names refer to. Think of it as a custom dictionary. In fact, g implements common dict methods like .get(), .pop(), .setdefault(), and .clear(), e.g. context.g.get('answer', 42). You can even represent it as a dict by dict(context.g), or check if an attribute is set by answer in context.g.

In the example below, functions.py defines three functions representing the functionality described above. g is available to the functions from context after the import on line 1. Notice that only sum_list() takes arguments, and that the other functions operate on data in g. The workflow is defined in main.py, and is run by typing caliber workflow run main.json after creating main.json with py main.py.

Try experimenting with setting different values on line 9 in main.py to see the workflow operate on another set of input. Also, try commenting out line 6 in functions.py to trigger the else branch in print_results().

functions.py

from caliber import context, print


def sum_list(*args):
    """Sum numbers in a list."""
    context.g.input = args
    context.g.sum = sum(args)


def square():
    """Calculate the square of a number."""
    context.g.square = context.g.sum**2


def print_result():
    """Prints the input and output of the workflow."""
    # Get dict representation of g
    g_dct = dict(context.g)

    # Print input
    if 'input' in context.g:
        print('The input:')
        print(' '.join(f'{number:.2f}' for number in g_dct['input']))
    else:
        print('No input stored in g')

    # Print output
    print(f'The sum:\n{g_dct["sum"]:.2f}')
    print(f'The square:\n{g_dct["square"]:.2f}')

main.py

import caliber
import functions

# Define tasks
sums = caliber.Task(
    function=functions.sum_list,
    name='Calculate sum',
    args=[
        1.0, 4.2, 3.14, 6.3
    ],
)

squares = caliber.Task(
    function=functions.square,
    name='Calculate square of sum',
)

prints = caliber.Task(
    function=functions.print_result,
    name='Print input and output',
)

# Collect tasks in process
calculation = caliber.Process(
    name='Calculation',
    tasks=[
        sums,
        squares,
        prints,
    ],
)

# Create workflow
do_calculation = caliber.Workflow(
    process=calculation,
    name='Calculate square of sum of list',
)

do_calculation.to_json('main.json')

Potential footgun

When one function does changes on g as part of it’s execution, these changes will be available to other functions. A common use case is where one function creates input to another, e.g. a list with data. Other functions might need to operate on this list, e.g. adding or subtracting values, or even appending or popping items. Lists or dicts that are accessed from g should therefore be turned into instances that are local to the scope of the function through list or dict comprehension, to avoid introducing hard-to-detect bugs by accidentally editing data that other functions rely on.