Integrate Python console scripts as tools

Now, let’s add a new build step that generates code, which is a common requirement in real-life projects. For today, we’ll use a popular templating engine called Jinja. Bazel is not opinionated about what tools you use, AND it’s not necessary to write any custom rules or macros to call existing tools using their CLI. In fact, it’s quite easy.

Fetch the jinja2 tool

Add to the requirements/tools.in file, which is where you organized dependencies that are only used at build time.

echo "jinja2-cli" >> requirements/tools.in

Repin the transitive dependencies again: ./tools/repin

Tell Bazel how to run it

Open the BUILD file in the tools folder. There’s already a py_console_script_binary rule which you used to run the copier tool. Make an identical one for jinja2.

The package name with a hyphen was converted by rules_python to be an underscore here! That’s not necessarily the same thing as the string you use in import.

Also add a visibility attribute, since Bazel enforces that the dependency graph doesn’t get tangled by default.

py_console_script_binary(
    name = "jinja2",
    pkg = "@pip//jinja2_cli",
    visibility = ["//visibility:public"],
)

Now check that you can run it:

% bazel run tools:jinja2
...
INFO: Build completed successfully, 5 total actions
INFO: Running command line: bazel-bin/tools/jinja2
Usage: rules_python_entry_point_jinja2.py [options] <input template> <input data>

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  --format=FORMAT       format of input variables: auto, env, ini, json,
                        querystring
  -e EXTENSIONS, --extension=EXTENSIONS
                        extra jinja2 extensions to load
  -D key=value          Define template variable in the form of key=value
  -s SECTION, --section=SECTION
                        Use only this section from the configuration
  --strict              Disallow undefined variables to be used within the
                        template
  -o FILE, --outfile=FILE
                        File to use for output. Default is stdout.

This shows that you can run the tool under Bazel. It also prints the help text for the CLI, which is critical for the next step: invoking it with the right arguments.

Declare the code generation

The library already came with a jinja2 template file in mylib/header.tmpl.txt. You can see it has a {{TIMESTAMP}} placeholder to fill in. To help out, the library also has a run_binary rule in the mylib/BUILD file. This is a basic building block which uses a tool and some declared inputs, then runs it when needed to produce the declared outputs. You can see the args already match the command-line flags from the jinja2 help output above. The tool = "TODO" line is ready for you to fill in with the jinja2 CLI you just setup. Replace that with the label //tools:jinja2. Now you can run a build to check the codegen:

bazel build mylib:header.txt; cat bazel-bin/mylib/header.txt 
INFO: Analyzed target //mylib:header.txt (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //mylib:header.txt up-to-date:
  bazel-bin/mylib/header.txt
INFO: Elapsed time: 0.146s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
-- Built at <unstamped> --
I'm a cow, I found this on the internet

It worked!

Why does the value say <unstamped> though? This is “determinism” where the output of a build step should only depend on the inputs. If the current timestamp appears here, the output will be different every time, causing cache misses for any action that depends on this one, even transitively. That slows down builds! You’ll see how to stamp artifacts for production later.

Depend on the codegen

Finally, use the new templated header file in the application. The mylib library included a second function, stamped, so change the marvin function in app/__main__.py to use that instead. When you try to run the application now you get an error:

bazel run //app:app_bin

Traceback (most recent call last):
  File "execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/app/app_bin.runfiles/_main/app/__main__.py", line 6, in <module>
    say.stamped(x.text)
  File "execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/app/app_bin.runfiles/_main/mylib/say.py", line 14, in stamped
    with open(path.join(FOLDER, "header.txt"), "r") as header:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'mylib/header.txt'

This error says that you didn’t declare the header.txt file as a dependency. And because the reference isn’t an import statement, BUILD file generation is not smart enough to know what to do. To fix this, you need to edit the BUILD file by hand. Since it’s a runtime dependency, you need to list it as a data dependency. How do you know where it belongs though? The answer is the “locality principle”. You should declare the dependency locally to where the reference appears. There should always be a symmetry between the symbol reference in the code, and the BUILD definition for that file. Since the stack trace points to say.py , you should add the data dependency next to the srcs=["say.py"] declaration:

py_library(
    name = "mylib",
    srcs = ["say.py"],
    data = ["header.txt"],
    visibility = ["//:__subpackages__"],
    deps = ["@pip//cowsay"],
)

Now run the app and see the header appears:

​Fetch the jinja2 tool

​Tell Bazel how to run it

​Declare the code generation

​Depend on the codegen

Fetch the jinja2 tool

Tell Bazel how to run it

Declare the code generation

Depend on the codegen