Create documentation for bazel-skylib analysis testing framework RELNOTES: None. PiperOrigin-RevId: 236160153
diff --git a/site/docs/skylark/testing.md b/site/docs/skylark/testing.md index 5bb5de6..fe4cfc2 100644 --- a/site/docs/skylark/testing.md +++ b/site/docs/skylark/testing.md
@@ -16,8 +16,8 @@ [Skylib](https://github.com/bazelbuild/bazel-skylib) has a test framework called [`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/master/lib/unittest.bzl) for checking the analysis-time behavior of rules, such as their actions and -providers. It is currently the best option for tests that need to access the -inner workings of rules. +providers. Such tests are called "analysis tests" and are currently the best +option for testing the inner workings of rules. Some caveats: @@ -32,12 +32,15 @@ first. It helps to [keep in mind](concepts.md#evaluation-model) which code runs during the loading phase and which code runs during the analysis phase. -* It cannot easily test for expected failures. +* Analysis tests are intended to be fairly small and lightweight. Certain + features of the analysis testing framework are restricted to verifying + targets with a maximum number of transitive dependencies (currently 500). + This is due to performance implications of using these features with larger + tests. The basic principle is to define a testing rule that depends on the rule-under-test. This gives the testing rule access to the rule-under-test’s -providers. There is experimental support for passing along action information -in the form of an additional provider. +providers. The testing rule’s implementation function carries out assertions. If there are any failures, these are not raised immediately by calling `fail()` (which would @@ -84,31 +87,28 @@ ```python -load("@bazel_skylib//:lib.bzl", "asserts", "unittest") +load("@bazel_skylib//lib:unittest.bzl", "asserts", "analysistest") load(":myrules.bzl", "myrule", "MyInfo") # ==== Check the provider contents ==== def _provider_contents_test_impl(ctx): # Analysis-time test logic; place assertions here. Always begins with begin() - # and ends with end(). If you forget to call end(), you will get an error - # about the test result file not having a generating action. - env = unittest.begin(ctx) - asserts.equals(env, "some value", ctx.attr.dep[MyInfo].val) + # and ends with returning end(). If you forget to return end(), you will get an + # error about an analysis test needing to return an instance of AnalysisTestResultInfo. + env = analysistest.begin(ctx) + target_under_test = analysistest.target_under_test(env) + asserts.equals(env, "some value", target_under_test[MyInfo].val) # You can also use keyword arguments for readability if you prefer. asserts.equals(env, - expected="some value", - actual=ctx.attr.dep[MyInfo].val) - return unittest.end(env) + expected="some value", + actual=target_under_test[MyInfo].val) + return analysistest.end(env) # Create the testing rule to wrap the test logic. Note that this must be bound # to a global variable due to restrictions on how rules can be defined. Also, # its name must end with "_test". -provider_contents_test = unittest.make(_provider_contents_test_impl, - attrs={"dep": attr.label()}) -# You can use a different attrs dict if you need to take in multiple rules for -# the same unit test, or if you need to test an aspect, or if you want to -# parameterize the assertions with different expected results. +provider_contents_test = analysistest.make(_provider_contents_test_impl) # Macro to setup the test. def test_provider_contents(): @@ -116,8 +116,9 @@ myrule(name = "provider_contents_subject") # Testing rule. provider_contents_test(name = "provider_contents", - dep = ":provider_contents_subject") - + target_under_test = ":provider_contents_subject") + # Note the target_under_test attribute is how the test rule depends on + # the real rule target. # Entry point from the BUILD file; macro for running each test case's macro and # declaring a test suite that wraps them together. @@ -142,7 +143,7 @@ * The tests themselves, each of which consists of 1) an analysis-time implementation function for the testing rule, 2) a declaration of the testing - rule via `unittest.make()`, and 3) a loading-time function (macro) for + rule via `analysistest.make()`, and 3) a loading-time function (macro) for declaring the rule-under-test (and its dependencies) and testing rule. If the assertions do not change between test cases, 1) and 2) may be shared by multiple test cases. @@ -172,96 +173,97 @@ Note that the labels of all targets can conflict with other labels in the same BUILD package, so it’s helpful to use a unique name for the test. -### Actions example +### Failure Testing -To check that the `ctx.actions.write()` line works correctly, the above example -is modified as follows. +It may be useful to verify that a rule fails given certain inputs or in +certain state. This can be done using the analysis test framework: -`//mypkg/myrules.bzl`: +Firstly, the test rule created with `analysistest.make` should specify `expect_failure`: ```python -... - -myrule = rule( - implementation = _myrule_impl, - outputs = {"out": "%{name}.out"}, - # This enables the Actions provider for this rule. - _skylark_testable = True, +failure_testing_test = analysistest.make( + _failure_testing_test_impl, + expect_failure = True, ) ``` -`//mypkg/myrules_test.bzl`: +Secondly, the test rule implementation should make assertions on the nature +of the failure that took place (specifically, the failure message): ```python -... +def _failure_testing_test_impl(ctx): + env = analysistest.begin(ctx) -# ==== Check the emitted file_action ==== + asserts.expect_failure(env, "This rule should never work") -def _file_action_test_impl(ctx): - env = unittest.begin(ctx) - dep = ctx.attr.dep - # Retrieve the Actions provider. - actions = dep[Actions] - # Retrieve the generating action for the output file. - action = actions.by_file[dep.out] - # Check the content that is to be written by the action. - asserts.equals(env, action.content, "abc") - return unittest.end(env) - -file_action_test = unittest.make(_file_action_test_impl, - attrs={"dep": attr.label()}) - -def test_file_action(): - myrule(name = "file_action_subject") - file_action_test(name = "file_action", - dep = ":file_action_subject") - -... - -def myrules_test_suite(): - # Call all test functions and wrap their targets in a suite. - test_provider_contents() - test_file_action() - # ... - - native.test_suite( - name = "myrules_test", - tests = [ - ":provider_contents", - ":file_action", - # ... - ] -), + return analysistest.end(env) ``` -The flag `"_skylark_testable = True"` is needed on any rule whose actions are to -be tested. This triggers the creation of the `Actions` provider. (The leading -underscore is because this API is still experimental.) The test logic for -actions makes use of the following API. +### Verifying Registered Actions -### Actions API - -The [`Actions`](lib/globals.html#Actions) provider is retrieved like any other -(non-legacy) provider: +You may want to write tests which make assertions about the actions that your +rule registers, for example, using `ctx.actions.run()`. This can be done in +your analysis test rule implementation function. An example: ```python -ctx.attr.foo[Actions] +def _inspect_actions_test_impl(ctx): + env = analysistest.begin(ctx) + + actions = analysistest.target_actions(env) + asserts.equals(env, 1, len(actions)) + action_output = actions[0].outputs.to_list()[0] + asserts.equals(env, "out.txt", action_output.basename) + return analysistest.end(env) ``` -The returned object has a single field, `by_file`, which holds a dictionary -mapping each of the rule’s output files to its generating action. (Actions that -do not have output files, in particular those generated by -`ctx.actions.do_nothing()`, cannot be retrieved.) +Note that `analysistest.target_actions(env)` returns a list of +[`Action`](lib/Action.html) objects which represent actions registered by the +target under test. -The interface of the actions stored in the `by_file` map is documented -[here](lib/Action.html). +### Verifying Rule Behavior Under Different Flags -Finally, there is support for testing helper functions that are not rules, but -that take in a rule’s `ctx` in order to create actions on it. Use -`ctx.created_actions()` to get an `Actions` provider that has information about -all actions created on `ctx` up to the point that this function was called. For -this to work, the testing rule itself must have `"_skylark_testable=True"` set. -Testing rules created using `unittest.make()` automatically have this flag set. +You may want to verify your real rule behaves a certain way given certain +build flags. For example, your rule may behave differently if a user specifies: + +```python +bazel build //mypkg:real_target -c opt +``` +versus +```python +bazel build //mypkg:real_target -c dbg +``` + +At first glance, this could be done by testing the target under test using +the desired build flags: +```python +bazel test //mypkg:myrules_test -c opt +``` + +But then it becomes impossible for your test suite to simultaneously contain a +test which verifies the rule behavior under `-c opt` and another test which +verifies the rule behavior under `-c dbg`. Both tests would not be able to run +in the same build! + +This can be solved by specifying the desired build flags when defining +the test rule: + +```python +myrule_c_opt_test = analysistest.make( + _myrule_c_opt_test_impl, + config_settings = { + "//command_line_option:c": "opt", + }, +) +``` + +Normally, a target under test is analyzed given the current build flags. +Specifying `config_settings` overrides the values of the specified command line +options. (Any unspecified options will retain their values from the actual +command line). + +In the specified `config_settings` dictionary, command line flags must be +prefixed with a special placeholder value `//command_line_option:`, as is shown +above. ## For validating artifacts @@ -396,12 +398,13 @@ ## For testing Starlark utilities -The same framework that was used to test rules can also be used to test utility -functions (i.e., functions that are neither macros nor rule implementations). -There is no need to pass an `attrs` argument to `unittest.make()`, and there is -no special loading-time setup code to instantiate any rules-under-test. The -convenience function `unittest.suite()` can be used to reduce boilerplate in -this case. +[Skylib](https://github.com/bazelbuild/bazel-skylib)'s +[`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/master/lib/unittest.bzl) +framework can be used to test utility functions (that is, functions that are neither +macros nor rule implementations). Instead of using `unittest.bzl`'s `analysistest` +library, `unittest` may be used. +For such test suites, the convenience function `unittest.suite()` can be used to +reduce boilerplate. `//mypkg/BUILD`: @@ -422,7 +425,7 @@ ```python -load("@bazel_skylib//:lib.bzl", "asserts", "unittest") +load("@bazel_skylib//lib:unittest.bzl", "asserts", "unittest") load(":myhelpers.bzl", "myhelper") def _myhelper_test_impl(ctx): @@ -446,10 +449,3 @@ For more examples, see Skylib’s own [tests](https://github.com/bazelbuild/bazel-skylib/blob/master/tests/BUILD). -This can also be used when the utility function takes in a rule’s `ctx` object -as a parameter. If the behavior of the utility function requires that the rule -be defined in a certain way, you may have to pass in an `attrs` parameter to -`unittest.make()` after all, or you may have to declare the rule manually using -`rule()`. To test helpers that create actions, make the unit test rule set -`"_skylark_testable=True"` (if it is not created via `unittest.make()`) and -write assertions on the result of `ctx.created_actions()`, as described above.