| --- | 
 | layout: documentation | 
 | title: Test encyclopedia | 
 | --- | 
 | <h1>Test encyclopedia</h1> | 
 |  | 
 | <p class="lead">An Exhaustive Specification of the Test Execution Environment</p> | 
 |  | 
 | <h2>Background</h2> | 
 |  | 
 | <p>The Bazel BUILD language includes rules which can be used to define | 
 | automated test programs in many languages.</p> | 
 |  | 
 | <p>Tests are run using <code><a href="user-manual.html#test">bazel test</a></code>. | 
 |  | 
 | Users may also execute test binaries directly. This is allowed but not endorsed, as such | 
 | an invocation will not adhere to the mandates described below.</p> | 
 |  | 
 | <p>Tests should be <i>hermetic</i>: that is, they ought to access only those | 
 | resources on which they have a declared dependency. If tests are not properly | 
 | hermetic then they do not give historically reproducible results. This could be a | 
 | significant problem for culprit finding (determining which change broke a test), | 
 | release engineering auditability, and resource isolation of tests (automated | 
 | testing frameworks ought not DDOS a server because some tests happen to | 
 | talk to it).<p> | 
 |  | 
 | <h2>Objective</h2> | 
 |  | 
 | <p>The goal of this document is to formally establish the runtime environment | 
 | for and expected behavior of Bazel tests.  It will also impose requirements on | 
 | the test runner and the build system. | 
 |  | 
 | Our intent is to help test authors avoid relying on unspecified | 
 | behavior, and thus give the testing infrastructure more freedom to make | 
 | implementation changes.  We will also take the opportunity to tighten up some | 
 | holes which currently allow many tests to pass despite not being | 
 | properly hermetic, deterministic, and reentrant.</p> | 
 |  | 
 | <p>This document is intended to be both normative and authoritative.  If | 
 | this specification and the implemented behavior of test runner disagree, the | 
 | specification takes precedence.</p> | 
 |  | 
 |  | 
 | <h2>Purpose of tests</h2> | 
 |  | 
 | <p>The purpose of Bazel tests is to confirm some property of the source files | 
 | checked into the repository.  (In this document, "source files" includes test data, | 
 | golden outputs, and anything else kept under version control.)  One | 
 | user writes a test to assert an invariant which they expect to be maintained. | 
 | Other users execute the test later to check whether the invariant has been | 
 | broken.  If the test depends on any variables other than source files | 
 | (non-hermetic), its value is diminished, because the later users cannot be sure | 
 | their changes are at fault when the test stops passing.</p> | 
 |  | 
 | <p>Therefore the outcome of a test must depend only on:</p> | 
 | <ul> | 
 |   <li>source files on which the test has a declared dependency</li> | 
 |   <li>products of the build system on which the test has a declared dependency</li> | 
 |   <li>resources whose behavior is guaranteed by the test runner to remain constant</li> | 
 | </ul> | 
 |  | 
 | <p>Currently, such behavior is not enforced. However, test runners reserve the | 
 | right to add such enforcement in the future.</p> | 
 |  | 
 | <h2>Role of the build system</h2> | 
 |  | 
 | <p>Test rules are analogous to binary rules in that each must yield an | 
 | executable program.  For some languages, this is a stub program which combines | 
 | a language-specific harness with the test code.  Test rules must produce other | 
 | outputs as well.  In addition to the primary test executable, the test runner | 
 | will need a manifest of <b>runfiles</b>, input files which should be made | 
 | available to the test at runtime, and it may need information about the type, | 
 | size, and tags of a test.</p> | 
 |  | 
 | <p>The build system may use the runfiles to deliver code as well as data.  (This | 
 | might be used as an optimization to make each test binary smaller by sharing | 
 | files across tests, e.g. through the use of dynamic linking.)  The build system | 
 | should ensure that the generated executable loads these files via the runfiles | 
 | image provided by the test runner, rather than hardcoded references to absolute | 
 | locations in the source or output tree.</p> | 
 |  | 
 | <h2>Role of the test runner</h2> | 
 |  | 
 | <p>From the point of view of the test runner, each test is a program which can | 
 | be invoked with <code>execve()</code>.  There may be other ways to execute | 
 | tests; for example, an IDE might allow the execution of Java tests in-process. | 
 | However, the result of running the test as a standalone process must be | 
 | considered authoritative.  If a test process runs to completion and terminates | 
 | normally with an exit code of zero, the test has passed.  Any other result is | 
 | considered a test failure.  In particular, writing any of the strings | 
 | <code>PASS</code> or <code>FAIL</code> to stdout has no significance to the test | 
 | runner.</p> | 
 |  | 
 | <p>If a test takes too long to execute, exceeds some resource limit, or the test | 
 | runner otherwise detects prohibited behavior, it may choose to kill the test | 
 | and treat the run as a failure.  The runner must not report the test as passing | 
 | after sending a signal to the test process or any children thereof.</p> | 
 |  | 
 | <p id="timeout">The whole test target (not individual methods or tests) is given a | 
 | limited amount of time to run to completion. The time limit for a test is based | 
 | on its <a href="be/common-definitions.html#test.timeout"><code>timeout</code></a> | 
 | attribute according to the following table:</p> | 
 |  | 
 | <table class="table table-bordered table-striped table-condensed"> | 
 |   <thead> | 
 |     <tr> | 
 |       <th><a href="be/common-definitions.html#test.timeout"><code>timeout</code></a></th> | 
 |       <th>Time Limit (sec.)</th> | 
 |     </tr> | 
 |   </thead> | 
 |   <tbody> | 
 |     <tr><td><code>short</code></td><td>60</td></tr> | 
 |     <tr><td><code>moderate</code></td><td>300</td></tr> | 
 |     <tr><td><code>long</code></td><td>900</td></tr> | 
 |     <tr><td><code>eternal</code></td><td>3600</td></tr> | 
 |   </tbody> | 
 | </table> | 
 |  | 
 | <p id="size">Tests which do not explicitly specify a timeout have one implied based on the | 
 | test's <a href="be/common-definitions.html#test.size"><code>size</code></a> as follows:</p> | 
 |  | 
 | <table class="table table-bordered table-striped table-condensed"> | 
 |   <thead> | 
 |     <tr> | 
 |       <th><a href="be/common-definitions.html#test.size"><code>size</code></a></th> | 
 |       <th>Implied timeout label</th> | 
 |     </tr> | 
 |   </thead> | 
 |   <tbody> | 
 |     <tr><td><code>small</code></td><td>short</td></tr> | 
 |     <tr><td><code>medium</code></td><td>moderate</td></tr> | 
 |     <tr><td><code>large</code></td><td>long</td></tr> | 
 |     <tr><td><code>enormous</code></td><td>eternal</td></tr> | 
 |   </tbody> | 
 | </table> | 
 | <p>For example a "large" test with no explicit timeout setting will be allotted | 
 | 900 seconds to run. A "medium" test with a timeout of "short" will be allotted | 
 | 60 seconds.</p> | 
 |  | 
 | <p>Unlike <code>timeout</code>, the <code>size</code> also determines the | 
 | limits for other resources like RAM, as described in | 
 | <a href="be/common-definitions.html#common-attributes-tests">Common | 
 | definitions</a>.</p> | 
 |  | 
 | <p>All combinations of <code>size</code> and <code>timeout</code> labels are | 
 | legal, so an "enormous" test may be declared to have a timeout of "short". | 
 | Presumably it would do some really horrible things very quickly.</p> | 
 | <p>Tests may return arbitrarily fast regardless of timeout.  A test is not | 
 | penalized for an overgenerous timeout, although a warning may be issued: you | 
 | should generally set your timeout as tight as you can without incurring any | 
 | flakiness.</p> | 
 |  | 
 | <p>The test timeout can be overridden with the <code>--test_timeout</code> | 
 | bazel flag, e.g. for manually running under conditions which are known to be | 
 | slow. The <code>--test_timeout</code> values are in seconds. For example, | 
 | <code>--test_timeout=120</code> will set the test timeout to two minutes.</p> | 
 |  | 
 | <p>There is also a recommended lower bound for test timeouts as follows: </p> | 
 |  | 
 | <table class="table table-bordered table-striped table-condensed"> | 
 |   <thead> | 
 |     <tr><th>timeout</th><th>Time minimum (sec.)</th></tr> | 
 |   </thead> | 
 |   <tbody> | 
 |     <tr><td><code>short</code></td><td>0</td></tr> | 
 |     <tr><td><code>moderate</code></td><td>30</td></tr> | 
 |     <tr><td><code>long</code></td><td>300</td></tr> | 
 |     <tr><td><code>eternal</code></td><td>900</td></tr> | 
 |   </tbody> | 
 | </table> | 
 |  | 
 | <p>For example, if a "moderate" test completes in 5.5s, consider setting | 
 | <code>timeout = "short"</code> or <code>size = "small"</code>. Using the bazel | 
 | <code>--test_verbose_timeout_warnings</code> command line option will show the | 
 | tests whose specified size is too big.</p> | 
 |  | 
 | <p>Test sizes and timeouts are specified in the BUILD file according to the specification | 
 | <a href="be/common-definitions.html#common-attributes-tests">here</a>. | 
 | If unspecified, a test's size will default to "medium".</p> | 
 |  | 
 | <p>If the main process of a test exits, but some of its children are still | 
 | running, the test runner should consider the run complete and count it as a | 
 | success or failure based on the exit code observed from the main process.  The | 
 | test runner may kill any stray processes.  Tests should not leak processes in | 
 | this fashion.</p> | 
 |  | 
 | <h2 id="test-sharding">Test sharding</h2> | 
 |  | 
 | <p>Tests can be parallelized via test sharding. See | 
 | <a href="user-manual.html#sharding-strategy"><code>--test_sharding_strategy</code></a> | 
 | and | 
 | <a href="be/common-definitions.html#common-attributes-tests"><code>shard_count</code></a> | 
 | to enable test sharding. When sharding is enabled, the test runner is launched | 
 | once per shard. The environment variable | 
 | <a href="#initial-conditions"><code>TEST_TOTAL_SHARDS</code></a> is the number | 
 | of shards, and | 
 | <a href="#initial-conditions"><code>TEST_SHARD_INDEX</code></a> is the shard | 
 | index, beginning at 0. Runners use this information to select which tests to | 
 | run - for example, using a round-robin strategy. Not all test runners support sharding. | 
 | If a runner supports sharding, it must create or update the last modified date | 
 | of the file specified by | 
 | <a href="#initial-conditions"><code>TEST_SHARD_STATUS_FILE</code></a>. | 
 | Otherwise, Bazel assumes it does not support sharding and will not launch | 
 | additional runners.</p> | 
 |  | 
 | <h2 id="initial-conditions">Initial conditions</h2> | 
 |  | 
 | <p>When executing a test, the test runner must establish certain initial | 
 | conditions.</p> | 
 |  | 
 | <p>The test runner must invoke each test with the path to the test | 
 | executable in <code>argv[0]</code>.  This path must be relative and | 
 | beneath the test's current directory (which is in the runfiles tree, | 
 | see below). | 
 | The test runner should not pass any other arguments to a | 
 | test unless the user explicitly requests it.</p> | 
 |  | 
 | <p>The initial environment block shall be composed as follows:</p> | 
 |  | 
 | <table class="table table-bordered table-striped table-condensed"> | 
 |   <thead> | 
 |     <tr><th>Variable</th><th>Value</th><th>Status</th></tr> | 
 |   </thead> | 
 |   <tbody> | 
 |  | 
 |     <tr><td><code>HOME</code></td><td>value of <code>$TEST_TMPDIR</code></td><td>recommended</td></tr> | 
 |     <tr><td><code>LANG</code></td><td><i>unset</i></td><td>required</td></tr> | 
 |     <tr><td><code>LANGUAGE</code></td><td><i>unset</i></td><td>required</td></tr> | 
 |     <tr><td><code>LC_ALL</code></td><td><i>unset</i></td><td>required</td></tr> | 
 |     <tr><td><code>LC_COLLATE</code></td><td><i>unset</i></td><td>required</td></tr> | 
 |     <tr><td><code>LC_CTYPE</code></td><td><i>unset</i></td><td>required</td></tr> | 
 |     <tr><td><code>LC_MESSAGES</code></td><td><i>unset</i></td><td>required</td></tr> | 
 |     <tr><td><code>LC_MONETARY</code></td><td><i>unset</i></td><td>required</td></tr> | 
 |     <tr><td><code>LC_NUMERIC</code></td><td><i>unset</i></td><td>required</td></tr> | 
 |     <tr><td><code>LC_TIME</code></td><td><i>unset</i></td><td>required</td></tr> | 
 |     <tr><td><code>LD_LIBRARY_PATH</code></td><td>colon-separated list of directories containing shared libraries</td><td>optional</td></tr> | 
 |     <tr><td><code>JAVA_RUNFILES</code></td><td>value of <code>$TEST_SRCDIR</code></td><td>deprecated</td></tr> | 
 |  | 
 |     <tr><td><code>LOGNAME</code></td><td>value of <code>$USER</code></td><td>required</td></tr> | 
 |  | 
 |     <tr><td><code>PATH</code></td><td><code>/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.</code></td><td>recommended</td></tr> | 
 |     <tr><td><code>PWD</code></td><td><code>$TEST_SRCDIR/<i>workspace-name</i></code></td><td>recommended</td></tr> | 
 |     <tr><td><code>SHLVL</code></td><td><code>2</code></td><td>recommended</td></tr> | 
 |     <tr><td><code>TEST_INFRASTRUCTURE_FAILURE_FILE</code></td><td>absolute path to a private file in a writable directory (This file should only be used to report failures originating from the testing infrastructure, not as a general mechanism for reporting flaky failures of tests. In this context, testing infrastructure is defined as systems or libraries that are not test-specific, but can cause test failures by malfunctioning. The first line is the name of the testing infrastructure component that caused the failure, the second one a human-readable description of the failure. Additional lines are ignored.)</td><td>optional</td></tr> | 
 |     <tr><td><code>TEST_LOGSPLITTER_OUTPUT_FILE</code></td><td>absolute path to a private file in a writable directory (used to write Logsplitter protobuffer log)</td><td>optional</td></tr> | 
 |     <tr><td><code>TEST_PREMATURE_EXIT_FILE</code></td><td>absolute path to a private file in a writable directory (used for catching calls to exit())</td><td>optional</td></tr> | 
 |     <tr> | 
 |       <td><code>TEST_RANDOM_SEED</code></td> | 
 |       <td>If the <code class='flag'>--runs_per_test</code> option is used, TEST_RANDOM_SEED | 
 |       is set to the <var>run number</var> (starting with 1) for each individual test run.</td> | 
 |       <td>optional</td> | 
 |     </tr> | 
 |     <tr><td><code>TEST_TARGET</code></td><td>The name of the target being tested</td><td>optional</td></tr> | 
 |     <tr><td><code>TEST_SIZE</code></td><td>The test <a href="#size"><code>size</code></a></td><td>optional</td></tr> | 
 |     <tr><td><code>TEST_TIMEOUT</code></td><td>The test <a href="#timeout"><code>timeout</code></a> in seconds</td><td>optional</td></tr> | 
 |     <tr><td><code>TEST_SHARD_INDEX</code></td><td>shard index, if <a href="#test-sharding">sharding</a> is used</td><td>optional</td></tr> | 
 |     <tr><td><code>TEST_SHARD_STATUS_FILE</code></td><td>path to file to touch to indicate support for <a href="#test-sharding">sharding</a></td><td>optional</td></tr> | 
 |     <tr><td><code>TEST_SRCDIR</code></td><td>absolute path to the base of the runfiles tree</td><td>required</td></tr> | 
 |     <tr><td><code>TEST_TOTAL_SHARDS</code></td><td>total <a href="be/common-definitions.html#test.shard_count">shard count</a>, if <a href="#test-sharding">sharding</a> is used</td><td>optional</td></tr> | 
 |     <tr><td><code>TEST_TMPDIR</code></td><td>absolute path to a private writable directory</td><td>required</td></tr> | 
 |  | 
 |     <tr><td><code>TEST_WORKSPACE</code></td><td>the local repository's workspace name</td><td>optional</td></tr> | 
 |     <tr><td><code>TEST_UNDECLARED_OUTPUTS_DIR</code></td><td>absolute path to a private writable directory (used to write undeclared test outputs)</td><td>optional</td></tr> | 
 |  | 
 |     <tr><td><code>TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR</code></td><td>absolute path to a private writable directory (used to write undeclared test output annotation .part files). | 
 |     </td><td>optional</td></tr> | 
 |  | 
 |     <tr><td><code>TEST_WARNINGS_OUTPUT_FILE</code></td><td>absolute path to a private file in a writable directory (used to write test target warnings)</td><td>optional</td></tr> | 
 |     <tr><td><code>TESTBRIDGE_TEST_ONLY</code></td><td>value of <a href="user-manual.html#flag--test_filter"><code>--test_filter</code></a>, if specified</td><td>optional</td></tr> | 
 |  | 
 |     <tr><td><code>TZ</code></td><td><code>UTC</code></td><td>required</td></tr> | 
 |     <tr><td><code>USER</code></td><td>value of <code>getpwuid(getuid())->pw_name</code></td><td>required</td></tr> | 
 |  | 
 |     <tr> | 
 |       <td><code>XML_OUTPUT_FILE</code></td> | 
 |       <td>Location of the test result XML output file. | 
 |         The XML schema is based on the | 
 |         <a href="https://windyroad.com.au/dl/Open%20Source/JUnit.xsd">JUnit test result | 
 |         schema</a>. | 
 |       </td> | 
 |       <td>optional</td> | 
 |     </tr> | 
 |   </tbody> | 
 | </table> | 
 | <br> | 
 | <p>The environment may contain additional entries.  Tests should not depend on the | 
 | presence, absence, or value of any environment variable not listed above.</p> | 
 |  | 
 | <p>The initial working directory shall be <code>$TEST_SRCDIR/$TEST_WORKSPACE</code>.</p> | 
 | <p> The current process id, process group id, session id, and parent process | 
 | id are unspecified.  The process may or may not be a process group leader or a | 
 | session leader.  The process may or may not have a controlling terminal.  The | 
 | process may have zero or more running or unreaped child processes.  The process | 
 | should not have multiple threads when the test code gains control.</p> | 
 |  | 
 | <p>File descriptor 0 (stdin) shall be open for reading, but what it is attached | 
 | to is unspecified.  Tests must not read from it.  File descriptors 1 (stdout) | 
 | and 2 (stderr) shall be open for writing, but what they are attached to is | 
 | unspecified.  It could be a terminal, a pipe, a regular file, or anything else | 
 | to which characters can be written.  They may share an entry in the open file | 
 | table (meaning that they cannot seek independently).  Tests should not inherit | 
 | any other open file descriptors.</p> | 
 |  | 
 | <p>The initial umask shall be 022 or 027.</p> | 
 |  | 
 | <p>No alarm or interval timer shall be pending.</p> | 
 |  | 
 | <p>The initial mask of blocked signals shall be empty.  All signals shall be set | 
 | to their default action.</p> | 
 |  | 
 | <p>The initial resource limits, both soft and hard, should be set as follows:</p> | 
 |  | 
 | <table class="table table-bordered table-striped table-condensed"> | 
 |   <thead> | 
 |     <tr><th>Resource</th><th>Limit</th></tr> | 
 |   </thead> | 
 |   <tbody> | 
 |     <tr><td>RLIMIT_AS</td><td>unlimited</td></tr> | 
 |     <tr><td>RLIMIT_CORE</td><td>unspecified</td></tr> | 
 |     <tr><td>RLIMIT_CPU</td><td>unlimited</td></tr> | 
 |     <tr><td>RLIMIT_DATA</td><td>unlimited</td></tr> | 
 |     <tr><td>RLIMIT_FSIZE</td><td>unlimited</td></tr> | 
 |     <tr><td>RLIMIT_LOCKS</td><td>unlimited</td></tr> | 
 |     <tr><td>RLIMIT_MEMLOCK</td><td>unlimited</td></tr> | 
 |     <tr><td>RLIMIT_MSGQUEUE</td><td>unspecified</td></tr> | 
 |     <tr><td>RLIMIT_NICE</td><td>unspecified</td></tr> | 
 |     <tr><td>RLIMIT_NOFILE</td><td>at least 1024</td></tr> | 
 |     <tr><td>RLIMIT_NPROC</td><td>unspecified</td></tr> | 
 |     <tr><td>RLIMIT_RSS</td><td>unlimited</td></tr> | 
 |     <tr><td>RLIMIT_RTPRIO</td><td>unspecified</td></tr> | 
 |     <tr><td>RLIMIT_SIGPENDING</td><td>unspecified</td></tr> | 
 |     <tr><td>RLIMIT_STACK</td><td>unlimited, or 2044KB <= rlim <= 8192KB</td></tr> | 
 |   </tbody> | 
 | </table> | 
 |  | 
 | <p>The initial process times (as returned by <code>times()</code>) and resource | 
 | utilization (as returned by <code>getrusage()</code>) are unspecified.</p> | 
 |  | 
 | <p>The initial scheduling policy and priority are unspecified.</p> | 
 |  | 
 | <h2>Role of the host system</h2> | 
 |  | 
 | <p>In addition to the aspects of user context under direct control of the | 
 | test runner, the operating system on which tests execute must satisfy certain | 
 | properties for a test run to be valid.</p> | 
 |  | 
 | <h4>Filesystem</h4> | 
 |  | 
 | <p> | 
 | The root directory observed by a test may or may not be the real root directory.<br> | 
 | <code>/proc</code> shall be mounted.<br> | 
 | All build tools shall be present at the absolute paths under <code>/usr</code> used by a local installation.<br> | 
 | Paths starting with <code>/home</code> may not be available. Tests should not access any such paths.<br> | 
 | <code>/tmp</code> | 
 | shall be writable, but tests should avoid using these paths.<br> | 
 |  | 
 | Tests must not assume that any constant path is available for their exclusive use.<br> | 
 | Tests must not assume that atimes are enabled for any mounted filesystem.<br> | 
 | </p> | 
 |  | 
 | <h4>Users and groups</h4> | 
 |  | 
 | <p>The users root, nobody, and unittest must exist. The groups root, nobody, | 
 | and eng must exist.</p> | 
 |  | 
 | <p>Tests must be executed as a non-root user.  The real and effective user ids | 
 | must be equal; likewise for group ids.  Beyond this, the current user id, group | 
 | id, user name, and group name are unspecified.  The set of supplementary group | 
 | ids is unspecified.</p> | 
 |  | 
 | <p>The current user id and group id must have corresponding names which can be | 
 | retrieved with <code>getpwuid()</code> and <code>getgrgid()</code>.  The same | 
 | may not be true for supplementary group ids.</p> | 
 |  | 
 | <p>The current user must have a home directory. It may not be writable.  Tests | 
 | must not attempt to write to it.</p> | 
 |  | 
 | <h4>Networking</h4> | 
 |  | 
 | <p>The hostname is unspecified. It may or may not contain a dot. Resolving | 
 | the hostname must give an IP address of the current host. Resolving the | 
 | hostname cut after the first dot must also work. The hostname localhost must | 
 | resolve.</p> | 
 |  | 
 | <h4>Other resources</h4> | 
 |  | 
 | <p>Tests are granted at least one CPU core. Others may be available but this | 
 | is not guaranteed. Other performance aspects of this core are not specified. | 
 | You can increase the reservation to a higher number of CPU cores by adding the | 
 | tag "cpu:n" (where n is a positive number) to a test rule. If a machine has | 
 | less total CPU cores than requested, Bazel will still run the test. If a test | 
 | uses <a href="#test-sharding">sharding</a>, each individual shard will reserve | 
 | the number of CPU cores specified here.</p> | 
 |  | 
 | <p>Tests may create subprocesses, but not process groups or sessions.</p> | 
 |  | 
 | <p>There is a limit on the number of input files a test may consume. This limit | 
 | is subject to change, but is currently in the range of tens of thousands of inputs. | 
 |  | 
 | </p> | 
 |  | 
 | <h4>Time and date</h4> | 
 |  | 
 | <p>The current time and date are unspecified. The system timezone is unspecified. | 
 |  | 
 | </p> | 
 |  | 
 | <p>X Windows may or may not be available. Tests that need an X server should | 
 | start Xvfb.</p> | 
 |  | 
 | <h2>Test interaction with the filesystem</h2> | 
 | <p>All file paths specified in test environment variables point to | 
 | somewhere on the local filesystem, unless otherwise specified.</p> | 
 |  | 
 | <p> | 
 | Tests should create files only within the directories specified by | 
 | <code>$TEST_TMPDIR</code> and <code>$TEST_UNDECLARED_OUTPUTS_DIR</code> | 
 | (if set).<br> | 
 | These directories will be initially empty.<br> | 
 | Tests must not attempt to remove, chmod, or otherwise alter these directories.<br> | 
 | These directories may be a symbolic links.<br> | 
 | The filesystem type of <code>$TEST_TMPDIR/.</code> remains unspecified.<br> | 
 | Tests may also write .part files to the <code>$TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR</code> | 
 | to annotate undeclared output files.</p> | 
 |  | 
 | <p>In rare cases, a test may be forced to create files in <code>/tmp</code>. For | 
 | example, <a href="https://serverfault.com/questions/641347">path length limits | 
 | for Unix domain sockets</a> typically require creating the socket under | 
 | <code>/tmp</code>. Bazel will be unable to track such files; the test itself must | 
 | take care to be hermetic, to use unique paths to avoid colliding with other, | 
 | simultaneously running tests and non-test processes, and to clean up the files it | 
 | creates in <code>/tmp</code>.</p> | 
 |  | 
 | <p>Tests must access inputs through the <b>runfiles</b> mechanism, or other | 
 | parts of the execution environment which are specifically intended to make | 
 | input files available. | 
 |  | 
 | Tests must not access other outputs of the | 
 | build system at paths inferred from the location of their own executable.</p> | 
 |  | 
 | <p>It is unspecified whether the runfiles tree contains regular files, symbolic | 
 | links, or a mixture. The runfiles tree may contain symlinks to directories. | 
 | Tests should avoid using paths containing <code>..</code> components within the | 
 | runfiles tree.</p> | 
 |  | 
 | <p>No directory, file, or symlink within the runfiles tree (including paths | 
 | which traverse symlinks) should be writable. (It follows that the initial | 
 | working directory should not be writable.) Tests must not assume that any part | 
 | of the runfiles is writable, or owned by the current user (i.e. chmod and chgrp | 
 | may fail).</p> | 
 |  | 
 | <p>The runfiles tree (including paths which traverse symlinks) must not change | 
 | during test execution.  Parent directories and filesystem mounts must not change | 
 | in any way which affects the result of resolving a path within the runfiles | 
 | tree.</p> | 
 |  | 
 | <p>In order to catch early exit, a test may create a file at the path specified by | 
 | <code>TEST_PREMATURE_EXIT_FILE</code> upon start and remove it upon exit.  If | 
 | Bazel sees the file when the test finishes, it will assume that the test exited | 
 | prematurely and mark it as having failed.</p> | 
 |  | 
 | <h2 id="tag-conventions">Tag conventions</h2> | 
 |  | 
 | <p> | 
 |   Some tags in the test rules have a special | 
 |   meaning. See also the <a href="be/common-definitions.html#common.tags">Bazel | 
 |   Build Encyclopedia on the <code>tags</code> attribute</a>. | 
 | </p> | 
 |  | 
 | <table class="table table-bordered table-striped table-condensed"> | 
 |   <thead> | 
 |     <tr><th>Tag</th><th>Meaning</th></tr> | 
 |   </thead> | 
 |   <tbody> | 
 |     <tr> | 
 |       <th><code>exclusive</code></th> | 
 |  | 
 |       <td>run no other test at the same time</td> | 
 |     </tr> | 
 |     <tr> | 
 |       <th><code>external</code></th> | 
 |       <td>test has an external dependency; disable test caching</td> | 
 |     </tr> | 
 |     <tr> | 
 |       <th><code>large</code></th> | 
 |       <td><code>test_suite</code> convention; suite of large tests</td> | 
 |     </tr> | 
 |  | 
 |     <tr> | 
 |       <th><code>manual</code>*</th> | 
 |  | 
 |       <td>don't include test target in wildcard target patterns like <code>:...</code>, <code>:*</code>, or <code>:all</code>)</td> | 
 |     </tr> | 
 |     <tr> | 
 |       <th><code>medium</code></th> | 
 |  | 
 |       <td><code>test_suite</code> convention; suite of medium tests</td> | 
 |     </tr> | 
 |  | 
 |     <tr> | 
 |       <th><code>small</code></th> | 
 |       <td><code>test_suite</code> convention; suite of small tests</td> | 
 |     </tr> | 
 |  | 
 |     <tr> | 
 |       <th><code>smoke</code></th> | 
 |       <td> | 
 |         <code>test_suite</code> convention; means it should be run before committing code changes | 
 |         into the version control system | 
 |       </td> | 
 |     </tr> | 
 |  | 
 |   </tbody> | 
 | </table> | 
 |  | 
 | *Note: bazel <code>query</code> does not respect the manual tag. | 
 | <h2>Runfiles</h2> | 
 |  | 
 | <p>In the following, assume there is a *_binary() rule labeled <code>//foo/bar:unittest</code>, | 
 | with a run-time dependency on the rule labeled <code>//deps/server:server</code>.</p> | 
 |  | 
 | <h4>Location</h4> | 
 | <p>The runfiles directory for a target <code>//foo/bar:unittest</code> is the directory | 
 | <code>$(WORKSPACE)/$(BINDIR)/foo/bar/unittest.runfiles</code>. This path is referred to as the | 
 | <code>runfiles_dir</code>.</p> | 
 |  | 
 | <h4>Dependencies</h4> | 
 | <p>The runfiles directory is declared as a compile-time dependency of the *_binary() rule. | 
 | The runfiles directory itself depends on the set of BUILD files that affect the *_binary() rule | 
 | or any of its compile-time or run-time dependencies. Modifying source files does not affect the | 
 | structure of the runfiles directory, and thus does not trigger any rebuilding.</p> | 
 |  | 
 | <h4>Contents</h4> | 
 | <p>The runfiles directory contains the following:</p> | 
 | <ul> | 
 |   <li><b>Symlinks to run-time dependencies</b>: each OutputFile and CommandRule that is a run-time | 
 | dependency of the *_binary() rule is represented by one symlink in the runfiles directory. | 
 | The name of the symlink is <code>$(WORKSPACE)/package_name/rule_name</code>. For example, the | 
 | symlink for server would be named <code>$(WORKSPACE)/deps/server/server</code>, and the full path | 
 | would be <code>$(WORKSPACE)/foo/bar/unittest.runfiles/$(WORKSPACE)/deps/server/server</code>. | 
 | The destination of the symlink is the OutputFileName() of the OutputFile or CommandRule, | 
 | expressed as an absolute path. Thus, the destination of the symlink might be | 
 | <code>$(WORKSPACE)/linux-dbg/deps/server/42/server</code>.</li> | 
 |   <li><b>Symlinks to sub-runfiles</b>: for every *_binary() Z that is a run-time depdendency of | 
 | *_binary() C, there is a second link in the runfiles directory of C to the runfiles of Z. | 
 | The name of the symlink is <code>$(WORKSPACE)/package_name/rule_name.runfiles</code>. | 
 | The target of the symlink is the runfiles directory. I.e. all subprograms share a common | 
 | runfiles directory.</li> | 
 | </ul> | 
 |  |