commit 8f16879813383dad7026757e41111aac6bd8e7a1
author Googler <noreply@google.com> Mon Jul 09 01:36:01 2018 -0700
committer Copybara-Service <copybara-piper@google.com> Mon Jul 09 01:37:03 2018 -0700
tree a03c83a4b48fcdb4d3f14534b359f36156cc8c4e
parent f4c494d5f7f59d7d380e459ee496ecfb3f87526f

Make loading/analysis phase a lot faster.

There is a fundamental bottleneck in the way targets are currently configured:
The ConfiguredTargetFunction needs to load all dependent targets before it can
configure its own target. Now, outgoing deps of a target can be in three
states:
1. Package of dependency target not yet loaded
2. Dependency target not configured, but package already loaded
3. Dependency target configured

State #3 is good, we don't need to do any more work before we can configure the
current target. Now, most targets will have a combination of deps in states #1
and #2. The current behavior is to schedule the target configuration of deps in
state #2 and the package loading in state #1 (to get to state #3 and #2,
respectively). Only after *all* of these are computed will the corresponding
ConfiguredTargetFunction be rescheduled upon which the remaining deps that were
in state #1 before are now in state #2 and the corresponding target
configuration can be started.

This creates a substantial parallelization bottleneck by artifically sync'ing
the state transitions (#1->#2 and #2->#3) for all deps. In turn, this makes the
critical path artifically long. Instead, settle for just loading dependent
packages as long as there are any. This is a fast single-step operation.
Postpone the configuration of dependent targets to later as it can require long
dependency chains to be configured.

In theory, a better solution would be to split the target configuration in a
way so that it is possible to ask for configured targets without yet knowing
their package. However, that would require pulling config transitions apart (as
some of them happen because of the origin target/config and some of them happen
because of the destination target. Thus we'd need to change the
ConfiguredTargetKey so that the configuration stands for the configuration that
we request the target in, with target-caused transitions already applied. Now
that itself leads to inefficiencies. E.g. we would start requestiong InputFiles
in several configurations just to underneath always load them with a null
configuration.

RELNOTES: Faster analysis by improved parallelization.
PiperOrigin-RevId: 203725209