Baseline: ead44955

Cherry picks:

   + 99a85e79:
     Automated rollback of commit
     92f7b586.
   + 16523615:
     Fix a race condition in scheduling code in
     `FileSystemValueCheckerInferringAncestors`.
   + ed251187:
     Add incompatible flag to guard top-level aspects dependencies

Incompatible changes:

  - Removing java_common.javac_jar Starlark call.
  - native.existing_rule now returns select values in a form that is
    accepted by rule instantiation. This is a breaking API change
    because there is some code that relies on the precise type
    returned, including brittle workarounds for this bug specifically
    and insufficiently flexible workarounds for other issues with the
    intersection of select and native.existing_rule.
  - flipped incompatible_use_toolchain_resolution_for_java_rules, see
    #7849
  - Query output=xml/proto/location for source files will now show
    the location of line 1 of the source file (as the new default)
    instead of its location in the BUILD file.
  - Specifying a target pattern underneath a directory specified by
    .bazelignore will now emit a warning, not an error.
  - Query `--order_output=auto` will now sort lexicographically.
    However, when `somepath` is used as a top level function (e.g.
    `query 'somepath(a, b)'`), it will continue to output in
    dependency order. If you do not want the lexicographical output
    ordering, specify another `--order_output` value (`no`, `deps` or
    `full`) based on what ordering you require.
  - In the build event stream,
    BuildMetrics.TargetMetrics.targets_loaded is no longer populated.
    Its value was always mostly meaningless.
    BuildMetrics.TargetMetrics.targets_configured and
    BuildMetrics.ActionSummary.actions_created now include configured
    aspect data.
  - //visibility:legacy_public has been removed.
  - Flip and remove incompatible_dont_collect_so_artifacts
    (https://github.com/bazelbuild/bazel/issues/13043).
  - Remove flag --experimental_no_product_name_out_symlink: it is
    always true.
  - The Starlark method generate_dsym in objc fragment has
    been deleted.  Please use the equivalent apple_generate_dsym in
    cpp
    fragment instead.
  - Native libraries in data attribute are not collected. See
    https://github.com/bazelbuild/bazel/issues/13550 for details
  - Enforce the `--profile` path to be absolute.
  - Enforce the --memory_profile path to be absolute.
  - JavaToolchainInfo.jvm_opt returns Depset instead of a list.
  - --apple_sdk has been deleted.  It is a no-op.

Important changes:

  - Flag --incompatible_objc_compile_info_migration is removed.  See
    #10854.
  - Flag --incompatible_objc_compile_info_migration is removed.  See
    #10854.
  - Flag --incompatible_objc_compile_info_migration is removed.  See
    #10854.
  - none
    PAIR=cmita
  - The --incompatible_load_python_rules_from_bzl flag is now a no-op.
  - Filter all (instead of just C++) source files for coverage output
    according to --instrumentation_filter and
    --instrument_test_targets.
  - The `--incompatible_disable_native_apple_binary_rule` flag has
    been added which disables the native `apple_binary` rule. Users
    who need to use `apple_binary` directly (if they cannot use one
    of the more specific Apple rules) should load it from
    https://github.com/bazelbuild/rules_apple.
  - The Android rules' --use_singlejar_apkbuilder is now a no-op.
    SingleJar will always be used to build APKs.
  - dict.setdefault(key, ...) now fails if dict is frozen, even if it
    already contains key. This is an incompatible API change.
  - Flag --incompatible_objc_provider_remove_compile_info is removed.
     See #11359.
  - Starlark now permits def statements to be nested (closures).
  - native.existing_rule now returns select values in a form that is
    accepted by rule instantiation. This is a breaking API change,
    though the fallout is expected to be small.
  - Starlark now supports lambda (anonymous function) expressions.
  - The "test" and "coverage" commands no longer return 3 when a
    test action fails because of a system error. Instead, the exit
    code
    reflects the type of system error.
  - The undocumented ctx.expand feature no longer exists.
  - Make --legacy_dynamic_scheduler a no-op flag.
  - Multiplex persistent workers can now use the JSON protocol.
  - native.existing_rule now returns a mutable list, not a tuple, for
    a list-valued attributes. This is an incompatible API change.
  - Roll back change to have native.existing_rules use list instead
    of tuple.
  - BEP includes test suite expansions.
  - config_setting now honors `visibility` attribute (and defaults to
    `//visibility:public`)
  - Change the MultiArchSplitTransitionProvider to be based on
    platform type + CPU instead of fixed "ios_" + cpu.
  - enforce config_setting visibility. See
    https://github.com/bazelbuild/bazel/issues/12932 for details.
  - add a flag to build v4 signature file
  - Added _direct_source_jars output group to Java related targets.
    END_PUBLIC
  - pkg_deb is no longer part of @bazel_tools//build_defs/pkg:pkg.bzl.
    Use https://github.com/bazelbuild/rules_pkg/tree/main/pkg instead
  - Allowing the lipo operations to be conditional in the
    linkMultiArchBinary API for Apple binaries. Single architecture
    slices are now returned through AppleBinaryOutput and the
    Starlark API.
  - Release restriction for "-" in the package name for Python
    sources. Now `py_binary` and `py_test` targets can have main
    source file with "-" in the path.
  - Users consuming BEP may assume that a `named_set_of_files` event
    will
    appear before any event referencing that `named_set` by ID. This
    allows consumers
    to process the files for such events (eg. `TargetCompleted`)
    immediately.
  - BEP includes all files from successful actions in requested
    output groups.
    Previously, an output group's files were excluded if any file in
    the output group
    was not produced due to a failing action. Users can expect BEP
    output to be larger
    for failed builds.
  - In BEP, TargetComplete.output_group has a new field `incomplete`
    indicating that the file_sets field is missing one or more
    declared artifacts
    whose generating actions failed.
  - The flag `--toolchain_resolution_debug` now takes a regex
    argument, which is used to check which toolchain types should
    have debug info printed. You may use `.*` as an argument to keep
    the current behavior of debugging every toolchain type.
  - Add runfiles.merge_all() for merging a sequence of runfiles
    objects.
  - runfiles.merge() and merge_all() now respect
    --nested_set_depth_limit.
    If you hit the depth limit because you were calling merge() in a
    loop, use
    merge_all() on a sequence of runfiles objects instead.
  - Bazel will no longer create a bazel-out symlink if
    --symlink_prefix is specified: the directory pointed to via the
    bazel-out symlink is accessible via ${symlink_prefix}-out. If
    this causes problems for you, set
    --experimental_no_product_name_out_symlink=false in your builds
    and file an issue.
  - Updates worker protocol with cancellation fields, and adds
    experimental_worker_cancellation flag to control cancellation.
  - Simplify build failure output by always using `NNN arguments`.
  - trim_test_configuration now defaults to on
  - Mark genrule.srcs as a source attribute for coverage.
  - When using --allow_analysis_failures (for example, via
    bazel-skylib's
    analysistest with `expect_failure = True`), analysis-time
    failures in aspect
    implementation functions will now be propagated and saved in
    AnalysisFailureInfo, just like analysis-time failures in rules.
  - cquery --noimplicit_deps now correctly filters out resolved
    cc_toolchains
  - Sign apks deterministically.
  - Make gcov optional in cc_toolchain tools.
  - If --experimental_prefer_mutual_xcode is passed, Bazel will
    choose the local default (instead of the newest mutually
    available version) if it's available both locally and remotely.
  - Remove java_lite_proto_library.strict_deps attribute.
  - Generate proguard configurations deterministically.
  - Adds a new flag, `--incompatible_enable_cc_test_feature` which
    switches from the use of build variables to the feature of the
    same name.
  - Dropped fragile xz support from built in pkg_tar. Users requiring
    xz
    compression should switch to bazlebuild/rules_pkg.
  - If all strategies of one branch (the local or remote execution
    branch) of the `dynamic` strategy fail to even accept (via the
    response they give from `canExec`) the action, `dynamic` will now
    try to see if the other branch can accept it. (Trying to run it
    and it failing will still cause a failure if it was the first
    result, this is about strategies claiming they can't even try the
    action)
  - Add `disable_annotation_processing` option to
    `java_common.compile`, which disables any annotation processors
    passed to `plugins` or in `exported_plugins` of `deps`
  - Remove obsolete --incompatible_prohibit_aapt1
  - The minimum Android build tools version for the Android rules is
    now 30.0.0
  - Adds --experimental_reuse_sandbox_directories flag to reuse
    already-created non-worker sandboxes with cleanup.
  - --experimental_force_gc_after_build is deprecated and will be
    removed soon. Use --bep_publish_used_heap_size_post_build instead
  - Forward coverage-instrumented files from non-tool dependencies by
    default.
  - The used_heap_size_post_build field in BEP is populated when the
    --memory_profile flag is set
  - --run_validations defaults to true.
  - Consider label_keyed_string_dict attributes when gathering
    instrumented files for coverage.
  - Remove flag
    --experimental_forward_instrumented_files_info_by_default, now
    that this behavior is the default.
  - When using MemoryProfiler with multiple GCs via the
    --memory_profile_stable_heap_parameters flag, we do a more
    precise calculation of heap used at the end of the build. This
    will generally result in lower values.
  - --bep_publish_used_heap_size_post_build is deprecated. Use
    --memory_profile=/dev/null instead.

This release contains contributions from many people at Google, as well as Adam Liddell, Alex Eagle, Andrew Katson, Anthony Pratti, Austin Schuh, Benedek Thaler, Benjamin Lee, Benjamin Peterson, Benjamin Peterson, bromano, Cameron Mulhern, Christopher Peterson Sauer, Christopher Sauer, Cristian Hancila, Dan Bamikiya, Daniel McCarney, Daniel Wagner-Hall, Danny Wolf, Dave MacLachlan, Dave Nicponski, David Cummings, David, David Ostrovsky, Delwin9999, Denys Kurylenko, Dmitry Ivankov, dorranh, Ed Schouten, Eitan Adler, Elliotte Rusty Harold, erenon, Eric Cousineau, Ethan Steinberg, FaBrand, Felix Ehrenpfort, Finn Ball, Fredrik Medley, Garrett Holmstrom, Gautam Korlam, George Gensure, goodspark, Greg Estren, Grzegorz Lukasik, Jesse Chan, Joe Lencioni, Johannes Abt, John Laxson, Jonathan Schear, Justus Tumacder, Keith Smiley, Kevin Hogeland, Lauri Peltonen, Liu Liu, Lszl Csomor, m, Marc Zych, Mark Karpov, Masoud Koleini, Matt Mackay, Max Liu, Menny Even Danan, menny, Michael Chinen, Nathaniel Brough, Nick Korostelev, Niek Peeters, Nikolay Shelukhin, odisseus, Oleh Stolyar, Olek Wojnar, Olle Lundberg, Philipp Schrader, Pras Velagapudi, Qais Patankar, Rai, Ryan Beasley, samhowes, Samuel Giddins, Sebastian Olsson, Steve Siano, steve-the-bayesian, Stiopa Koltsov, susinmotion, tatiana, Thi Doan, ThomasCJY, Timothy Klim, Tobi, Torgil Svensson, Trustin Lee, Ulf Adams, Uri Baghin, Vaidas Pilkauskas, Vertexwahn, wisechengyi, Xavier Bonaventura, Yannic Bonenberger, Yannic Bonenberger, Yannic, Yury Evtikhov, Yuval Kaplan, Yuval, [zqzzq].

This CL introduces `incompatible_top_level_aspects_dependency` flag to enable aspect-on-aspect and requiring aspects for command line aspects. This can be needed because the new behavior can break builds where a relation between top-level aspects existed (based on required_aspect_providers and provides) but it was never actually applied as it was not supported.

One example of these cases:
blaze build //:main --aspects=/tools:my_def.bzl%a1,/tools:my_def.bzl%a2

If aspect a1 provides a1p provider, aspect a2 requires a1p provider and the rule of target `main` also provides a1p. Once top-level aspect-on-aspect is enabled this build will fail because a1p will be provided twice to a2 (from a1 applied on `main` and from `main` target rule). Previously the relation between a1 and a2 was not detected and a2 used to get only the value of a1p from `main`'s rule.

PiperOrigin-RevId: 390587290

`FileSystemValueCheckerInferringAncestors` works on the premise that we schedule all leaf entries from the diff and work our way up the path from there. The code which starts parallel processing detects leaves by looking at how many children to process each entry has. That is only correct if no processing is happening.

Fix the race condition happening due to the fact that we schedule the processing in a stream detecting the leaves -- this means that the detection algorithm works in parallel with processing. Add a materialization step in between to detect all leaves before any processing starts.

PiperOrigin-RevId: 390417452

*** Reason for rollback ***

Breakage in blaze nightly

*** Original change description ***

Get rid of unused j2objc module map

PiperOrigin-RevId: 389885762

The annotation processor is a java_plugin, which will stop returning JavaInfo and return only JavaPluginInfo.

PiperOrigin-RevId: 389691475

*** Reason for rollback ***

b/195550666: internal breakage

*** Original change description ***

Avoid touching the filesystem to see what the BUILD file for a Starlark extension was: we already have the package, so we can just read it out.

Genquery remains buggy here.

haxorz@ observed in http://b/123795023#comment6 that we only need to read these paths to determine if the package was defined in a BUILD or BUILD.bazel file. Since the package already knows that, we just ask it.

PiperOrigin-RevId: 389642637

Do visitation of graph for deletion in parallel, using ConcurrentHashMap#forEachEntry. Shaves ~1 second off benchmark build.

PiperOrigin-RevId: 389615354

grep was upgraded to 3.5 after updating MSYS on Winodws VM, which no longer matches CRLF line endings with `$`.

Fixes https://github.com/bazelbuild/bazel/issues/13816

RELNOTES: None
PiperOrigin-RevId: 389600750

Overloading the "re" module causes weird errors on our new Windows VM image, probably because Python itself uses it for some purpose now. I noticed this in our Bazel CI testing area, so the new image is not yet live.

Let's try to use a module that's less core to Python, but still in the standard library - I picked "mailbox" after looking at the list of standard library modules.

Here's an example log from the failing test on the new image:

```
** test_source_file_does_not_override_standard_library *************************
[...]
Error processing line 1 of C:\Python3\lib\site-packages\protobuf-3.17.3-py3.9-nspkg.pth:

Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "C:\Python3\lib\site.py", line 169, in addpackage
    exec(line)
  File "<string>", line 1, in <module>
  File "C:\Python3\lib\importlib\util.py", line 2, in <module>
    from . import abc
  File "C:\Python3\lib\importlib\abc.py", line 17, in <module>
    from typing import Protocol, runtime_checkable
  File "C:\Python3\lib\typing.py", line 2203, in <module>
    Pattern = _alias(stdlib_re.Pattern, 1)
AttributeError: module 're' has no attribute 'Pattern'
```
PiperOrigin-RevId: 389572773

Tracked in https://github.com/bazelbuild/bazel/issues/13816.

PiperOrigin-RevId: 389572724

PiperOrigin-RevId: 389421544

Implement a new diff checker which deduces affected `FileStateValue` and `DirectoryListingStateValue` keys from diffs which can skip ancestor directories in them. An extreme example of such diff would be one which only lists the leaf files when adding/deleting them (e.g. adding `a/b/c/d/file` where `a` does not currently exist).

Use the new algorithm for diffs marked as potentially missing ancestor directory entries.

The new logic has a very similar performance to the existing one with the difference of:

- adding new file with within new directories -- it does less stats since it deduces the directory from the leaf existence
- deleting a file -- it performs 1 extra stat since it cannot assume that the parent of a deleted file still exists
- unknown entries -- it tries to be lazy about stating those; we would only do that if that can prevent invalidation of a directory listing.

PiperOrigin-RevId: 389301057

Make action cache concurrency-friendly to avoid high contention. Use ConcurrentHashMap so there's no need for locking. Keep entries to be written to disk in a concurrent queue, and look their values up in the canonical map when writing them. This avoids races around writing data to the maps, although I don't know how big a deal that is anyway.

Doesn't seem to have much of a wall-time impact, most likely because contention just moves around. Hoping to tackle some of the ultimate causes soon.

PiperOrigin-RevId: 389272785

Introduce a new NodeEntry#removeReverseDepsFromDoneEntryDueToDeletion method and use it when deleting nodes from the Skyframe graph. We sacrifice some correctness checks for performance: rather than either keeping the rdeps as a set and removing from them one by one, or queuing the removed rdeps and doing the removal once (both of these use memory/are complex), we just filter the rdeps by the set of all keys we've visited during deletion.

Logs from sample build:

```
210806 11:55:35.183:I 57 [com.google.devtools.build.lib.analysis.BuildView.update:208] Starting analysis
210806 11:55:35.198:I 57 [com.google.devtools.build.lib.skyframe.SkyframeExecutor.handleAnalysisInvalidatingChange:877] Dropping configured target data
210806 11:55:35.245:I 4550 [com.google.devtools.build.lib.buildeventservice.BuildEventServiceUploader.publishBuildEvents:431] Starting publishBuildEvents: eventQueue=10
210806 11:55:38.100:I 57 [com.google.devtools.build.skyframe.InMemoryMemoizingEvaluator.delete:138] Spent 2880 milliseconds doing marking for deletion
210806 11:55:48.252:I 57 [com.google.devtools.build.skyframe.InvalidatingNodeVisitor$DeletingNodeVisitor.runInternal:276] Spent 8215 milliseconds doing invalidation graph traversal
210806 11:55:51.528:I 57 [com.google.devtools.build.skyframe.InvalidatingNodeVisitor$DeletingNodeVisitor.runInternal:285] Spent 3276 milliseconds doing reverse dep removal
210806 11:55:51.528:I 57 [com.google.devtools.build.skyframe.InvalidatingNodeVisitor.run:133] Spent 11684 milliseconds doing invalidation
```

The existence of ConcurrentHashMap#forEachEntry was brought up by haxorz@/nharmata@ in an unrelated chat.

PiperOrigin-RevId: 389246717

Roll forward of https://github.com/bazelbuild/bazel/commit/943c83aa58731c4f9561d79c458f254427a8f24c: Command line aspect-on-aspect

Supports aspect-on-aspect for command line aspects. Command line aspects specified via `--aspects` option will support a top-level aspect requiring aspect providers via `required_aspect_providers` to get their values from other top-level aspects advertising it that come before it in the `--aspects` list.

NEW:
- Add `incompatible_ignore_duplicate_top_level_aspects` flag to allow duplicates in `--aspects` list. The flag is set to true by default, otherwise a validation error will be thrown in case of duplicates in top-level aspects.
- Fix the error reporting for duplicate native aspects in `--aspects` list to be reported as a SkyFunction exception instead of crashing with assertion error.

Automated rollback of commit 7b4f9826.

*** Reason for rollback ***

Guard the validation against duplicate aspects in `--aspects` list by a flag to avoid breaking builds with duplicate aspects.

*** Original change description ***

Automated rollback of commit 7649f610.

*** Reason for rollback ***

The added validation to prevent duplicate aspects in --aspects list breaks //production/datapush/modular/implementations/build:buildtarget_test

*** Original change description ***

Roll forward of https://github.com/bazelbuild/bazel/commit/943c83aa58731c4f9561d79c458f254427a8f24c: Command line aspect-on-aspect

Supports aspect-on-aspect for command line aspects. Command line aspects specified via `--aspects` option will support a top-level aspect requiring aspect providers via `required_a...

***

PiperOrigin-RevId: 389217989

The flag has been replaced by the "_validation_transitive" special output group

PiperOrigin-RevId: 389174747

PiperOrigin-RevId: 389171146

PiperOrigin-RevId: 389165047

PiperOrigin-RevId: 389161568

Flip apple_static_library to Starlark implementation

Rollforward after failures fixed in unknown commit

RELNOTES:none
PiperOrigin-RevId: 389161430

Minimum versions with trailing 0 and more than 2 components are allowed.
The _lipo.a archive is now added to runfiles.
Repeated archives are not added to the command line

RELNOTES:none
PiperOrigin-RevId: 389157072

PiperOrigin-RevId: 389156141

PiperOrigin-RevId: 389152811

Add a '_validation_transitive' helper output group which overrides transitive validation output group propagation

PiperOrigin-RevId: 389148709

PiperOrigin-RevId: 389065107

We can do better both when include scanning is on, and when it's off.

PiperOrigin-RevId: 389039315

Suggested by nharmata@: use ForkJoinPool for DeletingNodeVisitor and enqueue work in #cpu batches for maximal efficiency. Also clean up some dead code.

Sadly, wall time for benchmarks is basically unchanged, but ~all the contention is gone except for InMemoryNodeEntry#removeReverseDep, which I'm hoping to tackle soon.

PiperOrigin-RevId: 389003979

PiperOrigin-RevId: 388993669

Don't interrupt the command thread on an abrupt exit/crash if we are already on the command thread: the interruption is to stop the command thread from proceeding with non-exit work, but if the command thread is the one noting the abrupt exit/crash, it's already done with non-exit work.

Test coverage that a non-command-thread interrupt does interrupt the main thread continues to be present via BlazeCommandDispatcherTest#crashPreventsNewCommand (times out without command thread interruption) and CommandInterruptionTest#callingExitOnceInterruptsAndOverridesExitCode.

PiperOrigin-RevId: 388963829

PiperOrigin-RevId: 388957142

PiperOrigin-RevId: 388949955

PiperOrigin-RevId: 388936929

RELNOTES: None.
PiperOrigin-RevId: 388908994

Problem

Signal handlers must only call signal-safe code. In particular the
functions it calls must be reentrant or uninterruptible by a signal.

For more information see
https://www.freebsd.org/cgi/man.cgi?query=sigaction&sektion=2&apropos=0&manpath=FreeBSD+13.0
or
https://man7.org/linux/man-pages/man7/signal-safety.7.html

fprintf is not such a function but is called inside of ForwardSignal
(via PRINT_DEBUG)

Solution

Remove the call.

Closes #13069.

PiperOrigin-RevId: 388908927

*** Reason for rollback ***

Causes failures in nightly
*** Original change description ***

Flip apple_static_library to Starlark implementation

RELNOTES:none
PiperOrigin-RevId: 388902965

PiperOrigin-RevId: 388899171

This is preparation for Bazel Starlark java_library.

PiperOrigin-RevId: 388897238

Extract the computation of the persistent worker key into a helper
class. This makes it easier to test and reuse (e.g., for remote
persistent workers).

Change-Id: Ia9aa073423281b67c16b4a351b13a812d9849e1c

Closes #13772.

Change-Id: Ib76e81612b7746a234dc0f04fe018c4d9bd493b7
PiperOrigin-RevId: 388894908

PiperOrigin-RevId: 388878860

Remote: Cleanup the code that determines whether the spawn should accept/upload results from/to remote cache.

Remote cache (and execution) for an action can be controlled with following tags:
  - `no-remote-cache`: prevent remote caching, but allow local caching
  - `no-remote-exec`: prevent remote execution, but allow remote caching
  - `no-remote`: combine no-remote-cache and no-remote-exec
  - `no-cache`: extend no-remote-cache to also prevent local caching

A combined cache is treated as remote cache, hence is disabled for actions that are tagged with `no-remote-cache` unless `--incompatible_remote_results_ignore_disk` is set in which case local component is enabled.

Fixes #13621.

Closes #13769.

PiperOrigin-RevId: 388875505