Baseline: e7bd0947

Cherry picks:

   + 04bed291:
     Disable shadowed_action part of action_aspect_test
   + 367b19eb:
     Fix starlark action with shadowed_action inputs cache test
   + 4acb9ae6:
     Automated rollback of commit
     03f6af62.

Incompatible changes:

  - Native libraries in data attribute are not collected. See
    https://github.com/bazelbuild/bazel/issues/13550 for details

Important changes:

  - Remove obsolete --incompatible_prohibit_aapt1
  - The minimum Android build tools version for the Android rules is
    now 30.0.0

This release contains contributions from many people at Google, as well as Cameron Mulhern, Delwin9999, Keith Smiley.

*** Reason for rollback ***

Potentially causes a performance regression: []

*** Original change description ***

Define a natural ordering for `ActionInput`.

`Artifact` defines natural ordering by comparing exec paths, which are defined at the level of `ActionInput`. Move that to `ActionInput` interface. Move `Artifact.EXEC_PATH_COMPARATOR` to `ActionInput` to allow convenient comparisons with null.

PiperOrigin-RevId: 380019680

The test was flaky mainly due to using `&` instead of `&&` to execute two consecutive commands in the aspect action.

This CL contains other improvements:
 - using timestamp in the test files instead of date-time to be more accurate.
 - splitting the test method into two tests to clarify the purpose of each of them.

PiperOrigin-RevId: 379793635

PiperOrigin-RevId: 379606378

PiperOrigin-RevId: 379325219

PiperOrigin-RevId: 379325097

This supposedly fixes the compiler error reported by the user:

`error: 'numeric_limits' is not a member of 'std'`

Thanks to samcom12 for the report and suggested fix.

RELNOTES: None.
PiperOrigin-RevId: 379291338

Remove unused methods from `MutableActionGraph` and use a simple `ConcurrentHashMap` instead of `ConcurrentMultimapWithHeadElement`.

Access to the secondary elements of the map is never required, so a multimap is unnecessary.

PiperOrigin-RevId: 379269613

PiperOrigin-RevId: 379267434

PiperOrigin-RevId: 379264356

*** Reason for rollback ***

Causes a significant performance regression for objc builds.

PiperOrigin-RevId: 379262567

Before we estimated all CppComppile actions with 250MB and 1 CPU. During investigation we find out the linear dependency between number of inputs and memory.
Using our data we made linear estimation of form C + K * inputs such that 95% of actions used less memory than estimated.

In case of memory overestimate  we will make our builds slower, because of large amount unused memory, which we could use for execution other actions.

In case of memory underestimate we could overload system and get OOMs during builds.

PiperOrigin-RevId: 379259864

PiperOrigin-RevId: 379238541

PiperOrigin-RevId: 379231315

We do not support relative paths for `sandbox_tmpfs_path` and currently Bazel
will crash when we pass one. Add an explicit check against relative paths
passed to the flag so that we generate a proper user-facing error.

PiperOrigin-RevId: 378979048

`Artifact` defines natural ordering by comparing exec paths, which are defined at the level of `ActionInput`. Move that to `ActionInput` interface. Move `Artifact.EXEC_PATH_COMPARATOR` to `ActionInput` to allow convenient comparisons with null.

PiperOrigin-RevId: 378956898

PiperOrigin-RevId: 378932578

`ActionInputHelper::fromPath` allows to create an action input from an exec path or a string representing it. In case of `PathFragment`, we will get the path string of it and recreate a new `PathFragment` in `getExecPath`. Store the `PathFragment` instead. Change callers which already have a `PathFragment` to use the override accepting it.

PiperOrigin-RevId: 378907363

RELNOTES: None
PiperOrigin-RevId: 378884478

PiperOrigin-RevId: 378868762

PiperOrigin-RevId: 378865320

This will start using the Starlark version of the rule's by default.

TGP is clean but if problems become apparent after a release, in order to avoid
a rollback, you can add --experimental_builtins_injection_override=-objc_library,-objc_import
to the blazerc.

TESTED=[]

RELNOTES:none
PiperOrigin-RevId: 378859899

Baseline: d571c897

Cherry picks:

   + c367a7e0:
     Fix places where nodes were still being enqueued at
     `Integer.MAX_VALUE` and add checks for overflow conditions.
   + 260015df:
     Automated rollback of commit
     f1b37b5d.

Important changes:

  - Add `disable_annotation_processing` option to
    `java_common.compile`, which disables any annotation processors
    passed to `plugins` or in `exported_plugins` of `deps`

This release contains contributions from many people at Google, as well as Dave MacLachlan, FaBrand, George Gensure, Jesse Chan, Keith Smiley, Kevin Hogeland, Uri Baghin, Xavier Bonaventura.

*** Reason for rollback ***

Partial roll-forward that includes jdeps files in Java*ProtoAspect but doesn't propagate JavaOutputs through java_*_proto_library rules to avoid changes resulting from doing the latter.

PiperOrigin-RevId: 378819435

RELNOTES: None
PiperOrigin-RevId: 378812390

Add a Caching state to ActionState. This state indicates the action is checking the cache and should be happened before Scheduling state.

Change ProgressStatus from enum to interface so that we can pass more data to the event handler (which is required to report upload/download details later).

Fixes https://github.com/bazelbuild/bazel/issues/13531.

Closes #13555.

PiperOrigin-RevId: 378800212

To support IDE use-cases, https://github.com/bazelbuild/bazel/commit/06c956623fdb81465f884c170df48bf38b59cadb changed target completion events to keep
a reference to the ActionInputMap in order to keep around metadata specifying if
an output file is a directory or not. This worked but caused an unexpected
increase in heap high-watermark, causing OOMs.

The unexpected heap usage was traced to large _validation output groups holding
tens of thousands of artifacts per target. This output group is not included in
BEP so the original change was in fact retaining far more memory than necessary
to implement the intended optimization.

With this change, a second ActionInputMap is prepared which includes only the
"important" artifacts and is sized appropriately. Important artifacts are those
which are not produced by internal output groups and are exactly the artifacts
whose URIs are written to BEP. Maintaining this second ActionInputMap costs
additional latency in CompletionFunction proportional to the number of
important artifacts, though care is taken to avoid doubling the runtime of
CompletionFunction in the case where all artifacts are important.

PiperOrigin-RevId: 378793974

See commit `b2dd6fb2` for context.

PiperOrigin-RevId: 378784115

This is necessary with recent JDK due to [JEP 396: Strongly Encapsulate JDK Internals by Default](https://openjdk.java.net/jeps/396)

PiperOrigin-RevId: 378780031

`InvocationPolicyEnforcer` [expands all flags before setting](https://github.com/bazelbuild/bazel/blob/60fc2d18fadec5f99849b78bce761da62dd78c32/src/main/java/com/google/devtools/common/options/InvocationPolicyEnforcer.java#L124-L125) them to later call into the `OptionsParser` to set their values, which [repeats the process for the flags with expansions/implicit requirements](https://github.com/bazelbuild/bazel/blob/60fc2d18fadec5f99849b78bce761da62dd78c32/src/main/java/com/google/devtools/common/options/OptionsParserImpl.java#L549-L556). Refactor `OptionsParser` to expose a method for setting flag value without the expansion so that `InvocationPolicyEnforcer` can skip the second, unnecessary expansion.

PiperOrigin-RevId: 378775962

I don't have domain knowledge on this. I just noticed this has no
references.

PiperOrigin-RevId: 378755726

This should only be used by Android rules during migration and is not for general use.

PiperOrigin-RevId: 378754338

Add remove_all_convenience_symlinks flag for the clean command. When this flag is set to true, the clean command will delete all symlinks in the workspace that have the symlink_prefix prefix and point into the output directory. Without the flag, only the symlinks with predefined suffixes are deleted.

Closes #11980

PiperOrigin-RevId: 378735373

PiperOrigin-RevId: 378731988

PiperOrigin-RevId: 378696513

This was already allowed for attributes.

PiperOrigin-RevId: 378695886

A make variable may contain another make variable.

RELNOTES:none
PiperOrigin-RevId: 378656678

This is a roll forward of https://github.com/bazelbuild/bazel/commit/526ea392ac50a0f11eb65fd29a6cde5962a08c97 , which was rolled back due to increased memory pressure.

The original change created a new .jdeps.stripped file for executor output, then preserved the existing .jdeps file for the final result in Bazel's output tree. Experimentation showed that the very act of adding a new output artifact to all Java compilations can contribute ~0.7% more memory to a build . While this is a small percentage, it's large enough to be problematic for large builds that are already close to OOMing.

This version keeps Bazel restricted to the existing .jdeps file. It distinguishes executor/Bazel content by time: 1) begin Java compilation action, 2) compile on the executor, write .jdeps, 3) locally rewrite .jdeps for Bazel, 4) complete the action.

Tests show this preserves the executor-caching results inspiring this work and saves the memory regression from the original version.

For https://github.com/bazelbuild/bazel/issues/6526.

PiperOrigin-RevId: 378649901

RELNOTES: None.
PiperOrigin-RevId: 378648914

Which when set, Bazel will save outputs whose digest does not match the expected value to the target directories.

Also use OutputDigestMismatchException to indicate the error and include output path in the error message. The message for such an error will become e.g.:

```
com.google.devtools.build.lib.remote.common.OutputDigestMismatchException: Output bazel-out/darwin-fastbuild/bin/output.txt download failed: Expected digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229' does not match received digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229'.
```

Used to support https://github.com/bazelbuild/continuous-integration/issues/1175.

Closes #13568.

PiperOrigin-RevId: 378624823