* [QT-637] Streamline our build pipeline (#24892)
Context
-------
Building and testing Vault artifacts on pull requests and merges is
responsible for about 1/3rd of our overall spend on Vault CI. Of the
artifacts that we ship as part of a release, we do Enos testing scenarios
on the `linux/amd64` and `linux/arm64` binaries and their derivative
artifacts. The extended build artifacts for non-Linux platforms or less
common machine architectures are not tested at this time. They are built,
notarized, and signed as part of every pull request update and merge. As
we don't actually test these artifacts, the only gain we get from this
rather expensive behavior is that we wont merge a change that would prevent
Vault from building on one of the extended targets. Extended platform or
architecture changes are quite rare, so performing this work as frequently
as we do is costly in both monetary and developer time for little relative
safety benefit.
Goals
-----
Rethink and implement how and when we build binaries and artifacts of Vault
so that we can spend less money on repetitive work and while also reducing
the time it takes for the build and test pipelines to complete.
Solution
--------
Instead of building all release artifacts on every push, we'll opt to build
only our testable (core) artifacts. With this change we are introducing a
bit of risk. We could merge a change that breaks an extended platform and
only find out after the fact when we trigger a complete build for a release.
We'll hedge against that risk by building all of the release targets on a
scheduled cadence to ensure that they are still buildable.
We'll make building all of the targets optional on any pull request by
use of a `build/all` label on the pull request.
Further considerations
----------------------
* We want to reduce the total number of workflows and runners for all of our
pipelines if possible. As each workflow runner has infrastructure cost and
runner time penalties, using a single runner over many is often preferred.
* Many of our jobs runners have been optimized for cost and performance. We
should simplify the choices of which runners to use.
* CRT requires us to use the same build workflow in both CE and Ent.
Historically that meant that modifying `build.yml` in CE would result in a
merge conflict with `build.yml` in Ent, and break our merge workflows.
* Workflow flow control in both `build.yml` and `ci.yml` can be quite
complicated, as each needs to maintain compatibility whether executed as CE
or Ent, and when triggered with various Github events like pull_request,
push, and workflow_call, each with their own requirements.
* Many jobs utilize similar patterns of flow control and metadata but are not
reusable.
* Workflow call depth has a maximum of four, so we need to be quite
considerate when calling other workflows.
* Called workflows can only have 10 inputs.
Implementation
--------------
* Refactor the `build.yml` workflow to be agnostic to whether or not it is
executing in CE or Ent. That makes future updates to the build much easier
as we won't have to worry about merge conflicts when the change is merged
downstream.
* Extract common steps in workflows into composite actions that we can reuse.
* Fix bugs where some but not all workflows would use different Git
references when building and testing a pull request.
* We rewrite the application, docs, and UI change helpers as a composite
action. This allows us to re-use this logic to make consistent behavior
choices across build and CI.
* We combine several `build.yml` and `ci.yml` jobs into our final job.
This reduces the number of workflows required for the same behavior while
saving time overall.
* Update most of our action pins.
Results
-------
| Metric | Before | After | Diff |
|-------------------|----------|---------|-------|
| Duration: | ~14-18m | ~15-18m | ~ = |
| Workflows: | 43 | 18 | - 58% |
| Billable time: | ~1h15m | 16m | - 79% |
| Saved artifacts: | 34 | 12 | - 65% |
Infra costs should map closely to billable time.
Network I/O costs should map closely to the workflow count.
Storage costs should map directly with saved artifacts.
We could probably get parity with duration by getting more clever with
our UBI container build, as that's where we're seeing the increase. I'm
not yet concerned as it takes roughly the same time for this job to
complete as it did before.
While the CI workflow was not the focus on the PR, some shared
refactoring does show some marginal improvements there.
| Metric | Before | After | Diff |
|-------------------|----------|----------|--------|
| Duration: | ~24m | ~12.75m | - 15% |
| Workflows: | 55 | 47 | - 8% |
| Billable time: | ~4h20m | ~3h36m | - 7% |
Further focus on streamlining the CI workflows would likely result in a
few more marginal improvements, but nothing on the order like we've seen
with the build workflow.
[0] https://github.com/hashicorp/vault-enterprise/actions/runs/7875954928/job/21490054433?pr=5411#step:3:39
Signed-off-by: Ryan Cragun <me@ryan.ec>
* Implement user lockout log (#23140)
* implement user lockout logger
* formatting
* make user lockout log interval configurable
* create func to get locked user count, and fix potential deadlock
* fix test
* fix test
* add changelog
* fix panic when unlocking unlocked user (#23611)
[VAULT-14497] Ensure Role Governing Policies are only applied down the namespace hierarchy (#23090)
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
* add RequestResponseCallback to core/options
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* pass in router and apply function on requests
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* add callback
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* cleanup
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* Update vault/core.go
* bad typo...
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* use pvt interface, can't downcast to child struct
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* finer grained errors
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* trim path for backend
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* remove entire mount point instead of just the first part of url
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* Update vault/testing.go
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* add doc string
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* update docstring
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* reformat
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* added changelog
---------
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
This checks the request against the `read` permission for
`sys/events/subscribe/{eventType}` on the initial subscribe.
Future work includes moving this to its own verb (`subscribe`)
and periodically rechecking the request.
Tested locally by minting a token with the wrong permissions
and verifying that they are rejected as expected, and that
they work if the policy is adjusted to `sys/event/subscribe/*`
(or the specific topic name) with `read` permissions.
I had to change the `core.checkToken()` to be publicly accessible,
as it seems like the easiest way to check the token on the
`logical.Request` against all relevant policies, but without
going into all of the complex logic further in `handleLogical()`.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* named MFA method configurations
* fix a test
* CL
* fix an issue with same config name different ID and add a test
* feedback
* feedback on test
* consistent use of passcode for all MFA methods (#18611)
* make use of passcode factor consistent for all MFA types
* improved type for MFA factors
* add method name to login CLI
* minor refactoring
* only accept MFA method name with its namespace path in the login request MFA header
* fix a bug
* fixing an ErrorOrNil return value
* more informative error message
* Apply suggestions from code review
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* feedback
* test refactor a bit
* adding godoc for a test
* feedback
* remove sanitize method name
* guard a possbile nil ref
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* go get link proto @vault-11829-meta-get-cluster-status
* add HA status
* add HAEnabled method
* add raft config
* allocate HA nodes based on actual count
* add raft autopilot status
* add raft quorum warnings
* add ClusterID method
* add StorageType
* add ClusterID
* update github.com/hashicorp/vault/vault/hcp_link/proto
* add changelog entry
* fix raft config panic
* remove "Warning" quorum message prefix
* add error wrapping
* add Core.HAStateWithLock method
* reduce quorum warnings to single string
* fix HCP_API_HOST test env var check
* Revert "fix HCP_API_HOST test env var check"
This reverts commit 97c73c4798b77b84aea84f341f2c63c4d657914d.
* adding oss file changes
* check disabled and read values from config
* isUserLocked, getUserLockout Configurations, check user lock before login and return error
* remove stale entry from storage during read
* added failed login process workflow
* success workflow updated
* user lockouts external tests
* changing update to support delete
* provide access to alias look ahead function
* adding path alias lookahead
* adding tests
* added changelog
* added comments
* adding changes from ent branch
* adding lock to UpdateUserFailedLoginInfo
* fix return default bug
strings.ReplaceAll(s, old, new) is a wrapper function for
strings.Replace(s, old, new, -1). But strings.ReplaceAll is more
readable and removes the hardcoded -1.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* VAULT-6613 add DetermineRoleFromLoginRequest function to Core
* Fix body handling
* Role resolution for rate limit quotas
* VAULT-6613 update precedence test
* Add changelog
* VAULT-6614 start of changes for roles in LCQs
* Expiration changes for leases
* Add role information to RequestAuth
* VAULT-6614 Test updates
* VAULT-6614 Add expiration test with roles
* VAULT-6614 fix comment
* VAULT-6614 Protobuf on OSS
* VAULT-6614 Add rlock to determine role code
* VAULT-6614 Try lock instead of rlock
* VAULT-6614 back to rlock while I think about this more
* VAULT-6614 Additional safety for nil dereference
* VAULT-6614 Use %q over %s
* VAULT-6614 Add overloading to plugin backends
* VAULT-6614 RLocks instead
* VAULT-6614 Fix return for backend factory
