krd/vault-redux
1
0
Fork 0
Code
fc9dfa2a7d
vault-redux/Makefile
Ryan Cragun fc9dfa2a7d
[QT-358] Unify CRT and local builder workflows (#17766) 
Here we make the following major changes:

* Centralize CRT builder logic into a script utility so that we can share the
  logic for building artifacts in CI or locally.
* Simplify the build workflow by calling a reusable workflow many times
  instead of repeating the contents.
* Create a workflow that validates whether or not the build workflow and all
  child workflows have succeeded to allow for merge protection.

Motivation

* We need branch requirements for the build workflow and all subsequent
  integration tests (QT-353)
* We need to ensure that the Enos local builder works (QT-558)
* Debugging build failures can be difficult because one has to hand craft the
  steps to recreate the build
* Merge conflicts between Vault OSS and Vault ENT build workflows are quite
  painful. As the build workflow must be the same file and name we'll reduce
  what is contained in each that is unique. Implementations of building
  will be unique per edition so we don't have to worry about conflict
  resolution.
* Since we're going to be touching the build workflow to do the first two
  items we might as well try and improve those other issues at the same time
  to reduce the overhead of backports and conflicts.

Considerations

* Build logic for Vault OSS and Vault ENT differs
* The Enos local builder was duplicating a lot of what we did in the CRT build
  workflow
* Version and other artifact metadata has been an issue before. Debugging it
  has been tedious and error prone.
* The build workflow is full of brittle copy and paste that is hard to
  understand, especially for all of the release editions in Vault Enterprise
* Branch check requirements for workflows are incredibly painful to use for
  workflows that are dynamic or change often. The required workflows have to be
  configured in Github settings by administrators. They would also prevent us
  from having simple docs PRs since required integration workflows always have
  to run to satisfy branch requirements.
* Doormat credentials requirements that are coming will require us to modify
  which event types trigger workflows. This changes those ahead of time since
  we're doing so much to build workflow. The only noticeable impact will be
  that the build workflow no longer runs on pushes to non-main or release
  branches. In order to test other branches it requires a workflow_dispatch
  from the Actions tab or a pull request.

Solutions

* Centralize the logic that determines build metadata and creates releasable
  Vault artifacts. Instead of cargo-culting logic multiple times in the build
  workflow and the Enos local modules, we now have a crt-builder script which
  determines build metadata and also handles building the UI, Vault, and the
  package bundle. There are make targets for all of the available sub-commands.
  Now what we use in the pipeline is the same thing as the local builder, and
  it can be executed locally by developers. The crt-builder script works in OSS
  and Enterprise so we will never have to deal with them being divergent or with
  special casing things in the build workflow.
* Refactor the bulk of the Vault building into a reusable workflow that we can
  call multiple times. This allows us to define Vault builds in a much simpler
  manner and makes resolving merge conflicts much easier.
* Rather than trying to maintain a list and manually configure the branch check
  requirements for build, we'll trigger a single workflow that uses the github
  event system to determine if the build workflow (all of the sub-workflows
  included) have passed. We'll then create branch restrictions on that single
  workflow down the line.

Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 13:14:43 -07:00

302 lines
12 KiB
Makefile
Raw Blame History

# Determine this makefile's path.
# Be sure to place this BEFORE `include` directives, if any.
THIS_FILE := $(lastword $(MAKEFILE_LIST))
TEST?=$$($(GO_CMD) list ./... | grep -v /vendor/ | grep -v /integ)
TEST_TIMEOUT?=45m
EXTENDED_TEST_TIMEOUT=60m
INTEG_TEST_TIMEOUT=120m
VETARGS?=-asmdecl -atomic -bool -buildtags -copylocks -methods -nilfunc -printf -rangeloops -shift -structtags -unsafeptr
EXTERNAL_TOOLS_CI=\
golang.org/x/tools/cmd/goimports
EXTERNAL_TOOLS=\
github.com/client9/misspell/cmd/misspell
GOFMT_FILES?=$$(find . -name '*.go' | grep -v pb.go | grep -v vendor)
SED?=$(shell command -v gsed || command -v sed)
GO_VERSION_MIN=$$(cat $(CURDIR)/.go-version)
PROTOC_VERSION_MIN=3.21.7
GO_CMD?=go
CGO_ENABLED?=0
ifneq ($(FDB_ENABLED), )
CGO_ENABLED=1
BUILD_TAGS+=foundationdb
endif
default: dev
# bin generates the releasable binaries for Vault
bin: prep
@CGO_ENABLED=$(CGO_ENABLED) BUILD_TAGS='$(BUILD_TAGS) ui' sh -c "'$(CURDIR)/scripts/build.sh'"
# dev creates binaries for testing Vault locally. These are put
# into ./bin/ as well as $GOPATH/bin
dev: prep
@CGO_ENABLED=$(CGO_ENABLED) BUILD_TAGS='$(BUILD_TAGS)' VAULT_DEV_BUILD=1 sh -c "'$(CURDIR)/scripts/build.sh'"
dev-ui: assetcheck prep
@CGO_ENABLED=$(CGO_ENABLED) BUILD_TAGS='$(BUILD_TAGS) ui' VAULT_DEV_BUILD=1 sh -c "'$(CURDIR)/scripts/build.sh'"
dev-dynamic: prep
@CGO_ENABLED=1 BUILD_TAGS='$(BUILD_TAGS)' VAULT_DEV_BUILD=1 sh -c "'$(CURDIR)/scripts/build.sh'"
# *-mem variants will enable memory profiling which will write snapshots of heap usage
# to $TMP/vaultprof every 5 minutes. These can be analyzed using `$ go tool pprof <profile_file>`.
# Note that any build can have profiling added via: `$ BUILD_TAGS=memprofiler make ...`
dev-mem: BUILD_TAGS+=memprofiler
dev-mem: dev
dev-ui-mem: BUILD_TAGS+=memprofiler
dev-ui-mem: assetcheck dev-ui
dev-dynamic-mem: BUILD_TAGS+=memprofiler
dev-dynamic-mem: dev-dynamic
# Creates a Docker image by adding the compiled linux/amd64 binary found in ./bin.
# The resulting image is tagged "vault:dev".
docker-dev: prep
docker build --build-arg VERSION=$(GO_VERSION_MIN) --build-arg BUILD_TAGS="$(BUILD_TAGS)" -f scripts/docker/Dockerfile -t vault:dev .
docker-dev-ui: prep
docker build --build-arg VERSION=$(GO_VERSION_MIN) --build-arg BUILD_TAGS="$(BUILD_TAGS)" -f scripts/docker/Dockerfile.ui -t vault:dev-ui .
# test runs the unit tests and vets the code
test: prep
@CGO_ENABLED=$(CGO_ENABLED) \
VAULT_ADDR= \
VAULT_TOKEN= \
VAULT_DEV_ROOT_TOKEN_ID= \
VAULT_ACC= \
$(GO_CMD) test -tags='$(BUILD_TAGS)' $(TEST) $(TESTARGS) -timeout=$(TEST_TIMEOUT) -parallel=20
testcompile: prep
@for pkg in $(TEST) ; do \
$(GO_CMD) test -v -c -tags='$(BUILD_TAGS)' $$pkg -parallel=4 ; \
done
# testacc runs acceptance tests
testacc: prep
@if [ "$(TEST)" = "./..." ]; then \
echo "ERROR: Set TEST to a specific package"; \
exit 1; \
fi
VAULT_ACC=1 $(GO_CMD) test -tags='$(BUILD_TAGS)' $(TEST) -v $(TESTARGS) -timeout=$(EXTENDED_TEST_TIMEOUT)
# testrace runs the race checker
testrace: prep
@CGO_ENABLED=1 \
VAULT_ADDR= \
VAULT_TOKEN= \
VAULT_DEV_ROOT_TOKEN_ID= \
VAULT_ACC= \
$(GO_CMD) test -tags='$(BUILD_TAGS)' -race $(TEST) $(TESTARGS) -timeout=$(EXTENDED_TEST_TIMEOUT) -parallel=20
cover:
./scripts/coverage.sh --html
# vet runs the Go source code static analysis tool `vet` to find
# any common errors.
vet:
@$(GO_CMD) list -f '{{.Dir}}' ./... | grep -v /vendor/ \
| grep -v '.*github.com/hashicorp/vault$$' \
| xargs $(GO_CMD) vet ; if [ $$? -eq 1 ]; then \
echo ""; \
echo "Vet found suspicious constructs. Please check the reported constructs"; \
echo "and fix them if necessary before submitting the code for reviewal."; \
fi
# lint runs vet plus a number of other checkers, it is more comprehensive, but louder
lint:
@$(GO_CMD) list -f '{{.Dir}}' ./... | grep -v /vendor/ \
| xargs golangci-lint run; if [ $$? -eq 1 ]; then \
echo ""; \
echo "Lint found suspicious constructs. Please check the reported constructs"; \
echo "and fix them if necessary before submitting the code for reviewal."; \
fi
# for ci jobs, runs lint against the changed packages in the commit
ci-lint:
@golangci-lint run --deadline 10m --new-from-rev=HEAD~
# prep runs `go generate` to build the dynamically generated
# source files.
prep: fmtcheck
@sh -c "'$(CURDIR)/scripts/goversioncheck.sh' '$(GO_VERSION_MIN)'"
@$(GO_CMD) generate $($(GO_CMD) list ./... | grep -v /vendor/)
@if [ -d .git/hooks ]; then cp .hooks/* .git/hooks/; fi
# bootstrap the build by downloading additional tools needed to build
ci-bootstrap:
@for tool in $(EXTERNAL_TOOLS_CI) ; do \
echo "Installing/Updating $$tool" ; \
GO111MODULE=off $(GO_CMD) get -u $$tool; \
done
# bootstrap the build by downloading additional tools that may be used by devs
bootstrap: ci-bootstrap
go generate -tags tools tools/tools.go
# Note: if you have plugins in GOPATH you can update all of them via something like:
# for i in $(ls | grep vault-plugin-); do cd $i; git remote update; git reset --hard origin/master; dep ensure -update; git add .; git commit; git push; cd ..; done
update-plugins:
grep vault-plugin- go.mod | cut -d ' ' -f 1 | while read -r P; do echo "Updating $P..."; go get -v "$P"; done
static-assets-dir:
@mkdir -p ./http/web_ui
install-ui-dependencies:
@echo "--> Installing JavaScript assets"
@cd ui && yarn --ignore-optional
test-ember: install-ui-dependencies
@echo "--> Running ember tests"
@cd ui && yarn run test:oss
test-ember-enos: install-ui-dependencies
@echo "--> Running ember tests with a real backend"
@cd ui && yarn run test:enos
check-vault-in-path:
@VAULT_BIN=$$(command -v vault) || { echo "vault command not found"; exit 1; }; \
[ -x "$$VAULT_BIN" ] || { echo "$$VAULT_BIN not executable"; exit 1; }; \
printf "Using Vault at %s:\n\$$ vault version\n%s\n" "$$VAULT_BIN" "$$(vault version)"
ember-dist: install-ui-dependencies
@cd ui && npm rebuild node-sass
@echo "--> Building Ember application"
@cd ui && yarn run build
@rm -rf ui/if-you-need-to-delete-this-open-an-issue-async-disk-cache
ember-dist-dev: install-ui-dependencies
@cd ui && npm rebuild node-sass
@echo "--> Building Ember application"
@cd ui && yarn run build:dev
static-dist: ember-dist
static-dist-dev: ember-dist-dev
proto: bootstrap
@sh -c "'$(CURDIR)/scripts/protocversioncheck.sh' '$(PROTOC_VERSION_MIN)'"
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative vault/*.proto
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative vault/activity/activity_log.proto
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative helper/storagepacker/types.proto
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative helper/forwarding/types.proto
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative sdk/logical/*.proto
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative physical/raft/types.proto
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative helper/identity/mfa/types.proto
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative helper/identity/types.proto
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative sdk/database/dbplugin/*.proto
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative sdk/database/dbplugin/v5/proto/*.proto
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative sdk/plugin/pb/*.proto
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative vault/tokens/token.proto
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative sdk/helper/pluginutil/*.proto
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative vault/hcp_link/proto/node_status/*.proto
# No additional sed expressions should be added to this list. Going forward
# we should just use the variable names choosen by protobuf. These are left
# here for backwards compatability, namely for SDK compilation.
$(SED) -i -e 's/Id/ID/' vault/request_forwarding_service.pb.go
$(SED) -i -e 's/Idp/IDP/' -e 's/Url/URL/' -e 's/Id/ID/' -e 's/IDentity/Identity/' -e 's/EntityId/EntityID/' -e 's/Api/API/' -e 's/Qr/QR/' -e 's/Totp/TOTP/' -e 's/Mfa/MFA/' -e 's/Pingid/PingID/' -e 's/namespaceId/namespaceID/' -e 's/Ttl/TTL/' -e 's/BoundCidrs/BoundCIDRs/' helper/identity/types.pb.go helper/identity/mfa/types.pb.go helper/storagepacker/types.pb.go sdk/plugin/pb/backend.pb.go sdk/logical/identity.pb.go vault/activity/activity_log.pb.go
# This will inject the sentinel struct tags as decorated in the proto files.
protoc-go-inject-tag -input=./helper/identity/types.pb.go
protoc-go-inject-tag -input=./helper/identity/mfa/types.pb.go
fmtcheck:
@true
#@sh -c "'$(CURDIR)/scripts/gofmtcheck.sh'"
fmt:
find . -name '*.go' | grep -v pb.go | grep -v vendor | xargs gofumpt -w
semgrep:
semgrep --include '*.go' --exclude 'vendor' -a -f tools/semgrep .
semgrep-ci:
semgrep --error --include '*.go' --exclude 'vendor' -f tools/semgrep/ci .
assetcheck:
@echo "==> Checking compiled UI assets..."
@sh -c "'$(CURDIR)/scripts/assetcheck.sh'"
spellcheck:
@echo "==> Spell checking website..."
@misspell -error -source=text website/source
mysql-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/mysql-database-plugin ./plugins/database/mysql/mysql-database-plugin
mysql-legacy-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/mysql-legacy-database-plugin ./plugins/database/mysql/mysql-legacy-database-plugin
cassandra-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/cassandra-database-plugin ./plugins/database/cassandra/cassandra-database-plugin
influxdb-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/influxdb-database-plugin ./plugins/database/influxdb/influxdb-database-plugin
postgresql-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/postgresql-database-plugin ./plugins/database/postgresql/postgresql-database-plugin
mssql-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/mssql-database-plugin ./plugins/database/mssql/mssql-database-plugin
hana-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/hana-database-plugin ./plugins/database/hana/hana-database-plugin
mongodb-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/mongodb-database-plugin ./plugins/database/mongodb/mongodb-database-plugin
.PHONY: ci-config
ci-config:
@$(MAKE) -C .circleci ci-config
.PHONY: ci-verify
ci-verify:
@$(MAKE) -C .circleci ci-verify
.PHONY: bin default prep test vet bootstrap ci-bootstrap fmt fmtcheck mysql-database-plugin mysql-legacy-database-plugin cassandra-database-plugin influxdb-database-plugin postgresql-database-plugin mssql-database-plugin hana-database-plugin mongodb-database-plugin ember-dist ember-dist-dev static-dist static-dist-dev assetcheck check-vault-in-path packages build build-ci semgrep semgrep-ci
.NOTPARALLEL: ember-dist ember-dist-dev
# These crt targets are used for release builds by .github/workflows/build.yml
# and for artifact_source:local Enos scenario variants.
.PHONY: crt-build
crt-build:
@$(CURDIR)/scripts/crt-builder.sh build
.PHONY: crt-build-ui
crt-build-ui:
@$(CURDIR)/scripts/crt-builder.sh build-ui
.PHONY: crt-bundle
crt-bundle:
@$(CURDIR)/scripts/crt-builder.sh bundle
.PHONY: crt-get-artifact-basename
crt-get-artifact-basename:
@$(CURDIR)/scripts/crt-builder.sh artifact-basename
.PHONY: crt-get-date
crt-get-date:
@$(CURDIR)/scripts/crt-builder.sh date
.PHONY: crt-get-revision
crt-get-revision:
@$(CURDIR)/scripts/crt-builder.sh revision
.PHONY: crt-get-version
crt-get-version:
@$(CURDIR)/scripts/crt-builder.sh version
.PHONY: crt-get-version-base
crt-get-version-base:
@$(CURDIR)/scripts/crt-builder.sh version-base
.PHONY: crt-get-version-pre
crt-get-version-pre:
@$(CURDIR)/scripts/crt-builder.sh version-pre
.PHONY: crt-get-version-meta
crt-get-version-meta:
@$(CURDIR)/scripts/crt-builder.sh version-meta
.PHONY: crt-prepare-legal
crt-prepare-legal:
@$(CURDIR)/scripts/crt-builder.sh prepare-legal
View Git Blame Copy Permalink