* port SSCT OSS
* port header hmac key to ent and generate token proto without make command
* remove extra nil check in request handling
* add changelog
* add comment to router.go
* change test var to use length constants
* remove local index is 0 check and extra defer which can be removed after use of ExternalID
* feat: DB plugin multiplexing (#13734)
* WIP: start from main and get a plugin runner from core
* move MultiplexedClient map to plugin catalog
- call sys.NewPluginClient from PluginFactory
- updates to getPluginClient
- thread through isMetadataMode
* use go-plugin ClientProtocol interface
- call sys.NewPluginClient from dbplugin.NewPluginClient
* move PluginSets to dbplugin package
- export dbplugin HandshakeConfig
- small refactor of PluginCatalog.getPluginClient
* add removeMultiplexedClient; clean up on Close()
- call client.Kill from plugin catalog
- set rpcClient when muxed client exists
* add ID to dbplugin.DatabasePluginClient struct
* only create one plugin process per plugin type
* update NewPluginClient to return connection ID to sdk
- wrap grpc.ClientConn so we can inject the ID into context
- get ID from context on grpc server
* add v6 multiplexing protocol version
* WIP: backwards compat for db plugins
* Ensure locking on plugin catalog access
- Create public GetPluginClient method for plugin catalog
- rename postgres db plugin
* use the New constructor for db plugins
* grpc server: use write lock for Close and rlock for CRUD
* cleanup MultiplexedClients on Close
* remove TODO
* fix multiplexing regression with grpc server connection
* cleanup grpc server instances on close
* embed ClientProtocol in Multiplexer interface
* use PluginClientConfig arg to make NewPluginClient plugin type agnostic
* create a new plugin process for non-muxed plugins
* feat: plugin multiplexing: handle plugin client cleanup (#13896)
* use closure for plugin client cleanup
* log and return errors; add comments
* move rpcClient wrapping to core for ID injection
* refactor core plugin client and sdk
* remove unused ID method
* refactor and only wrap clientConn on multiplexed plugins
* rename structs and do not export types
* Slight refactor of system view interface
* Revert "Slight refactor of system view interface"
This reverts commit 73d420e5cd2f0415e000c5a9284ea72a58016dd6.
* Revert "Revert "Slight refactor of system view interface""
This reverts commit f75527008a1db06d04a23e04c3059674be8adb5f.
* only provide pluginRunner arg to the internal newPluginClient method
* embed ClientProtocol in pluginClient and name logger
* Add back MLock support
* remove enableMlock arg from setupPluginCatalog
* rename plugin util interface to PluginClient
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
* feature: multiplexing: fix unit tests (#14007)
* fix grpc_server tests and add coverage
* update run_config tests
* add happy path test case for grpc_server ID from context
* update test helpers
* feat: multiplexing: handle v5 plugin compiled with new sdk
* add mux supported flag and increase test coverage
* set multiplexingSupport field in plugin server
* remove multiplexingSupport field in sdk
* revert postgres to non-multiplexed
* add comments on grpc server fields
* use pointer receiver on grpc server methods
* add changelog
* use pointer for grpcserver instance
* Use a gRPC server to determine if a plugin should be multiplexed
* Apply suggestions from code review
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* add lock to removePluginClient
* add multiplexingSupport field to externalPlugin struct
* do not send nil to grpc MultiplexingSupport
* check err before logging
* handle locking scenario for cleanupFunc
* allow ServeConfigMultiplex to dispense v5 plugin
* reposition structs, add err check and comments
* add comment on locking for cleanupExternalPlugin
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* adding CRT to main branch
* cleanup
* um i dont know how that got removed but heres the fix
* add vault.service
Co-authored-by: Kyle Penfound <kpenfound11@gmail.com>
* copy over the webui
move web_ui to http
remove web ui files, add .gitkeep
updates, messing with gitkeep and ignoring web_ui
update ui scripts
gitkeep
ignore http/web_ui
Remove debugging
remove the jwt reference, that was from something else
restore old jwt plugin
move things around
Revert "move things around"
This reverts commit 2a35121850f5b6b82064ecf78ebee5246601c04f.
Update ui path handling to not need the web_ui name part
add desc
move the http.FS conversion internal to assetFS
update gitignore
remove bindata dep
clean up some comments
remove asset check script that's no longer needed
Update readme
remove more bindata things
restore asset check
update packagespec
update stub
stub the assetFS method and set uiBuiltIn to false for non-ui builds
update packagespec to build ui
* fail if assets aren't found
* tidy up vendor
* go mod tidy
* updating .circleci
* restore tools.go
* re-re-re-run make packages
* re-enable arm64
* Adding change log
* Removing a file
Co-authored-by: hamid ghaf <hamid@hashicorp.com>
Adds BUILD_TAGS to the docker build commands for docker-dev and
docker-dev-ui. Also changes the respective Dockerfile's to use double
quotes with ${BUILD_TAGS} so that it's interpolated.
* Update go version to 1.15.3
* Fix OU ordering for go1.15.x testing
* Fix CI version
* Update docker image
* Fix test
* packagespec upgrade -version 0.1.8
Co-authored-by: Sam Salisbury <samsalisbury@gmail.com>
This is part 1 of 4 for renaming the `newdbplugin` package. This copies the existing package to the new location but keeps the current one in place so we can migrate the existing references over more easily.
* Add new Database v5 interface with gRPC client & server
This is primarily for making password policies available to the DB engine, however since there are a number of other problems with the current interface this is getting an overhaul to a more gRPC request/response approach for easier future compatibility.
This is the first in a series of PRs to add support for password policies in the combined database engine
* add packagespec build system
- The majority of changes in this commit are files generated
by packagespec (everything in the packages-oss.lock directory).
* add .yamllint
* update to packagespec@fd54aea4
* ci: bust packagespec cache
- Change to packagespec results in package IDs that can use
git tag refs, not just commit refs.
* update to packagepsec@5fc121d0
- This busts all caches, because of a change to the way we
no longer traverse from tag refs to commit refs, due to
the potential confusion this can cause.
- See fd54aea482
for the original change to packagespec necessitating this.
* update to packagespec@5e6c87b6
- This completes the change to allowing git tag refs
to be used for source IDs, begun in f130b940a8fbe3e9398225b08ea1d63420bef7b6
* update to packagespec@4d3c9e8b
- This busts cache, needed to apply previous change.
* remove RELEASE_BUILD_WORKFLOW_NAME
* update packagespec, add watch-ci target
* fix package names (do not refer to EDITION)
* remove EDITION input from packages-oss.yml
* bump package cache, update packagespec
* update packagespec, add 'aliases' target
* update packagespec; less output noise
* ci: give release bundle file a better name
- When performing a release build, this will include the build ID
as part of the name, making it easier to distinguish from other
builds.
* ci: create aliases tarball artifact
* ci: cache package metadata files
* ci: add metadata artifact
* ci: bust circleci package cache
* Revert "ci: bust circleci package cache"
This reverts commit 1320d182613466f0999d63f5742db12ac17f8e92.
* ci: remove aliases artifact
* ci: use buildID not workflowName to id artifacts
* packages: add BUNDLE_NAME metadata
* do not cache package metadata with binaries
* ci: bump package cache
* ci: debugging
* ci: fix package cache; update packagespec
* ci: update packagespec to 10e7beb2
* ci: write package metadata and aliases
* ci: switch to .zip artifacts
* switch package bundle back to tar.gz (from zip)
- Because of the way zip works, the zip archive was over 2GB rather than under 750MB as with tar.gz.
* bump packagespec, adds list-staged-builds
* update packagespec
* add publish stub + general tidy up
* bump packagespec
* bump packagespec; add make publish-config
* Makefile: tidy up packagespec targets
* pass PRODUCT_REPO_ROOT to packagespec
* bump go to 1.14.6
* packages-oss.yml: use more explicit base image
* bump packagespec to b899b7c1
* bump packagespec to f040ce8f
* packages-oss.yml: pin base image to digest
- This digest is pointed to by debian:buster-20200720
- Using a specific digest ensures that builds use the same
base image in all contexts
* add release-repo; bump packagespec
* remove BUILD_TAGS and -tags flag
* bump packagespec to e444f742
* bump to go1.14.7
* ci: bump to go1.14.7
* UI update headless chrome flags to fix ember/chrome crash in test
* UI update references from test-oss to test:oss
* UI removed disable-gpu flag from headless Chrome
* UI and also remove the comment
* be more specific about node version, and specify a yarn version
* update ember, ember-cli, ember-data, ember-data-model-fragments
* use router handlers to access transition information
* fix shadowing of component helper
* update ivy-codemirror, ember-cli-inject-live-reload
* remove custom router service
* don't use transition.queryParams
* update ember-cli-deprecation-workflow
* refactor kv v1 to use 'path' instead of 'id' on creation
* fix auth-jwt-test and toolbar-link-test
* update ember composable helpers
* remove Ember.copy from test file
* no more deprecations in the workflow
* fix more secret tests
* fix remaining failed tests
* move select component to core because it's used by ttl-picker
* generate new model class for each test instead of reusing an existing one
* fix selectors on kmip tests
* refactor how control groups construct urls from the new transition objects
* add router service override back in, and have it be evented so that we can trigger router events on it
* move stories and markdown files to core if the component lives in core
* update ember-cli, ember-cli-babel, ember-auto-import
* update base64js, date-fns, deepmerge, codemirror, broccoli-asset-rev
* update linting rules
* fix test selectors
* update ember-api-actions, ember-concurrency, ember-load-initializers, escape-string-regexp, normalize.css, prettier-eslint-cli, jsdoc-to-markdown
* remove test-results dir
* update base64js, ember-cli-clipboard, ember-cli-sass, ember-cli-string-helpers, ember-cli-template-lint, ember-cli-uglify, ember-link-action
* fix linting
* run yarn install without restoring from cache
* refactor how tests are run and handle the vault server subprocess
* update makefile for new test task names
* update circle config to use the new yarn task
* fix writing the seal keys when starting the dev server
* remove optional deps from the lockfile
* don't ignore-optional on yarn install
* remove errant console.log
* update ember-basic-dropdown-hover, jsonlint, yargs-parser
* update ember-cli-flash
* add back optionalDeps
* update @babel/core@7.5.5, ember-basic-dropdown@1.1.3, eslint-plugin-ember@6.8.2
* update storybook to the latest release
* add a babel config with targets so that the ember babel plugin works properly
* update ember-resolver, move ember-cli-storybook to devDependencies
* revert normalize.css upgrade
* silence fetchadapter warning for now
* exclude 3rd party array helper now that ember includes one
* fix switch and entity lookup styling
* only add -root suffix if it's not in versions mode
* make sure drop always has an array on the aws role form
* fix labels like we did with the backport
* update eslintignore
* update the yarn version in the docker build file
* update eslint ignore
* ci: update Makefile
- Copied from c5c5149583/copythis.circleci/Makefile
* Makefile: make ci- targets phony and more explicit
* ci: add pre-flight checks
- Right now just validates ci config is up to date.
* ci: add --skip-update-check to circleci
* ci: remove cruft from Makefile
* ci: fix missing source dir error message
* ci: consolidate circleci path handling
* Work on raft backend
* Add logstore locally
* Add encryptor and unsealable interfaces
* Add clustering support to raft
* Remove client and handler
* Bootstrap raft on init
* Cleanup raft logic a bit
* More raft work
* Work on TLS config
* More work on bootstrapping
* Fix build
* More work on bootstrapping
* More bootstrapping work
* fix build
* Remove consul dep
* Fix build
* merged oss/master into raft-storage
* Work on bootstrapping
* Get bootstrapping to work
* Clean up FMS and node-id
* Update local node ID logic
* Cleanup node-id change
* Work on snapshotting
* Raft: Add remove peer API (#906)
* Add remove peer API
* Add some comments
* Fix existing snapshotting (#909)
* Raft get peers API (#912)
* Read raft configuration
* address review feedback
* Use the Leadership Transfer API to step-down the active node (#918)
* Raft join and unseal using Shamir keys (#917)
* Raft join using shamir
* Store AEAD instead of master key
* Split the raft join process to answer the challenge after a successful unseal
* get the follower to standby state
* Make unseal work
* minor changes
* Some input checks
* reuse the shamir seal access instead of new default seal access
* refactor joinRaftSendAnswer function
* Synchronously send answer in auto-unseal case
* Address review feedback
* Raft snapshots (#910)
* Fix existing snapshotting
* implement the noop snapshotting
* Add comments and switch log libraries
* add some snapshot tests
* add snapshot test file
* add TODO
* More work on raft snapshotting
* progress on the ConfigStore strategy
* Don't use two buckets
* Update the snapshot store logic to hide the file logic
* Add more backend tests
* Cleanup code a bit
* [WIP] Raft recovery (#938)
* Add recovery functionality
* remove fmt.Printfs
* Fix a few fsm bugs
* Add max size value for raft backend (#942)
* Add max size value for raft backend
* Include physical.ErrValueTooLarge in the message
* Raft snapshot Take/Restore API (#926)
* Inital work on raft snapshot APIs
* Always redirect snapshot install/download requests
* More work on the snapshot APIs
* Cleanup code a bit
* On restore handle special cases
* Use the seal to encrypt the sha sum file
* Add sealer mechanism and fix some bugs
* Call restore while state lock is held
* Send restore cb trigger through raft log
* Make error messages nicer
* Add test helpers
* Add snapshot test
* Add shamir unseal test
* Add more raft snapshot API tests
* Fix locking
* Change working to initalize
* Add underlying raw object to test cluster core
* Move leaderUUID to core
* Add raft TLS rotation logic (#950)
* Add TLS rotation logic
* Cleanup logic a bit
* Add/Remove from follower state on add/remove peer
* add comments
* Update more comments
* Update request_forwarding_service.proto
* Make sure we populate all nodes in the followerstate obj
* Update times
* Apply review feedback
* Add more raft config setting (#947)
* Add performance config setting
* Add more config options and fix tests
* Test Raft Recovery (#944)
* Test raft recovery
* Leave out a node during recovery
* remove unused struct
* Update physical/raft/snapshot_test.go
* Update physical/raft/snapshot_test.go
* fix vendoring
* Switch to new raft interface
* Remove unused files
* Switch a gogo -> proto instance
* Remove unneeded vault dep in go.sum
* Update helper/testhelpers/testhelpers.go
Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>
* Update vault/cluster/cluster.go
* track active key within the keyring itself (#6915)
* track active key within the keyring itself
* lookup and store using the active key ID
* update docstring
* minor refactor
* Small text fixes (#6912)
* Update physical/raft/raft.go
Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>
* review feedback
* Move raft logical system into separate file
* Update help text a bit
* Enforce cluster addr is set and use it for raft bootstrapping
* Fix tests
* fix http test panic
* Pull in latest raft-snapshot library
* Add comment
* ci: break config into separate files
* Untangle githooks
* githooks: fix whitespace
* .hooks/pre-commit: add ui -> lint-staged check
- We no longer require dependency on husky with this change.
* ui: remove husky dependency and config
- The previous commit obviates the need for it.
- We will now have to manage these hooks by hand, but this removes
the conflict between husky-installed hooks and those in the .hooks dir.
* ui: update yarn.lock with husky removed
* .hooks/pre-commit: always use subshell + docs
- Always use subshell means we consistently exit from the
same place which feels less complex.
- Docs are necessary for horrible bash like this I think...
* Makefile: remove old husky githooks
- Husky has installed a handler for every single git hook.
- This causes warnings on every git operation.
- Eventually we can remove this, but better not to confuse
people with these messages for now.
* ci: fix go build tags
* Makefile: improve compatibility of rm call
- Looks like the xargs in Travis does something different to the one
on my mac, this more verbose call should be safe everywhere.
* ci: fix make target names
* ci: fix test-ui invocation
* Makefile: simplify husky hook cleanup
* ci: more focussed readme
* ci: make ember-ci-test (browserstack) VLTES-28
* Update .circleci/config.yml
Co-Authored-By: Josh Freda <jfreda@users.noreply.github.com>
* Update .circleci/config.yml
Co-Authored-By: Josh Freda <jfreda@users.noreply.github.com>
* ci: rename test-ember -> test-ui-browserstack
* Makefile: ember-ci-test -> test-ui-browserstack
- Use the same name for this test everywhere to reduce cognitive load.
* browserstack: exit non-zero on failure to connect
* .travis.yml: ember-ci-test -> test-ui-browserstack
* browserstack: add vault bin to the path
* Makefile: browserstack: fail early w/clear msgs
This might save someone time later. The same checks could be applied
elsewhere too trivially.
* add browserstack
* check for data before removing root token
* fix root prefix and select by attributes for ie11
* use objectAt for ie11
* use blobs instead of files for ie11
* manually round cirucmference for ie11
* skip csp test on ie11
* skip tests in ie11
* include polyfill for CI
* remove on exit hooks
* update which browserstack tests are run
* remove ie check since we are not running these tests in ie
* remove ie check since we are not running these tests in ie
The static-assets target has a dependency on *either* ember-dist or
ember-dist-dev, so these targets must not execute in parallel. Since
this is an either/or dependency, it cannot be expressed as a regular
dependency unless the targets are refactored somehow.
Fixes: 7a312d7c37 ("Add Makefile/Dockerfile UI bits")
* add make targets to build the ui without minifying the source
* add polyfill for string.prototype.endswith and move to npm version of string.prototype.startswith
* replace the current history state with a / on the list secrets endpoint
* add test for list with id
* Add entity information request to system view
* fixing a few comments
* sharing types between plugin and logical
* sharing types between plugin and logical
* fixing output directory for proto
* removing extra replacement
* adding mount type lookup
* empty entities return nil instead of error
* adding some comments
