Golang: [1.16.2, 1.16.3]

No feature change

Golang [1.16]

Change

The harp-server plugin migrated to harp-plugins repository to lighten the release process and inherited dependencies.

SDK

• ruleset evaluation engine
  • A RuleSet is a set of Bundle secret constraints applied to a Package
  • It uses an extended CEL language to describe the bundle structure constraints
  • A RuleSet can be generated from an existing Bundle to enforce the Bundle structure in a separated deployment pipeline
• bundle lint
  • Evaluate a Bundle structure according to a given RuleSet
• Go 1.16 fs.FS
  • Experimental implementation of Bundle Filesystem using go 1.16 fs.FS interface exposed as github.com/elastic/harp/pkg/bundle/fs

Sample

Check that all packages are CSO compliant

apiVersion: harp.elastic.co/v1
kind: RuleSet
meta:
  name: harp-server
  description: Package and secret constraints for harp-server
  owner: [email protected]
spec:
  rules:
    - name: HARP-SRV-0001
      description: All package paths must be CSO compliant
      path: "*"
      constraints:
        - p.is_cso_compliant()

Lint an empty bundle will raise an error.

$ echo '{}' | harp from jsonmap \
  | harp bundle lint --spec test/fixtures/ruleset/valid/cso.yaml
{"level":"fatal","@timestamp":"2021-02-23T10:24:45.852Z","@caller":"[email protected]/command.go:856","@message":"unable to execute task","@appName":"harp-bundle-lint","@version":"","@revision":"8ebf40d","@appID":"BfGZbI8QYmSaXsBMWj8j0EASE67QcoP4OnC8nLl8xSXXtsY3PFEaABdfvm6c9yb3","@fields":{"error":"unable to validate given bundle: rule 'HARP-SRV-0001' didn't match any packages"}}

Lint valid bundle

$ echo '{"infra/aws/security/eu-central-1/ec2/ssh/default/authorized_keys":{"admin":"..."}}' \
  | harp from jsonmap \
  | harp bundle lint --spec test/fixtures/ruleset/valid/cso.yaml

No output and exit code (0) when everything is ok

Validate a secret structure

apiVersion: harp.elastic.co/v1
kind: RuleSet
meta:
  name: harp-server
  description: Package and secret constraints for harp-server
  owner: [email protected]
spec:
  rules:
    - name: HARP-SRV-0002
      description: Database credentials
      path: "app/qa/security/harp/v1.0.0/server/database/credentials"
      constraints:
        - p.has_all_secrets(['DB_HOST','DB_NAME','DB_USER','DB_PASSWORD'])

Lint an empty bundle will raise an error.

$ echo '{}' | harp from jsonmap \
  | harp bundle lint --spec test/fixtures/ruleset/valid/database-secret-validator.yaml
{"level":"fatal","@timestamp":"2021-02-23T10:31:05.792Z","@caller":"[email protected]/command.go:856","@message":"unable to execute task","@appName":"harp-bundle-lint","@version":"","@revision":"8ebf40d","@appID":"2kl6OWqgNTHkBumvlEtelxpJ4V1uDQCtE5MlOS1hXaUbOYtU1rrXbEL2zswx65y4","@fields":{"error":"unable to validate given bundle: rule 'HARP-SRV-0002' didn't match any packages"}}

Lint an invalid bundle

echo '{"app/qa/security/harp/v1.0.0/server/database/credentials":{}}' \
  | harp from jsonmap \
  | harp bundle lint --spec test/fixtures/ruleset/valid/database-secret-validator.yaml
{"level":"fatal","@timestamp":"2021-02-23T10:31:24.287Z","@caller":"[email protected]/command.go:856","@message":"unable to execute task","@appName":"harp-bundle-lint","@version":"","@revision":"8ebf40d","@appID":"7pflS7bCAAsDcAiPJWm36pypWY3nHhqOQwCc9Vp1ABCm8ZUWbmGinGL5zbP1EWvn","@fields":{"error":"unable to validate given bundle: package 'app/qa/security/harp/v1.0.0/server/database/credentials' doesn't validate rule 'HARP-SRV-0002'"}}

Generate a ruleset from a bundle

It will use the input bundle structure to generate a RuleSet.

harp ruleset from-bundle --in customer.bundle

api_version: harp.elastic.co/v1
kind: RuleSet
meta:
  description: Generated from bundle content
  name: vjz70BPFJuQhm_7quRGNt1ybocQU6DeXCn8h1o4aPm80CI4pM8lNwVBTDqH8SpW0W1r-8dXSVQK67pO-vtgS_Q
spec:
  rules:
  - constraints:
    - p.has_secret("API_KEY")
    name: LINT-vjz70B-1
    path: app/production/customer1/ece/v1.0.0/adminconsole/authentication/otp/okta_api_key
  - constraints:
    - p.has_secret("host")
    - p.has_secret("port")
    - p.has_secret("options")
    - p.has_secret("username")
    - p.has_secret("password")
    - p.has_secret("dbname")
    name: LINT-vjz70B-2
    path: app/production/customer1/ece/v1.0.0/adminconsole/database/usage_credentials
  - constraints:
    - p.has_secret("cookieEncryptionKey")
    - p.has_secret("sessionSaltSeed")
    - p.has_secret("jwtHmacKey")
    name: LINT-vjz70B-3
    path: app/production/customer1/ece/v1.0.0/adminconsole/http/session
  - constraints:
    - p.has_secret("API_KEY")
    name: LINT-vjz70B-4
    path: app/production/customer1/ece/v1.0.0/adminconsole/mailing/sender/mailgun_api_key
  - constraints:
    - p.has_secret("emailHashPepperSeedKey")
    name: LINT-vjz70B-5
    path: app/production/customer1/ece/v1.0.0/adminconsole/privacy/anonymizer
  - constraints:
    - p.has_secret("host")
    - p.has_secret("port")
    - p.has_secret("options")
    - p.has_secret("username")
    - p.has_secret("password")
    - p.has_secret("dbname")
    name: LINT-vjz70B-6
    path: app/production/customer1/ece/v1.0.0/userconsole/database/usage_credentials
  - constraints:
    - p.has_secret("privateKey")
    - p.has_secret("publicKey")
    name: LINT-vjz70B-7
    path: app/production/customer1/ece/v1.0.0/userconsole/http/certificate
  - constraints:
    - p.has_secret("cookieEncryptionKey")
    - p.has_secret("sessionSaltSeed")
    - p.has_secret("jwtHmacKey")
    name: LINT-vjz70B-8
    path: app/production/customer1/ece/v1.0.0/userconsole/http/session
  - constraints:
    - p.has_secret("user")
    - p.has_secret("password")
    name: LINT-vjz70B-9
    path: infra/aws/essp-customer1/us-east-1/rds/adminconsole/accounts/root_credentials
  - constraints:
    - p.has_secret("API_KEY")
    - p.has_secret("ca.pem")
    name: LINT-vjz70B-10
    path: platform/production/customer1/us-east-1/billing/recurly/vendor_api_key
  - constraints:
    - p.has_secret("username")
    - p.has_secret("password")
    name: LINT-vjz70B-11
    path: platform/production/customer1/us-east-1/postgresql/admiconsole/admin_credentials
  - constraints:
    - p.has_secret("username")
    - p.has_secret("password")
    name: LINT-vjz70B-12
    path: platform/production/customer1/us-east-1/zookeeper/accounts/admin_credentials
  - constraints:
    - p.has_secret("privateKey")
    - p.has_secret("publicKey")
    name: LINT-vjz70B-13
    path: product/ece/v1.0.0/artifact/signature/key

Golang [1.15.8, 1.16]

Fix

• Patch application raised an error when package names are duplicated. - https://github.com/elastic/harp/pull/35
• Possible overflow fixed discovered by CodeQL - https://github.com/elastic/harp/pull/37

Change

SDK

• Template functions
  • fromJwk convert a JWK to a native key
  • keyToBytes encodes the given key as a byte array
  • encryptJwe encrypts the given payload using JWE encoding (PBES2_HS256_A128KW)
  • decryptJwe decrypts the given payload encoded using JWE encoding
  • toJws create a JWS token
  • bech32enc/bech32dec to encode/decode a Bech32 encoded string
• Value Transformers
  • aes.Transformer is now aead.AESGCM (breaking change)
  • aead.AESGCM now support 128, 192, 256 according to given key size
  • aead.Chacha20Poly1305 added to support chacha20poly1305 encryption
  • aead.XChacha20Poly1305 added to support xchacha20poly1305 encryption
  • aead.AESPMACSIV added to support AES-PMAC-SIV encryption
• Container Identities
  • Public keys are now encoded using Bech32 encoding instead of Base64. Bech32 allows the developer to add a human-readable prefix in front of the encoded value.
    Sample public key - security1fxhzk3k9mvpc77pu80qv44xh09snj6hpw574tcr5jx2w2zzckgzs0a8cfx
  • Sealing process now uses this format as input in order to specify identities.
• BundlePatch
  • Allow package creation via patch - https://github.com/elastic/harp/pull/36

Build

• go 1.16 is the default toolchain version
• Nancy dependency check added via GHA
• Unit tests are now execution on Linux, Mac, Windows via GHA
• darwin/arm64 artifact is produced

Sample

Generate an identity

$ harp container identity --description security-recovery --passphrase $(harp passphrase) | jq
{
  "@apiVersion": "harp.elastic.co/v1",
  "@kind": "ContainerIdentity",
  "@timestamp": "2021-02-16T10:06:31.126671Z",
  "@description": "security-recovery",
  "public": "security-recovery1jjq095c68kjz4e3ck5cvu97qrgf8npm7ck2qfex24nw7zfk2g5jqxkzzwt",
  "private": {
    "encoding": "jwe",
    "content": "eyJhbGciOiJQQkVTMi1IUzUxMitBMjU2S1ciLCJjdHkiOiJqd2sranNvbiIsImVuYyI6IkEyNTZHQ00iLCJwMmMiOjUwMDAwMSwicDJzIjoiVUVVNFdIbHFRMGxEYjI1dWRHWnJiZyJ9.d4qhmOsCNseGI_oyTvOKP6LVdOfEYdKkoplZ0kZuDA1ncUjaKoZOvw.3DmFEueug6zvNkbC.5mvVIkFEBQf9GQulf6BL4TeMfMJcSxQI3sJx3lo0Cf7EJ6ZF1v1U3YaQMB7smG3t9emZNvij5FI8g0DwPd0NHT4BNwuG_-oSbdmHZyD4ilkMdAZYHO9ZctNjLS-0dqV1wG7-uiF40g8FKZbx8UbQ9NDd5UutUTIWfaf8FxhYaf4.xIIn95CNXWAFQd2QCg-tiA"
  }
}

Seal a container using the identity

$ echo '{}' | harp from jsonmap | harp container seal \
  --identity security-recovery1jjq095c68kjz4e3ck5cvu97qrgf8npm7ck2qfex24nw7zfk2g5jqxkzzwt \
  --identity harp-server12qgw2lc4my3d9avahv7jd8w46zez90dumjf6ktq8t66zvr2hggasnw6kvy \
  --out sealed.bundle

Golang [1.15.7, 1.15.8, 16rc1]

Changes

SDK

• Prepare to go1.16 migration, x509.DecryptPEM deprecation replaced by PKCS#8 (PBES2 AES only) - https://github.com/elastic/harp/pull/20
• oplog is sorted to get a stable output - https://github.com/elastic/harp/pull/30
• Fixed an issue when passing a json object as a template value which is not deserializable as map[string]interface{}. When it's the case error is raised and inform the user that he could use an explicit prefix - https://github.com/elastic/harp/pull/30
• Generate a YAML BundlePatch from bundle diff - https://github.com/elastic/harp/pull/31
• from oplog can be used to create a Bundle from JSON OpLog format - https://github.com/elastic/harp/pull/33

Build

• Add darwin/arm64 target for M1 native support when compiling with go1.16 only - https://github.com/elastic/harp/pull/30
• Dependencies / Tools upgraded - https://github.com/elastic/harp/pull/30
• Taskfile rework to support conditional build - https://github.com/elastic/harp/pull/32

Golang [1.15.7, 1.15.8, 1.16rc1]

Changes

• bundle diff now expose an oplog as a JSONPatch like output - https://github.com/elastic/harp/pull/28
• from vault can pull a specific secret version by adding <path>?version=<version> - https://github.com/elastic/harp/pull/26
• Vault secret metadata are now exported as annotations when a bundle is created from Vault

Changes

• Fix a random test issue with fuzz test and memguard - https://github.com/elastic/harp/pull/23 https://github.com/elastic/harp/pull/24
• Fix golangci-lint dependency pull error - https://github.com/elastic/harp/pull/22
• Fix errorlint findings in harp-server - https://github.com/elastic/harp/pull/25

Changes

• bundle dump and from dump use wire agnostic value serialization format (JSON) so that it can be used to migrate data from container versions. - https://github.com/elastic/harp/pull/16
• from vault / to vault - Vault error was masked by an invalid errgroup usage with context, the error was overridden by context cancellation error. Now the real underlying error is raised and displayed to the user. - https://github.com/elastic/harp/pull/18
• Vault authentication is now explicitly tested before starting any operation.
• Vault parallel workers now use explicit worker count, modifiable via flags and don't rely on runtime to prevent docker CPU count detection problem. - https://github.com/elastic/harp/pull/15

Harp v0.1.7 - https://github.com/elastic/harp

Changes

• Go toolchain constraints updated to [1.15.6, 1.15.7]
• os/exec vulnerability mitigation - https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3115 (not disclosed yet)
• Fix container reader usage for bundle diff/dump
• Dependencies have been updated
• Tools have been updated (rebuild tools base image via mage docker:tools)
• errorwrapping is now fixed and enforced by linter
• CodeQL is executed on each PR + main
• Snyk vulnerability scan for dependency is executed on each PR + main

Reference(s)

• https://blog.golang.org/path-security
• https://groups.google.com/g/golang-announce/c/mperVMGa98w