The parent project for OpenZiti. Here you will find the executables for a fully zero trust, application embedded, programmable network @OpenZiti
APACHE-2.0 License
Bot releases are visible (Hide)
appId
and appVersion
in sdkInfo
ziti ps
) in ziti-tunnelziti edge db start-check-integrity
ziti edge db check-integrity-status
services.list
ziti log-format
or ziti lf
for short, for formating JSON log output as something moreterminationTimeoutSeconds
and routeTimeoutSeconds
added together
getSessionTimeout
is specified in the router config under listeners: options:
There's a new REST endpoint /current-api-session/service-updates, which will return the last time
services were changed. If there have been no service updates since the api session was established,
the api session create date/time will be returned. This endpoint can be polled to see if services
need to be refreshed. This will save network and cpu utilization on the client and controller.
The controller configuration file now supports a httpTimeouts
section under
edge.api
. The section and all of its fields are optional and default to the values of previous
versions.
For production environments these values should be tuned for the networks intended userbase. The
quality and latency of the underlay between the networks endpoints/routers and controller should be
taken into account.
edge:
...
api:
...
httpTimeouts:
# (optional, default 5s) readTimeoutMs is the maximum duration for reading the entire request, including the body.
readTimeoutMs: 5000
# (optional, default 0) readHeaderTimeoutMs is the amount of time allowed to read request headers.
# The connection's read deadline is reset after reading the headers. If readHeaderTimeoutMs is zero, the value of
# readTimeoutMs is used. If both are zero, there is no timeout.
readHeaderTimeoutMs: 0
# (optional, default 10000) writeTimeoutMs is the maximum duration before timing out writes of the response.
writeTimeoutMs: 100000
# (optional, default 5000) idleTimeoutMs is the maximum amount of time to wait for the next request when keep-alives are enabled
idleTimeoutMs: 5000
ziti log-format
or ziti lf
for short, for formating JSON log output as something moreterminationTimeoutSeconds
and routeTimeoutSeconds
added together
getSessionTimeout
is specified in the router config under listeners: options:
The controller configuration file now supports a httpTimeouts
section under
edge.api
. The section and all of its fields are optional and default to the values of previous
versions.
For production environments these values should be tuned for the networks intended userbase. The
quality and latency of the underlay between the networks endpoints/routers and controller should be
taken into account.
edge:
...
api:
...
httpTimeouts:
# (optional, default 5s) readTimeoutMs is the maximum duration for reading the entire request, including the body.
readTimeoutMs: 5000
# (optional, default 0) readHeaderTimeoutMs is the amount of time allowed to read request headers.
# The connection's read deadline is reset after reading the headers. If readHeaderTimeoutMs is zero, the value of
# readTimeoutMs is used. If both are zero, there is no timeout.
readHeaderTimeoutMs: 0
# (optional, default 10000) writeTimeoutMs is the maximum duration before timing out writes of the response.
writeTimeoutMs: 100000
# (optional, default 5000) idleTimeoutMs is the maximum amount of time to wait for the next request when keep-alives are enabled
idleTimeoutMs: 5000
appId
and appVersion
in sdkInfo
ziti ps
) in ziti-tunnelziti edge db start-check-integrity
ziti edge db check-integrity-status
services.list
host
mode, if youThere's a new REST endpoint /current-api-session/service-updates, which will return the last time
services were changed. If there have been no service updates since the api session was established,
the api session create date/time will be returned. This endpoint can be polled to see if services
need to be refreshed. This will save network and cpu utilization on the client and controller.
When the tunneler hosts services there was previously no way to specify the precedence and cost
associated with those services.
See Ziti XT documentation
for an overview of how precedence and cost relate to HA and load balancing.
There are now two new fields on identity:
default
, required
and failed
. Defaultsdefault
.When hosting a service via the tunneler, the terminator for the SDK hosted service will be created
with the precedence and cost of the identity used by the tunneler.
NOTE: This means all services hosted by an identity will have the same precedence and cost.
We'll likely add support for service specific overrides in the future if/when use cases arise which
call for it. In the meantime, a work-around is to use multiple identities if you need different
values for different services.
The ziti CLI supports setting the default hosting precedence and cost when creating identities
The GO SDK has a new API method GetCurrentIdentity() (*edge.CurrentIdentity, error)
which lets SDK
users retrieve the currently logged in identity, including the default hosting precedence and cost.
This could be used by other SDK applications which may want to use the fields for the same reason
when hosting services.
The go tunneler now supports health checks. Support for health checks may be added to other
tunnelers (such as ziti-edge-tunnel) in the future, but that is not guaranteed.
Health checks can be configured in the service configuration using the ziti-tunneler-server.v1
config type. Support in the host.v1
config type will be added when support for that config type is
added to the go tunneler.
The tunneler supports two types of health check.
Port checks look to see if a host/port can be dialed. This is simple check which just ensures that
something is listening on a give host/port.
Port checks have the following properties:
Http checks a specific URL. They support the following properties:
GET
, POST
, PUT
or PATCH
. Defaults to GET
Each health check may specify actions to execute when a health check runs.
Each action may specify:
pass
or fail
. Specifies if the action should run when the check ismark healthy
- the terminator precedence will be set to the default hosting precedence ofmark unhealthy
- the terminator precedence will be set to failed
increase cost N
- the terminator cost will be increased by N. This will only happen whiledecrease cost N
- the terminator cost will be decrease by N to a minimuim. The terminatorThe following config defines a TCP service which can be reach at port 8171 on localhost
. It has a
port check defined which runs every 5 seconds, with a timeout of 500 milliseconds. The following
actions are defined on the health check:
{
"protocol" : "tcp",
"hostname" : "localhost",
"port" : 8171,
"portChecks" : [
{
"interval" : "5s",
"timeout" : "500ms",
"address" : "localhost:8171",
"actions": [
{
"action": "mark unhealthy",
"consecutiveEvents": 10,
"trigger": "fail"
},
{
"action": "increase cost 100",
"trigger": "fail"
},
{
"action": "mark healthy",
"duration": "10s",
"trigger": "pass"
},
{
"action": "decrease cost 25",
"trigger": "pass"
}
]
}
]
}
The ziti-tunnel can now be run in a mode where it will only host services and will not intercept any
services.
Ex: ziti-tunnel host -i /path/to/identity.json
For reference, here is the full, updated ziti-tunneler-server.v1
schema:
{
"$id": "http://edge.openziti.org/schemas/ziti-tunneler-server.v1.config.json",
"additionalProperties": false,
"definitions": {
"action": {
"additionalProperties": false,
"properties": {
"action": {
"pattern": "(mark (un)?healthy|increase cost [0-9]+|decrease cost [0-9]+)",
"type": "string"
},
"consecutiveEvents": {
"maximum": 65535,
"minimum": 0,
"type": "integer"
},
"duration": {
"$ref": "#/definitions/duration"
},
"trigger": {
"enum": [
"fail",
"pass"
],
"type": "string"
}
},
"required": [
"trigger",
"action"
],
"type": "object"
},
"actionList": {
"items": {
"$ref": "#/definitions/action"
},
"maxItems": 20,
"minItems": 1,
"type": "array"
},
"duration": {
"pattern": "[0-9]+(h|m|s|ms)",
"type": "string"
},
"httpCheck": {
"additionalProperties": false,
"properties": {
"actions": {
"$ref": "#/definitions/actionList"
},
"body": {
"type": "string"
},
"expectInBody": {
"type": "string"
},
"expectStatus": {
"maximum": 599,
"minimum": 100,
"type": "integer"
},
"interval": {
"$ref": "#/definitions/duration"
},
"method": {
"$ref": "#/definitions/method"
},
"timeout": {
"$ref": "#/definitions/duration"
},
"url": {
"type": "string"
}
},
"required": [
"interval",
"timeout",
"url"
],
"type": "object"
},
"httpCheckList": {
"items": {
"$ref": "#/definitions/httpCheck"
},
"type": "array"
},
"method": {
"enum": [
"GET",
"POST",
"PUT",
"PATCH"
],
"type": "string"
},
"portCheck": {
"additionalProperties": false,
"properties": {
"actions": {
"$ref": "#/definitions/actionList"
},
"address": {
"type": "string"
},
"interval": {
"$ref": "#/definitions/duration"
},
"timeout": {
"$ref": "#/definitions/duration"
}
},
"required": [
"interval",
"timeout",
"address"
],
"type": "object"
},
"portCheckList": {
"items": {
"$ref": "#/definitions/portCheck"
},
"type": "array"
}
},
"properties": {
"hostname": {
"type": "string"
},
"httpChecks": {
"$ref": "#/definitions/httpCheckList"
},
"port": {
"maximum": 65535,
"minimum": 0,
"type": "integer"
},
"portChecks": {
"$ref": "#/definitions/portCheckList"
},
"protocol": {
"enum": [
"tcp",
"udp"
],
"type": [
"string",
"null"
]
}
},
"required": [
"hostname",
"port"
],
"type": "object"
}
Logs generated by Ziti components written in Go (Controller, Router, SDK) will no longer output ANSI
color control characters by default. Color logs can be enabled by setting in the environment
variable PFXLOG_USE_COLOR
to any truthy value: 1, t, T, TRUE, true, True, 0, f, F, FALSE, false,
False.
All authentication mechanisms can now bootstrap key pairs via an authenticated session using API
Session Certificates. These key pairs involve authenticating, preparing an X509 Certificate Signing
Request (CSR), and then submitting the CSR for processing. The output is an ephemeral certificate
tied to that session that can be used to connect to Edge Routers for session dial/binds.
New Endpoints:
csr
field)API Session Certificates have a 12hr life span. New certificates can be created before previous ones
expire and be used for reconnection.
host
mode, if youWhen the tunneler hosts services there was previously no way to specify the precedence and cost
associated with those services.
See Ziti XT documentation
for an overview of how precedence and cost relate to HA and load balancing.
There are now two new fields on identity:
default
, required
and failed
. Defaultsdefault
.When hosting a service via the tunneler, the terminator for the SDK hosted service will be created
with the precedence and cost of the identity used by the tunneler.
NOTE: This means all services hosted by an identity will have the same precedence and cost.
We'll likely add support for service specific overrides in the future if/when use cases arise which
call for it. In the meantime, a work-around is to use multiple identities if you need different
values for different services.
The ziti CLI supports setting the default hosting precedence and cost when creating identities
The GO SDK has a new API method GetCurrentIdentity() (*edge.CurrentIdentity, error)
which lets SDK
users retrieve the currently logged in identity, including the default hosting precedence and cost.
This could be used by other SDK applications which may want to use the fields for the same reason
when hosting services.
The go tunneler now supports health checks. Support for health checks may be added to other
tunnelers (such as ziti-edge-tunnel) in the future, but that is not guaranteed.
Health checks can be configured in the service configuration using the ziti-tunneler-server.v1
config type. Support in the host.v1
config type will be added when support for that config type is
added to the go tunneler.
The tunneler supports two types of health check.
Port checks look to see if a host/port can be dialed. This is simple check which just ensures that
something is listening on a give host/port.
Port checks have the following properties:
Http checks a specific URL. They support the following properties:
GET
, POST
, PUT
or PATCH
. Defaults to GET
Each health check may specify actions to execute when a health check runs.
Each action may specify:
pass
or fail
. Specifies if the action should run when the check ismark healthy
- the terminator precedence will be set to the default hosting precedence ofmark unhealthy
- the terminator precedence will be set to failed
increase cost N
- the terminator cost will be increased by N. This will only happen whiledecrease cost N
- the terminator cost will be decrease by N to a minimuim. The terminatorThe following config defines a TCP service which can be reach at port 8171 on localhost
. It has a
port check defined which runs every 5 seconds, with a timeout of 500 milliseconds. The following
actions are defined on the health check:
{
"protocol" : "tcp",
"hostname" : "localhost",
"port" : 8171,
"portChecks" : [
{
"interval" : "5s",
"timeout" : "500ms",
"address" : "localhost:8171",
"actions": [
{
"action": "mark unhealthy",
"consecutiveEvents": 10,
"trigger": "fail"
},
{
"action": "increase cost 100",
"trigger": "fail"
},
{
"action": "mark healthy",
"duration": "10s",
"trigger": "pass"
},
{
"action": "decrease cost 25",
"trigger": "pass"
}
]
}
]
}
The ziti-tunnel can now be run in a mode where it will only host services and will not intercept any
services.
Ex: ziti-tunnel host -i /path/to/identity.json
For reference, here is the full, updated ziti-tunneler-server.v1
schema:
{
"$id": "http://edge.openziti.org/schemas/ziti-tunneler-server.v1.config.json",
"additionalProperties": false,
"definitions": {
"action": {
"additionalProperties": false,
"properties": {
"action": {
"pattern": "(mark (un)?healthy|increase cost [0-9]+|decrease cost [0-9]+)",
"type": "string"
},
"consecutiveEvents": {
"maximum": 65535,
"minimum": 0,
"type": "integer"
},
"duration": {
"$ref": "#/definitions/duration"
},
"trigger": {
"enum": [
"fail",
"pass"
],
"type": "string"
}
},
"required": [
"trigger",
"action"
],
"type": "object"
},
"actionList": {
"items": {
"$ref": "#/definitions/action"
},
"maxItems": 20,
"minItems": 1,
"type": "array"
},
"duration": {
"pattern": "[0-9]+(h|m|s|ms)",
"type": "string"
},
"httpCheck": {
"additionalProperties": false,
"properties": {
"actions": {
"$ref": "#/definitions/actionList"
},
"body": {
"type": "string"
},
"expectInBody": {
"type": "string"
},
"expectStatus": {
"maximum": 599,
"minimum": 100,
"type": "integer"
},
"interval": {
"$ref": "#/definitions/duration"
},
"method": {
"$ref": "#/definitions/method"
},
"timeout": {
"$ref": "#/definitions/duration"
},
"url": {
"type": "string"
}
},
"required": [
"interval",
"timeout",
"url"
],
"type": "object"
},
"httpCheckList": {
"items": {
"$ref": "#/definitions/httpCheck"
},
"type": "array"
},
"method": {
"enum": [
"GET",
"POST",
"PUT",
"PATCH"
],
"type": "string"
},
"portCheck": {
"additionalProperties": false,
"properties": {
"actions": {
"$ref": "#/definitions/actionList"
},
"address": {
"type": "string"
},
"interval": {
"$ref": "#/definitions/duration"
},
"timeout": {
"$ref": "#/definitions/duration"
}
},
"required": [
"interval",
"timeout",
"address"
],
"type": "object"
},
"portCheckList": {
"items": {
"$ref": "#/definitions/portCheck"
},
"type": "array"
}
},
"properties": {
"hostname": {
"type": "string"
},
"httpChecks": {
"$ref": "#/definitions/httpCheckList"
},
"port": {
"maximum": 65535,
"minimum": 0,
"type": "integer"
},
"portChecks": {
"$ref": "#/definitions/portCheckList"
},
"protocol": {
"enum": [
"tcp",
"udp"
],
"type": [
"string",
"null"
]
}
},
"required": [
"hostname",
"port"
],
"type": "object"
}
Terminator events are now generated and can be found the fabric events/ package along with other
fabric events. They can also be emitted in json or plain text to a file or stdout, same as other
events. Events are generated when:
A terminator event will have the following properties:
fabric.terminators
Example: To register for json events
events:
jsonLogger:
subscriptions:
- type: fabric.terminators
Example JSON output:
{
"namespace": "fabric.terminators",
"event_type": "updated",
"timestamp": "2021-01-08T16:26:08.0005535-05:00",
"service_id": "49Gc41SuL",
"terminator_id": "y8qR",
"router_id": "T-8CFqqtB",
"router_online": true,
"precedence": "required",
"static_cost": 1100,
"dynamic_cost": 0,
"total_terminators": 1,
"usable_default_terminators": 1,
"usable_required_terminators": 0
}
Logs generated by Ziti components written in Go (Controller, Router, SDK) will
no longer output ANSI color control characters by default. Color logs can be
enabled by setting in the environment variable PFXLOG_USE_COLOR
to any
truthy value: 1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False.
All authentication mechanisms can now bootstrap key pairs via an authenticated session
using API Session Certificates. These key pairs involve authenticating, preparing an
X509 Certificate Signing Request (CSR), and then submitting the CSR for processing.
The output is an ephemeral certificate tied to that session that can be used to
connect to Edge Routers for session dial/binds.
New Endpoints:
csr
field)API Session Certificates have a 12hr life span. New certificates can be created
before previous ones expire and be used for reconnection.
ziti-tunnel enroll
should set non-zeroziti edge
would fail when terminators referenced pureThis rewrite fixed several deadlocks observed at high throughput. It also tries to ensure that slow
clients attached to a router can't block traffic/processing for faster clients. It does this by
dropping data for a client if the client isn't handling incoming traffic quickly enough. Dropped
payloads will be retransmitted. The new xgress implementation uses similar windowing and
retransmission strategies to the upcoming transwarp work.
0.18+ routers will probably work with older router versions, but probably not well. 0.18+ xgress
instances expect to get round trip times and receive buffer sizes on ack messages. If they don't get
them then retransmission will likely be either too agressive or not aggressive enough.
Mixing 0.18+ routers with older router versions is not recommended without doing more testing first.
Added
REMOVED: The retransmission option is no longer available. Retransmission can't be toggled off
anymore as that would lead to lossy connections.
New metrics were introduced as part of the rewrite.
NOTE: Some of these metrics were introduced to try and find places where tuning was required.
They may not be interesting or useful in the long term and may be removed in a future release.
The new metrics include:
New Meters
New Histograms
New Timers
New Gauges
The fabric will now create two channels for each link, one for data and the other for acks. When
establishing links the dialing side will attach headers indicating the channel type and a shared
link ID. If the receiving side doesn't support split links then it will treat both channels as
regular links and send both data and acks over both.
If an older router dials a router expecting split links it won't have the link type and will be
treated as a regular, non-split link.
The service terminator strategies use dial failures to adjust terminator weights and/or mark
terminators as failed. Previously SDK applications didn't have a way to mark a dial as failed. If
the SDK was hosting an application, this was generally not a problem. If the application could be
reached, it wouldn't want to mark an incoming connection as failed. However, the tunneler is just
proxying connections. It wants to be able to reach out to another application when the service is
dialed and proxy data. If the dial fails, it wants to be able to notify the controller that the
application wasn't reachable. The golang SDK now has the capability.
There is a new API on edge.Listener
.
AcceptEdge() (Conn, error)
The Conn
returned here is an edge.Conn
(which extends net.Conn
). edge.Conn
has two new APIs.
CompleteAcceptSuccess() error
CompleteAcceptFailed(err error)
If ListenWithOptions
is called with the ManualStart: true
in the provided options, the
connection won't be established until CompleteAcceptSuccess
is called. Writing or reading the
connection before call that method will have undefined results.
The ziti-tunnel has been updated to use this API, and so should now work correctly with the various
terminator strategies.
When establishing a new virtual connection to hosted SDK application the router had to execute the
following steps:
If the connection id could be established on the router, we could simplify things as follows
We didn't do this previously because the sdk controls ids for outbound connection. To enable this we
have split the 32 bit id range in half. The top half is now reserved for hosted connection ids. This
behavior is controlled by the SDK, which requests it when it binds uisng a boolean flag. The new
flag is:
RouterProvidedConnId = 1012
If the bind result from the router has the same flag set to true, then the sdk will expect Dial
messages from the router to have a connection id provided in the header keyed with the same 1012
.
This means that this feature should be both backwards and forward compatible.
Builds have been moved from travis.org to Github Actions
IDs generated for entities in the Edge no longer use underscores and instead use periods to avoid issues when used as a common name in CSRs
edge#424 Authenticated, non-admin, clients can query service terminators
sdk-golang#112 Process checks for Windows are case-insensitive
The CLI agent now runs over unix sockets and is enabled by default. See doc/ops-agent.md for details in the ziti repository.
ziti#245 Make timeout used by CLI's internal REST client configurable via cmd line arg
All ziti edge controller
subcommands now support the --timeout=n
flag which controls the internal REST-client timeout used when communicating with the Controller. The timeout resolution is in seconds. If this flag is not specified, the default is 5
. Prior to this release, the the REST-client timeout was always 2
. You now have the opportunity to increase the timeout if necessary (e.g. if large amounts of data are being queried).
All ziti edge controller
subcommands now support the --verbose
flag which will cause internal REST-client to emit debugging information concerning HTTP headers, status, raw json response data, and more. You now have the opportunity to see much more information, which could be valuable during trouble-shooting.