Name |
Date |
Size |
#Lines |
LOC |
||
---|---|---|---|---|---|---|
.. | - | - | ||||
plugins/ssp-h1url/ | 03-May-2024 | - | 41 | 23 | ||
protocols/ | 03-May-2024 | - | 1,210 | 746 | ||
system/ | 03-May-2024 | - | 437 | 277 | ||
README.md | D | 03-May-2024 | 13.3 KiB | 455 | 303 | |
policy.c | D | 03-May-2024 | 23.2 KiB | 979 | 742 | |
private-lib-secure-streams.h | D | 03-May-2024 | 9.5 KiB | 348 | 197 | |
secure-streams-client.c | D | 03-May-2024 | 12.4 KiB | 575 | 373 | |
secure-streams-process.c | D | 03-May-2024 | 12.8 KiB | 534 | 315 | |
secure-streams-serialize.c | D | 03-May-2024 | 20.7 KiB | 927 | 621 | |
secure-streams.c | D | 03-May-2024 | 12.8 KiB | 582 | 378 |
README.md
1# Secure Streams 2 3Secure Streams is a client api that strictly separates payload from any metadata. 4That includes the endpoint address for the connection, the tls CA and even the 5protocol used to connect to the endpoint. 6 7The user api just receives and transmits payload, and receives advisory connection 8state information. 9 10The details about how the connections for different types of secure stream should 11be made are held in JSON "policy database" initially passed in to the context 12creation, but able to be updated from a remote copy. 13 14![overview](../doc-assets/ss-explain.png) 15 16# JSON Policy Database 17 18Example JSON policy... formatting is shown for clarity but whitespace can be 19omitted in the actual policy. 20 21Ordering is not critical in itself, but forward references are not allowed, 22things must be defined before they are allowed to be referenced later in the 23JSON. 24 25 26``` 27{ 28 "release": "01234567", 29 "product": "myproduct", 30 "schema-version": 1, 31 "retry": [{ 32 "default": { 33 "backoff": [1000, 2000, 3000, 5000, 10000], 34 "conceal": 5, 35 "jitterpc": 20 36 } 37 }], 38 "certs": [{ 39 "isrg_root_x1": "MIIFazCCA1OgAw...AnX5iItreGCc=" 40 }, { 41 "LEX3_isrg_root_x1": "MIIFjTCCA3WgAwIB...WEsikxqEt" 42 }], 43 "trust_stores": [{ 44 "le_via_isrg": ["isrg_root_x1", "LEX3_isrg_root_x1"] 45 }], 46 "s": [{ 47 "mintest": { 48 "endpoint": "warmcat.com", 49 "port": 4443, 50 "protocol": "h1get", 51 "aux": "index.html", 52 "plugins": [], 53 "tls": true, 54 "opportunistic": true, 55 "retry": "default", 56 "tls_trust_store": "le_via_isrg" 57 } 58 }] 59} 60``` 61 62### `Release` 63 64Identifies the policy version 65 66### `Product` 67 68Identifies the product the policy should apply to 69 70### `Schema-version` 71 72The minimum version of the policy parser required to parse this policy 73 74### `via-socks5` 75 76Optional redirect for Secure Streams client traffic through a socks5 77proxy given in the format `address:port`, eg, `127.0.0.1:12345`. 78 79### `retry` 80 81A list of backoff schemes referred to in the policy 82 83### `backoff` 84 85An array of ms delays for each retry in turn 86 87### `conceal` 88 89The number of retries to conceal from higher layers before giving errors. If 90this is larger than the number of times in the backoff array, then the last time 91is used for the extra delays 92 93### `jitterpc` 94 95Percentage of the delay times mentioned in the backoff array that may be 96randomly added to the figure from the array. For example with an array entry of 971000ms, and jitterpc of 20%, actual delays will be chosen randomly from 1000ms 98through 1200ms. This is to stop retry storms triggered by a single event like 99an outage becoming synchronized into a DoS. 100 101### `certs` 102 103Certificates needed for validation should be listed here each with a name. The 104format is base64 DER, which is the same as the part of PEM that is inside the 105start and end lines. 106 107### `trust_stores` 108 109Chains of certificates given in the `certs` section may be named and described 110inside the `trust_stores` section. Each entry in `trust_stores` is created as 111a vhost + tls context with the given name. Stream types can later be associated 112with one of these to enforce validity checking of the remote server. 113 114Entries should be named using "name" and the stack array defined using "stack" 115 116### `s` 117 118These are an array of policies for the supported stream type names. 119 120### `endpoint` 121 122The DNS address the secure stream should connect to. 123 124This may contain string symbols which will be replaced with the 125corresponding streamtype metadata value at runtime. Eg, if the 126streamtype lists a metadata name "region", it's then possible to 127define the endpoint as, eg, `${region}.mysite.com`, and before 128attempting the connection setting the stream's metadata item 129"region" to the desired value, eg, "uk". 130 131### `port` 132 133The port number as an integer on the endpoint to connect to 134 135### `protocol` 136 137The wire protocol to connect to the endpoint with. Currently supported 138streamtypes are 139 140|Wire protocol|Description| 141|---|---| 142|h1|http/1| 143|h2|http/2| 144|ws|http/1 Websockets| 145|mqtt|mqtt 3.1.1| 146 147### `plugins` 148 149Array of plugin names to apply to the stream, if any 150 151### `tls` 152 153Set to `true` to enforce the stream travelling in a tls tunnel 154 155### `client cert` 156 157Set if the stream needs to authenticate itself using a tls client certificate. 158Set to the certificate index counting from 0+. The certificates are managed 159using lws_sytstem blobs. 160 161### `opportunistic` 162 163Set to `true` if the connection may be left dropped except when in use 164 165### `nailed_up` 166 167Set to `true` to have lws retry if the connection carrying this stream should 168ever drop. 169 170### `retry` 171 172The name of the policy described in the `retry` section to apply to this 173connection for retry + backoff 174 175### `tls_trust_store` 176 177The name of the trust store described in the `trust_stores` section to apply 178to validate the remote server cert. 179 180## http transport 181 182### `http_method` 183 184HTTP method to use with http-related protocols, like GET or POST. 185Not required for ws. 186 187### `http_url` 188 189Url path to use with http-related protocols 190 191The URL path can include metatadata like this 192 193"/mypath?whatever=${metadataname}" 194 195${metadataname} will be replaced by the current value of the 196same metadata name. The metadata names must be listed in the 197"metadata": [ ] section. 198 199### `http_auth_header` 200 201The name of the header that takes the auth token, with a trailing ':', eg 202 203``` 204 "http_auth_header": "authorization:" 205``` 206 207### `http_dsn_header` 208 209The name of the header that takes the dsn token, with a trailing ':', eg 210 211``` 212 "http_dsn_header": "x-dsn:" 213``` 214 215### `http_fwv_header` 216 217The name of the header that takes the firmware version token, with a trailing ':', eg 218 219``` 220 "http_fwv_header": "x-fw-version:" 221``` 222 223### `http_devtype_header` 224 225The name of the header that takes the device type token, with a trailing ':', eg 226 227``` 228 "http_devtype_header": "x-device-type:" 229``` 230 231### `http_auth_preamble` 232 233An optional string that precedes the auth token, eg 234 235``` 236 "http_auth_preamble": "bearer " 237``` 238 239### `auth_hexify` 240 241Convert the auth token to hex ('A' -> "41") before transporting. Not necessary if the 242auth token is already in printable string format suitable for transport. Needed if the 243auth token is a chunk of 8-bit binary. 244 245### `nghttp2_quirk_end_stream` 246 247Set this to `true` if the peer server has the quirk it won't send a response until we have 248sent an `END_STREAM`, even though we have sent headers with `END_HEADERS`. 249 250### `h2q_oflow_txcr` 251 252Set this to `true` if the peer server has the quirk it sends an maximum initial tx credit 253of 0x7fffffff and then later increments it illegally. 254 255### `http_multipart_name` 256 257Indicates this stream goes out using multipart mime, and provides the name part of the 258multipart header 259 260### `http_multipart_filename` 261 262Indicates this stream goes out using multipart mime, and provides the filename part of the 263multipart header 264 265### `http_multipart_content_type` 266 267The `content-type` to mark up the multipart mime section with if present 268 269### `http_www_form_urlencoded` 270 271Indicate the data is sent in `x-www-form-urlencoded` form 272 273### `rideshare` 274 275For special cases where one logically separate stream travels with another when using this 276protocol. Eg, a single multipart mime transaction carries content from two or more streams. 277 278## ws transport 279 280### `ws_subprotocol` 281 282Name of the ws subprotocol to use. 283 284### `ws_binary` 285 286Use if the ws messages are binary 287 288## MQTT transport 289 290### `mqtt_topic` 291 292Set the topic this streamtype uses for writes 293 294### `mqtt_subscribe` 295 296Set the topic this streamtype subscribes to 297 298### `mqtt qos` 299 300Set the QOS level for this streamtype 301 302### `mqtt_keep_alive` 303 30416-bit number representing MQTT keep alive for the stream. 305 306This is applied at connection time... where different streams may bind to the 307same underlying MQTT connection, all the streams should have an identical 308setting for this. 309 310### `mqtt_clean_start` 311 312Set to true if the connection should use MQTT's "clean start" feature. 313 314This is applied at connection time... where different streams may bind to the 315same underlying MQTT connection, all the streams should have an identical 316setting for this. 317 318### `mqtt_will_topic` 319 320Set the topic of the connection's will message, if any (there is none by default). 321 322This is applied at connection time... where different streams may bind to the 323same underlying MQTT connection, all the streams should have an identical 324setting for this. 325 326### `mqtt_will_message` 327 328Set the content of the connect's will message, if any (there is none by default). 329 330This is applied at connection time... where different streams may bind to the 331same underlying MQTT connection, all the streams should have an identical 332setting for this. 333 334### `mqtt_will_qos` 335 336Set the QoS of the will message, if any (there is none by default). 337 338This is applied at connection time... where different streams may bind to the 339same underlying MQTT connection, all the streams should have an identical 340setting for this. 341 342### `mqtt_will_retain` 343 344Set to true if the connection should use MQTT's "will retain" feature, if there 345is a will message (there is none by default). 346 347This is applied at connection time... where different streams may bind to the 348same underlying MQTT connection, all the streams should have an identical 349setting for this. 350 351## Loading and using updated remote policy 352 353If the default, hardcoded policy includes a streamtype `fetch_policy`, 354during startup when lws_system reaches the POLICY state, lws will use 355a Secure Stream of type `fetch_policy` to download, parse and update 356the policy to use it. 357 358The secure-streams-proxy minimal example shows how this is done and 359fetches its real policy from warmcat.com at startup using the built-in 360one. 361 362## Stream serialization and proxying 363 364By default Secure Streams expects to make the outgoing connection described in 365the policy in the same process / thread, this suits the case where all the 366participating clients are in the same statically-linked image. 367 368In this case the `lws_ss_` apis are fulfilled locally by secure-streams.c and 369policy.c for policy lookups. 370 371However it also supports serialization, where the SS api can be streamed over 372another transport such as a Unix Domain Socket connection. This suits the case 373where the clients are actually in different processes in, eg, Linux or Android. 374 375In those cases, you run a proxy process (minimal-secure-streams-proxy) that 376listens on a Unix Domain Socket and is connected to by one or more other 377processes that pass their SS API activity to the proxy for fulfilment (or 378onward proxying). 379 380In this case the proxy uses secure-streams.c and policy.c as before to fulfil 381the inbound proxy streams, but uses secure-streams-serialize.c to serialize and 382deserialize the proxied SS API activity. The proxy clients define 383LWS_SS_USE_SSPC either very early in their sources before the includes, or on 384the compiler commandline... this causes the lws_ss_ apis to be replaced at 385preprocessor time with lws_sspc_ equivalents. These serialize the api action 386and pass it to the proxy over a Unix Domain Socket for fulfilment, the results 387and state changes etc are streamed over the Unix Domain Socket and presented to 388the application exactly the same as if it was being fulfilled locally. 389 390To demonstrate this, some minimal examples, eg, minimal-secure-streams and 391mimimal-secure-streams-avs build themselves both ways, once with direct SS API 392fulfilment and once with Unix Domain Socket proxying and -client appended on the 393executable name. To test the -client variants, run minimal-secure-streams-proxy 394on the same machine. 395 396## Complicated scenarios with secure streams proxy 397 398As mentioned above, Secure Streams has two modes, by default the application 399directly parses the policy and makes the outgoing connections itself. 400However when configured at cmake with 401 402``` 403-DLWS_WITH_SOCKS5=1 -DLWS_WITH_SECURE_STREAMS=1 -DLWS_WITH_SECURE_STREAMS_PROXY_API=1 -DLWS_WITH_MINIMAL_EXAMPLES=1 404``` 405 406and define `LWS_SS_USE_SSPC` when building the application, applications forward 407their network requests to a local or remote SS proxy for fulfilment... and only 408the SS proxy has the system policy. By default, the SS proxy is on the local 409machine and is connected to via a Unix Domain Socket, but tcp links are also 410possible. (Note the proxied traffic is not encrypyed by default.) 411 412Using the configuration above, the example SS applications are built two ways, 413once for direct connection fulfilment (eg, `./bin/lws-minimal-secure-streams`), 414and once with `LWS_SS_USE_SSPC` also defined so it connects via an SS proxy, 415(eg, `./bin/lws-minimal-secure-streams-client`). 416 417## Testing an example scenario with SS Proxy and socks5 proxy 418 419``` 420 [ SS application ] --- tcp --- [ socks 5 proxy ] --- tcp --- [ SS proxy ] --- internet 421``` 422 423In this scenario, everything is on localhost, the socks5 proxy listens on :1337 and 424the SS proxy listens on :1234. The SS application connects to the socks5 425proxy to get to the SS proxy, which then goes out to the internet 426 427### 1 Start the SS proxy 428 429Tell it to listen on lo interface on port 1234 430 431``` 432$ ./bin/lws-minimal-secure-streams-proxy -p 1234 -i lo 433``` 434 435### 2 Start the SOCKS5 proxy 436 437``` 438$ ssh -D 1337 -N -v localhost 439``` 440 441The -v makes connections to the proxy visible in the terminal for testing 442 443### 3 Run the SS application 444 445The application is told to make all connections via the socks5 proxy at 446127.0.0.1:1337, and to fulfil its SS connections via an SS proxy, binding 447connections to 127.0.0.1 (ipv4 lo interface, -1), to 127.0.0.1:1234 (-a/-p). 448 449``` 450socks_proxy=127.0.0.1:1337 ./bin/lws-minimal-secure-streams-client -p 1234 -i 127.0.0.1 -a 127.0.0.1 451``` 452 453You can confirm this goes through the ssh socks5 proxy to get to the SS proxy 454and fulfil the connection. 455