Lines Matching full:the

3 This document gives an overview of the browser's lower-layers for networking.
5 Networking in the browser ranges from high level Javascript APIs like
6 `fetch()`, all the way down to writing encrypted bytes on a socket.
8 This document assumes that requests for URLs are mediated through the browser's
9 [Network Service](../../services/network/README.md), and focuses on all the
10 layers below the Network Service, including key points of integration.
12 It's particularly targeted at people new to the Chrome network stack, but
13 should also be useful for team members who may be experts at some parts of the
15 through how a basic request issued by another process works its way through the
22 # Anatomy of the Network Stack
24 The network stack is located in //net/ in the Chrome repo, and uses the
26 generally be assumed it's in //net/ and is in the net namespace.
28 The top-level network stack object is the URLRequestContext. The context has
29 non-owning pointers to everything needed to create and issue a URLRequest. The
33 The primary use of the URLRequestContext is to create URLRequest objects using
34 URLRequestContext::CreateRequest(). The URLRequest is the main interface used
35 by direct consumers of the network stack. It manages loading URLs with the
42 The HttpNetworkSession is another major network stack object. It owns the
43 HttpStreamFactory, the socket pools, and the HTTP/2 and QUIC session pools. It
44 also has non-owning pointers to the network stack objects that more directly
48 above the HttpStreamFactory, objects often grab their dependencies from the
49 URLRequestContext, while the HttpStreamFactory and layers below it generally
50 get their dependencies from the HttpNetworkSession.
55 A URLRequest informs the consumer of important events for a request using two
56 main interfaces: the URLRequest::Delegate interface and the NetworkDelegate
59 The URLRequest::Delegate interface consists of a small set of callbacks needed
60 to let the embedder drive a request forward. The NetworkDelegate is an object
61 pointed to by the URLRequestContext and shared by all requests, and includes
62 callbacks corresponding to most of the URLRequest::Delegate's callbacks, as
65 # The Network Service and Mojo
67 The network service, which lives in //services/network/, wraps //net/ objects,
68 and provides cross-process network APIs and their implementations for the rest
69 of Chrome. The network service uses the namespace "network" for all its classes.
70 The Mojo interfaces it provides are in the network::mojom namespace. Mojo is
72 proxy object in the consumer's process which also implements
73 the network::mojom::Foo interface. When the proxy object's methods are invoked,
74 it passes the call and all its arguments over a Mojo IPC channel, using a
75 `mojo::Receiver<network::mojom::Foo>`, to an implementation of the
76 network::mojom::Foo interface in the network service (the implementation is
78 another thread in the consumer's process, or even the same thread in the
81 The network::NetworkService object is singleton that is used by Chrome to create
82 all other network service objects. The primary objects it is used to create are
83 the network::NetworkContexts, each of which owns its own mostly independent
86 types of requests, depending on what's making the request. Here are the main
89 * The system NetworkContext, created and owned by Chrome's
98 the browser process, and created by a Profile's ProfileNetworkContextService.
100 installed on that Profile. As with the main NetworkContext, these may have
101 on-disk data, depending on the Profile and the App.
107 a network::URLLoader in the network service (which, on desktop platform, is
108 typically in its own process). The URLLoader then creates a URLRequest to
109 drive the network request. That job first checks the HTTP cache, and then
110 creates a network transaction object, if necessary, to actually fetch the data.
112 it creates a new one. Once it has established a connection, the HTTP request is
113 dispatched, the response read and parsed, and the result returned back up the
114 stack and sent over to the caller.
118 Consider a simple request issued by some process other than the network
119 service's process. Suppose it's an HTTP request, the response is uncompressed,
120 no matching entry in the cache, and there are no idle sockets connected to the
121 server in the socket pool.
130 * In the browser process, the network::mojom::NetworkContext interface is used
132 * A consumer (e.g. the content::ResourceDispatcher for Blink, the
135 network::mojom::URLLoaderClient Mojo channel to the
138 * Mojo sends the network::ResourceRequest over an IPC pipe to a
139 network::URLLoaderFactory in the network process.
144 themselves, apart from the network service (Which either runs in its own
145 process, or potentially in the browser process to conserve RAM). There are
146 multiple types of child processes (renderer, GPU, plugin, network, etc). The
147 renderer processes are the ones that layout webpages and run HTML.
149 The browser process creates the top level network::mojom::NetworkContext
150 objects. The NetworkContext interface is privileged and can only be accessed
151 from the browser process. The browser process uses it to create
153 privileged processes to allow them to load resources using the NetworkContext.
155 is passed to the NetworkContext to configure fields that other processes are
158 One such field is the net::IsolationInfo field, which includes:
159 * A net::NetworkIsolationKey, which is used to enforce the
161 in the network stack, separating network resources used by different sites in
165 accessible when that site is the top-level site.
168 A consumer, either in the browser process or a child process, that wants to
169 make a network request gets a URLLoaderFactory from the browser process through
170 some manner, assembles a bunch of parameters in the large
172 channel for the network::mojom::URLLoader to use to talk back to it, and then
173 passes them all to the URLLoaderFactory, which returns a URLLoader object that
174 it can use to manage the network request.
176 ### network::URLLoaderFactory sets up the request in the network service
181 * network::URLLoader uses the network::NetworkContext's URLRequestContext to
184 The network::URLLoaderFactory, along with all NetworkContexts and most of the
185 network stack, lives on a single thread in the network service. It gets a
186 reconstituted ResourceRequest object from the network::mojom::URLLoaderFactory
187 Mojo pipe, does some checks to make sure it can service the request, and if so,
188 creates a URLLoader, passing the request and the NetworkContext associated with
189 the URLLoaderFactory.
191 The URLLoader then calls into the NetworkContext's net::URLRequestContext to
192 create the URLRequest. The URLRequestContext has pointers to all the network
193 stack objects needed to issue the request over the network, such as the cache,
194 cookie store, and host resolver. The URLLoader then calls into the
195 network::ResourceScheduler, which may delay starting the request, based on
196 priority and other activity. Eventually, the ResourceScheduler starts the
199 ### Check the cache, request an HttpStream
203 * The URLRequest asks the URLRequestJobFactory to create a URLRequestJob,
205 * The URLRequestHttpJob asks the HttpCache to create an HttpTransaction, and
207 * The HttpCache::Transaction sees there's no cache entry for the request,
209 * The HttpNetworkTransaction calls into the HttpStreamFactory to request an
212 The URLRequest then calls into the URLRequestJobFactory to create a
214 (historically, non-network URL schemes were also disptched through the
215 network stack, so there were a variety of job types.) The
216 URLRequestHttpJob attaches cookies to the request, if needed. Whether or
217 not SameSite cookies are attached depends on the IsolationInfo's
218 SiteForCookies, the URL, and the URLRequest's request_initiator field.
220 The URLRequestHttpJob calls into the HttpCache to create an
221 HttpCache::Transaction. The cache checks for an entry with the same URL
222 and NetworkIsolationKey. If there's no matching entry, the
223 HttpCache::Transaction will call into the HttpNetworkLayer to create an
224 HttpNetworkTransaction, and transparently wrap it. The HttpNetworkTransaction
225 then calls into the HttpStreamFactory to request an HttpStream to the server.
232 * HttpStreamFactory::Job calls into the TransportClientSocketPool to
237 * TransportClientSocketPool puts the StreamSocket in the ClientSocketHandle,
240 ownership of the ClientSocketHandle.
241 * It returns the HttpBasicStream to the HttpNetworkTransaction.
243 The HttpStreamFactory::Job creates a ClientSocketHandle to hold a socket,
244 once connected, and passes it into the ClientSocketPoolManager. The
245 ClientSocketPoolManager assembles the TransportSocketParams needed to
246 establish the connection and creates a group name ("host:port") used to
249 The ClientSocketPoolManager directs the request to the
250 TransportClientSocketPool, since there's no proxy and it's an HTTP request. The
251 request is forwarded to the pool's ClientSocketPoolBase<TransportSocketParams>'s
253 are available socket slots, the ClientSocketPoolBaseHelper will create a new
254 TransportConnectJob using the aforementioned params object. This Job will do the
255 actual DNS lookup by calling into the HostResolverImpl, if needed, and then
258 Once the socket is connected, ownership of the socket is passed to the
259 ClientSocketHandle. The HttpStreamFactory::Job is then informed the
261 takes ownership of the ClientSocketHandle. It then passes ownership of the
262 HttpBasicStream back to the HttpNetworkTransaction.
264 ### Send request and read the response headers
268 * HttpNetworkTransaction gives the request headers to the HttpBasicStream,
269 and tells it to start the request.
270 * HttpBasicStream sends the request, and waits for the response.
271 * The HttpBasicStream sends the response headers back to the
273 * The response headers are sent up through the URLRequest, to the
275 * They're then sent to the network::mojom::URLLoaderClient via Mojo.
277 The HttpNetworkTransaction passes the request headers to the HttpBasicStream,
278 which uses an HttpStreamParser to (finally) format the request headers and body
279 (if present) and send them to the server.
281 The HttpStreamParser waits to receive the response and then parses the HTTP/1.x
282 response headers, and then passes them up through both the
283 HttpNetworkTransaction and HttpCache::Transaction to the URLRequestHttpJob. The
284 URLRequestHttpJob saves any cookies, if needed, and then passes the headers up
285 to the URLRequest and on to the network::URLLoader, which sends the data over
286 a Mojo pipe to the network::mojom::URLLoaderClient, passed in to the URLLoader
293 * network::URLLoader creates a raw Mojo data pipe, and passes one end to the
295 * The URLLoader requests shared memory buffer from the Mojo data pipe.
296 * The URLLoader tells the URLRequest to write to the memory buffer, and tells
297 the pipe when data has been written to the buffer.
298 * The last two steps repeat until the request is complete.
300 Without waiting to hear back from the network::mojom::URLLoaderClient, the
301 network::URLLoader allocates a raw mojo data pipe, and passes the client the
302 read end of the pipe. The URLLoader then grabs an IPC buffer from the pipe,
303 and passes a 64KB body read request down through the URLRequest all the way
304 down to the HttpStreamParser. Once some data is read, possibly less than 64KB,
305 the number of bytes read makes its way back to the URLLoader, which then tells
306 the Mojo pipe the read was complete, and then requests another buffer from the
307 pipe, to continue writing data to. The pipe may apply back pressure, to limit
308 the amount of unconsumed data that can be in shared memory buffers at once.
309 This process repeats until the response body is completely read.
315 * When complete, the network::URLLoaderFactory deletes the network::URLLoader,
316 which deletes the URLRequest.
317 * During destruction, the HttpNetworkTransaction determines if the socket is
318 reusable, and if so, tells the HttpBasicStream to return it to the socket pool.
320 When the URLRequest informs the network::URLLoader the request is complete, the
321 URLLoader passes the message along to the network::mojom::URLLoaderClient, over
322 its Mojo pipe, before telling the URLLoaderFactory to destroy the URLLoader,
323 which results in destroying the URLRequest and closing all Mojo pipes related to
324 the request.
326 When the HttpNetworkTransaction is being torn down, it figures out if the
327 socket is reusable. If not, it tells the HttpBasicStream to close the socket.
328 Either way, the ClientSocketHandle returns the socket is then returned to the
329 socket pool, either for reuse or so the socket pool knows it has another free
334 A sample of the object relationships involved in the above process is
339 There are a couple of points in the above diagram that do not come
342 * The method that generates the filter chain that is hung off the
343   URLRequestJob is declared on URLRequestJob, but the only current
344   implementation of it is on URLRequestHttpJob, so the generation is
355 The HttpCache::Transaction sits between the URLRequestHttpJob and the
356 HttpNetworkTransaction, and implements the HttpTransaction interface, just like
357 the HttpNetworkTransaction. The HttpCache::Transaction checks if a request can
358 be served out of the cache. If a request needs to be revalidated, it handles
359 sending a conditional revalidation request over the network. It may also break a
363 The HttpCache::Transaction uses one of three disk_cache::Backends to actually
364 store the cache's index and files: The in memory backend, the blockfile cache
365 backend, and the simple cache backend. The first is used in incognito. The
368 One important detail is that it has a read/write lock for each URL. The lock
370 always grabs the lock for writing and reading before downgrading it to a read
371 only lock, all requests for the same URL are effectively done serially. The
372 renderer process merges requests for the same URL in many cases, which mitigates
376 cache, which has no relation to the cache implemented in net/, which lives in
377 the network service.
381 A consumer can cancel a request at any time by deleting the
382 network::mojom::URLLoader pipe used by the request. This will cause the
386 figures out if the socket the HttpStream owns can potentially be reused, based
387 on the protocol (HTTP / HTTP/2 / QUIC) and any received headers. If the socket
389 read any remaining body bytes of the HttpStream, if any, before returning the
390 socket to the SocketPool. If this takes too long, or there's an error, the
391 socket is closed instead. Since this all happens at the layer below the cache,
392 any drained bytes are not written to the cache, and as far as the cache layer is
397 The URLRequestHttpJob checks if headers indicate a redirect when it receives
398 them from the next layer down (typically the HttpCache::Transaction). If they
399 indicate a redirect, it tells the cache the response is complete, ignoring the
400 body, so the cache only has the headers. The cache then treats it as a complete
401 entry, even if the headers indicated there will be a body.
403 The URLRequestHttpJob then checks with the URLRequest if the redirect should be
404 followed. The URLRequest then informs the network::URLLoader about the redirect,
405 which passes information about the redirect to network::mojom::URLLoaderClient,
406 in the consumer process. Whatever issued the original request then checks
407 if the redirect should be followed.
409 If the redirect should be followed, the URLLoaderClient calls back into the
410 URLLoader over the network::mojom::URLLoader Mojo interface, which tells the
411 URLRequest to follow the redirect. The URLRequest then creates a new
412 URLRequestJob to send the new request. If the URLLoaderClient chooses to
413 cancel the request instead, it can delete the network::mojom::URLLoader
414 pipe, just like the cancellation case discussed above. In either case, the
415 old HttpTransaction is destroyed, and the HttpNetworkTransaction attempts to
416 drain the socket for reuse, as discussed in the previous section.
418 In some cases, the consumer may choose to handle a redirect itself, like
419 passing off the redirect to a ServiceWorker. In this case, the consumer cancels
420 the request and then calls into some other network::mojom::URLLoaderFactory
421 with the new URL to continue the request.
425 When the URLRequestHttpJob receives headers, it sends a list of all
427 chain of filters. As body bytes are received, they're passed through the
428 filters at the URLRequestJob layer and the decoded bytes are passed back to the
431 Since this is done above the cache layer, the cache stores the responses prior
432 to decompression. As a result, if files aren't compressed over the wire, they
433 aren't compressed in the cache, either.
437 The ClientSocketPoolManager is responsible for assembling the parameters needed
438 to connect a socket, and then sending the request to the right socket pool.
440 ClientSocketHandle, and a "group name". The params object contains all the
442 different types of socket pools take different params types. The
444 return it to the socket pool when done. All connections with the same group name
445 in the same pool can be used to service the same connection requests, so it
447 sockets in the goup.
449 All socket pool classes derive from the ClientSocketPoolBase<SocketParamType>.
450 The ClientSocketPoolBase handles managing sockets - which requests to create
454 which establishes a connection using the socket params, before the pool hands
455 out the connected socket.
462 example, each socket in the SSLClientSocketPool is layered on top of a socket
463 in the TransportClientSocketPool. There are a couple additional complexities
466 From the perspective of the lower layer pool, all of its sockets that a higher
467 layer pools owns are actively in use, even when the higher layer pool considers
472 Since sockets in the higher layer pool are also in a group in the lower layer
474 instance, SSL and HTTP connections won't be grouped together in the
475 TcpClientSocketPool, which the SSLClientSocketPool sits on top of.
479 The relationships between the important classes in the socket pools is
480 shown diagrammatically for the lowest layer socket pool
485 The ClientSocketPoolBase is a template class templatized on the class
486 containing the parameters for the appropriate type of socket (in this
487 case TransportSocketParams). It contains a pointer to the
488 ClientSocketPoolBaseHelper, which contains all the type-independent
489 machinery of the socket pool.
494 ClientSocketPoolBase::ConnectJobFactory templatized by the same type
495 as the ClientSocketPoolBase. (In the case of the diagram above, that
500 passed to the initialization of the ClientSocketPoolBaseHelper. This
501 allows the helper to create connect jobs while preserving a type-safe
502 API to the initialization of the socket pool.
506 When an SSL connection is needed, the ClientSocketPoolManager assembles the
507 parameters needed both to connect the TCP socket and establish an SSL
508 connection. It then passes them to the SSLClientSocketPool, which creates
509 an SSLConnectJob using them. The SSLConnectJob's first step is to call into the
512 Once a connection is established by the lower layered pool, the SSLConnectJob
513 then starts SSL negotiation. Once that's done, the SSL socket is passed back to
514 the HttpStreamFactory::Job that initiated the request, and things proceed
515 just as with HTTP. When complete, the socket is returned to the
523 SSLSocketPool between the the HttpProxyClientSocketPool and the
524 TransportSocketPool, since they can talk SSL to both the proxy and the
527 The first step the HttpStreamFactory::Job performs, just before calling
528 into the ClientSocketPoolManager to create a socket, is to pass the URL to the
530 for that URL. Then when the ClientSocketPoolManager tries to get a socket for
531 the Job, it uses that list of proxies to direct the request to the right socket
538 HTTP/2 negotation is performed as part of the SSL handshake, so when
540 as well. When it gets a socket with HTTP/2 negotiated as well, the Job creates a
541 SpdySession using the socket and a SpdyHttpStream on top of the SpdySession.
542 The SpdyHttpStream will be passed to the HttpNetworkTransaction, which drives
543 the stream as usual.
545 The SpdySession will be shared with other Jobs connecting to the same server,
546 and future Jobs will find the SpdySession before they try to create a
559 server that has advertised QUIC support in the past, it will create a second
561 The two Jobs (one for QUIC, one for all versions of HTTP) will be raced against
565 other Jobs connecting to the same server, and future Jobs will just reuse the
573 * The ResourceScheduler lives outside net/, and in some cases, delays starting
575 * DNS lookups are initiated based on the highest priority request for a lookup.
577 when a socket becomes idle, it will be assigned to the highest priority request
578 for the server it's connected to, even if there's a higher priority request to
580 * HTTP/2 and QUIC both support sending priorities over-the-wire.
582 At the socket pool layer, sockets are only assigned to socket requests once the
584 a higher priority request for a group reaches the socket pool before a
585 connection is established, the first usable connection goes to the highest
591 upgrade header. Once the handshake completes successfully, the connection
592 is used as a full-duplex communication channel to the server for WebSocket
594 own Mojo interfaces and //net classes, but internally they reuse the full
595 URLRequest machinery up to the point headers are received from the server.
596 Then the connection is handed off to the WebSocket code to manage.
599 is not part of the network service. Standard schemes like file:// and blob://
600 are handled by the content layer and its dependencies
603 chrome-extension:// are often handled by a URLLoaderFactory in the chrome layer,
607 process requests a data:// subresource, the renderer typically decodes it
609 Navigations are a bit different. To navigate to a URL, the browser process
610 creates a URLLoader and passes it over to a renderer process. So in the
612 content::DataURLLoaderFactory that lives in the browser process, and then a
613 mojo::Remote for the browser-hosted URLLoader is passed to a renderer
616 about:blank is similarly often handled in the renderer, though there is a
617 factory for that used in the case of navigations as well. Other about: URLs
618 are mapped to the corresponding Chrome URLs by the navigation code, rather