Lines Matching +full:storage +full:- +full:repo +full:- +full:branch +full:- +full:coverage
3 This CI is used on-top of (not in replacement of) AOSP's TreeHugger.
4 It gives early testing signals and coverage on other OSes and older Android
12 
32 It uses a non-privileged gmail account and has no meaningful voting power.
36 - It periodically (every 5s) polls Gerrit for CLs updated in the last 24h.
37 - It checks the list of CLs against the list of already known CLs in the DB.
38 - For each new CL it enqueues `N` new jobs in the database, one for each
39 configuration defined in [config.py](/infra/ci/config.py) (e.g. `linux-debug`,
40 `android-release`, ...).
41 - It monitors the state of jobs. When all jobs for a CL have been completed,
42 it posts a comment and adds the vote if the CL is marked as `Presubmit-Ready`.
43 - It does some other less-relevant bookkeeping.
44 - AppEngine is highly reliable and self-healing. If a task fails (e.g. because
45 of a Gerrit 500) it will be automatically re-tried with exponential backoff.
53 - It's an almost fully static website based on HTML and Javascript.
54 - The only backend-side code ([frontend.py](/infra/ci/frontend/frontend.py))
57 - Such XHR requests are GET-only and anonymous.
58 - The frontend python code also serves as a memcache layer for Gerrit requests
65 The GCE instance is running a CrOS-based
66 [Container-Optimized](https://cloud.google.com/container-optimized-os/docs/) OS.
68 The whole system image is read-only. The VM itself is stateless. No state is
69 persisted outside of the DB and Google Cloud Storage (only for UI artifacts).
78 associated. Workers are always alive (they work in polling-mode), while
79 sandboxes are started and stopped by the worker on-demand.
95 - Poll for an available job from the `/jobs_queued` sub-tree of the DB.
96 - Move such job into `/jobs_running`.
97 - Start the sandbox container, passing down the job config and the git revision
99 - Stream the sandbox stdout to the `/logs` sub-tree of the DB.
100 - Terminate the sandbox container prematurely in case of timeouts or job
102 - Upload UI artifacts to GCS.
103 - Update the DB to reflect completion of jobs, removing the entry from
109 (for git pull / install-build-deps) but they cannot impersonate the GCE service
116 - Checkout the code at the revision specified in the job config.
117 - Run one of the [test/ci/](/test/ci/) scripts which will build and run tests.
118 - Return either a success (0) or fail (!= 0) exit code.
121 the semi-ephemeral `/ci/cache` mount-point. This mount-point is tmpfs-based
129 It is a high-scale JSON object accessible via a simple REST API.
130 Clients can GET/PUT/PATCH/DELETE individual sub-nodes without having a local
131 full-copy of the DB.
135 # For post-submit jobs.
137 /master-20190626000853
138 # ┃ ┗━ Committer-date of the HEAD of the branch.
139 # ┗━ Branch name
143 subject: "Cloud-based CI"
144 time_committed: "2019-07-06T02:35:14Z"
147 20190708153242--branches-master-20190626000853--android-...: 0
148 20190708153242--branches-master-20190626000853--linux-...: 0
152 /master-20190701235742 {...}
154 # For pre-submit jobs.
156 /1000515-65
159 time_queued: "2019-07-08T15:32:42Z"
160 time_ended: "2019-07-08T15:33:25Z"
165 20190708153242--cls-1000515-65--android-clang: 0
167 20190708153242--cls-1000515-65--ui-clang: 0
170 /1000515-66 {...}
172 /1011130-3 {...}
178 /1000515-65: 0
181 /20190708153242--cls-1000515-65--android-clang-arm-debug:
183 # ┃ ┗━ Path of the CL or branch object.
186 src: "cls/1000515-66"
194 time_ended: "2019-07-07T12:47:22Z"
195 time_queued: "2019-07-07T12:34:22Z"
196 time_started: "2019-07-07T12:34:25Z"
197 type: "android-clang-arm-debug"
198 worker: "zqz2-worker-2"
200 /20190707123422--cls-1000515-66--android-clang-arm-rel {..}
205 /20190708153242--cls-1000515-65--android-clang-arm-debug: 0
210 /20190707123422--cls-1000515-66--android-clang-arm-rel
213 /20190707123422--cls-1000515-66--android-clang-arm-rel
214 /00a053-0000: "+ chmod 777 /ci/cache /ci/artifacts"
218 # ┗━ Hex-encoded timestamp, relative since start of test.
219 /00a053-0001: "+ chown perfetto.perfetto /ci/ramdisk"
229 make -C /infra/ci worker-start
232 [GCE] # From /infra/ci/worker/gce-startup-script.sh
233 docker run worker-1 ...
235 docker run worker-N ...
237 [worker-X] # From /infra/ci/worker/Dockerfile
239 ┗━ docker run sandbox-X ...
241 [sandbox-X] # From /infra/ci/sandbox/Dockerfile
258 ### [gce-startup-script.sh](/infra/ci/worker/gce-startup-script.sh)
260 - Is ran once per GVE vm, at (re)boot.
261 - It prepares the tmpfs mountpoint for the shared ccache.
262 - It wipes the SSD scratch disk for the build artifacts
263 - It pulls the latest {worker, sandbox} container images from
265 - Sets up Docker and `iptables` (for the sandboxed network).
266 - Starts `N` worker containers in Docker.
270 - It polls the DB to retrieve a job.
271 - When a job is retrieved starts a sandbox container.
272 - It streams the container stdout/stderr to the DB.
273 - It upload the build artifacts to GCS.
277 - It is pinned in the container image. Does NOT depend on the particular
279 - Checks out the repo at the revision specified (by the Controller) in the
281 - Sets up ccache
282 - Deals with caching of buildtools/.
283 - Runs the test script specified in the job config from the checkout.
287 - Are NOT pinned in the container and are ran from the checked out revision.
288 - Finally build and run the test.
294 Test-locally: `make -C infra/ci/frontend test`
296 Deploy with `make -C infra/ci/frontend deploy`
300 Deploy with `make -C infra/ci/controller deploy`
302 It is possible to try locally via the `make -C infra/ci/controller test`
305 - Manually stopping the production AppEngine instance via the Cloud Console
307 - Downloading the testing service credentials `test-credentials.json`
314 `make -C infra/ci build push`
318 `make -C infra/ci restart-workers`
323 - Stop the workers: `make -C infra/ci stop-workers`
324 - Open https://console.firebase.google.com/u/0/project/perfetto-ci/database/perfetto-ci/data/~2Fci
325 - Delete the `jobs_running`, `jobs_queued`, `workers` subtrees
326 - Restart the workers: `make -C infra/ci start-workers`
330 - Both the Firebase DB and the gs://perfetto-artifacts GCS bucket are
331 world-readable and writable by the GAE and GCE service accounts.
333 - The GAE service account also has the ability to log into Gerrit using a
336 - Overall, no account in this project has any interesting privilege:
337 - The Gerrit account used for commenting on CLs is just a random gmail account
339 - The service accounts of GAE and GCE don't have any special capabilities
342 - This CI deals only with functional and performance testing and doesn't deal
345 - Presubmit jobs are only triggered if at least one of the following is true:
346 - The owner of the CL is a @google.com account.
347 - The user that applied the Presubmit-Ready label is a @google.com account.
349 - Sandboxes are not too hard to escape (Docker is the only boundary) and can
352 - As such neither pre-submit nor post-submit build artifacts are considered
356 - Binaries built by the CI are not ran on any other machines outside of the
359 - The only build artifacts that are retained (for up to 30 days) and uploaded to
363 - UI artifacts are served from a different origin (the GCS per-bucket API) than