Lines Matching +full:run +full:- +full:parallel
1 # -*- coding: utf-8 -*-
3 # Use of this source code is governed by a BSD-style license that can be
43 resulting in temporary directories (pymp-xxx) not being cleaned
86 # many top-level paths in /tmp (see crbug.com/945523).
88 # calls parallel first, and some other user calls it later.
89 tmp_dir = '/tmp/chromite.parallel.%d' % os.geteuid()
97 # when the enclosing with-statement exits.
125 """Run a task in the background.
127 This task may be the 'Run' function from a buildbot stage or just a plain
128 function. It will be run in the background. Output from this task is saved
152 # How long we allow debug commands to run (so we don't hang will trying to
164 steps that are run in the background. This can be used to limit the
165 number of simultaneous parallel tasks.
168 task: The task (a functor) to run in the background.
188 # TODO(davidjames): Use python-2.7 syntax to simplify this.
195 """Swallow any exception run raises.
206 return cros_build_lib.run(cmd, **kwargs).output
211 # Debug commands to run in gdb. A class member so tests can stub it out.
215 'thread apply all py-list',
216 'thread apply all py-bt',
226 ('pstree', '-Apals', pid),
227 ('lsof', '-p', pid),
231 log_output=True, encoding='utf-8')
235 cmd = ('gdb', '--nx', '-q', '-p', pid, '-ex', 'set prompt',)
237 log_output=True, input=stdin, encoding='utf-8')
257 ('pgrep', '-P', ppid), debug_level=log_level, print_cmd=False,
387 tmp_dir = '/tmp/chromite.parallel.%d' % os.geteuid()
390 delete=False, dir=tmp_dir, prefix='chromite-parallel-')
394 def run(self): member in _BackgroundTask
395 """Run the list of steps."""
415 logging.error('pre-kill notification (SIGXCPU); traceback:\n%s',
447 cls.SILENT_TIMEOUT -= cls.SILENT_TIMEOUT_STEP
499 time_left = end_time - time.time()
502 task = bg_tasks[-1]
511 """Run a list of functions in parallel.
517 they were run in sequence.
520 them after all parallel tasks have finished running. Further, a
524 steps: A list of functions to run.
525 max_parallel: The maximum number of simultaneous tasks to run in parallel.
526 By default, run all tasks in parallel.
567 # contextlib ignores caught exceptions unless explicitly re-raised.
575 """Run task(*input) for each input in the queue.
578 If exceptions occur, save them off and re-raise them as a
582 queue: A queue of tasks to run. Add tasks to this queue, and they will
583 be run.
584 task: Function to run on each queued input.
585 onexit: Function to run after all inputs are processed.
617 # Run exit handlers.
628 """Run a list of functions in parallel.
633 they were run in sequence.
636 them after all parallel tasks have finished running. Further, a
640 # This snippet will execute in parallel:
649 steps: A list of functions to run.
650 max_parallel: The maximum number of simultaneous tasks to run in parallel.
651 By default, run all tasks in parallel.
697 """Run the specified task on each queued input in a pool of processes.
701 workers run task(*args + *input, **kwargs). Note that certain kwargs will
706 as if the tasks were run in sequence.
709 them after all parallel tasks have finished running. Further, a
713 # This will run somefunc(1, 'small', 'cow', foo='bar') in the background
726 task: Function to run on each queued input.
727 queue: A queue of tasks to run. Add tasks to this queue, and they will
728 be run in the background. If None, one will be created on the fly.
730 onexit: Function to run in each background process after all inputs are
764 """Run the specified function with each supplied input in a pool of processes.
771 as if the tasks were run in sequence.
774 them after all parallel tasks have finished running. Further, a
778 # This snippet will execute in parallel:
793 task: Function to run on each input.
796 onexit: Function to run in each background process after all inputs are
803 # - Use >=16 processes by default, in case it's a network-bound operation.
804 # - Try to use all of the CPUs, in case it's a CPU-bound operation.
812 # Micro-optimization: Setup the queue so that BackgroundTaskRunner