1[/ 2 Copyright Oliver Kowalke 2016. 3 Distributed under the Boost Software License, Version 1.0. 4 (See accompanying file LICENSE_1_0.txt or copy at 5 http://www.boost.org/LICENSE_1_0.txt 6] 7 8[/ import path is relative to this .qbk file] 9[import ../examples/work_sharing.cpp] 10 11[#migration] 12[section:migration Migrating fibers between threads] 13 14[heading Overview] 15 16Each fiber owns a stack and manages its execution state, including all 17registers and CPU flags, the instruction pointer and the stack pointer. That 18means, in general, a fiber is not bound to a specific thread.[footnote The 19["main] fiber on each thread, that is, the fiber on which the thread is 20launched, cannot migrate to any other thread. Also __boost_fiber__ implicitly 21creates a dispatcher fiber for each thread [mdash] this cannot migrate 22either.][superscript,][footnote Of course it would be problematic to migrate a 23fiber that relies on [link thread_local_storage thread-local storage].] 24 25Migrating a fiber from a logical CPU with heavy workload to another 26logical CPU with a lighter workload might speed up the overall execution. 27Note that in the case of NUMA-architectures, it is not always advisable to 28migrate data between threads. Suppose fiber ['f] is running on logical CPU 29['cpu0] which belongs to NUMA node ['node0]. The data of ['f] are allocated on 30the physical memory located at ['node0]. Migrating the fiber from ['cpu0] to 31another logical CPU ['cpuX] which is part of a different NUMA node ['nodeX] 32might reduce the performance of the application due to increased latency of 33memory access. 34 35Only fibers that are contained in __algo__[s] ready queue can migrate between 36threads. You cannot migrate a running fiber, nor one that is __blocked__. You 37cannot migrate a fiber if its [member_link context..is_context] method returns 38`true` for `pinned_context`. 39 40In __boost_fiber__ a fiber is migrated by invoking __context_detach__ on the 41thread from which the fiber migrates and __context_attach__ on the thread to 42which the fiber migrates. 43 44Thus, fiber migration is accomplished by sharing state between instances of a 45user-coded __algo__ implementation running on different threads. The fiber[s] 46original thread calls [member_link algorithm..awakened], passing the 47fiber[s] [class_link context][^*]. The `awakened()` implementation calls 48__context_detach__. 49 50At some later point, when the same or a different thread calls [member_link 51algorithm..pick_next], the `pick_next()` implementation selects a ready 52fiber and calls __context_attach__ on it before returning it. 53 54As stated above, a `context` for which `is_context(pinned_context) == true` 55must never be passed to either __context_detach__ or __context_attach__. It 56may only be returned from `pick_next()` called by the ['same] thread that 57passed that context to `awakened()`. 58 59[heading Example of work sharing] 60 61In the example [@../../examples/work_sharing.cpp work_sharing.cpp] 62multiple worker fibers are created on the main thread. Each fiber gets a 63character as parameter at construction. This character is printed out ten times. 64Between each iteration the fiber calls __yield__. That puts the fiber in the 65ready queue of the fiber-scheduler ['shared_ready_queue], running in the current 66thread. 67The next fiber ready to be executed is dequeued from the shared ready queue 68and resumed by ['shared_ready_queue] running on ['any participating thread]. 69 70All instances of ['shared_ready_queue] share one global concurrent queue, used 71as ready queue. This mechanism shares all worker fibers between all instances 72of ['shared_ready_queue], thus between all participating threads. 73 74 75[heading Setup of threads and fibers] 76 77In `main()` the fiber-scheduler is installed and the worker fibers and the 78threads are launched. 79 80[main_ws] 81 82The start of the threads is synchronized with a barrier. The main fiber of 83each thread (including main thread) is suspended until all worker fibers are 84complete. When the main fiber returns from __cond_wait__, the thread 85terminates: the main thread joins all other threads. 86 87[thread_fn_ws] 88 89Each worker fiber executes function `whatevah()` with character `me` as 90parameter. The fiber yields in a loop and prints out a message if it was migrated 91to another thread. 92 93[fiber_fn_ws] 94 95 96[heading Scheduling fibers] 97 98The fiber scheduler `shared_ready_queue` is like `round_robin`, except that it 99shares a common ready queue among all participating threads. A thread 100participates in this pool by executing [function_link use_scheduling_algorithm] 101before any other __boost_fiber__ operation. 102 103The important point about the ready queue is that it[s] a class static, common 104to all instances of shared_ready_queue. 105Fibers that are enqueued via __algo_awakened__ (fibers that are ready to be 106resumed) are thus available to all threads. 107It is required to reserve a separate, scheduler-specific queue for the thread[s] 108main fiber and dispatcher fibers: these may ['not] be shared between threads! 109When we[,]re passed either of these fibers, push it there instead of in the 110shared queue: it would be Bad News for thread B to retrieve and attempt to 111execute thread A[s] main fiber. 112 113[awakened_ws] 114 115When __algo_pick_next__ gets called inside one thread, a fiber is dequeued from 116['rqueue_] and will be resumed in that thread. 117 118[pick_next_ws] 119 120 121The source code above is found in 122[@../../examples/work_sharing.cpp work_sharing.cpp]. 123 124[endsect] 125