�
{ fc @ sT d g Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z m Z m Z d d l
m Z m Z d Z
d Z d Z e j � Z d � Z d e f d
� � YZ e f e d � Z d e f d � � YZ d
e f d � � YZ e Z d e f d � � YZ d e f d � � YZ d e f d � � YZ d e f d � � YZ d S( t Pooli����N( t Processt cpu_countt TimeoutError( t Finalizet debugi i i c C s
t | � S( N( t map( t args( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt mapstar@ s t MaybeEncodingErrorc B s) e Z d Z d � Z d � Z d � Z RS( sV Wraps possible unpickleable errors, so they can be
safely sent through the socket.c C sA t | � | _ t | � | _ t t | � j | j | j � d S( N( t reprt exct valuet superR t __init__( t selfR R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR K s c C s d | j | j f S( Ns( Error sending result: '%s'. Reason: '%s'( R R ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt __str__P s c C s d t | � S( Ns <MaybeEncodingError: %s>( t str( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt __repr__T s ( t __name__t
__module__t __doc__R R R ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR G s c C s� | j } | j } t | d � r> | j j � | j j � n | d k rW | | � n d } xI| d k s~ | r�| | k r�y
| � } Wn" t t f k
r� t d � Pn X| d k r� t d � Pn | \ } }
} } }
y t
| | |
� f } Wn t k
r} t | f } n Xy | | |
| f � WnM t k
r�} t
| | d � } t d | � | | |
t | f f � n Xd } } } } } }
| d 7} q` Wt d | � d S( Nt _writeri s) worker got EOFError or IOError -- exitings worker got sentinel -- exitingi s0 Possible encoding error while sending result: %ss worker exiting after %d tasks( t putt gett hasattrR t closet _readert Nonet EOFErrort IOErrorR t Truet Exceptiont FalseR ( t inqueuet outqueuet initializert initargst maxtasksR R t completedt taskt jobt it funcR t kwdst resultt et wrapped( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt workerX s@
!
c B s e Z d Z e Z d d d d d � Z d � Z d � Z d � Z d � Z d i d � Z
d d � Z d d � Z d d
� Z
d i d d � Z d d d � Z e d
� � Z e d � � Z e d � � Z e d � � Z d � Z d � Z d � Z d � Z e d � � Z e d � � Z RS( sH
Class which supports an async version of the `apply()` builtin
c C s/ | j � t j � | _ i | _ t | _ | | _ | | _ | | _ | d k r| y
t
� } Wq| t k
rx d } q| Xn | d k r� t d � � n | d k r� t
| d � r� t d � � n | | _ g | _ | j � t j d t j d | f � | _ t | j _ t | j _ | j j � t j d t j d | j | j | j | j | j f � | _ t | j _ t | j _ | j j � t j d t j d | j | j | j f � | _ t | j _ t | j _ | j j � t! | | j" d | j | j# | j | j | j | j | j | j f d d �| _$ d S( Ni s& Number of processes must be at least 1t __call__s initializer must be a callablet targetR t exitpriorityi (% t
_setup_queuest Queuet
_taskqueuet _cachet RUNt _statet _maxtasksperchildt _initializert _initargsR R t NotImplementedErrort
ValueErrorR t TypeErrort
_processest _poolt _repopulate_poolt threadingt ThreadR t _handle_workerst _worker_handlerR t daemont startt
_handle_taskst
_quick_putt _outqueuet
_task_handlert _handle_resultst
_quick_gett _result_handlerR t _terminate_poolt _inqueuet
_terminate( R t processesR$ R% t maxtasksperchild( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR � sX
c C sw t } xj t t t | j � � � D]M } | j | } | j d k r" t d | � | j � t } | j | =q" q" W| S( s� Cleanup after any worker processes which have exited due to reaching
their specified lifetime. Returns True if any workers were cleaned up.
s cleaning up worker %dN(
R! t reversedt ranget lenRA t exitcodeR R t joinR ( R t cleanedR* R0 ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt _join_exited_workers� s "
c
C s� x� t | j t | j � � D]� } | j d t d | j | j | j | j | j
f � } | j j | � | j j
d d � | _ t | _ | j � t d � q Wd S( s� Bring the number of pool processes up to the specified number,
for use after reaping workers which have exited.
R2 R R t
PoolWorkers added workerN( RV R@ RW RA R R0 RQ RK R; R< R: t appendt namet replaceR RG RH R ( R R* t w( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRB � s #
c C s | j � r | j � n d S( sE Clean up any exited workers and start replacements for them.
N( R[ RB ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt _maintain_pool� s c C sP d d l m } | � | _ | � | _ | j j j | _ | j j j | _ d S( Ni ( t SimpleQueue(
t queuesRb RQ RK R t sendRJ R t recvRN ( R Rb ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR4 � s
c C s | j | | | � j � S( s1
Equivalent of `apply()` builtin
( t apply_asyncR ( R R+ R R, ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt apply� s c C s | j | | | � j � S( s/
Equivalent of `map()` builtin
( t map_asyncR ( R R+ t iterablet chunksize( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR � s i c s� | d k rQ t | j � � | j j � � f d � t | � D� � j f � � St j � | | � } t | j � � | j j � f d � t | � D� � j f � d � � D� Sd S( sZ
Equivalent of `itertools.imap()` -- can be MUCH slower than `Pool.map()`
i c 3 s0 | ]& \ } } � j | � | f i f Vq d S( N( t _job( t .0R* t x( R+ R- ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr> s c 3 s0 | ]& \ } } � j | t | f i f Vq d S( N( Rk R ( Rl R* Rm ( R- ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr>
s c s s" | ] } | D] } | Vq
q d S( N( ( Rl t chunkt item( ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr> s N( t IMapIteratorR7 R6 R t enumeratet _set_lengthR t
_get_tasks( R R+ Ri Rj t task_batches( ( R+ R- s, /usr/lib64/python2.7/multiprocessing/pool.pyt imap� s c s� | d k rQ t | j � � | j j � � f d � t | � D� � j f � � St j � | | � } t | j � � | j j � f d � t | � D� � j f � d � � D� Sd S( sK
Like `imap()` method but ordering of results is arbitrary
i c 3 s0 | ]& \ } } � j | � | f i f Vq d S( N( Rk ( Rl R* Rm ( R+ R- ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr> s c 3 s0 | ]& \ } } � j | t | f i f Vq d S( N( Rk R ( Rl R* Rm ( R- ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr> s c s s" | ] } | D] } | Vq
q d S( N( ( Rl Rn Ro ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr>! s N( t IMapUnorderedIteratorR7 R6 R Rq Rr R Rs ( R R+ Ri Rj Rt ( ( R+ R- s, /usr/lib64/python2.7/multiprocessing/pool.pyt imap_unordered s c C sA t | j | � } | j j | j d | | | f g d f � | S( s>
Asynchronous equivalent of `apply()` builtin
N( t ApplyResultR7 R6 R Rk R ( R R+ R R, t callbackR- ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRf # s +c s� t | d � s t | � } n | d k rh t t | � t | j � d � \ } } | rh | d 7} qh n t | � d k r� d } n t j | | | � } t | j | t | � | � � | j
j � f d � t | � D� d f � � S( s<
Asynchronous equivalent of `map()` builtin
t __len__i i i c 3 s0 | ]& \ } } � j | t | f i f Vq d S( N( Rk R ( Rl R* Rm ( R- ( s, /usr/lib64/python2.7/multiprocessing/pool.pys <genexpr>= s N(
R t listR t divmodRW RA R Rs t MapResultR7 R6 R Rq ( R R+ Ri Rj Ry t extraRt ( ( R- s, /usr/lib64/python2.7/multiprocessing/pool.pyRh , s ( c C so t j � } xB | j t k s6 | j rP | j t k rP | j � t j d � q W| j j
d � t d � d S( Ng�������?s worker handler exiting(
RC t current_threadR9 R8 R7 t TERMINATERa t timet sleepR6 R R R ( t poolt thread( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRE A s *
c C s
t j � } x�t | j d � D]q\ } } d } d } zGy� x� t | � D]� \ } } | j rm t d � Pn y | | � WqJ t k
r� }
| d \ } } y | | j | t
|
f � Wq� t k
r� q� XqJ XqJ W| r� t d � | | d � n w PWn} t k
r|}
| r | d n d \ } } | | k rX| | j | d t
|
f � n | r}t d � | | d � q}n XWd d } } } Xq Wt d � y@ t d � | j d � t d � x | D] } | d � q�WWn t
k
r�t d
� n Xt d � d S(
Ni����s' task handler found thread._state != RUNi s doing set_length()i i s task handler got sentinels/ task handler sending sentinel to result handlers( task handler sending sentinel to workerss/ task handler got IOError when sending sentinelss task handler exiting( i i ( RC R t iterR R Rq R9 R R t _setR! t KeyErrorR R ( t taskqueueR R# R� t cacheR� t taskseqt
set_lengthR( R* R. R) t indt ext p( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRI N sR
!
c C s� t j � } x� y
| � } Wn" t t f k
r@ t d � d SX| j rX t d � Pn | d k rr t d � Pn | \ } } } y | | j | | � Wn t k
r� n Xd } } } q Wx� | rq| j t k rqy
| � } Wn" t t f k
rt d � d SX| d k r%t d � q� n | \ } } } y | | j | | � Wn t k
r_n Xd } } } q� Wt
| d � r�t d � y5 x. t d � D] } | j j
� s�Pn | � q�WWq�t t f k
r�q�Xn t d t | � | j � d S( Ns. result handler got EOFError/IOError -- exitings, result handler found thread._state=TERMINATEs result handler got sentinels& result handler ignoring extra sentinelR s" ensuring that outqueue is not fulli
s7 result handler exiting: len(cache)=%s, thread._state=%s( RC R R R R R9 R R� R� R� R RV R t pollRW ( R# R R� R� R( R) R* t obj( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRM � sZ
c c sD t | � } x1 t t j | | � � } | s1 d S| | f Vq Wd S( N( R� t tuplet itertoolst islice( R+ t itt sizeRm ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRs � s c C s t d � � d S( Ns: pool objects cannot be passed between processes or pickled( R= ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt
__reduce__� s c C s5 t d � | j t k r1 t | _ t | j _ n d S( Ns closing pool( R R9 R8 t CLOSERF ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR � s
c C s- t d � t | _ t | j _ | j � d S( Ns terminating pool( R R� R9 RF RR ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt terminate� s
c C sS t d � | j j � | j j � | j j � x | j D] } | j � q; Wd S( Ns joining pool( R RF RY RL RO RA ( R R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRY � s
c C sW t d � | j j � x9 | j � rR | j j � rR | j j � t j d � q Wd S( Ns7 removing tasks from inqueue until task handler finishedi ( R t _rlockt acquiret is_aliveR R� Re R� R� ( R" t task_handlerR� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt _help_stuff_finish� s
c
C s� t d � t | _ t | _ t d � | j | | t | � � t | _ | j d � t d � t j � | k r� | j d � n | r� t
| d d � r� t d � x- | D]" } | j d k r� | j � q� q� Wn t d � t j � | k r | j d � n t d � t j � | k r,| j d � n | r�t
| d d � r�t d
� x; | D]0 } | j
� rVt d | j � | j � qVqVWn d S( Ns finalizing pools& helping task handler/workers to finishs joining worker handlerg}Ô%�I�Ti R� s terminating workerss joining task handlers joining result handlers joining pool workerss cleaning up worker %d( R R� R9 R� RW R R RC R RY R RX R� R� t pid(
t clsR� R" R# R� t worker_handlerR� t result_handlerR� R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRP � s6
N( ( ( ( R R R R R R R[ RB Ra R4 Rg R Ru Rw Rf Rh t staticmethodRE RI RM Rs R� R R� RY R� t classmethodRP ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR � s0 :
4< Rx c B sD e Z d � Z d � Z d � Z d d � Z d d � Z d � Z RS( c C sS t j t j � � | _ t j � | _ | | _ t | _ | | _
| | | j <d S( N( RC t Conditiont Lockt _condt job_countert nextRk R7 R! t _readyt _callback( R R� Ry ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR s c C s | j S( N( R� ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt ready& s c C s | j S( N( t _success( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt
successful) s c C sB | j j � z | j s, | j j | � n Wd | j j � Xd S( N( R� R� R� t waitt release( R t timeout( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� - s
c C s< | j | � | j s t � n | j r/ | j S| j � d S( N( R� R� R R� t _value( R R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR 5 s
c C s� | \ | _ | _ | j r7 | j r7 | j | j � n | j j � z t | _ | j j � Wd | j j � X| j | j
=d S( N( R� R� R� R� R� R R� t notifyR� R7 Rk ( R R* R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� > s
N( R R R R� R� R R� R R� ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRx s R} c B s e Z d � Z d � Z RS( c C s t j | | | � t | _ d g | | _ | | _ | d k r` d | _ t | _ | | j =n | | t
| | � | _ d S( Ni ( Rx R R R� R R� t
_chunksizet _number_leftR� Rk t bool( R R� Rj t lengthRy ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR R s
c C s | \ } } | r� | | j | | j | d | j +| j d 8_ | j d k r| j rn | j | j � n | j | j =| j j � z t | _ | j j
� Wd | j j � XqnW t | _
| | _ | j | j =| j j � z t | _ | j j
� Wd | j j � Xd S( Ni i ( R� R� R� R� R7 Rk R� R� R R� R� R� R! R� ( R R* t success_resultt successR- ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� ^ s* "
( R R R R� ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR} P s Rp c B s> e Z d � Z d � Z d d � Z e Z d � Z d � Z RS( c C sk t j t j � � | _ t j � | _ | | _ t j � | _
d | _ d | _
i | _ | | | j <d S( Ni ( RC R� R� R� R� R� Rk R7 t collectionst dequet _itemst _indexR t _lengtht _unsorted( R R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR s c C s | S( N( ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt __iter__� s c C s� | j j � z� y | j j � } Wn� t k
r� | j | j k rN t � n | j j | � y | j j � } Wq� t k
r� | j | j k r� t � n t � q� Xn XWd | j j
� X| \ } } | r� | S| � d S( N( R� R� R� t popleftt
IndexErrorR� R� t
StopIterationR� R R� ( R R� Ro R� R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� � s&
c C s� | j j � z� | j | k r� | j j | � | j d 7_ xJ | j | j k r� | j j | j � } | j j | � | j d 7_ qA W| j j � n
| | j | <| j | j k r� | j | j
=n Wd | j j � Xd S( Ni ( R� R� R� R� R] R� t popR� R� R7 Rk R� ( R R* R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� � s
c C s^ | j j � z< | | _ | j | j k rH | j j � | j | j =n Wd | j j � Xd S( N( R� R� R� R� R� R7 Rk R� ( R R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRr � s
N( R R R R� R R� t __next__R� Rr ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRp } s
Rv c B s e Z d � Z RS( c C st | j j � zR | j j | � | j d 7_ | j j � | j | j k r^ | j | j =n Wd | j j � Xd S( Ni (
R� R� R� R] R� R� R� R7 Rk R� ( R R* R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� � s
( R R R� ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyRv � s t
ThreadPoolc B sB e Z d d l m Z d d d d � Z d � Z e d � � Z RS( i ( R c C s t j | | | | � d S( N( R R ( R RS R$ R% ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR � s c C s@ t j � | _ t j � | _ | j j | _ | j j | _ d S( N( R5 RQ RK R RJ R RN ( R ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR4 � s c C sW | j j � z5 | j j � | j j d g | � | j j � Wd | j j � Xd S( N( t not_emptyR� t queuet cleart extendR t
notify_allR� ( R" R� R� ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� � s
N( ( R R t dummyR R R R4 R� R� ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyR� � s ( t __all__RC R5 R� R� R� t multiprocessingR R R t multiprocessing.utilR R R8 R� R� t countR� R R R R R0 t objectR Rx t AsyncResultR} Rp Rv R� ( ( ( s, /usr/lib64/python2.7/multiprocessing/pool.pyt <module># s, ,� �.-I
All system for education purposes only. For more tools: Telegram @jackleet