Thread in Python 学习笔记

    这段时间做IO的testing，发现如果我主进程P1里启动一个线程T1，T1里popen一个dd的命令，然后P1又继续popen一个sg_reset的命令，有小概率可能P1执行完popen之后会卡住不再继续。kernel的同事说kernel显示都schedule了，那问题可能就出在管道上，log显示的是wait for read。查看了代码说如果popen参数设置了subprocess.PIPE，如果输出太多可能导致卡住。可是我的dd命令是不会有输出的似乎，所以不太像这个原因。试过好种办法，效果都不太好，于是换了multiprocessing来跑dd。虽然效率差了，但是能保证UT的稳定性，也就作罢。现在有时间，研究一下python的thread。

首先把相关的module都列出来：
threading
multiprocessing
subprocess

multiprocessing-with-python-presentation是一个很好的slides对比。

看到了很多例子，由于GIL （global interpreter lock)，python thread里是可以用popen的，内核同事说C thread用popen是不对的：threads and fork-think twice before using them

锁

2.5 引入了with statement
pep-343阐述了大概原理。大意是 __enter__, __exit__完成了with的工作

• The expression is evaluated and should result in an object called a ``context manager''. The context manager must have __enter__() and __exit__() methods.
• The context manager's __enter__() method is called. The value returned is assigned to VAR. If no 'as VAR' clause is present, the value is simply discarded.
• The code in BLOCK is executed.
• If BLOCK raises an exception, the __exit__(type, value, traceback) is called with the exception details, the same values returned by sys.exc_info(). The method's return value controls whether the exception is re-raised: any false value re-raises the exception, and True will result in suppressing it. You'll only rarely want to suppress the exception, because if you do the author of the code containing the 'with' statement will never realize anything went wrong.
• If BLOCK didn't raise an exception, the __exit__() method is still called, but type, value, and traceback are all None.

这里可以看到，不管有没有异常，__exit__会帮们处理好，所以

lock = Threading.Lock()
with lock:
    statements

这样的片段是exception安全的，不会因为异常而没有release锁。

Pool:

Pool是一个进程池，可以指定若干个 worker进程来做事情，也分为sync和async的版本，同时可以获得返回值来做一些同步。需要注意的是，在调用pool.close()之后，pool就结束了，如果不小心把pool.close()放在了一个for循环里，或者在apply()之前调用了，比如

def func(msg):
    for i in xrange(3):
        print "func:"+msg
        time.sleep(1)
    return "hello func %s" % msg

if __name__ == "__main__":
    pool = multiprocessing.Pool(processes=2)
    result = []
    for i in xrange(3):
        msg = "hello %d" %(i)
        result.append(pool.apply_async(func, [msg]))
        pool.close()   <--- done.="" es="" for="" in="" obj.get="" obj="" pool.join="" pre="" print="" result:="" tab="" timeout="1)" ub-process="">
会得到一个断言异常：AssertionError: assert self._state == RUN


正确写法是：
def func(msg):
    for i in xrange(3):
        print "func:"+msg
        time.sleep(1)
    return "hello func %s" % msg

if __name__ == "__main__":
    pool = multiprocessing.Pool(processes=2)
    result = []
    for i in xrange(3):
        msg = "hello %d" %(i)
        result.append(pool.apply_async(func, [msg]))
    pool.close()
    pool.join()
    print "Sub-process(es) done."
    for obj in result:
        print obj.get(timeout=1)


数据访问和共享