poolboy是一个erlang编写的线程池框架,代码很简洁,项目地址:git://github.com/devinus/poolboy.git
具体用法,项目上有对应范例。
现在查看一下源码
首先看看初始化部分:
init({PoolArgs, WorkerArgs}) -> process_flag(trap_exit, true), Waiting = queue:new(), Monitors = ets:new(monitors, [private]), init(PoolArgs, WorkerArgs, #state{waiting = Waiting, monitors = Monitors}). init([{worker_module, Mod} | Rest], WorkerArgs, State) when is_atom(Mod) -> {ok, Sup} = poolboy_sup:start_link(Mod, WorkerArgs), init(Rest, WorkerArgs, State#state{supervisor = Sup}); init([{size, Size} | Rest], WorkerArgs, State) when is_integer(Size) -> init(Rest, WorkerArgs, State#state{size = Size}); init([{max_overflow, MaxOverflow} | Rest], WorkerArgs, State) when is_integer(MaxOverflow) -> init(Rest, WorkerArgs, State#state{max_overflow = MaxOverflow}); init([_ | Rest], WorkerArgs, State) -> init(Rest, WorkerArgs, State); init([], _WorkerArgs, #state{size = Size, supervisor = Sup} = State) -> Workers = prepopulate(Size, Sup), {ok, State#state{workers = Workers}}. prepopulate(N, _Sup) when N < 1 -> queue:new(); prepopulate(N, Sup) -> prepopulate(N, Sup, queue:new()). prepopulate(0, _Sup, Workers) -> Workers; prepopulate(N, Sup, Workers) -> prepopulate(N-1, Sup, queue:in(new_worker(Sup), Workers)).
初始化两个队列:
1.workers队列,进程池中闲置可选择的进程队列。
2.waiting队列,当我们用阻塞方式调用checkout进程时,而这时已经没有进程可以选择时,则进入此队列中,当有进程归还时,进行处理。
初始化一个pool_sup作为supervisor,来挂接管理进程池中的每一个进程。
初始化一个私有的ets,保存调用checkout的进程,以便归还,或异常结束时候作为依据进行的资源处理。
link进程池中的每一个子进程。
接下来,看看checkout的操作,选择闲置进程的操作:
handle_call({checkout, Block, Timeout}, {FromPid, _} = From, State) -> #state{supervisor = Sup, workers = Workers, monitors = Monitors, overflow = Overflow, max_overflow = MaxOverflow} = State, case queue:out(Workers) of {{value, Pid}, Left} -> Ref = erlang:monitor(process, FromPid), true = ets:insert(Monitors, {Pid, Ref}), {reply, Pid, State#state{workers = Left}}; {empty, Empty} when MaxOverflow > 0, Overflow < MaxOverflow -> {Pid, Ref} = new_worker(Sup, FromPid), true = ets:insert(Monitors, {Pid, Ref}), {reply, Pid, State#state{workers = Empty, overflow = Overflow + 1}}; {empty, Empty} when Block =:= false -> {reply, full, State#state{workers = Empty}}; {empty, Empty} -> Waiting = add_waiting(From, Timeout, State#state.waiting), {noreply, State#state{workers = Empty, waiting = Waiting}} end;
进程A调用checkout池中的闲置进程时,
若workers队列有闲置进程可以选择,从workers队列选取进程, 同时,对进程A进行监视。
若队列已为空,
1. 则根据maxoverflow的限制,确定是否可以创建新的扩展进程,若没超过上限,则添加新的进程。
2. 若已经不能进行扩展,则按照调用checkout方式进行处理,
阻塞调用时候,否则,将进程A进入waiting队列,
非阻塞调用时候,直接返回full。
不管是否使用了新创建的扩展进程,如果checkout成功的时候,poolboy都会对进程A进行监视,并以{Pid, Ref}的形式插入私有ets中。以便于处理在尚没有归还进程,就出现进程A结束的情况。
再看看checking,归还进程的操作:
handle_cast({checkin, Pid}, State = #state{monitors = Monitors}) -> case ets:lookup(Monitors, Pid) of [{Pid, Ref}] -> true = erlang:demonitor(Ref), true = ets:delete(Monitors, Pid), NewState = handle_checkin(Pid, State), {noreply, NewState}; [] -> {noreply, State} end; handle_checkin(Pid, State) -> #state{supervisor = Sup, waiting = Waiting, monitors = Monitors, overflow = Overflow} = State, case queue:out(Waiting) of {{value, {{FromPid, _} = From, Timeout, StartTime}}, Left} -> case wait_valid(StartTime, Timeout) of true -> Ref1 = erlang:monitor(process, FromPid), true = ets:insert(Monitors, {Pid, Ref1}), gen_server:reply(From, Pid), State#state{waiting = Left}; false -> handle_checkin(Pid, State#state{waiting = Left}) end; {empty, Empty} when Overflow > 0 -> ok = dismiss_worker(Sup, Pid), State#state{waiting = Empty, overflow = Overflow - 1}; {empty, Empty} -> Workers = queue:in(Pid, State#state.workers), State#state{workers = Workers, waiting = Empty, overflow = 0} end.
假设进程A调用的checkout方法,则在checkin时候,则不再对进程A进行监视,同时销毁在ets的记录。
归还进程时候:
首先查看waiting队列是否为空,若有任务正在阻塞中,则取出,查看响应时间是否超时,若没有则返回进程,
若waiting队列为空,如果进程池仍然有扩展进程,则销毁该进程。以致进程池数量回复到正常的数量,否则,直接置入workers队列。
接下来,我们看看,当进程池中进程结束或者,或者正在调用checkout的进程结束时的处理:
handle_info({'DOWN', Ref, _, _, _}, State) -> case ets:match(State#state.monitors, {'$1', Ref}) of [[Pid]] -> Sup = State#state.supervisor, ok = supervisor:terminate_child(Sup, Pid), %% Don't wait for the EXIT message to come in. %% Deal with the worker exit right now to avoid %% a race condition with messages waiting in the %% mailbox. true = ets:delete(State#state.monitors, Pid), NewState = handle_worker_exit(Pid, State), {noreply, NewState}; [] -> {noreply, State} end; handle_info({'EXIT', Pid, _Reason}, State) -> #state{supervisor = Sup, monitors = Monitors} = State, case ets:lookup(Monitors, Pid) of [{Pid, Ref}] -> true = erlang:demonitor(Ref), true = ets:delete(Monitors, Pid), NewState = handle_worker_exit(Pid, State), {noreply, NewState}; [] -> case queue:member(Pid, State#state.workers) of true -> W = queue:filter(fun (P) -> P =/= Pid end, State#state.workers), {noreply, State#state{workers = queue:in(new_worker(Sup), W)}}; false -> {noreply, State} end end; handle_worker_exit(Pid, State) -> #state{supervisor = Sup, monitors = Monitors, overflow = Overflow} = State, case queue:out(State#state.waiting) of {{value, {{FromPid, _} = From, Timeout, StartTime}}, LeftWaiting} -> case wait_valid(StartTime, Timeout) of true -> MonitorRef = erlang:monitor(process, FromPid), NewWorker = new_worker(State#state.supervisor), true = ets:insert(Monitors, {NewWorker, MonitorRef}), gen_server:reply(From, NewWorker), State#state{waiting = LeftWaiting}; _ -> handle_worker_exit(Pid, State#state{waiting = LeftWaiting}) end; {empty, Empty} when Overflow > 0 -> State#state{overflow = Overflow - 1, waiting = Empty}; {empty, Empty} -> Workers = queue:in( new_worker(Sup), queue:filter(fun (P) -> P =/= Pid end, State#state.workers) ), State#state{workers = Workers, waiting = Empty} end.
我们知道,进程池中的每个进程都已经link,所以当我们监听到'EXIT'消息,
首先判断该进程的状态
1. checkout中,则对ets中记录等进行销毁,
2. 尚在闲置中,则直接从workers进行移除。
在此之后创建一个新进程放入workers队列中。
当监听到一个持有闲置进程的进程结束时候,而尚未来的及向池checkin时, 除了停止supervisor管理的进程,
同时销毁私有ets中的checkout记录,清理对该进程的监视。