This document was last updated on 2008-02-17.
CL-MUPROC is a Common Lisp library which strives to offer some of the multiprocessing abstractions found in the Erlang programming language. In CL-MUPROC
At this point, CL-MUPROC works on Lispworks, OpenMCL, SBCL, CMUCL, and Allegro Common Lisp, but we're certainly interested in patches adding support for other implementaions.
Please report any problems you encounter with CL-MUPROC!
Although CL-MUPROC has been developed with distributed operation in mind, at this point all muprocs in a CL-MUPROC based system must execute within the same Lisp image.
CL-MUPROC is available under a BSD license.
If you are interested in using or contributing to CL-MUPROC, consider joining the cl-muproc-devel at common-lisp.net mailing list. If you're in getting just announcements relating to CL-MUPROC you may wish to join the cl-muproc-announce at common-lisp.net mailing list instead.
This document is the only available documentation of CL-MUPROC at this time, as you will see below, it is still quite incomplete. In addition, a presentation held by Klaus Harbo at the European Common Lisp Meeting in Hamburg, April 2006 provides an overview of some of CL-MUPROC's features.
The most recent release of CL-MUPROC can be downloaded as cl-muproc.tar.gz
Feel free to contact me if you have questions regarding cl-muproc.
Klaus Harbo <klaus at harbo.net>
muproc-spawn creates a new muproc whose muproc-name will be name, by calling init-fn with the arguments in arglist. It is an error if a muproc already exists with the same name.
Every muproc must have an errorstream to which it may write log messages. If errorstream is given, the errorstream messages of the created muproc is written to that stream. Inside muprocs the errorstream is accessible via the *muproc-errorstream*, which has dynamic extent coinciding with the lifetime of the muproc. When a muproc is spawned by another muproc, and no errorstream argument is given to muproc-spawn, the errorstream is inherited from the 'parent' muproc.
Using the inport argument, the input port of a newly created muproc can be specified. This is only advisable under very special circumstances.
A set of bindings that exist at the start of the muproc can be given using the initial-bindings argument, which should be a alist whose car element is the name of variable to be bound, and whose cdr element is the value to which it should be bound. Since muprocs should not share mutable state, initial-bindings is often a convenient way to given muprocs the equivalent of special variables, because each muproc will thus have their own instance of the variable.
The link argument is used to control whether the spawning muproc and the spawned muproc is linked to each other. When to muprocs are linked, they are informed if the other muproc terminates (cf. the trap-exits flag and muproc-set-trap-exits to learn more about how the fact that a muproc has terminated is communicated to the muprocs to which it is linked). Iff link is NIL, the spawning and spawn muprocs are linked, otherwise they are not.
The trap-exits argument controls how the spawned muproc handles the termination of a muproc to which it is linked. By default, a muproc terminates if a muproc to which it is linked terminates. However, if a muproc traps exits, it is sent a special mumsg informing it of the termination of the linked muproc. This mumsg contains two fields, terminated is the pid of the terminated muproc and reason is the value the terminated muproc gave as argument to muproc-exit when it terminated. Sometimes the muproc run-time system decides to terminate a muproc, in which cas the terminated field in the termination message sent if the muproc is trapping exits is NIL.
EXAMPLE — in which a muproc is spawned, which spawns a child to which it is linked, and with which it 'co-terminates' due to the linkage,
MUPROC> (muproc-spawn :spawn-test (lambda (max) (flet ((child () (unwind-protect (loop (muproc-log-errorstream "Still here.~%") (sleep 1)) (muproc-log-errorstream "Done.~%")))) (muproc-spawn :child #'child nil :link t) (loop repeat max with start = (get-universal-time) for count from 1 for since = (- (get-universal-time) start) do (muproc-log-errorstream "Count=~d since=~d~%" count since) do (sleep 2)) (muproc-log-errorstream "Done.~%"))) (list 3) :errorstream *trace-output*) 20:09:34.879 SPAWN-TEST Count=1 since=0 20:09:34.884 CHILD Still here. #<MP:PROCESS Name "muproc-6[:SPAWN-TEST]" Priority 850000 State "Running"> 20:09:35.888 CHILD Still here. 20:09:36.884 SPAWN-TEST Count=2 since=2 20:09:36.892 CHILD Still here. 20:09:37.896 CHILD Still here. 20:09:38.888 SPAWN-TEST Count=3 since=4 20:09:38.900 CHILD Still here. 20:09:39.900 CHILD Still here. 20:09:40.892 SPAWN-TEST Done. 20:09:40.893 CHILD Done. MUPROC>
SEE ALSO — muproc-exit, muproc-kill
muproc-exit is used to terminate a muproc, causing muproc to terminate with reason as the exit value. The default muproc to terminate is the muproc calling muproc-exit. An error occurs if muproc is not a muproc.
EXAMPLE 1 — in which a muproc is spawn, which spawns a child to which it is linked, and from which it receives a termination message which the child decides to terminate,
MUPROC> (muproc-spawn :exit-test (lambda () (flet ((child () (unwind-protect (progn (sleep 2) (muproc-exit :i-am-done)) (muproc-log-errorstream "Done.~%")))) (unwind-protect (progn (muproc-spawn :child #'child nil :link t) (mumsg-receive (from) ((terminated reason) t (muproc-log-errorstream "Got termination from ~a: ~a." terminated reason)))) (muproc-log-errorstream "Done.~%")))) nil :trap-exits t :errorstream *trace-output*) #<MP:PROCESS Name "muproc-8[:EXIT-TEST]" Priority 850000 State "Running"> 20:28:23.260 CHILD Done. 20:28:23.263 EXIT-TEST Got termination from #<MP:PROCESS Name :DEAD-PROCESS Priority 0 State "Dead">: I-AM-DONE. 20:28:23.267 EXIT-TEST Done. MUPROC>
EXAMPLE 2 — in which a muproc is spawned, which spawns a child that it decides to terminate, and from which it receives a termination message,
MUPROC> (muproc-spawn :exit-test (lambda () (flet ((child () (unwind-protect (loop (muproc-log-errorstream "Still here.") (sleep 1)) (muproc-log-errorstream "Done.~%")))) (unwind-protect (let ((child (muproc-spawn :child #'child nil :link t))) (sleep 3) (muproc-exit :you-are-done child) (mumsg-receive (from) ((terminated reason) t (muproc-log-errorstream "Got termination from ~a: ~a." terminated reason)))) (muproc-log-errorstream "Done.~%")))) nil :trap-exits t :errorstream *trace-output*) 20:31:25.236 CHILD Still here. #<MP:PROCESS Name "muproc-10[:EXIT-TEST]" Priority 850000 State "Running"> 20:31:26.244 CHILD Still here. 20:31:27.248 CHILD Still here. 20:31:28.236 CHILD Done. 20:31:28.237 EXIT-TEST Got termination from #<MP:PROCESS Name :DEAD-PROCESS Priority 0 State "Dead">: YOU-ARE-DONE. 20:31:28.237 EXIT-TEST Done. MUPROC>
SEE ALSO — muproc-kill, muproc-spawn
mumsg is used to build mumsg objects, which are used as the basis of muproc's pattern-matching message sending protocol. However, mumsg is not used very often, since mumsg-send a 'built-in' support for create mumsg objects 'on-the-fly'.
NOTE — The representation of mumsg objects is not part of the interface and may change — use only the functions exported from the muproc package to access mumsg field values.
EXAMPLE
MUPROC> (mumsg :a 1 :b 2) (:MUMSG (:A . 1) (:B . 2)) MUPROC>
SEE ALSO — mumsg-send, mumsg-receive
mumsg-send is used to send mumsg messages between muprocs (obviously), i.e. it is used for sending messages which are intended to be received using the message pattern matching performed by mumsg-receive. A mumsg object is constructed with the named fields and values found in the plist argument, and sent to to, which is either a pid or a registered (named) port.
NOTE — mumsg-send can only be used to send mumsg objects. To send other kinds of objects, use muproc-send, on top of which mumsg-send is built.
NOTE — cl-muproc is based on the notion that multiprocessing should be done using shared-nothing message-passing. However, since muprocs running in the same Lisp image exchange regular Lisp data structures, they are not – strictly speaking – shared-nothing, quite the contrary actually. The intention in cl-muproc, however, is that data objects exchanged between muprocs must be treated as immutable. That is, muproc applications should take care to either
EXAMPLE — in which two muprocs send messages back and forth to each other a specified number of times,
MUPROC> (labels ((out (&rest args) (apply #'muproc-log-errorstream args)) (pong () (loop (mumsg-receive (from) ((ping) t (out "PONG -- Got ping: ~a." ping) (mumsg-send from :pong (* 100 ping))) ((done) t (out "PONG -- Got done.") (muproc-exit :done))))) (ping (pong count) (loop for iter from 1 to count do (mumsg-send pong :ping iter) do (mumsg-receive (from) ((pong) t (out "PING -- Got pong: ~a." pong)))) (mumsg-send pong :done :dummy)) (ping-pong (count) (let ((pong (muproc-spawn 'pong #'pong () :errorstream *trace-output*))) (muproc-spawn 'ping #'ping (list pong count) :errorstream *trace-output*)))) (ping-pong 3)) 13:27:51.305 PONG PONG -- Got ping: 1. 13:27:51.305 PING PING -- Got pong: 100. 13:27:51.306 PONG PONG -- Got ping: 2. 13:27:51.306 PING PING -- Got pong: 200. 13:27:51.306 PONG PONG -- Got ping: 3. 13:27:51.307 PING PING -- Got pong: 300. 13:27:51.307 PONG PONG -- Got done. #<MP:PROCESS Name "muproc-17[PING]" Priority 850000 State "Running"> MUPROC>
SEE ALSO — mumsg-receive, with-registered-port, mumsg, muproc-send
mumsg-receive is used to receive messages using message pattern matching. Using mumsg-receive, it is possible to selectively look for packets in the muproc's input queue. This allows different kinds of inbound data to be processed at different times.
mumsg-receive processes the messages in the input queue is the order the messages arrived, trying to match each message in turn against the patterns specified in clauses, as described below.
mumsg-receive processes each message in the input queue, trying each clause in turn. For each clause, mumsg-receive looks for a field for each of the field names mentioned in the list-of-field-names. Iff a field is found in the message, the value of each field is bound to the field name, and the predicate is evaluated. Iff the predicate evaluates to a non-NIL value, the entire mumsg-receive form returns the result of evaluating the boyd (using progn). If the predicate evaluates to NIL, the next clause is tried. If a field is NOT found for each field name, the next clause is tried. If no clause can be found which can match the message, the next message is tried. If there are no more messages to trie, mumsg-receive block until a message arrives, at which point the matching process continues.
A mumsg-receive form has the following high-level structure
(mumsg-receive (from) . list-of-clauses)where each clause has the form
(list-of-field-names predicate . body-forms)
Looking at an example of a mumsg-receive form,
(mumsg-receive (from) ((field-a) (integerp field-a) :integer ((field-a) t :not-an-integer (() t :no-field-a))
The above mumsg-receive example has 3 clauses whose parts are
| Clause | Field names | Predicate | Body |
| 1 | field-a | (integerp field-a) | :integer |
| 2 | field-a | t | :not-an-integer |
| 3 | () | t | :no-field-a |
In the above, mumsg-receive will always match the first message in the input queue because the last clause requires no fields to be present and the predicate always returns t.
NOTE — Most of the time, the clause specifies all the fields necessary for performing message processing, but sometimes this is not possible, in which case the contents of the message can be accessed after message matching using muproc-get-field, *muproc-mumsg*, and/or *muproc-packet*.
NOTE — Use muproc-with-timeout if mumsg-receive should block for a limited period of time.
EXAMPLE — Cf. mumsg-send.
SEE ALSO — muproc-send, mumsg, muproc-with-timeout, muproc-discard-all-pending-input, muproc-get-field, *muproc-packet*, *muproc-mumsg*
Every muproc has a stream designated as it's 'errorstream'. Normally, muprocs should not write to their errorstream, but rather use application-specific ways of communicating messages to its environment. However, for debugging it is very useful to manage explicitly where to send strings.
muproc-log-errorstream is used to write strings to the muproc's errostream. format-string is a formatting string with the same sematics as those given to format, against which args are matched.
Note — Since errorstreams are mutable and usually shared between muprocs (by default, a muproc inherits the errorstream of its parent), a lock is employed to serialize access. As there is just on lock controlling access to all errorstreams, using muproc-log-errorstream can potentially cause performance loss. Therefore, muproc-log-errorstream should be used only sparingly.
SEE ALSO — *muproc-errorstream*
muproc-with-message-tag generates a 'statistically unique' message tag, and binds it to tag-name in body.
In general, messages may arrive in a muproc's input at any time. Very often, however, a muproc want to send some request to another muproc, wait for the reply to come back, and then proceed. In this situation, it is essential that the reply we get is the reply to the request we just sent, and not a reply we sent some time ago, but whose reply we never got to process (due to a timeout, perhaps, see NOTE below).
A common idiom for dealing with this issue is the use of unique message tags which a put into requests and returned by the muproc at the other end. By remembering the unique message tag, we can look for the reply to the request we just sent using mumsg-receive pattern matching. This is best illustrated by the following example:
(muproc-with-message-tag (mytag)
(mumsg-send some-muproc :the-request :bla :tag mytag)
(mumsg-receive (from)
((the-reply tag) (muproc-msgtag= mytag tag)
(list :reply-to-what-we-just-sent the-reply))
((the-reply tag) t
(list :reply-to-some-old-reply the-reply)))) NOTE — muprocs are responsible for 'keeping their input queues clean' themselves. That is, care must be taken that muprocs do not accumulate messages in their input queues because some kinds of messages are not matched by any mumsg-receive clause (or by muproc-receive if lower-level primitives are used). How this is done best, depends on the application (which is why cl-muproc does not attempt to deal with this issue). If care is not take to process any message which may arrive in a muproc, the queue will eventually fill up, which will lead to an error whivh causes the muproc to terminate. Very long input queues also affect performance, since pattern matching on input always starts with the oldest message in the queue.
SEE ALSO — muproc-msgtag=, mumsg-receive, mumsg-send
muproc-msgtag= compares to message tags and returns t iff they are identical.
Messages can be sent to muprocs using two kinds of 'addresses', either their pid or a registered port name, which must a Lisp keyword. By registering a port name, a muproc can ensure that other muprocs can send it messages easily, even if they do not know that muproc's pid.
The usual way to use a register a port name is using muproc-with-registered-port, which registers name while evaluating body.
EXAMPLE — in which one muproc registers a port name to which another sends a few times,
MUPROC> (labels ((done () (muproc-log-errorstream "DONE.~%")) (server (count) (unwind-protect ;; REGISTER PORT NAME -- (muproc-with-registered-port (:SERVER) (loop repeat count do (mumsg-receive (from) ((request) t (mumsg-send from :reply (* request 2)))))) (done))) (client (count) (unwind-protect (progn (sleep .1) ;; wait for server to be ready (loop for req from 1 repeat count do (progn ;; SEND TO REGISTERED PORT NAME -- (mumsg-send :SERVER :request req) (mumsg-receive (from) ((reply) t (muproc-log-errorstream "Reply(~a)=~a.~%" req reply)))))) (done))) (parent (count) (muproc-spawn :srv #'server (list count) :errorstream *trace-output*) (muproc-spawn :cli #'client (list count) :errorstream *trace-output*))) (parent 4)) #<MP:PROCESS Name "muproc-69[:CLI]" Priority 850000 State "Running"> 16:14:19.510 CLI Reply(1)=2. 16:14:19.511 CLI Reply(2)=4. 16:14:19.511 CLI Reply(3)=6. 16:14:19.512 SRV DONE. 16:14:19.512 CLI Reply(4)=8. 16:14:19.513 CLI DONE. MUPROC>
SEE ALSO — muproc-register-port-name, muproc-unregister-port-name
Evaluate body for a maximum of timeout seconds. If timeout occurs, return the results of evaluationg timeout-forms.
EXAMPLE —
MUPROC> (muproc-with-timeout (1 :timed-out) (progn (sleep 2) :done)) :TIMED-OUT MUPROC> (muproc-with-timeout (2 :timed-out) (progn (sleep 1) :done)) :DONE MUPROC>
Call function at universal-time. Iff repeat-period is given, repeat every repeat-period seconds.
NOTE — All timers created by a muproc are destroyed (unscheduled) when it terminates.
EXAMPLE — in which we see a timer scheduled using universal time repeating every 1 seconds, and we note that the timer created by the muproc is unscheduled upon its exit,
MUPROC> (muprocn (5 :muprocn-timed-out)
(let ((start (get-universal-time)))
(flet ((f () (muproc-log-errorstream "Time passed: ~d" (- (get-universal-time) start))))
(muproc-schedule #'f (+ start 2) 1)
(sleep 8)
(muproc-log-errorstream "Done."))))
19:38:43.365 MUPROCN-7587 Time passed: 2
19:38:44.365 MUPROCN-7587 Time passed: 3
19:38:45.365 MUPROCN-7587 Time passed: 4
19:38:46.366 MUPROCN-7587 Time passed: 5
:MUPROCN-TIMED-OUT
19:38:47.365 MUPROCN-7587 Time passed: 6
19:38:48.365 MUPROCN-7587 Time passed: 7
19:38:49.365 MUPROCN-7587 Time passed: 8
19:38:49.365 MUPROCN-7587 Done.
MUPROC>
SEE ALSO — muproc-schedule-relative, muproc-unschedule-timer
Call function after expiry-time seconds. Iff repeat-period is given, repeat every repeat-period seconds.
NOTE — All timers created by a muproc are destroyed (unscheduled) when it terminates.
EXAMPLE — in which we see a timer scheduled relatively repeating every 1 seconds, and we note that the timer created by the muproc is unscheduled upon its exit,
MUPROC> (muprocn (5 :muprocn-timed-out) (let ((start (get-universal-time))) (flet ((f () (muproc-log-errorstream "Time passed: ~d" (- (get-universal-time) start)))) (muproc-schedule-relative #'f 1 1) (sleep 8) (muproc-log-errorstream "Done.")))) 19:13:12.615 MUPROCN-5504 Time passed: 1 19:13:13.615 MUPROCN-5504 Time passed: 2 19:13:14.615 MUPROCN-5504 Time passed: 3 19:13:15.615 MUPROCN-5504 Time passed: 4 19:13:16.616 MUPROCN-5504 Time passed: 5 :MUPROCN-TIMED-OUT 19:13:17.615 MUPROCN-5504 Time passed: 6 19:13:18.615 MUPROCN-5504 Time passed: 7 19:13:19.615 MUPROCN-5504 Time passed: 8 19:13:19.615 MUPROCN-5504 Done. MUPROC>
SEE ALSO — muproc-schedule
Link muproc1 and muproc2 to each other. The default value of muproc2 is the muproc calling muproc-link. When two muprocs are linked, one process will be notified if the other terminates, and vice versa. Iff the muproc being notified traps exits, the termination notification is by sending a termination notification mumsg to the muproc, otherwise muproc-exit is called for the muproc.
A notification message has two fields: terminated whose value is the muproc which terminated, and reason which is the value the muproc gave upon exit.
NOTE — Muproc linking is a symmetric relationship between two muprocs. By contrast muproc monitoring is an asymmetric relationship, where one muproc is notified is the other terminates, but NOT vice versa (except, of course, if the other happens to be monitoring the former).
SEE ALSO — muproc-set-trap-exits, muproc-monitor
Let subject-muproc monitor object-muproc. When a muproc monitors another, it is notified if the monitored muproc terminates Iff the muproc being notified traps exits, the termination notification is by sending a termination notification mumsg to the muproc, otherwise muproc-exit is called for the muproc.
A notification message has two fields: terminated whose value is the muproc which terminated, and reason which is the value the muproc gave upon exit.
NOTE — Muproc monitoring is an asymmetric relationship, where one muproc is notified is the other terminates, but NOT vice versa (except, of course, if the other happens to be monitoring the former). By contrast muproc linking is a symmetric relationship between two muprocs.
SEE ALSO — muproc-set-trap-exits, muproc-link
Set whether muproc traps exits or not. The default value of muproc is the current muproc. muproc does NOT trap exits if new-value is NIL, and DOES trap exits otherwise.
When a muproc traps exits, it receives notification messages when linked or monitored muprocs terminated. Otherwise muproc-exit used for notification (which results in the termination of the linked or monitoring muproc also).
SEE ALSO — muproc-trap-exits-p, muproc-link, muproc-monitor, muproc-spawn
Returns the name of muproc. The default value of muproc is the current muproc.
NOTE — A muproc is given its name when it is created.
SEE ALSO — muproc-spawn
Returns a list of curredntly running muprocs.
SEE ALSO — muproc-find, muproc-spawn, muproc-kill
Searches current muprocs returning those which fullfil criteria, which is either the name of the muproc (a symbol), or a string representing a regular expression to be used for matching the names of current muprocs.
If as-list is NIL, it is an error for a search match more than one muproc, and muproc-find simply returns this process; otherwise muproc-find returns a list with the muprocs matching criteria.
NOTE 1 — Edi Weitz's CL-PPCRE library must be loaded before compiling CL-MUPROC for regular expression matching to work.
NOTE 2 — muproc-find should probably be made more flexible.
SEE ALSO — muproc-spawn
Terminate muproc with reason as exit reason, without giving it the opportunity to 'trap' the exit. That is, as opposed to muproc-exit which either causes a termination message or termination signal to be sent (depending on the value of muproc-trap-exits-p), muproc-kill termination is always signalled (i.e. cannot be trapped).
NOTE — muproc-kill is mostly appropriate under quite exceptional circumstances, e.g. when developing and debugging code. Under normal circumstances, muprocs' wishes to trap exits should probably be respected.
SEE ALSO — muproc-exit, muproc-set-trap-exits, muproc-trap-exits-p
Returns T iff obj is a mumsg object, and NIL otherwise.
SEE ALSO — mumsg, mumsg-send, mumsg-receive
Returns T iff obj is a muproc, and NIL otherwise.
SEE ALSO — muproc-spawn, muproc-exit, in-muproc-p
Returns T iff called in a muproc, and NIL otherwise.
SEE ALSO — muproc-spawn, muproc-exit, muproc-p
Returns T iff obj is a mupacket object, and NIL otherwise. mupackets are the representation of messages exchanged between muprocs. Under most circumstances, muproc users do not need to manipulate mupackets directly.
SEE ALSO — muproc-send, muproc-receive
Returns T iff obj is a cl-muproc address object, and NIL otherwise. A muproc address is the destination of muproc messages, and can be a muproc, a muproc port object, or a keyword symbol referring to a registered port name.
SEE ALSO — mumsg-send, mumsg-receive, muproc-send, muproc-receive, muproc-register-port-name, muproc-unregister-port-name, muproc-with-registered-port
Returns T iff obj is a valid muproc name, and NIL otherwise. Muproc names can be any symbol except NIL. Every muproc must have a unique name.
SEE ALSO — muproc-spawn, muproc-name
Returns T iff obj is a valid muproc port name, and NIL otherwise. Muproc port names must be a keyword symbol.
SEE ALSO — muproc-register-port-name, muproc-unregister-port-name, muproc-with-registered-port
Returns T iff muproc traps exits, and NIL otherwise. The default value of muproc is the muproc making the call.
SEE ALSO — muproc-set-trap-exits, muproc-spawn
Returns T iff the calling muproc has unprocessed messages in its input queue, and NIL otherwise. Under most circumstances muprocs do not need to use this function.
SEE ALSO — mumsg-receive, muproc-receive
muprocn spawns a muproc and evaluates a list of forms inside it. The name muprocn is intended as analogous to progn.
The value of evaluating the last form given to muprocn is returned. However, since a new muproc is spawned and since it may never terminate, an upper bound may be specified in timeout. Iff a timeout occurs, the value of evaluating timeout-action is returned. The default value of timeout is 10, and the default value of timeout-action is :timeout.
muprocn is very useful for interactively communicating with a muproc, e.g. for debugging or ad hoc queries.
Since operators like mumsg-send and mumsg-receive can only be used inside muprocs, muprocn can sometimes be useful for creating simple interfaces between non-muproc and muproc parts of a system. Note, however, that (in current muproc), spawning a muproc is relatively expensive operation, so using muprocn as the basis of a high-volume interface is generally a bad idea.
EXAMPLE — in which we see how to use muprocn interactively,
CL-MUPROC> (in-muproc-p) NIL CL-MUPROC> (muprocn () (in-muproc-p)) T CL-MUPROC> (handler-case (muproc-log-errorstream "Hi") (error (err) (format nil "~a" err))) "The assertion (IN-MUPROC-P) failed." CL-MUPROC> (handler-case (muprocn () (muproc-log-errorstream "Hi")) (error (err) (format nil "~a" err))) 22:05:48.188 MUPROCN-1353 Hi NIL CL-MUPROC>