The Schemer's Guide to Programming Gerbils
by Dimitris Vyzovitis [vyzo at media.mit.edu]

# update: <2009-04-29 Wed>
# update: <2009-04-28 Tue>
# create: <2009-04-27 Mon>

* #lang gerbil

Gerbil is a distributed dialect of scheme implemented as a 
plt-scheme [<http://plt-scheme.org>] macro language. The language inherits
all the traits of plt scheme and provides abstractions and libraries
for actor-oriented distributed programming.

This document is a quick introduction to gerbil. It assumes that you are 
a merely eccentric schemer who can find his way around plt-scheme
[s/his/her as needed]. 
It is probably best to read this in emacs using org-mode.

*Note*
This document is by nature incomplete; when in doubt you must go to the source.

** Getting started

You can get the most recent release version of gerbil from
<http://web.media.mit.edu/~vyzo/gerbil>

For the moment gerbil has to be installed as a collection.
Just unpack the gerbil tarball and link path-to-gerbil/gerbil 
to $HOME/.plt-scheme/X.X.X.X/collects.

You can then compile with mzc for faster loading using setup-plt
#<
$ setup-plt -l gerbil
#>

You can run the test suite in the mzscheme repl:
#<
$ mzscheme
> (require gerbil/_test)
; ...
#>
This may take a while in the first run, as dependencies are pulled
from planet.

*Note*
I have only tested on GNU/Linux; If you run into trouble in some other 
platform, let me know.

*** Dependencies

A recent plt-scheme, 4.1.5.x should do.

The mzsocket [<http://web.media.mit.edu/~vyzo/mzsocket>] 
and mzcrypto [<http://web.media.mit.edu/~vyzo/mzcrypto>] libraries. 
The required versions are automatically pulled from planet in gerbil/depend, 
unless there is a locally installed version.

*** License

Gerbil: A Distributed Scheme Dialect
Copyright (C) 2007-2009 Dimitris Vyzovitis [vyzo at media.mit.edu]

Gerbil is free software: you can redistribute it and/or modify
it under the terms of the GNU (Lesser) General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

Gerbil is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU (Lesser) General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.


** Using gerbil

*** REPL

The easiest way to get acquainted with gerbil is the REPL. You can
get a gerbil repl from mzscheme using gerbil/interactive:
#<
$ mzscheme 
Welcome to MzScheme v4.1.5.4 [3m], Copyright (c) 2004-2009 PLT Scheme Inc.
> (require gerbil/interactive)
> (gerbil)
gerbil> ...
#>

There is a wrapper script in <src>/etc/gerbil that runs the gerbil repl 
directly:
#<
$ cd <path-to-gerbil>
$ ./etc/gerbil
gerbil>  ...
#>

*** Emacs

There is a simple emacs mode in <src>/etc/gerbil.el. 
gerbil-mode extends scheme-mode with indentation and highlight rules,
and hooks for cmuscheme repl interaction.

If you want to use gerbil as your repl with cmuscheme, then
M-x gerbil-set-scheme-program will set scheme-program-name to the gerbil 
wrapper script.

*** Module Bindings

You can use gerbil as a module language with the #lang reader macro.
The language reader uses gerbil/main as initial module bindings for 
both runtime and expansion time.

This is a valid hello world module:
#<
$ cat /tmp/hello-world.ss 
#lang gerbil

(printf "hello world\n")

$ gerbil /tmp/hello-world.ss 
hello world
$ ...
#>

*** Macrology

Gerbil is a macro language and invariably a fair amount of macrology is
involved when writing a gerbil program. In general gerbil macros are
composable, building upon a small set of primitive macros.

Gerbil departs slightly from the usually austere scheme syntactic conventions. 
Some patterns that you will frequently encounter have special syntactic
support. Notable is the use of square brackets as shorthand list expressions
and squiglies as method expressions.

**** Syntactic Sugar

The generic expansion of #|{f hd e ...}| is #|((f hd) e ...)|,
but certain macros may provide a more specific method dispatch implementation.
For instance, protocol methods expand to perform a synchronous method
call when used with curly braces.

List expression expand with a syntax similar to ml list expressions:
#<
[]           ; null
[1 2 3]      ; (list 1 2 3)
[1 . 2]      ; (cons 1 2)
[1 :: [2 3]] ; (cons 1 (list 2 3))
[1 ++ [2 3]] ; (append (list 1) (list 2 3)) 
#>

The normal plt-scheme identifier conventions are followed, but for two
exceptions: underscores and "dot-first" identifiers.
Using _ in binding position indicates an anonymous binding.
"Dot-first" identifiers, that means identifiers that match "[.][^.]\sw*",
are reader expanded to dynamic object access. This interpretation is only
for syntax, and you can use bar-quotes to escape it.

Furthermore, when reading syntax, the gerbil reader wraps #|[]| and #|{}| 
expressions with #|@list:| and #|@method:| respectively. 
These are special purpose macros that force the paren-shape property for 
expansion, and are used to ensure correct paren-shape propagation in
compiled code. You can disable wrapping with the reader-wrap-shape
parameter.

**** Bindings

Bindings are introduced with #|def|, while syntax is defined with #|defsyntax|.
More commonly, the #|defrule| and #|defrules| macros are used, which define
syntax declaratively with the syntax-rules pattern language.

Eg:
#<
(def (f x) (+ x 1))

> (f 1)
=> 2

(defrule (g x) (+ x 1))
; macro expansion: (g 1) -> (+ 1 1)

> (g 1)
=> 2

(defrules add ()
  ((_) 0)
  ((_ x) x)
  ((self x xs ...) (+ x (self xs ...))))
; macro expansion:
;  (add 1 2 3)
;  -> (+ 1 (add 2 3))
;  -> (+ 1 (+ 2 3))

> (add 1 2 3)
=> 6
#>

**** Setf

set! behaves like setf:
#<
(set! (foo x) 1)
; expands to (foo-set! x 1)
#>

**** Structs

Structs are defined with defstruct:
#<
(defstruct test (x y))
; definition can be customized with keyword hooks
; constructor
> (test 1 2) 
=> #<test>
; accessor
> (test.x (test 1 2))
=> 1
#>

Structs are by default immutable; provide a #|#:mutable| body argument
to specify a mutable struct (or specific fields).

**** Pattern matching

Pattern matching is used extensively throughout gerbil. The primitive semantics
of pattern matching are derived from the plt match macro, but extended
for simpler destructuring and splicing composition.


*** Gerbil and Scheme

Gerbil is a plt scheme language, so you can (and should) freely 
intermingle gerbil code with code written in any other plt language 
through the module system.

Nonetheless, there is a rough edge to be aware of:
Gerbil struct metadata are currently incompatible with scheme/base struct
metadata. In short, you can define a gerbil struct (with defstruct)
that extends a scheme struct but not vice versa. Also, gerbil structs
cannot be destructured with the plt match macro.


** Hacking gerbil

Some quick notes to aid your hacking and compensate for missing
documentation.  
Patches are always appreciated, even if it is just a typo fix.

*** Airdrop map

The gerbil source tree is roughly organized like this:
#<
<src>/            ; gerbil tree root
+ doc/             
  guide.org       ; this guide
  tutorial.ss     ; interactive tutorial snippets

+ etc/
  gerbil          ; wrapper script
  gerbil.el       ; gerbil-mode

+ gerbil/         ; gerbil implementation
  main.ss         ; #lang gerbil
  interactive.ss  ; gerbil repl
  boot.ss         ; prelingua
  core.ss         ; core macros
  misc.ss         ; miscellaneous macros
  struct.ss       ; defstruct
  match.ss        ; pattern matching macros
  actor.ss        ; actor primitives
  proto.ss        ; protocol primitives [defproto and friends]
  object.ss       ; a lightweight prototype object system
  package.ss      ; meta-syntactic sasuages
  export.ss       ; require/provide macros
  reader.ss       ; gerbil reader
  stx.ss          ; macro utilities [for-syntax]
  type.ss         ; macro meta [for-syntax]
  match-expand.ss ; match parser [for-syntax]

  + proto/        ; distributed programming support
    xdr.ss        ; XDR [RFC 4506/1832] and proto type system
    remote.ss     ; gerbil rpc
    + remote/     ; ...
      ...
    simnet.ss     ; network simulation
    + simnet/     ; ...
      ...
    + misc/       ; misc proto libs
      stun.ss     ; STUN [RFC3489/5389] client support
      upnp.ss     ; UPNP IGDP client support
    packet.ss     ; raw packet protocol macros
    eventt.ss     ; local event synchronizer protocol

  + etc/          ; misc libs
    ...
  + depend/       ; external dependencies
    ...

  + examples/     ; tutorial examples
    ...
#>

*** Zen or something or other

There isn't any. 
Just speak with a lisp and write natural code. 
When you do something twice, write a macro [but look around, because if 
it is common enough then there is one already whp].

Uhm, that came out dangerously close to some "beautiful is better than ugly" 
platitude.

At any rate, be prepared for macro-heavy code. <src>/gerbil/main.ss, 
which packages the initial language bindings, should make a good first
airdrop target.


* TODO Gerbils and Actors 

actor: abstract, 
 composable concurrent algorithm, 
 abstraction of physical processor
gerbil: actor instance as a thread

Messages and Events
Action and Reaction
Reactive Loops

Protocols and Composition
Message classes
Message redirection
 handles
Remote gerbils
 remote
The RPC subsystem
 abstract rpc
 rpc implementations
 network simulation


* Programming Gerbils

There is a collection of examples in gerbil/examples, which illustrate
basic gerbil programming through a series of growingly complex protocols.

You can interactively explore gerbil by trying them out in the repl;
a few sample interactions follow in tutorial fashion.

The notation #|A>| indicates a repl in "node" or "host" A, which here 
is a seperate process.
Naturally that can be in a different machine, but you'll have to adjust  
addresses accordingly. 
A comment of the form "tX" indicates interactions taking place at logical 
time X.

In every host, you should require gerbil/examples/tutorial:
#<
> (require gerbil/examples/tutorial)
#>
which imports the necessary symbols to follow the tutorial.


** Hello world

The hello world of distributed programming is the echo server. Thou shall beat
it to death.

*** The primitive way

Before we make it distributed, we can do it in process using only actor
primitives:
#<
> (def primechod (& (lambda () (do _ (<- (x (-> x)))))))
> (-> primechod 'hello)
> (<-)
=> hello
   #<thread:primechod>
#>

 #|primechod| here is a thread spawned with #|&|; this gerbil loops 
reflecting every message it receives. 

The loop core is implemented in the #|<-| expression: #|<-| is the
primitive reaction macro in gerbil. 
The syntax allows you to define (and compose) message and event reaction 
rules.

The rules are specified using pattern matching. In the example above
the reaction is equivalent to
#<
(let (msg (thread-receive))
  (syntax-parameterize (...)
    (match msg (x (-> x)))))
#>

 #|->| is the send operator; it accepts 1-3 arguments. 

In the full form, the arguments specify the routing envelope passing 
the source and destination actors consed on the message data.
When the source argument is missing, the syntactic parameter #|@self| is
used, which by default is the current thread.

When in reaction context, the destination can be infered as a reply to
the current peer as in the example above. In that case, the action
expression recursively expands using the #|@peer| syntactic parameter:
#<
(-> @self @peer x)
#>

*** The protocol way

As a protocol, echo has a single method that says hello with an identity call:
#<
(defproto echo
  (say (_) _))
#>

After macro expansion of #|defproto|, this expression defines message classes 
and metadata for protocol implementation. The expansion results in 
roughly the following:
#<
(def @echo.say ...)       ; return contract
(defstruct !echo.say ...) ; call class (with contract)
(defcall echo.say ...)    ; method macro
(defsyntax echo ...)      ; protocol metadata
#>

The underscores in the definition indicate a contract free protocol. Specific
contracts can be supplied as predicates (with #|_| interpreted as any) to
be dynamically checked at interface boundaries.

The method macro, #|echo.say|, handles construction and optionally synchronous
semantics for message dispatch.

Next we need a concrete a implementation of the echo protocol:
#<
(defproto/rpc echo.rpc
  #:uid example.echo
  (echo (say _ :: _)))
#>

This expression defines an implemntation using the native gerbil rpc backend:
#<
(def echo.rpc (implement-proto ...))
#>

 #|echo.rpc| is a protocol object, used internally by the rpc subsystem,
that specifies dispatch rules and external data representation.
Multiple protocols can be composed in a single #|defproto/rpc| 
expression; The method arguments supply XDR types that can be composed
through the gerbil type for the implementation
with #|_| interpreted as any primitive type.

Using this machinery, the echo gerbil itself is quite simple:
#<
(def (echo::server rpcd)
  {proxy.register rpcd echo.rpc}
  (react 
    ((echo.say k x) 
     (printf "~a says ~a~n" @peer x)
     {!value k x})))
#>

echo::server again implements a looping gerbil. On entry, the actor registers
with the rpc router as implementing the echo protocol. Then, it
enters a reaction loop where it responds to echo.say messages.

The #|echo.say| message carries a context cookie, which denotes 
an asynchronous one-shot continuation context held by the router in the
source node. The context is resumed when a #|!value| or #|!error| message 
with the same coookie is received. In this case, the gerbil never errs 
and always returns the sole argument.

*** Local echo

When the echo gerbil lives in the local process, we can send a message
to it directly:
#<
> (def rpc (& rpc::dgram))
> (def echod (& echo::server rpc))
> {echo.say echod 'hello}
 #<thread:mzscheme> says hello
=> 'hello
#>

The first line creates a router gerbil. This one IP-routes
using UDP for transport. #|echod| registers with it, but in this 
case send we circumvent it by sending the message directly to echod.

*** UDP echo

This time around, we involve the rpc routers by running #|echod| in a separate
host process:

#<
> (def srv (inet4 #"127.0.0.1" 4999))

;; t0
A> (def rpc1 (& rpc::dgram srv))
A> (def echod (& echo::server rpc1))

;; t1
B> (def rpc2 (& rpc::dgram))
#>

The rpc router in #|A| binds to the server address, while the server in 
 #|B| doesn't bind a static address. The node is still reachable through
the Internet, but peers will have to obtain the dynamic address through prior
communication.

We then perform a #|echo.say| remote call:
#<
;; t2
B> {echo.say (remote rpc2 echo.rpc srv) 'hello}
 => 'hello

A>   
; rpc:bf4db8e9ad1940ac@#<inet4 127.0.0.1:54875> says hello
#>

The call sends a #|echo.say| message, routed to #|echod@A| through #|rpc2@B| 
by means of a remote handle. The handle is constructed with arguments 
the local router, the protocol implementation object, and a network address
for reaching #|A|.

The UDP router also supports IP multicast, although this is probably
a poor choice for routing remote calls:
#<
> (def mcast (inet4 #"239.255.255.101" 4999)) ; arbitrary multicast address

;; t0
A> (def rpc1 (& rpc::mcast mcast))
A> (def echod (& echo::server rpc1))

;; t1
B> (def rpc2 (& rpc::dgram))
B> {echo.say (remote rpc2 echo.rpc mcast) 'hello}
 => 'hello

A>
; rpc:bf4db8e9ad1940ac@#<inet4 192.168.1.5:59744> says hello
#>

Here, the call is performed synchronously. The method
macro implements synchronous semantics for the call using a fresh cookie.
Code that uses actor-level primitives to handle context synchronization
would look like this:
#<
B> (let (x (!cookie @echo.say))
     (-> (remote rpc2 echo.rpc srv) (echo.say x 'hello))
     (<- ... ((!value (eq? x) v) v)))
 => hello
#>

The ellipsis in the reaction indicates a forward match in the thread's
message box. The expression blocks in the current thread's mailbox implicitly
looping until a matching message is received, while buffering other messages. 
Note that the code above does not handle error responses; a similar reaction 
rule that triggers on #|!error| would be used for this purpose.

*** TCP echo

The same code applies when our gerbil prefers to use TCP as the transport.
The only change is the router implementation choice, this time #|rpc::stream|
using #|SOCK_STREAM| sockets as transport.

#<
;; t0
A> (def rpc1 (& rpc::stream srv))
A> (def echod (& echo::server rpc1))

;; t1
B> (def echod (& echo::server rpc1))
B> (def rpc2 (& rpc::stream))
B> {echo.say (remote rpc2 echo.rpc srv) 'hello}
; [rpc::stream] connect #<inet4 127.0.0.1:60824> -> #<inet4 127.0.0.1:4999>
 => 'hello

A>
; [rpc1] [#<inet4 127.0.0.1:4999>]: accept #<inet4 127.0.0.1:4999> <- #<inet4 127.0.0.1:60824>
; rpc:bf4db8e9ad1940ac@#<inet4 127.0.0.1:60824> says hello

*** The gory details

It might seem that even the simplest expression touches upon 
stacked layers of macros. Well, it's true, and sooner or later you will
want to look at expanded code. 
For such cases, the utility macro #|@expand| can be helpful for
(partially) expanding syntax in the repl.

For instance, this is the car expansion of the #|echo| protocol 
definition:
#<
> (@expand (defproto echo (say (_) _)))
(begin
  (def @echo.say (call-guard _))
  (defstruct
   !echo.say
   (_0)
   #:property
   (prop:proto.call @echo.say)
   #:property
   (prop:proto.eid 'echo.say)
   #:property
   (prop:custom-write (/write-message '!echo.say))
   #:transparent)
  (defcall echo.say 'echo.say !echo.say @echo.say)
  (defsyntax
   echo
   (let (cert (syntax-local-certifier))
     (make-metaproto
      'echo
      (list
       (list
        'say
        'echo.say
        (cert (quote-syntax echo.say))
        (cert (quote-syntax !echo.say))
        (cert (quote-syntax @echo.say))))
      (list)))))
=> #<syntax>
#>

The expansion of the method application goes like this:
#<
> (@expand {echo.say echod 'hello})
(let-values (((peer) echod) ((x) (!cookie (call-guard @echo.say))))
  (if (-> peer (echo.say x 'hello))
    (with-peer
     _
     (<- ... ((!value (eq? x) v) v) ((!error (eq? x) e) (raise e))))
    (raise (exn:fail:proto "dead actor"))))
=> #<syntax>
#>

The macro takes an optional argument that controls recursive expansion.
It can be a stop list, cutting expansion when
certain identifiers are encountered at the car position. When it is #|#f| 
(default) then expansion proceeds while the syntax car is a macro.


** Stateful gerbils

After hello world comes factorial in the natural order of things. 
In distributed programming this ought to be the "registry" protocol.

*** The registry protocol

A registry is a stateful actor that responds to two messages: get and put!.
put! creates a key-value binding, and get retrieves a value given a key:
#<
(defproto registry
  (put! (_ _))
  (get (_) _))

(defproto/rpc registry.rpc
  #:uid examples.registry
  (registry
   (put! _ _)
   (get _ :: _)))

#>

*** Implementation

registry::server provides a direct loop implementation:
#<
(def (registry::server rpcd)
  (def keys (make-hash))
  {proxy.register rpcd registry.rpc}
  (react
    ((registry.put! key value)
     (hash-put! keys key value))
    ((registry.get k key)
     (cond
      ((hash-get keys key) => (lambda (value) {!value k value}))
      (else {!error k (exn:fail:contract* "no such key: ~a" key)})))))
#>

The registry gerbil maintains a private store of key-value mapping as
a hash table. 
put! is an event that stores a new mapping in the hash table.
get is a call that retrieves an existing mapping, raising an exception 
if there is none.

*** Interaction

Using UDP as the transport, here is a simple interaction between a registry
gerbil and two registry clients:
#<
(def srv (inet4 #"127.0.0.1" 4999))

;; t0
A> (def rpc1 (& rpc::dgram srv))
A> (def regd (& registry::server rpc1))

;; t1
B> (def rpc2 (& rpc::dgram))
B> (def cli2 (remote rpc2 registry.rpc srv))
B> {registry.get cli2 'a}
ERROR: no such key: a

;; t2
C> (def rpc3 (& rpc::dgram))
C> (def cli3 (remote rpc3 registry.rpc srv))
C> {registry.put! cli3 'a 1}

;; t3
B> {registry.get cli2 'a}
=> 1
#>


** Asynchronous notifications

A natural extension of the registry protocol is to implement a form
of "publish-subscribe". We can do this by defining an "observer"
protocol:
#<
(defproto observer
  (notify (_ _)))
(defproto/rpc observer.client.rpc
  (observer
   (notify _ _ #:ref)))
#>

Clients that implement the observer protocol can pass an (optional) reference 
cookie in get messages. A registry that understands the observer 
protocol can then asynchronously respond to a get for an unbound key by 
emitting an #|observer.notify| message. We specify this by defining a new 
instance of the registry protocol:
#<
(defproto/rpc observer.rpc
  #:uid example.registry ; plain registry compatible
  (registry
   (put! _ _)
   (get _ #:ref :: _)))
#>

*** An Observable registry

The implementation of the registry gerbil is easily adapted to 
support observers:
#<
(def (observer::server rpcd)
  (def keys (make-hash))
  (def pend (make-hash))
  {proxy.register rpcd observer.rpc}
  (react
    ((registry.put! key value)
     (hash-put! keys key value)
     (alet (xs (hash-get pend key))
       (for ((next xs))
         (with ([peer . ref] next)
           {observer.notify peer key value #:ref ref}))
       (hash-remove! pend key)))
    ((registry.get k key #:ref ref)
     (cond
      ((hash-get keys key) => (lambda (value) {!value k value}))
      (else
        (when ref
          (hash-update! pend key 
            (lambda (lst) [[(observer-client @peer) . ref] . lst]) []))
        {!error k (exn:fail:contract* "no such key: ~a" key)})))))
#>

This registry implementation closes over a second hash-table which stores
pending notifications for unbound keys.

When reacting to a get message for an unbound key, the server checks for 
the presence of a reference cookie. If it is not nil, then an entry is 
stored to the pend table. 
The reaction to put! is similarly augmented to dispatch pending notifications.

*** Interaction

The intereaction grows more complex now. We use 4 nodes: the registry
gerbil resides in A while B,C, and D are client gerbils.
#<
(def srv (inet4 #"127.0.0.1" 4999))

;; t0
A> (def rpc1 (& rpc::dgram srv))
A> (def obsd (& observer::server rpc1))

;; t1
B> (def rpc2 (& rpc::dgram))
B> (def cli2 (remote rpc2 observer.rpc srv))
B> {proxy.register rpc2 observer.client.rpc}
B> {registry.get cli2 'a #:ref 'A}
ERROR: no such key: a

C> (def rpc3 (& rpc::dgram))
C> (def cli3 (remote rpc3 observer.rpc srv))
C> {proxy.register rpc3 observer.client.rpc}
C> {registry.get cli3 'a #:ref 'B}
ERROR: no such key: a

;; t2
D> (def rpc4 (& rpc::dgram))
D> (def cli4 (remote rpc4 observer.rpc srv))
D> {registry.put! cli4 'a 1}

;; t3
B> (<-)
=> #(struct:!event
     (!observer.notify a 1)
     observer.observer.notify
     #hash((#:ref . B)))
   rpc:b1ca2306b23877f3@#<inet4 127.0.0.1:4999>
C> (<-)
=> #(struct:!event
     (!observer.notify a 1)
     observer.observer.notify
     #hash((#:ref . A)))
   rpc:b1ca2306b23877f3@#<inet4 127.0.0.1:4999>
#>


** The blabbers

At this point we can examine a more realistic example: 
a peer-to-peer chatroom. 

The chatroom is implemented by a number of peer (client) nodes, who
form a connected overlay. The overlay is bootstrapped through a directory
gerbil acting as a presence service.

*** The directory gerbil

The directory protocol is a variation of the registry protocol, maintaining
a soft list of mappings from nicknames to remote gerbils. 
The mapping is refreshed through #|chat.directory.put!| messages, with
individual entries expiring after a period of inactivity.

#<
(defproto chat.directory
  (list () list?)
  (put! (symbol?) _)
  (get (symbol?) actor?))
#>

*** Chatting gerbils

Client gerbils implement the chatroom as a peer-to-peer overlay. In 
each network node, a client gerbil runs as a router for chatroom messages
[for a particular chatroom]. 

The client protocol consists of two event methods:
#<
(defproto/rpc chat.client.rpc
  (chat.client
   (hello &symbol)
   (say &symbol &string)))
#>

The client's actor role is best explained through code:
#<
(def (chat::client rpcd local #:nick nick #:directory dir ...)
  (defstruct !client ...)
  (def TICK ...)
  (def (refresh-evt) ...)

  {proxy.register rpcd chat.client.rpc}
  ;; retrieve peer list from directory and loop
  (let lp ((evt always-evt)
           (peers (foldl (lambda/match ([who . peer] r) 
                           (hash-put r who (!client peer)))
                    (hasheq) 
                    {chat.directory.list dir})))
    (react* ...)))
#>

The routing functionality is implemented at the body of the reactive
loop. The loop mainains the overlay as a soft routing table
initialized with the directory list.

Chatting is performed using chat.client.say events.  
Local messages are forwarded to the overlay, and remote messages 
are routed to the designated actor.  A local actor can broadcast a message
to the chatroom by sending a chat.client.say message with #f as the
target. A 'private' message can be sent by specifying a nickname as
the target:

#<
(def (chat::client ...)
  ...
  (let lp ((evt ...) (peers ...))
    (defrule (client-tick who) ...)
    (defrule (client-tock cli what) ...)
    
    (react*
      ;; route message
      ((chat.client.say who what)
       (match* (@peer who)
         (((remote _ (eq? chat.client.rpc) _) _) ; incoming
          (-> @peer local @message)
          (client-tick who))
         ((_ (? not))                   ; outgoing broadcast
          (for (((who cli) peers)) (client-tock cli what))
          (lp evt peers))
         (else                          ; outgoing private
           (cond
            ((hash-get peers who) => (cut client-tock <> what))
            (else (warn "unknown peer: ~a~n" who)))
           (lp evt peers))))
      ;; peer presence
      ((chat.client.hello who) 
       (when (remote? @peer)
         (client-tick who)))
      ;; refresh presence
      (! evt
         {chat.directory.put! dir nick #:ttl ttl}
         (lp (refresh-evt)
             (hash-fold peers
               (lambda/match (who (and cli (!client peer fresh? ticks)) r)
                 (cond
                  (fresh?               ; no need to say hello
                   (set! (!client.fresh cli) #f)
                   (set! (!client.ticks cli) (1- ticks))
                   (hash-put r who cli))
                  ((> ticks 0)          ; still ticking
                   {chat.client.hello peer nick}
                   (set! (!client.ticks cli) (1- ticks))
                   (hash-put r who cli))
                  (else r)))            ; stale
               (hasheq))))))
#>

The protocol uses chat.client.hello as keepalive messages
when there is no traffic from a particular peer. If the maximum number
of heatbeats ellapses without hearing from a peer then it is declared
dead and removed from the local routing table.

The helper macros perform book-keeping:
#<
(def (chat::client ...)
  ...
  (let lp ((evt ...) (peers ...))
    (defrule (client-tick who)
      (match (hash-get peers who)
         ((and cli (!client (equal? @peer) _ _))
          (set! (!client.ticks cli) TICK)
          (lp evt peers))
         (else (lp evt (hash-put peers who (!client @peer))))))
    
    (defrule (client-tock cli what)
      (begin
        (set! (!client.fresh cli) #t)
        {chat.client.say (!client.peer cli) nick what}))
    ...))
#>

*** Interaction

We use again 4 nodes: A, acting as the directory, and B,C, and D acting
as peer gerbils (Wrickwrackrum!):
#<
> (def srv (inet4 #"127.0.0.1" 4999))

;; t0
A> (def rpc1 (& rpc::dgram srv))
A> (def dird (& chat::server rpc1))

;; t1
B> (def rpc2 (& rpc::dgram))
B> (def dir2 (remote rpc2 chat.directory.rpc srv))
B> (def cli2 (& chat::client rpc2 @self #:nick 'B #:directory dir2))

C> (def rpc3 (& rpc::dgram))
C> (def dir3 (remote rpc3 chat.directory.rpc srv))
C> (def cli3 (& chat::client rpc3 @self #:nick 'C #:directory dir3))

D> (def rpc4 (& rpc::dgram))
D> (def dir4 (remote rpc4 chat.directory.rpc srv))
D> (def cli4 (& chat::client rpc4 @self #:nick 'D #:directory dir4))

;; t2
A> {chat.directory.list dird}
=> ((C . rpc:932d79889ee518d5@#<inet4 127.0.0.1:53141>)
    (D . rpc:932d79889ee518d5@#<inet4 127.0.0.1:55211>)
    (B . rpc:932d79889ee518d5@#<inet4 127.0.0.1:42768>))

;; t3
B> {chat.client.say cli2 #f "hello"}

;; t4
C> (<-)
=> #(struct:!event #(struct:!chat.client.say B "hello") 
                  chat.chat.client.say #f)
   rpc:08855787e63f4654@#<inet4 127.0.0.1:42768>

D> (<-)
=> #(struct:!event #(struct:!chat.client.say B "hello") 
                   chat.chat.client.say #f)
   rpc:08855787e63f4654@#<inet4 127.0.0.1:42768>
#>


** Virtual gerbils

Perhaps the most salient feature of actor-oriented programming is that
physical processors are abstracted away. After all Hewitt devised the 
actor model as a physical processor abstraction (and scheme herself 
evolved out of the actor formalism).

In gerbil practice, this translates to a very simple idea: complete protocols
can be simulated in virtual networks without changing a line of code or
needing more than a single physical processor. For this purpose, Gerbil 
provides a virtualized rpc implementation.

This is a simple echo simulation script:
#<
(def (echo/simnet #:trace (simdata (trace.capture))
                  #:world (world (world.trace world.nil simdata))
                  #:net (net (network.trace network.nil simdata))
                  #:end (end (system-idle-evt)))
  (def root (& simnet::root #:world world #:net net))
  (def n1 {simnet.create root 'A})
  (def echod {simnet.spawn root n1 echo::server})
  (def (test::echo rpcd srv top)
    (let (cli (remote rpcd echo.rpc srv))
      (-> top {echo.say cli 'hello})))

  (def n2 {simnet.create root 'B})
  (def cli1 {simnet.spawn root n2 (cute test::echo <> n1 @self)})
  (sync end)
  {simnet.shutdown root}
  (sync (system-idle-evt))
  {.get-data simdata})
#>

Running in the nil network generates this result [hello virtual world!]:
#<
> (echo/simnet #:trace (trace.display))
(start)
(world.create A #f)
(world.create B #f)
(network.send (B A (packet (call 32f358dc42dba84c (!echo.say hello) ()) 28)))
(network.enque (B (packet (call 32f358dc42dba84c (!echo.say hello) ()) 28)) up 0 #f)
(network.route (B A) 0)
(network.enque (A (packet (call 32f358dc42dba84c (!echo.say hello) ()) 28)) down 0 #f)
(network.recv (B A (packet (call 32f358dc42dba84c (!echo.say hello) ()) 28)))
rpc:bf4db8e9ad1940ac@B says hello
(network.send (A B (packet (value 32f358dc42dba84c hello) 16)))
(network.enque (A (packet (value 32f358dc42dba84c hello) 16)) up 0 #f)
(network.route (A B) 0)
(network.enque (B (packet (value 32f358dc42dba84c hello) 16)) down 0 #f)
(network.recv (A B (packet (value 32f358dc42dba84c hello) 16)))
(world.destroy B)
(world.destroy A)
(stop)
=> #f
#>

A more interesting simulation, tracing a second of real time simulation 
in a random network:
#<
> (let (simdata (trace.timestamp (trace.capture)))
    (echo/simnet
     #:trace simdata
     #:world (world.trace world.nil simdata)
     #:net (network.trace 
            (network (network.core.consistent (network.core.random)) 
                     network.edge.nil) 
            simdata)
     #:end (alarm-evt* 1)))
rpc:bf4db8e9ad1940ac@B says hello
=> 
((start)
 (0.18505859375 world.create A #f)
 (0.862060546875 world.create B #f)
 (1.535888671875
  network.send
  (B A (packet (call 829c606590bd527d (!echo.say hello) ()) 28)))
 (1.5400390625
  network.enque
  (B (packet (call 829c606590bd527d (!echo.say hello) ()) 28))
  up
  0
  #f)
 (1.825927734375 network.route (B A) 0.02785576787870371)
 (30.470947265625
  network.enque
  (A (packet (call 829c606590bd527d (!echo.say hello) ()) 28))
  down
  0
  #f)
 (30.488037109375
  network.recv
  (B A (packet (call 829c606590bd527d (!echo.say hello) ()) 28)))
 (30.952880859375
  network.send
  (A B (packet (value 829c606590bd527d hello) 16)))
 (30.95703125
  network.enque
  (A (packet (value 829c606590bd527d hello) 16))
  up
  0
  #f)
 (30.971923828125 network.route (A B) 0.02785576787870371)
 (59.074951171875
  network.enque
  (B (packet (value 829c606590bd527d hello) 16))
  down
  0
  #f)
 (59.091064453125
  network.recv
  (A B (packet (value 829c606590bd527d hello) 16)))
 (59.509033203125 world.destroy B)
 (1000.242919921875 world.destroy A)
 (stop))
#>




* Release history
2009-04-28: bug fixes
 rewrote keyword-rules to avoid 3D macro embedding
 reader macro for list/method expressions
 update crypto for ffi/provide bug workaround in 4.1.5

2009-04-27: first stable release
 tcp and simnet integration
 bug fixes and cleanup

2009-02-23: gerbil prerelease
 core gerbil
 only udp support