marius a. eriksen2015-08-28T21:12:30-07:00http://monkey.org/~mariusmarius a. eriksenmarius@monkey.orgHints for writing Unix tools2014-10-20T00:00:00-07:00http://monkey.org/~marius//unix-tools-hints<p><em>Note: this article has been translated into <a href="http://postd.cc/unix-tools-hints/">Japanese</a></em></p>
<p>The workaday world of a modern programmer abounds with Unix tools,
stitched together in myriad ways.
While good tools integrate seamlessly with your own environment,
bad ones will constantly frustrate your efforts.
Good tools have a seemingly limitless application,
constrained only by your own imagination.
Bad tools,
on the other hand,
will often require that you deploy a salvo of brittle hacks
to keep them barely working in your own environment.</p>
<blockquote class="twitter-tweet" lang="en"><p>“One thing well” misses the point: it should be “One thing well AND COMPOSES WELL”</p>— marius eriksen (@marius) <a href="https://twitter.com/marius/status/256034150052683776">October 10, 2012</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>I don’t want to attempt to explain what makes for good design;
this has been <a href="http://www.amazon.com/Programming-Environment-Prentice-Hall-Software-Series/dp/013937681X">discussed</a> <a href="http://harmful.cat-v.org/cat-v/unix_prog_design.pdf">elsewhere</a>.
Instead,
I want to outline a few established customs
that you should take care to follow when writing new tools.
While making a truly <em>good</em> tool can be an elusive goal,
it isn’t difficult to avoid making a truly bad one.
Unix demands good citizenry from its tools:
it relies on a set of conventions to make things work,
and importantly, to compose,
well.
Here follows a few key customs, often violated.
These aren’t absolute requirements,
but you should think long and hard before violating them.</p>
<p><strong>Consume input from stdin, produce output to stdout.</strong>
Put another way, your program should be a <a href="http://en.wikipedia.org/wiki/Filter_(software)#Unix">filter</a>.
Filters are easily integrated into
<a href="http://cm.bell-labs.com/who/dmr/mdmpipe.pdf">shell pipelines</a>,
arguably the most important utility for Unix tools composition.</p>
<p><strong>Output should be free from headers or other decoration.</strong>
Superflous output will frustrate users who are trying to parse tool output.
Headers and decoration tend to be less regular
and more idiosyncratic
than the structured data you’re really trying to get at.
Don’t do it.</p>
<p><strong>Output should be simple to parse and compose.</strong>
This usually means representing each record as a single,
plain-text formatted line of output
whose columns are separated by whitespace.
(No JSON, please.)
Most venerable Unix tools—grep, sort, and sed among them—assume this.
As a simple example, consider the following output from a benchmark suite.
It is formatted by starting each record with the benchmark name,
followed by a set of key-value pairs associated with the named benchmark.
This is a flexible structure to work with as it allows you to add or remove
keys at will without violating the output format.</p>
<pre class="code">
$ ./runbenchmarks
Benchmark: fizzbuzz
Time: 10 ns/op
Alloc: 32 bytes/op
Benchmark: fibonnacci
Time: 13 ns/op
Alloc: 40 bytes/op
...
$
</pre>
<p>While convenient, it is quite clumsy to work with in Unix.
Consider a very common thing we might want to do:
look up the timing results for a single benchmark.
Here’s how you do it.</p>
<pre class="code">
$ ./runbenchmarks | awk '/^Benchmark:/ { bench = $2} bench=="fizzbuzz"'
Benchmark: fizzbuzz
Time: 10 ns/op
lloc: 32 bytes/op
$
</pre>
<p>If instead each line presents exactly one record,
where columns are separated by whitespace,
this becomes a much simpler task.</p>
<pre class="code">
$ ./runbenchmarks
fizzbuzz 10 32
fibonnaci 13 40
...
$ ./runbenchmarks | grep '^fizzbuzz'
fizzbuzz 10 32
$
</pre>
<p>The advantage becomes even more evident when reordering
or aggregating the input.
For example,
when the output is record-per-line,
sorting the results by time spent is a simple matter of invoking sort:</p>
<pre class="code">
$ ./runbenchmarks | sort -n -r -k2,2
fibonnaci 13 40
fizzbuzz 10 32
...
$
</pre>
<p><strong>Treat a tool’s output as an API.</strong>
Your tool <em>will</em> be used in contexts beyond your own imagination.
If a tool’s output format is changed,
other tools that compose or otherwise build on its output
will invariably break—you have broken the API contract.</p>
<p><strong>Place diagnostics output on stderr.</strong>
Diagnostics output includes anything that is not the primary
data output of your tool.
Among these are:
progress indicators,
debugging output,
log messages,
error messages,
and usage information.
When diagnostics output is intermingled with data,
it is very difficult to parse, and thus compose, the tool’s output.
What’s more, stderr makes diagnostics output more useful since,
even if stdout is being filtered or redirected,
stderr keeps printing to the user’s terminal—the ultimate target of diagnostics output.</p>
<p><strong>Signal failure with an exit status.</strong>
If your tool fails,
exit with a status other than 0.
This allows for simple integration shells,
and also simpler error handling in scripts.
Consider the difference between two tools that build binaries.
We’d like to build upon this tool to execute the built binary
only if the build succeeds.
Badbuild prints the word ‘FAILED’ as the last line when it fails.</p>
<pre class="code">
$ ./badbuild binary
...
FAILED
$ echo $?
0
$ # Run binary on successful build.
$ test "$(./badbuild binary | tail -1)" != "FAILED" && ./binary
$
</pre>
<p>Goodbuild sets its exit status appropriately.</p>
<pre class="code">
$ ./goodbuild
$ echo $?
1
$ # Run binary on successful build.
$ ./goodbuild binary && ./binary
$
</pre>
<p><strong>Make a tool’s output portable.</strong>
Put another way,
a tool’s output should stand on its own,
requiring as little context as possible to parse and interpret.
For example,
you should use absolute paths to represent files,
and fully qualified hostnames to name internet hosts.
Portable output is directly usable by other tools
without further context.
A frequent violator of this is build tools.
For example, both the GCC and Clang compilers
try to be clever by reporting paths that are relative
to your working directory.
In this example,
the source file paths are presented relative to the current working directory
when the compiler was invoked.</p>
<pre class="code">
$ cc tmp/bad/x.c
tmp/bad/x.c:1:1: error: unknown type name 'INVALID_C'
INVALID_C
^
tmp/bad/x.c:1:10: error: expected identifier or '('
INVALID_C
^
2 errors generated.
$
</pre>
<p>This cleverness breaks down quickly.
For example if I use <a href="http://linux.die.net/man/1/make">make(1)</a> with the -C flag.</p>
<pre class="code">
$ cat tmp/bad/Makefile
all:
cc x.c
$ make -C tmp/bad
cc x.c
x.c:1:1: error: unknown type name 'INVALID_C'
INVALID_C
^
x.c:1:10: error: expected identifier or '('
INVALID_C
^
2 errors generated.
make: *** [all] Error 1
$
</pre>
<p>Now the output is less useful:
to which file does “x.c” refer?
Other tools that build on this need additional context,
the -C argument,
in order to interpret the compiler’s
output—the output does not stand on its own.</p>
<p><strong>Omit needless diagnostics.</strong>
Resist the temptation to inform the user of everything that is being done.
(But if you must, do it on stderr.)
A good tool is quiet when all is well,
but produces useful diagnostics output when things go wrong.
Excessive diagnostics conditons users to ignore all diagnostics;
useful diagnostics output does not require the user to grub around
in endless log files to discern what went wrong, and where.
There’s nothing wrong with having a verbose mode
(typically enabled by a ‘-v’ flag)
in order to aid development and debugging,
but do not make this the default.</p>
<p><strong>Avoid making interactive programs.</strong>
Tools should be usable without user interaction
beyond what’s provided by the user’s shell.
Unix programs are expected to run without user input:
it allows programs to be run in non-interactively by cron,
or to be easily distributed for execution by a remote machine.
Even a single interaction forfeits this very useful capability.
Interactivity also makes composition more difficult.
Since Unix’s program composition model does not distinguish
the output of the various programs involved,
it isn’t always clear which program a user is even interacting with.
A common use of interactive programs is to ask the user to confirm some
dangerous action.
This is easily avoided by asking the user instead to supply
a flag on the command line to the appropriate tool.</p>
<p>I wrote this because I find myself continually frustrated by
attempting to use and compose bad tools—bad tools that waste time
and limit their own usefulness.
Most of these tools could be made a lot better
by following the above advice.</p>
<p>For a more general discussion of Unix tools design,
I encourage you to read Kernighan and Pike’s “<a href="http://www.amazon.com/Programming-Environment-Prentice-Hall-Software-Series/dp/013937681X">The Unix Programming Environment </a>.”</p>
<p>Discussion on <a href="https://news.ycombinator.com/item?id=8484718">Hacker News</a>.</p>
SOSP 2013 Trip report, and a note on systems research2013-11-09T00:00:00-08:00http://monkey.org/~marius//sosp<p>The <a href="http://sigops.org/sosp/sosp13/">twenty-fourth ACM Symposium for on Operating Systems
Principles</a> (SOSP) was held at the <a href="http://www.nemacolin.com">Nemacolin
Woodlands</a> resort in Western Pennsylvania. (Nemacolin
is a luxury resort, and it wants you to know it. It’s sort of like a
Disneyland rendition of the palace of Versailles.) SOSPs are usually
held in slightly isolated settings. It forces everyone to stay
together for the entire affair—an all-inclusive conference. This
works: your time there is consumed by SOSP activities. There is one
track of sessions during the day, immediately followed by receptions,
poster sessions, BoFs, and WIPs—open bars abound—followed by banquet
dinners. This is Woodstock for systems researchers.</p>
<p>On the Sunday before the main conference I attended the
<a href="http://plosworkshop.org/2013/">PLOS’13</a> workshop where I presented a paper,
<span class="sidenote-number">“<a href="http://plosworkshop.org/2013/preprint/eriksen.pdf">Your Server as
a Function</a>”</span>
<span class="sidenote">Slides are <a href="http://plosworkshop.org/2013/presentations/eriksen.pdf">available</a>;
the talk was not recorded.
</span>
about some of the core
abstractions and software systems that we use to construct our server
software—i.e. our systems software stack. <a href="http://swtch.com/~rsc/">Russ Cox</a> gave the
workshop keynote, on <a href="http://golang.org/">Go</a>. Russ did a great job motivating the
philosophy behind Go’s type system, concurrency constructs, and its
package system. I’ve had an eye on Go for a while: its focus on
simplicity is a healthy response to the complexity creep that seems to
plague just about every other “industry” language. I’m curious to see
how well Go’s approach will hold up to time, and whether they truly
manage to keep it simple in the face of larger code bases, greater
demand for generic code, and a changing hardware and software
landscape. Go’s designers have a healthy attitude, though. <a href="https://twitter.com/avsm">Anil
Madhavapeddy</a> maintained a <a href="http://www.syslog.cl.cam.ac.uk/2013/11/03/liveblog-from-programming-languages-and-operating-systems-2013/">liveblog of the entire
event</a>.</p>
<p>Monday through Wednesday featured the main conference. SOSP
is a single track conference, and the talks are very well attended. There
are lots of good questions—some are asked in anger!—and the whole
affair is quite engaging. The talks were all well-rehearsed; there were
few awkward pauses or speakers who stumbled.</p>
<p>My 3 favorite talks were also the best paper award winners.</p>
<p>“<a href="http://dl.acm.org/citation.cfm?doid=2517349.2522712">The Scalable Commutativity Rule: Designing Scalable Software for
Multicore Processors</a>” describes a tool,
<span class="newthought">commuter</span>, which uses
symbolic execution on models (effectively, “pseudo-code” Python
implementations of the underlying operations) to determine what
operations and argument combinations <em>commute</em>—that is, which pairs
of operations may be executed independently of order?
<span class="newthought">commuter</span> allows
you to statically analyze systems—more accurately: models of
systems—and determine where operations may execute concurrently. The
authors implement a filesystem in order to show the applicability of
<span class="newthought">commuter</span>.</p>
<p>“<a href="http://dl.acm.org/citation.cfm?doid=2517349.2522728">Towards Optimization-Safe Systems: Analyzing the Impact of Undefined
Behavior</a>” describes a tool,
<span class="newthought">stack</span>, which sniffs out
where applications rely on behavior that is undefined in C. (Did you
know that pointer arithmetic overflow is undefined? Me neither.) The
authors created a model for understanding such unstable code, and
developed a static checker to find such code. Finally they ran
<span class="newthought">stack</span>
on a number of open source projects (Kerberos, Postgres, the Linux
kernel) finding a large amount of unstable code. This was a fun
presentation and paper.</p>
<p>Perhaps my favorite paper was “<a href="http://dl.acm.org/citation.cfm?doid=2517349.2522738">Naiad: a timely dataflow
system</a>.” Naiad is a low-latency dataflow system that admits
<em>cycles</em>, the upshot of which is support for dataflow iteration. For example,
you can mix computing pagerank with more standard <em>mapreduce</em>-like
data processing. It presents a single, unified model for doing so.
Naiad uses a vertex-communication model similar to <a href="http://kowshik.github.io/JPregel/pregel_paper.pdf">Pregel</a> but
without synchronous iteration: incoming messages are computed
incrementally. Naiad maintains a versioning scheme and distinguishes
loop ingress- and egress-nodes. The story for fault recovery is a bit
weak—currently a synchronous recovery mechanism is employed—and may be
its achilles heel; I’m curious to see it develop.</p>
<p>I also enojyed Gernot Heiser’s <a href="http://dl.acm.org/citation.cfm?doid=2517349.2522720">microkernel retrospective</a>.
It’s great to see such long and impactful lines of research, and
better still to attempt to draw lessons from the process.</p>
<p>The first day of the conference was <a href="http://www.syslog.cl.cam.ac.uk/2013/11/04/live-blog-from-sosp-2013/">liveblogged</a>.</p>
<h2 id="a-note-on-systems-research">A note on systems research</h2>
<p>As an industry participant, it’s dissapointing to see a number of very
important problems given short shrift.</p>
<p><span class="newthought">The distributed systems tooling gap</span>.
We have good tools for
understanding behavior of single-system processes: Debugging and
diagnostics tools like <a href="http://dtrace.org/blogs/">Dtrace</a> and <a href="https://perf.wiki.kernel.org">perf</a>, are among these.
Equivalent tooling for distributed systems are weak to nonexistent.
Google’s <a href="http://research.google.com/pubs/pub36356.html">Dapper</a> is one of the only published systems that come to
mind which attempts to tackle parts of this problem. In reality,
almost all organizations that deploy sizable distributed systems have
their own set of tools and systems—aggregation and display of system
statistics, RPC tracing, alerting, etc.—but these are rather
primitive and unwieldy compared to tools like Dtrace and perf. I think
there are a lot of important (and interesting!) problems to be solved
here, but it seems to be beneath the radar of the academy.</p>
<p><span class="newthought">Cost control and isolation in “the cloud”</span>.
Cost control is
paramount in large systems deployments. It’s important to understand
the cost structure of your system, and the trade-offs involved in
optimizing. This is especially important in large multi-tenant
systems. <a href="http://en.wikipedia.org/wiki/Service-oriented_architecture">Service-oriented architectures</a>, where different uses
(“features”) share common systems, exacerbate this further. We have
good, coarse-grained, isolation in operating systems—processes,
virtual machines, containers, etc.—but <em>vertical isolation</em> in
distributed systems is poor to nonexistent.</p>
<p><span class="newthought">Distributed systems modularity</span>.
We reach for many tools when
structuring software—classes, mixins, the ML module system, Go
packages and interfaces, etc.—but in distributed systems we have far
fewer, far more primitive ones. The state of the art is pretty much
using some form of interface definition langauge (e.g. <a href="http://thrift.apache.org">thrift</a>,
<a href="https://code.google.com/p/protobuf/">protobufs</a>) to define module boundaries. These are then manually
glued together. This is like having only C header files and
<a href="http://linux.die.net/man/3/dlopen">dlopen(3)</a>. Worse, because services tend to <a href="http://en.wikipedia.org/wiki/Conway's_law">follow
organizational boundaries</a>, and because it’s tedious to
maintain a large number of systems, these interfaces tend to become
kitchen-sinks comprising everything a particular team is responsible
for, whether or not these functionalities are even related. It’s
considered good programming practice to focus on <em>compositionality</em>:
build software out of small, well-defined modules that <em>combine</em> to
give rise to other modules with different behaviors. This is simply
too difficult to do in distributed systems. Why?</p>
<p><span class="newthought">Authentication</span>.
<a href="http://en.wikipedia.org/wiki/Kerberos_(protocol)">Kerberos</a> showed us how to do authentication in
distributed systems. It’s still a good model, but a lot has changed
since <a href="ftp://athena-dist.mit.edu/pub/kerberos/doc/usenix.PS">1988</a>. In particular, service-oriented
architectures make session authentication less useful: usually a
request is performed on someone else’s behalf. Additionally, because
central scheduling systems (e.g. <a href="http://mesos.apache.org">Mesos</a>, <a href="http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf">Omega</a>) are actually
responsible for provisioning and managing processes, there needs to be
a chain of trust. Above all, such a system must be simple to operate
and easy to understand and audit.</p>
You should participate in CUFP this year2013-06-02T00:00:00-07:00http://monkey.org/~marius//cufp<p>The <a href="http://cufp.org">Workshop for Commercial Users of Functional Programming
2013</a> (CUFP) has been meeting annually for nearly
10 years. These same years have seen a staggering growth in the
commercial application of functional programming languages
and techniques — their influence is felt in even the
most established
<a href="http://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html">languages</a>, <a href="http://code.google.com/p/guava-libraries/">libraries</a>,
and communities. Attendance has grown nearly ten-fold.</p>
<blockquote>
<p>The deadline for submitting a proposal to present is <strong>June 29</strong>.
See the <a href="http://cufp.org/2013cfp">call for presentations</a> to details.</p>
</blockquote>
<p>Many of the largest internet properties are using a functional language
for their primary service development — <a href="https://twitter.com">Twitter</a>,
<a href="http://www.tumblr.com">Tumblr</a>,
and <a href="http://www.foursquare.com">Foursquare</a> among them. Functional
language have always had a toe-hold in the financial industry, and its
influence there is continuing to grow — in 2011
<a href="http://anil.recoil.org/papers/2011-cufp-scribe-preprint.pdf">Lennart Augustsson from Standard Charter</a> gave the CUFP keynote address about <code>Mu</code>,
a dialect of Haskell used there for trading and analysis; Yaron Minsky
highlighted Jane Street’s use of OCaml with <a href="http://cufp.org/videos/jane-street-status-report">a talk in 2012</a>.</p>
<p>The CUFP workshop, which I am chairing together with <a href="http://www.deinprogramm.de/sperber/">Michael Sperber</a> this year, comprises a day of talks and
two days of tutorials. The talks focus on the use of functional languages
in a practical setting: lessons learned, novel applications and techniques,
organizational adaptation and, as well, what didn’t <em>didn’t</em> work?</p>
<p>I recommend perusing past workshop <a href="http://cufp.org/conference">schedules</a>
and <a href="http://cufp.org/videos">videos</a> on the
<a href="http://cufp.org">CUFP</a> website. Some of my recorded favorites are
Bryan O’Sullivan’s 2009 keynote,
“<a href="http://cufp.org/videos/keynote-real-world-haskell">Real World Haskell</a>” —
where Bryan talks about the value of community — and from 2011, Gregory Wright’s
“<a href="http://cufp.org/videos/fourteen-days-haskell">Fourteen Days of Haskell</a>”
describes a heroic effort to implement antenna control software for low power
devices in a mere 2 weeks — a problem domain that turned out to be a great
fit for Haskell.</p>
<p>The CUFP tutorials are high quality affairs — often taught by the
people that literally wrote the book — and aim to instill much knowledge
in a small amount of time. <a href="http://cufp.org/conference/schedule">This year</a>, we have a <a href="http://cufp.org/conference/sessions/2013/haskell-day-1">Haskell tutorial</a> taught by luminaries
Andres Löh and Simon Marlow; Yaron Minsky and Anil Madhavapeddy are
taking a break from writing <a href="http://shop.oreilly.com/product/0636920024743.do">Real World Ocaml</a> to give <a href="http://cufp.org/conference/sessions/2013/ocaml-tutorial">a tutorial</a> about the very same subject;
<a href="http://okmij.org/ftp/">Oleg Kiselyov</a>, a long-time
<a href="http://haskell.spreadshirt.com/oleg-already-did-it-A6499531">pillar of the FP community</a> is giving a <a href="http://cufp.org/conference/sessions/2013/systematic-generation-optimal-code-metaocaml">tutorial</a> on <a href="http://www.metaocaml.org">MetaOCaml</a>.
Erlang gets double treatment: <a href="http://www.cs.kent.ac.uk/~sjt/">Simon Thompson</a>
opens with an <a href="http://cufp.org/conference/sessions/2013/erlang-101-your-introduction-concurrency-and-multi">Erlang tutorial</a> focusing on concurrency and
multi-core programming; Steve Vinoski, of Basho, finishes by
<a href="http://cufp.org/conference/sessions/2013/erlang-web-frameworks">teaching us Erlang web programming</a>.
<a href="http://clojure.org">Clojure</a> is also represented beginning with a
<a href="http://cufp.org/conference/sessions/2013/clojure-tutorial">half-day language tutorial</a> from Luke Vander Hart, followed by
<a href="http://www.leonardoborges.com/">Leonardo Borges</a> teaching one on
<a href="http://cufp.org/conference/sessions/2013/bending-clojure-your-will-macros-and-domain-specif">Clojure macros and DSLs</a>. Finally, Dean Wampler is giving a double
feature on Scala, beginning with a <a href="http://cufp.org/conference/sessions/2013/seductions-scala">language tutorial</a> focusing on functional
programming and concurrency; Dean will then <a href="http://cufp.org/conference/sessions/2013/scalding-scala-tool-data-analytics-hadoop-systems">hold court</a> on <a href="https://github.com/twitter/scalding">Scalding</a>, a powerful Scala-based tool for doing analytics over
large datasets.</p>
<p>No workshop would be complete without a keynote:
<a href="http://www.davethomas.net">“SmallTalk” Dave Thomas</a> is giving our
keynote this year on a as-yet-unrevealed topic relating to functional
programming.</p>
<p>As well as the main
<a href="http://icfpconference.org/icfp2013/index.html">ICFP conference</a>, there are
other affiliated events that are well worth attending. The
<a href="http://www.haskell.org/haskell-symposium/2013/">Haskell Symposium</a> is
held the day after the CUFP talks, and the
<a href="http://ocaml.org/meetings/ocaml/2013/">OCaml workshop</a> the day
after that.</p>
<p>Submissions for the main workshop (see the <a href="http://cufp.org/2013cfp">CFP</a>)
are due by <em>29th of June, 2013</em>. Please consider making a submission if you
have experience you want to share, technologies or techniques you want to
talk about, or just about anything else you think would be compelling to the
audience. To get a feel for the nature of the talks, I recommend reading the
<a href="http://anil.recoil.org/papers/2011-cufp-scribe-preprint.pdf">scribe report from 2011</a>.</p>
<p>Follow <a href="https://twitter.com/cufpconference/">@cufpconference</a> on Twitter
for more updates on the workshop, and please do not hestitate to contact
Mike or I if you have any questions:</p>
<pre><code>marius(at)monkey(dot)org
sperber(at)deinprogramm(dot)de
</code></pre>
<p>See you in Boston!</p>
Futures aren't ersatz threads2013-04-02T00:00:00-07:00http://monkey.org/~marius//futures-arent-ersatz-threads<p>(Originally published at <a href="http://aboutwhichmorelater.tumblr.com/post/46862769549/futures-arent-ersatz-threads">aboutwhichmorelater.tumblr.com</a>.)</p>
<p>Concurrent programming is an increasingly important topic in the context of building modern distributed systems. Most such systems have a large amount of <em>inherent</em> concurrency. For example: To accommodate for large corpus sizes, a search engine splits its index into many small pieces. In order to efficiently satisfy queries, it must issue requests to each of these <em>shards</em> concurrently.</p>
<p>Threads (or processes) are a common abstraction for programming concurrent systems: there are multiple <em>threads of execution</em>, each of which has its own stack. The system manages how these threads are mapped onto physical hardware, and how they are preempted and interleaved. This allows the operating system to suspend execution of a thread while it is waiting for I/O operations, allowing other threads to proceed. Traditionally threads have had both high fixed costs (stacks must be allocated in increments of page sizes) as well high context switching costs. The emergence of <em>event based</em> programming addressed these issues: there is only one thread, with a run-loop, and I/O events are explicitly dispatched (eg. data is ready to be read, a write completed, etc.). While this reduces the cost of threads (you have only one stack, you don’t need the OS to interleave your executions), the programming model is awkward: the programmer has to split the program up into pieces delimited by I/O operations. In C, in particular, you have to declare separate functions for each of these pieces. The <a href="http://www.wangafu.net/~nickm/libevent-book/01_intro.html">introduction of the libevent book</a> has more details. This is also the model of <code>node.js</code> though, because javascript has first-class closures, the problem is ameliorated somewhat: by nesting callbacks, you can easily share context. This improves brevity but not modularity: each sequence of actions is fixed in place, errors must be handled very carefully. Nontrivial applications become difficult to reason about very quickly.</p>
<p>The assumptions upon which event-based programming is based have largely withered: context switches aren’t very expensive anymore, and memory has become plentiful. However, the degree to which our systems need to be concurrent has also increased. It is not uncommon to handle 100s of thousands, if not millions, of operations simultaneously. The archetypical example of this sort of application are so-called <a href="http://www.briandrought.com/blog/?p=56">“long-polling”</a> web servers. </p>
<p>Languages like <a href="http://www.haskell.org/haskellwiki/Haskell">Haskell</a> and <a href="http://golang.org">Go</a> implement lightweight threads in their runtimes, allowing for cheaply maintaining millions of threads of execution, with the runtime itself multiplexing I/O operations. Go manages this by using <em>segmented stacks</em>: it allocates stack space as needed. This requires both the complicity of the compiler and a different ABI, but that’s the beauty of being able to start over.</p>
<p>So, clearly, “events vs. threads” is a false dichotomy; the statement doesn’t even make much sense, and conflates two separate concerns: They denote two concrete programming models, differing in (1) how context is encoded (heap vs. stack), and (2) where multiplexing is done (in a library, or runtime/OS).</p>
<p>In Finagle and elsewhere, we use <a href="http://twitter.github.com/finagle/guide/Futures.html">composable futures</a> as the basis of our concurrent programming model (these are quite different than the venerable <code>java.util.Future</code> and its brethren). Futures, it is often argued, make up for the deficiencies of traditional “event programming”: callbacks are tedious, compromise modularity, and make for spaghetti-like code that is difficult to understand. Futures correct this by introducing constructs that make callbacks manageable.</p>
<p>This misses the point.</p>
<p>Futures offer a concurrent programming model that translates naturally to the types of concerns that arise when building distributed systems. Take for instance remote procedure calls; RPCs are inherently <em>asynchronous</em>. This means that each component operates at their own speed, and the components fail independently of each other. There is no shared clock, and no shared bus. RPCs usually have dramatically different latency characteristics than a typical local operation. In fact, this holds for any sort of I/O, not just RPCs. </p>
<p><span class="newthought">Futures model the real world truthfully.</span>
A <code>Future[T]</code> represents a <code>T</code>-typed result, which may or may not be delivered at some time in the future, or which may fail altogether. Like real-world asynchronous operations, a future is always in one of 3 states: incomplete, completed successfully, or completed with a failure. </p>
<p><span class="newthought">Futures provide a firewall between synchronous and asynchronous operations.</span>
Because futures are typed differently than regular values (a value furnished by a future has type <code>Future[T]</code>, not <code>T</code>), you can easily, at a glance, discern which operations imply asynchronous semantics. This allows you to reason more easily about the runtime semantics of your code: not only are such operations slower, but they have very different failure semantics. Furthermore, futures are “tainting”: if you want to incorporate the result of a future, you have to “lift” other values in to them. That is, if you are computing anything that requires an asynchronous, the operation itself becomes asynchronous - its type must be <code>Future[T]</code>. The upshot is that semantics are witnessed by types; we can use the type system to model and enforce behavior. Put another way: <em>we can use the compiler to guarantee certain behavior</em>.</p>
<p><span class="newthought">Futures compose well.</span>
Futures, like physically asynchronous operations, are closed under composition. This is important: the composition of two asynchronous operations can in turn be described as an asynchronous operation. Futures work the same way: I might compose two Futures <a href="http://twitter.github.com/finagle/guide/Futures.html#sequential-composition">sequentially</a> (eg. because there is a data dependency) or <a href="http://twitter.github.com/finagle/guide/Futures.html#concurrent-composition">concurrently</a> (because there is not), both of which produce a new future. When composing Futures, failure handling is baked in: When composed sequentially, the sequence of operations short circuit at the first failure; when composed in a concurrent configuration, any underlying failure also fails the composed operation.</p>
<p><span class="newthought">Futures are <a href="http://en.wikipedia.org/wiki/Persistent_data_structure">persistent</a></span>
A <code>Future[T]</code> always means the same thing, refers to the same underlying operation, no matter how it is later used. To string operations together (concurrently or sequentially), <em>new</em> futures are produced, with new semantics. This makes reasoning about Futures very simple; you needn’t worry about how they might be used or modified elsewhere.</p>
<p><span class="newthought">Futures liberate semantics from mechanics</span>.
By stringing futures together via the various combinators, the programmer is expressing the <em>semantics</em> of an operation, not its execution mechanics. In this example, adapted from the <a href="http://twitter.github.com/finagle/guide/Futures.html#concurrent-composition">Finagle User’s Guide</a>, we fetch all images in a web page:</p>
<pre class="code">
val allImages: Future[Seq[Image]] =
fetchUrl(url) flatMap { bytes =>
val fetches = findImageUrls(bytes) map { url =>
fetchUrl(url)
}
Future.collect(fetches)
}
</pre>
<p>This can be read thus:</p>
<blockquote>
<p><code>allImages</code> is the operation that first fetches <code>url</code>, then for each image URL in the body of the page that was fetched, the result of fetching all of those URLs as images in turn.</p>
</blockquote>
<p>The code is fairly close to the English description of what we want to do. We have described the operation, but not how to perform it. In the description, there is inherent <em>concurrency</em>: the subsequent image URLs may all be fetched at the same time; this translates naturally into the concurrent combinators used with Futures, in this case <code>Future.collect</code>. Failure handling is omitted from the description, and is also omitted from the code. In this case, it is clear that we cannot proceed if any operation fails, and those are also the semantics of the composite operation (<code>allImages</code>). The data dependencies, inherent to the problem being solved, are all-revealing.</p>
<p>This separation enhances the ease of reasoning: the meaning of the operation isn’t intertwined instructions for how to go about executing it. No threads are spun up, no errors explicitly handled, no explicit coordination mechanism is required to gather the result of the image fetches. Instead, there are only <em>data dependencies</em> — to fetch all of the images, we need to fetch their URLs; to get the URLs we need to fetch the web page. </p>
<p>This property also allows us to separately consider the runtime behavior of such operations. The Future implementation might choose to limit the number of concurrent operations, for example, or to exploit inherent locality (like biasing connection pools so that a given connection is always handled by one underlying thread). A library might thread through tracing data to diagnose operations. This is not hypothetical: <a href="http://twitter.github.com/finagle/">Finagle</a> does all of this, and it’s done entirely as a library without any special runtime support.</p>
<h2 id="qed">Q.E.D.?</h2>
<p>Futures present a concurrent programming model that is appealing on its own. They should not be seen as a poor-man’s version of threads.</p>
<p>Ultimately such abstractions exist in the service of writing robust, safe, performant, and modular software. Futures have served us very well in this regard at Twitter, and I think they’ll have a long shelf-life as a go-to abstraction for constructing concurrent systems. We’re in the <a href="http://www.sauria.com/blog/2009/10/05/the-cambrian-period-of-concurrency/">Cambrian period of concurrency</a>, and I predict futures will emerge as one of the survivors.</p>
<p>This isn’t to say that other models aren’t also good. I’m personally a huge fan of Go/Haskell’s cheap threads model: because they are built into the language and runtime, their usage is enforced. This is perhaps the main sticking point of futures. Because they are implemented as 3rd-party libraries, their usage can be spotty, and you may find yourself always writing little bridges or adapters. At Twitter, we’re lucky in that we have a <a href="http://twitter.github.com/finagle/">common system</a> upon which our systems are built, so this is less of a problem. But this isn’t inherent to the model: C# and F# has it into the language and runtime. Scala’s <a href="http://docs.scala-lang.org/sips/pending/futures-promises.html">SIP-14</a> also promises to unify that ecosystem.</p>
Implementing python style generators with delimited continuations2010-09-12T00:00:00-07:00http://monkey.org/~marius//implementing-python-style-generators-with-delimited-continuations<p><a href="http://ambassadortothecomputers.blogspot.com/2010/08/mixing-monadic-and-direct-style-code.html">Jake’s article</a>
on implementing LWT-compatible fibers with
<a href="http://okmij.org/ftp/continuations/caml-shift.pdf">Oleg’s delimited continuation library</a>
got me thinking about one of my favorite Python features:
<a href="http://docs.python.org/tutorial/classes.html#generators">generators</a>.</p>
<h2 id="delimited-continuations">Delimited continuations</h2>
<p>Delimited continuations have a very simple API (note: I’m going to
consider only the high level API provided by <code>delimcc</code> here consistent
with the abstract view of delimited continuations. Jake describes the
low-level API in
<a href="http://ambassadortothecomputers.blogspot.com/2010/08/mixing-monadic-and-direct-style-code.html">his article</a>).
The core API is:</p>
<div class="highlight"><pre><code class="language-ocaml" data-lang="ocaml"><span class="k">val</span> <span class="n">new_prompt</span> <span class="o">:</span> <span class="kt">unit</span> <span class="o">-></span> <span class="k">'</span><span class="n">a</span> <span class="n">prompt</span>
<span class="k">val</span> <span class="n">push_prompt</span> <span class="o">:</span> <span class="k">'</span><span class="n">a</span> <span class="n">prompt</span> <span class="o">-></span> <span class="o">(</span><span class="kt">unit</span> <span class="o">-></span> <span class="k">'</span><span class="n">a</span><span class="o">)</span> <span class="o">-></span> <span class="k">'</span><span class="n">a</span>
<span class="k">val</span> <span class="n">shift0</span> <span class="o">:</span> <span class="k">'</span><span class="n">a</span> <span class="n">prompt</span> <span class="o">-></span> <span class="o">((</span><span class="k">'</span><span class="n">b</span> <span class="o">-></span> <span class="k">'</span><span class="n">a</span><span class="o">)</span> <span class="o">-></span> <span class="k">'</span><span class="n">a</span><span class="o">)</span> <span class="o">-></span> <span class="k">'</span><span class="n">b</span></code></pre></div>
<p><code>new_prompt</code> simply creates the “handle”, <code>push_prompt</code> (traditionally
called <code>reset</code>) sets the limit of the continuation (how far up the
stack we will go), <code>shift0</code> captures the continuation and gives it to
the passed function. Crucially, the continuation captured by <code>shift0</code>
is <em>delimited</em> by the outer <code>push_prompt</code>. The “region” of the
continuation delimited by <code>push_prompt</code> and <code>shift0</code> consists of the
stack frames between the two (this also means you cannot call <code>shift0</code>
anywhere outside the scope of <code>push_prompt</code>).</p>
<p>An example should clarify. Let’s first do something illegal.</p>
<div class="highlight"><pre><code class="language-ocaml" data-lang="ocaml"><span class="o">#</span> <span class="k">open</span> <span class="nc">Delimcc</span><span class="o">;;</span>
<span class="o">#</span> <span class="k">let</span> <span class="n">p</span> <span class="o">=</span> <span class="n">new_prompt</span> <span class="bp">()</span><span class="o">;;</span>
<span class="k">value</span> <span class="n">p</span> <span class="o">:</span> <span class="nn">Delimcc</span><span class="p">.</span><span class="n">prompt</span> <span class="k">'</span><span class="o">_</span><span class="n">a</span> <span class="o">=</span> <span class="o"><</span><span class="n">abstr</span><span class="o">></span>
<span class="o">#</span> <span class="n">push_prompt</span> <span class="n">p</span> <span class="n">identity</span><span class="o">;;</span>
<span class="o">-</span> <span class="o">:</span> <span class="kt">unit</span> <span class="o">=</span> <span class="bp">()</span>
<span class="o">#</span> <span class="n">shift0</span> <span class="n">p</span> <span class="o">(</span><span class="k">fun</span> <span class="n">k</span> <span class="o">-></span> <span class="n">k</span> <span class="bp">()</span><span class="o">);;</span>
<span class="nc">Exception</span><span class="o">:</span> <span class="nc">Failure</span> <span class="s2">"No prompt was set"</span><span class="o">.</span></code></pre></div>
<p>Here, we tried to <code>shift0</code> outside of the <code>push_prompt</code>. Nope, can’t
do that.</p>
<div class="highlight"><pre><code class="language-ocaml" data-lang="ocaml"><span class="o">#</span> <span class="n">push_prompt</span> <span class="n">p</span> <span class="o">(</span><span class="k">fun</span> <span class="bp">()</span> <span class="o">-></span> <span class="n">shift0</span> <span class="n">p</span> <span class="o">(</span><span class="k">fun</span> <span class="n">k</span> <span class="o">-></span> <span class="n">k</span> <span class="mi">123</span><span class="o">));;</span>
<span class="o">-</span> <span class="o">:</span> <span class="kt">int</span> <span class="o">=</span> <span class="mi">123</span></code></pre></div>
<p>Here, the inner function (passed to <code>shift0</code>) was given the captured
continuation <code>k</code> (as delimited by <code>push_prompt</code>), calling it with the
argument <code>123</code> which was subsequently returned.</p>
<p>The neat thing about delimited continuations is that they <em>capture the
state entire delimited stack</em>. That is, calling <code>k ARG</code> from within
<code>shift0</code> is equivalent to replacing the <code>shift0</code> statement with <code>ARG</code>,
returning the result of <code>push_prompt</code>. Furthermore, it’s a first class
continuation, so we may use it multiple times. To wit:</p>
<div class="highlight"><pre><code class="language-ocaml" data-lang="ocaml"><span class="o">#</span> <span class="n">push_prompt</span> <span class="n">p</span> <span class="o">(</span><span class="k">fun</span> <span class="bp">()</span> <span class="o">-></span> <span class="mi">10</span> <span class="o">*</span> <span class="n">shift0</span> <span class="n">p</span> <span class="o">(</span><span class="k">fun</span> <span class="n">k</span> <span class="o">-></span> <span class="n">k</span> <span class="mi">1</span> <span class="o">+</span> <span class="n">k</span> <span class="mi">3</span><span class="o">));;</span>
<span class="o">-</span> <span class="o">:</span> <span class="kt">int</span> <span class="o">=</span> <span class="mi">40</span></code></pre></div>
<p>Here, <code>k</code> is equivalent to the function <code>( * ) 10</code>. The argument &
return types needn’t be uniform:</p>
<div class="highlight"><pre><code class="language-ocaml" data-lang="ocaml"><span class="o">#</span> <span class="n">push_prompt</span> <span class="n">p</span> <span class="o">(</span><span class="k">fun</span> <span class="bp">()</span> <span class="o">-></span> <span class="n">string_of_int</span> <span class="o">(</span><span class="n">shift0</span> <span class="n">p</span> <span class="o">(</span><span class="k">fun</span> <span class="n">k</span> <span class="o">-></span> <span class="s2">"hey "</span> <span class="o">^</span> <span class="n">k</span> <span class="mi">123</span><span class="o">)));;</span>
<span class="o">-</span> <span class="o">:</span> <span class="kt">string</span> <span class="o">=</span> <span class="s2">"hey 123"</span></code></pre></div>
<h2 id="leaking-continuations">Leaking continuations</h2>
<p>What makes delimited continuations yet more interesting is that the
continuations themselves can escape the scope of <code>shift0</code>, and be
restarted from outside.</p>
<p>Because of the type signatures of <code>shift0</code>, we need to unify the
types, so let’s first define an ADT to do so:</p>
<div class="highlight"><pre><code class="language-ocaml" data-lang="ocaml"><span class="o">#</span> <span class="k">type</span> <span class="n">t</span> <span class="o">=</span> <span class="nc">Done</span> <span class="o">|</span> <span class="nc">More</span> <span class="k">of</span> <span class="o">(</span><span class="kt">unit</span> <span class="o">-></span> <span class="n">t</span><span class="o">);;</span></code></pre></div>
<p>Note that we can’t just use <code>option</code> here, it seems, because we need a
recursive type.</p>
<div class="highlight"><pre><code class="language-ocaml" data-lang="ocaml"><span class="o">#</span> <span class="k">let</span> <span class="n">count</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">;;</span>
<span class="o">#</span> <span class="k">let</span> <span class="nc">More</span> <span class="n">k</span> <span class="o">=</span> <span class="n">push_prompt</span> <span class="n">p</span> <span class="o">(</span><span class="k">fun</span> <span class="bp">()</span> <span class="o">-></span>
<span class="k">while</span> <span class="bp">true</span> <span class="k">do</span>
<span class="n">count</span> <span class="o">:=</span> <span class="o">!</span><span class="n">count</span> <span class="o">+</span> <span class="mi">1</span><span class="o">;</span>
<span class="n">shift0</span> <span class="n">p</span> <span class="o">(</span><span class="k">fun</span> <span class="n">k</span> <span class="o">-></span> <span class="nc">More</span> <span class="n">k</span><span class="o">)</span>
<span class="k">done</span><span class="o">;</span>
<span class="nc">Done</span><span class="o">);;</span></code></pre></div>
<p>Here we are using our unified type trick to return the continuation
itself (note how <code>push_prompt</code> and <code>shift</code> have the same return
type). Let’s play!</p>
<div class="highlight"><pre><code class="language-ocaml" data-lang="ocaml"><span class="o">#</span> <span class="o">!</span><span class="n">count</span><span class="o">;;</span>
<span class="o">-</span> <span class="o">:</span> <span class="kt">int</span> <span class="o">=</span> <span class="mi">1</span>
<span class="o">#</span> <span class="n">k</span> <span class="bp">()</span><span class="o">;;</span>
<span class="o">-</span> <span class="o">:</span> <span class="n">t</span> <span class="o">=</span> <span class="nc">More</span> <span class="o"><</span><span class="k">fun</span><span class="o">></span>
<span class="o">#</span> <span class="n">k</span> <span class="bp">()</span><span class="o">;;</span>
<span class="o">-</span> <span class="o">:</span> <span class="n">t</span> <span class="o">=</span> <span class="nc">More</span> <span class="o"><</span><span class="k">fun</span><span class="o">></span>
<span class="o">#</span> <span class="o">!</span><span class="n">count</span><span class="o">;;</span>
<span class="o">-</span> <span class="o">:</span> <span class="kt">int</span> <span class="o">=</span> <span class="mi">3</span></code></pre></div>
<h2 id="generators">Generators</h2>
<p>We now have a pretty good idea of what’s needed to implement
generators with an API similar to:</p>
<div class="highlight"><pre><code class="language-ocaml" data-lang="ocaml"><span class="k">let</span> <span class="n">producer</span> <span class="bp">()</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">state</span> <span class="o">=</span> <span class="o">...</span> <span class="k">in</span>
<span class="n">gen</span> <span class="k">begin</span> <span class="k">fun</span> <span class="n">yield</span> <span class="o">-></span>
<span class="n">yield</span> <span class="n">state</span><span class="o">;</span> <span class="o">...;</span> <span class="n">yield</span> <span class="n">state</span><span class="o">;</span> <span class="o">...;</span> <span class="n">yield</span> <span class="n">state</span>
<span class="k">end</span></code></pre></div>
<p>So we need a function <code>gen</code> that takes a “yielder” (in Python
terminology) that needs to capture the continuation, communicate the
value and resume execution when the next value is asked for by the
consumer:</p>
<div class="highlight"><pre><code class="language-ocaml" data-lang="ocaml"><span class="k">let</span> <span class="n">consumer</span> <span class="bp">()</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">g</span> <span class="o">=</span> <span class="n">producer</span> <span class="bp">()</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">v0</span> <span class="o">=</span> <span class="n">g</span> <span class="bp">()</span> <span class="k">in</span>
<span class="o">...</span>
<span class="k">let</span> <span class="n">vN</span> <span class="o">=</span> <span class="n">g</span> <span class="bp">()</span></code></pre></div>
<p>It turns out that this is a very natural fit, and we can implement
full duplex generators in just a few lines, using the ideas developed
above. Here is the full source code:</p>
<div class="highlight"><pre><code class="language-ocaml" data-lang="ocaml"><span class="k">type</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">)</span> <span class="n">t</span> <span class="o">=</span> <span class="nc">Done</span> <span class="o">|</span> <span class="nc">More</span> <span class="k">of</span> <span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">b</span> <span class="o">-></span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">)</span> <span class="n">t</span><span class="o">)</span>
<span class="k">let</span> <span class="n">gen</span> <span class="n">f</span> <span class="o">=</span>
<span class="c">(*</span>
<span class="c"> * Note: the first value to yield gets thrown away as the generator</span>
<span class="c"> * has not yet started.</span>
<span class="c"> *)</span>
<span class="k">let</span> <span class="n">start</span> <span class="o">_</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">p</span> <span class="o">=</span> <span class="nn">Delimcc</span><span class="p">.</span><span class="n">new_prompt</span> <span class="bp">()</span> <span class="k">in</span>
<span class="nn">Delimcc</span><span class="p">.</span><span class="n">push_prompt</span> <span class="n">p</span> <span class="k">begin</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-></span>
<span class="n">f</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-></span> <span class="nn">Delimcc</span><span class="p">.</span><span class="n">shift0</span> <span class="n">p</span> <span class="o">(</span><span class="k">fun</span> <span class="n">k</span> <span class="o">-></span> <span class="nc">More</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">k</span><span class="o">)));</span> <span class="nc">Done</span>
<span class="k">end</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">next</span> <span class="o">=</span> <span class="n">ref</span> <span class="n">start</span> <span class="k">in</span>
<span class="k">fun</span> <span class="n">rv</span> <span class="o">-></span>
<span class="k">match</span> <span class="o">!</span><span class="n">next</span> <span class="n">rv</span> <span class="k">with</span>
<span class="o">|</span> <span class="nc">More</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">k</span><span class="o">)</span> <span class="o">-></span> <span class="n">next</span> <span class="o">:=</span> <span class="n">k</span><span class="o">;</span> <span class="nc">Some</span> <span class="n">x</span>
<span class="o">|</span> <span class="nc">Done</span> <span class="o">-></span> <span class="nc">None</span></code></pre></div>
<p>And sample use:</p>
<div class="highlight"><pre><code class="language-ocaml" data-lang="ocaml"><span class="k">let</span> <span class="k">rec</span> <span class="n">take_all</span> <span class="n">t</span> <span class="n">count</span> <span class="o">=</span>
<span class="k">match</span> <span class="n">t</span> <span class="n">count</span> <span class="k">with</span>
<span class="o">|</span> <span class="nc">Some</span> <span class="n">i</span> <span class="o">-></span> <span class="n">printf</span> <span class="s2">"took: %d</span><span class="se">\n</span><span class="s2">"</span> <span class="n">i</span><span class="o">;</span> <span class="n">take_all</span> <span class="n">t</span> <span class="o">(</span><span class="n">count</span> <span class="o">+</span> <span class="mi">1</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">None</span> <span class="o">-></span> <span class="bp">()</span>
<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">take</span> <span class="o">=</span> <span class="n">gen</span> <span class="k">begin</span> <span class="k">fun</span> <span class="n">yield</span> <span class="o">-></span>
<span class="k">for</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">to</span> <span class="mi">10</span> <span class="k">do</span>
<span class="k">let</span> <span class="n">rv</span> <span class="o">=</span> <span class="n">yield</span> <span class="n">i</span> <span class="k">in</span>
<span class="n">printf</span> <span class="s2">"got: %d</span><span class="se">\n</span><span class="s2">"</span> <span class="n">rv</span>
<span class="k">done</span>
<span class="k">end</span> <span class="k">in</span>
<span class="n">take_all</span> <span class="n">take</span> <span class="mi">100</span></code></pre></div>
<p>The only additional work we have to do is to type it for
bi-directional communication, and to store the next continuation in a
ref cell for the next invocation. Given that the generators developed
here are full duplex, we can implement co-routines as outlined in
<a href="http://www.python.org/dev/peps/pep-0342/">PEP 342</a>.</p>
<p>You can also get it as a gist <a href="http://gist.github.com/576922">here</a>. I
also highly recommend
<a href="http://ambassadortothecomputers.blogspot.com/2010/08/mixing-monadic-and-direct-style-code.html">Jake’s article</a>
which explores <code>delimcc</code> from a lower level of abstraction.</p>
Self-contained emacs2010-02-21T00:00:00-08:00http://monkey.org/~marius//self-contained-emacs<p>One annoying thing about using emacs on remote systems is the absence
of your own initialization & configuration code. It’s of course
possible to push your own, but often it’s annoying. Configuration
often gets rather involved: my own <code>.emacs.d</code> directory contains many
packages and libraries that are not shipped with standard emacs
distributions. Another common scenario is that you use shared
accounts, making editor configurations rather intrusive.</p>
<p>So, I sought to simplify the situation.</p>
<p><a href="http://github.com/mariusaeriksen/make-emacs"><code>make-emacs</code></a> creates for you a simple, self-contained
and relocatable script that allows you to invoke emacs with your own
configuration anywhere. For example:</p>
<pre><code> $ make-emacs ~/.emacs.d /tmp/e
$ scp /tmp/e remoteserver:
$ ssh remoteserver
$ ./e MYFILE
extracting emacs.d..
<emacs bliss>
</code></pre>
<p>It uses <a href="http://en.wikipedia.org/wiki/Shar">shar</a> to create a self-extracting archive and wraps
that extraction code to invoke emacs properly. It also caches the
extracted configuration files, so that it only has to perform a
potentially costly <code>shar</code> extraction once.</p>
<p>Get it on <a href="http://github.com/">GitHub</a> <a href="http://github.com/mariusaeriksen/make-emacs">here</a>.</p>
Emacs as a tiling window manager2010-01-26T00:00:00-08:00http://monkey.org/~marius//emacs-as-a-tiling-window-manager<p>I’ve been using <code>emacs</code> in in full-screen mode lately. It provides a
nice, distraction-free environment. If you’re using <a href="http://homepage.mac.com/zenitani/emacs-e.html">carbon
emacs</a> it’s quite easy:</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">(defun mac-toggle-max-window ()
(interactive)
(set-frame-parameter nil 'fullscreen
(if (frame-parameter nil 'fullscreen)
nil
'fullboth)))
(global-set-key (kbd "<C-M-return>") 'mac-toggle-max-window)</code></pre></div>
<p>I also really enjoy the efficacy of navigation provided by tiling
window managers such as <a href="http://xmonad.org/">xmonad</a>. So, why not make
emacs behave like it?</p>
<p><a href="http://gist.github.com/287633">emacsd-tile.el</a> is a really tiny
configuration snippet to provide just this.</p>
<p>It provides xmonad-inspired keyboard shortcuts:</p>
<table>
<tr><td><code>M-{j,k,h,l}</code></td><td style="padding: 3px;">→</td><td>move to window (down, up, left, right)</td></tr>
<tr><td><code>M-S-{j,k,h,l}</code></td><td style="padding: 3px;">→</td><td>enlarge/shrink horizontally/vertically</td></tr>
<tr><td><code>M-C-{j,k,h,l}</code></td><td style="padding: 3px;">→</td><td>swap with (down, up, left, right)</td></tr>
</table>
<p>This mostly uses just stuff that was already in <code>emacs</code>! The only part
I had to implement myself was window swapping.</p>
<p>Another really handy feature of emacs is the ability to <a href="http://www.chemie.fu-berlin.de/chemnet/use/info/emacs/emacs_14.html#SEC69">save window
configuration in
registers</a>. With
this, you can “stash” away an entire window configuration and recall
it later.</p>
<p>Here’s a <a href="http://monkey.org/~marius/emacs-tile.png">screenshot</a>,
though it’s not really descriptive because it doesn’t show the
keybindings in action, but it may convince you that emacs can at least
<em>look</em> like a tiling window manager :-)</p>
<p>Any other improvements or suggestions?</p>
Beautiful fixed-width fonts for OSX2010-01-10T00:00:00-08:00http://monkey.org/~marius//beautiful-fixed-width-fonts<p>(Or, <em>“an ode to 9x15”</em>)</p>
<p>I’ve always had great appreciation for the fixed-width fonts
<a href="http://www.cl.cam.ac.uk/~mgk25/ucs-fonts.html">distributed with X11</a>
(“<code>misc-fixed-*</code>”): 6x13, 7x14, and especially 9x15 (my all-time
favorite).</p>
<p><img src="images/9x15.png" alt="9x15 in action" /></p>
<p>I never really found any type so satisfying. It is extremely crisp and
legible, is not dull, and doesn’t go all fuzzy in its bold
variant. Its glyphs are extraordinarily well-suited to the kinds of
strange things we tend to do with the ASCII character set in code.</p>
<p><img src="images/9x15-code.png" alt="9x15 in code" /></p>
<p><a href="http://www.ccheever.com/blog/?page_id=2">Charlie Cheever</a> previously
converted <a href="http://www.ccheever.com/blog/?p=135">9x15 to an OSX dfont</a>,
so I followed suit to fill in with the rest of the sizes using
<a href="http://fontforge.sourceforge.net/">FontForge</a>.</p>
<p><a href="http://monkey.org/~marius/x11-misc-fixed.tar.gz">http://monkey.org/~marius/x11-misc-fixed.tar.gz</a></p>
<p>Included are 6x13, 7x14, 9x15 and 10x20. Make sure to use them only
with their respective sizes (ie. 13pt for 6x13, 15pt for 9x15 and so
on) and without anti aliasing turned on (bolds won’t render
correctly).</p>
<p>Enjoy!</p>
Haskell is beautiful in practice2009-11-05T00:00:00-08:00http://monkey.org/~marius//why-haskell-is-beautiful-in-practice<p>I’ve been writing a bit of <a href="http://www.haskell.org">Haskell</a> lately (see my <a href="http://github.com/mariusaeriksen/">GitHub page</a>), and it’s an absolutely beautiful experience. And I mean that in every way: The language itself of course, but also the community, the documentation, the package system and the <a href="http://hackage.haskell.org/platform/">Haskell platform</a> ultimately help make the practice of writing Haskell more pleasing than any other I’ve had any experience with.</p>
<p>I find that Haskell lets me translate so many ideas and abstractions naturally and without the fuzz of too much indirection. My hope is to give you a small taste of what I’m talking about. Please note that this is going to contain some deliberate (but mostly inconsequential) inaccuracies and I will also leave out details here and there. A lot of the code is also “paraphrased” to highlight the ideas behind it.</p>
<p>I recently wrote an implementation of <a href="http://bert-rpc.org">BERT</a> in Haskell (<a href="http://github.com/mariusaeriksen/bert/">github</a>). I had the need for an RPC mechanism, and it seemed to fit my requirements rather well. “BERTs” are a subset of <a href="http://www.erlang.org">Erlang</a> terms, which are composable and have a straightforward external representation (albeit not as space efficient as something like Google’s protocol buffers). So I figured this might be a good starting point for demonstrating some of the substantial ways in which Haskell provides facilities to create programs that are not only succinct but also readable and robust. At the same time, I hope to convince you that Haskell provides the kind of abstraction & encapsulation that is often much more natural than other approaches.</p>
<h2 id="types">Types</h2>
<p>We need to be able to represent BERT terms in Haskell. Thus we introduce an algebraic type representing a Term (<a href="http://github.com/mariusaeriksen/bert/blob/master/Data/BERT/Types.hs#L18">github</a>).</p>
<div class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="c1">-- | A single BERT term.</span>
<span class="kr">data</span> <span class="kt">Term</span>
<span class="c1">-- Simple (erlang) terms:</span>
<span class="ow">=</span> <span class="kt">IntTerm</span> <span class="kt">Int</span>
<span class="o">|</span> <span class="kt">FloatTerm</span> <span class="kt">Float</span>
<span class="o">|</span> <span class="kt">AtomTerm</span> <span class="kt">String</span>
<span class="o">|</span> <span class="kt">TupleTerm</span> <span class="p">[</span><span class="kt">Term</span><span class="p">]</span>
<span class="o">|</span> <span class="kt">BytelistTerm</span> <span class="kt">ByteString</span>
<span class="o">|</span> <span class="kt">ListTerm</span> <span class="p">[</span><span class="kt">Term</span><span class="p">]</span>
<span class="o">|</span> <span class="kt">BinaryTerm</span> <span class="kt">ByteString</span>
<span class="o">|</span> <span class="kt">BigintTerm</span> <span class="kt">Integer</span>
<span class="o">|</span> <span class="kt">BigbigintTerm</span> <span class="kt">Integer</span></code></pre></div>
<p>This is pretty straightforward to read: a <code>Term</code> can be either an <code>IntTerm</code>, a <code>FloatTerm</code> and so forth. The right column specifies the data for the type. The <code>IntTerm</code> carries an integer, etc. An important detail here is that these are not different types: they are all type <em>constructors</em> for the type <code>Term</code>. That is, these are different ways to create a <code>Term</code> type. Algebraic types may also be recursively defined. The <code>ListTerm</code> constructor represents a list of other <code>Term</code>s. Already with this simple declaration, we’ve expressed most of the semantics of BERT terms. We can now express composite BERT terms. For example BERT defines dictionaries to be represented as <code>{bert, dict, [{key, value}]</code>. In Haskell:</p>
<div class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="kt">TupleTerm</span> <span class="p">[</span> <span class="kt">AtomTerm</span> <span class="s">"bert"</span><span class="p">,</span> <span class="kt">AtomTerm</span> <span class="s">"dict"</span>
<span class="p">,</span> <span class="kt">ListTerm</span> <span class="p">[</span><span class="kt">TupleTerm</span> <span class="p">[</span><span class="kt">AtomTerm</span> <span class="s">"key"</span><span class="p">,</span> <span class="kt">AtomTerm</span> <span class="s">"value"</span><span class="p">]]]</span></code></pre></div>
<h2 id="typeclasses">Typeclasses</h2>
<p><code>Term</code>s encapsulate the BERT type representation, and we will write code that can encode <code>Term</code> values to a binary representation (as per the BERT spec). However, these values are not the most convenient to work with in other Haskell code. Furthermore, many <code>Term</code>s have a natural “more primitive” Haskell type (eg. <code>IntTerm</code> vs. <code>Int</code>, <code>ListTerm</code> vs. <code>[]</code>).</p>
<p>We would like to introduce <code>Term</code>s from these types and vice-versa. The most primitive way to do this is via <em>pattern matching</em> (this is ubiquitous in Haskell and other functional programming languages. Also see my <a href="pattern-matching-in-python.html">poor attempt at implementing pattern matching in Python</a>).</p>
<div class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="nf">listify</span> <span class="p">(</span><span class="kt">ListTerm</span> <span class="n">l</span><span class="p">)</span> <span class="ow">=</span> <span class="n">listify'</span> <span class="n">l</span>
<span class="nf">listify'</span> <span class="kt">[]</span> <span class="ow">=</span> <span class="kt">[]</span>
<span class="nf">listify'</span> <span class="p">((</span><span class="kt">IntTerm</span> <span class="n">x</span><span class="p">)</span><span class="kt">:</span><span class="n">xs</span><span class="p">)</span> <span class="ow">=</span> <span class="n">x</span><span class="kt">:</span><span class="n">listify'</span> <span class="n">xs</span></code></pre></div>
<p>This code introduces a list of integers (<code>[Int]</code>) from a <code>ListTerm</code> containing <code>IntTerm</code>s. Clearly going around creating little unpackers like this for everything is going to be quite cumbersome. Haskell provides a very nice solution. <em>Typeclasses</em> let you declare <em>common traits</em> to a set of types. For example, I can define a typeclass that declares the ability to translate a value of that a given type into or out of a <code>Term</code> (<a href="http://github.com/mariusaeriksen/bert/blob/master/Data/BERT/Term.hs#L116">github</a>).</p>
<div class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="kr">class</span> <span class="kt">BERT</span> <span class="n">a</span> <span class="kr">where</span>
<span class="c1">-- | Introduce a 'Term' from a Haskell value.</span>
<span class="n">showBERT</span> <span class="ow">::</span> <span class="n">a</span> <span class="ow">-></span> <span class="kt">Term</span>
<span class="c1">-- | Introduce a Haskell value from a 'Term'.</span>
<span class="n">readBERT</span> <span class="ow">::</span> <span class="kt">Term</span> <span class="ow">-></span> <span class="n">a</span></code></pre></div>
<p>This introduces two new functions. One to create a term from a Haskell value, and one to do the inverse. So what kinds of types can introduce Haskell values from BERT or vice-versa? A few are quite simple. The most trivial is for <code>Term</code> itself.</p>
<div class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="kr">instance</span> <span class="kt">BERT</span> <span class="kt">Term</span> <span class="kr">where</span>
<span class="n">showBERT</span> <span class="ow">=</span> <span class="n">id</span>
<span class="n">readBERT</span> <span class="ow">=</span> <span class="n">id</span></code></pre></div>
<p>To convert a <code>Term</code> to a <code>Term</code> is just the <code>id</code>entity function. We need to cover a few more primitive Haskell types. The <code>BERT</code> definition for lists is:</p>
<div class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="kr">instance</span> <span class="p">(</span><span class="kt">BERT</span> <span class="n">a</span><span class="p">)</span> <span class="ow">=></span> <span class="kt">BERT</span> <span class="p">[</span><span class="n">a</span><span class="p">]</span> <span class="kr">where</span>
<span class="n">showBERT</span> <span class="n">xs</span> <span class="ow">=</span> <span class="kt">ListTerm</span> <span class="p">(</span><span class="n">map</span> <span class="n">showBERT</span> <span class="n">xs</span><span class="p">)</span>
<span class="n">readBERT</span> <span class="p">(</span><span class="kt">ListTerm</span> <span class="n">xs</span><span class="p">)</span> <span class="ow">=</span> <span class="n">map</span> <span class="n">readBERT</span> <span class="n">xs</span>
<span class="n">readBERT</span> <span class="kr">_</span> <span class="ow">=</span> <span class="ne">error</span> <span class="s">"Invalid list type"</span></code></pre></div>
<p>Unlike Haskell lists, <code>BERT</code> lists can be heterogeneous: they are lists of <code>Term</code>s which, being an algebraic type, may eventually contain different types of data. Thus, to introduce a <code>BERT</code> list, we return a <code>ListTerm</code> that applies <code>showBERT</code> to every element in the list. This also explains the typeclass restriction on <code>a</code>: we require that the list we are encoding contains a type that also has a typeclass instance of <code>BERT</code>. Similarly, to decode a list, we call <code>readBERT</code> on each element, introducing a list with the decoded Haskell types. But what about the other way? Certainly we couldn’t introduce a Haskell list from a heterogeneous BERT list. The code above looks deceiving in this way, but <em>type inference</em> is going on in the background here. The type of <code>readBERT</code> is <code>Term -> [a]</code>. This inference is propagated when we pass <code>readBERT</code> to <code>map</code> as well; it’s equivalent to the following code:</p>
<div class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="nf">readBERT</span> <span class="ow">::</span> <span class="kt">Term</span> <span class="ow">-></span> <span class="p">[</span><span class="n">a</span><span class="p">]</span>
<span class="n">readBERT</span> <span class="p">(</span><span class="kt">ListTerm</span> <span class="n">xs</span><span class="p">)</span> <span class="ow">=</span> <span class="n">map</span> <span class="p">(</span><span class="n">readBERT</span> <span class="ow">::</span> <span class="kt">Term</span> <span class="ow">-></span> <span class="n">a</span><span class="p">)</span> <span class="n">xs</span></code></pre></div>
<p>Note that you can also create your own typeclass instances that suits your needs. For example if your application keeps some internal representation of some well defined value, you could create a typeclass instance to convert this representation to and from <code>Term</code>s.</p>
<h2 id="lazyness">Lazyness</h2>
<p>The RPC specification for BERT provides a rather opaque notion of a transport. The transport, in essence, is a channel through which you can send and receive BERT terms. The terms are wrapped in a 4-byte header specifying its length.</p>
<p>Haskell has a <code>Handle</code> type that is an opaque representation of a file-like object. Sockets can also be fronted by handles. A popular module, <code>Data.ByteString</code> provides a lazy bytestring type that can be backed by a handle. It’s a type that looks like a bytestring, smells like a bytestring, and acts like a bytestring, except that when it needs more data, it requests it from the handle it’s been constructed with. Similarly, our binary term decoder reads from such bytestrings, so it does not take much imagination to produce a list that represents the infinite stream of packets coming from the BERT peer (<a href="http://github.com/mariusaeriksen/bert/blob/master/Data/BERT/Packet.hs#L46">github</a>).</p>
<div class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="nf">packets</span> <span class="ow">::</span> <span class="kt">ByteString</span> <span class="ow">-></span> <span class="p">[</span><span class="kt">Packet</span><span class="p">]</span>
<span class="nf">packets</span> <span class="n">b</span>
<span class="o">|</span> <span class="n">null</span> <span class="n">b</span> <span class="ow">=</span> <span class="kt">[]</span>
<span class="o">|</span> <span class="n">otherwise</span> <span class="ow">=</span> <span class="n">p</span><span class="kt">:</span><span class="n">packets</span> <span class="n">b'</span>
<span class="kr">where</span> <span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">b'</span><span class="p">)</span> <span class="ow">=</span> <span class="n">parsePacket</span> <span class="n">b</span></code></pre></div>
<p>(In Haskell, <code>:</code> is the list constructor: it prepends an item to a list, eg. <code>1:[2,3] == [1,2,3]</code>.) <code>packets</code> reads like a declaration: if our bytestring, <code>b</code> is null, it is the empty list, otherwise, it is the first parsed packet from <code>b</code>, plus the list of packets represented by the remainder of the string. This works in Haskell because everything is evaluated <em>lazily</em>. This in effect means that, unless you specify otherwise, values are not computed (code is not evaluated) until needed. In practice, the Haskell runtime leaves a “promise” of a computation when the value is not needed immediately. The list constructor <code>:</code> is no different: its arguments are not evaluated until needed. In our example this means that the list defined by <code>packets</code> is lazily generated. So if we create such a value with <code>packet b</code>, not until we examine the resulting list do we even begin parsing (or generating the list, or reading from the socket). This is an example by which we use lazyness to provide an <em>abstraction</em>. It allows us to operate on the familiar list type while not being terribly concerned about how the list is being constructed. It also relieves us of having to create any further abstraction: we can translate our model of a transport almost perfectly into a primitive Haskell concept without having to go about creating unfamiliar interfaces.</p>
<h2 id="monadic-parsing">Monadic parsing</h2>
<p>We created an algebraic type to represent terms, but their construction (if you need to work with the term type) can be a little cumbersome. Our representation</p>
<div class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="kt">TupleTerm</span> <span class="p">[</span> <span class="kt">AtomTerm</span> <span class="s">"bert"</span><span class="p">,</span> <span class="kt">AtomTerm</span> <span class="s">"dict"</span>
<span class="p">,</span> <span class="kt">ListTerm</span> <span class="p">[</span><span class="kt">TupleTerm</span> <span class="p">[</span><span class="kt">AtomTerm</span> <span class="s">"key"</span><span class="p">,</span> <span class="kt">AtomTerm</span> <span class="s">"value"</span><span class="p">]]]</span></code></pre></div>
<p>would be represented more succinctly in the erlang grammar as:</p>
<div class="highlight"><pre><code class="language-erlang" data-lang="erlang"><span class="p">{</span><span class="n">bert</span><span class="p">,</span> <span class="n">dict</span><span class="p">,</span> <span class="p">[(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">)]}</span></code></pre></div>
<p>If just writing code, you probably don’t care too much about the difference: you typically don’t make big static constructs, but rather create types programmatically anyway. However, with the BERT implementation I wanted to have a command line tool that allowed for testing & inspection of results. For example:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>bert call localhost:8181 mod proc <span class="s2">"{1, test, [5,6,7]}"</span></code></pre></div>
<p>Luckily, Haskell provides an excellent facility for parsing: <a href="http://www.haskell.org/haskellwiki/Parsec">Parsec</a>. Parsec uses monadic actions to simultaneously lex & parse its input. I won’t go into monads here, but for our purposes here it is essentially a means by which you can run code in a certain context, and manipulate that context. It can provide a pure functional construct with which you can achieve the appearance of imperative programming. In the context of Parsec, its monad maintains the parser state (eg. where in the input are we, which rules have failed, which rules are yet to be tried, what is the output, etc.), and provides a set of functions that can operate within the context of the monad it provides. Here’s our parser for a term.</p>
<div class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="nf">p_term</span> <span class="ow">::</span> <span class="kt">Parser</span> <span class="kt">Term</span>
<span class="nf">p_term</span> <span class="ow">=</span> <span class="n">t</span> <span class="o"><*</span> <span class="n">spaces</span>
<span class="kr">where</span>
<span class="n">t</span> <span class="ow">=</span> <span class="kt">IntTerm</span> <span class="o"><$></span> <span class="n">p_num</span> <span class="p">(</span><span class="n">readSigned</span> <span class="n">readDec</span><span class="p">)</span>
<span class="o"><|></span> <span class="kt">FloatTerm</span> <span class="o"><$></span> <span class="n">p_num</span> <span class="p">(</span><span class="n">readSigned</span> <span class="n">readFloat</span><span class="p">)</span>
<span class="o"><|></span> <span class="kt">AtomTerm</span> <span class="o"><$></span> <span class="n">p_atom</span>
<span class="o"><|></span> <span class="kt">TupleTerm</span> <span class="o"><$></span> <span class="n">p_tuple</span>
<span class="o"><|></span> <span class="kt">BytelistTerm</span> <span class="o">.</span> <span class="kt">C</span><span class="o">.</span><span class="n">pack</span> <span class="o"><$></span> <span class="n">p_string</span>
<span class="o"><|></span> <span class="kt">ListTerm</span> <span class="o"><$></span> <span class="n">p_list</span>
<span class="o"><|></span> <span class="kt">BinaryTerm</span> <span class="o">.</span> <span class="kt">B</span><span class="o">.</span><span class="n">pack</span> <span class="o"><$></span> <span class="n">p_binary</span></code></pre></div>
<p>Again, reading very naturally: a term is <code>t</code> possibly followed by some whitespace. (<code>a <* b</code> means roughly “run a, then b, but the value of the expression is the value of a”). <code>t</code> in turn can be either a <code>p_num</code> wrapped with an <code>IntTerm</code>, and so forth. (In this context, <code><|></code> can very roughly be thought of as “otherwise, try..” and <code><$></code> as “wrap the results of the argument to the right with the thing on the left”.)</p>
<p>The various <code>p_*</code>s are other parsers. For example the one for tuples is</p>
<div class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="nf">p_tuple</span> <span class="ow">=</span>
<span class="n">between</span> <span class="p">(</span><span class="n">char</span> <span class="n">'</span><span class="p">{</span><span class="n">'</span> <span class="o">>></span> <span class="n">spaces</span><span class="p">)</span> <span class="p">(</span><span class="n">spaces</span> <span class="o">>></span> <span class="n">char</span> <span class="n">'</span><span class="p">}</span><span class="n">'</span><span class="p">)</span> <span class="o">$</span>
<span class="n">p_term</span> <span class="p">`</span><span class="n">sepBy</span><span class="p">`</span> <span class="p">(</span><span class="n">spaces</span> <span class="o">>></span> <span class="n">char</span> <span class="n">'</span><span class="p">,</span><span class="n">'</span> <span class="o">>></span> <span class="n">spaces</span><span class="p">)</span></code></pre></div>
<p><code>between</code> is one of the Parsec functions that means: between the parses of the following arguments, try to parse the third. The third argument states: “try and parse a <code>p_term</code> that is separated by maybe some whitespace, a comma and maybe some more whitespace.”</p>
<h2 id="section">…</h2>
<p>This is really just the tip of the ice berg. Most of my descriptions, especially those regarding the use of monads belie the rigor and generality of these constructs. If you’d like to explore further, the <a href="http://www.realworldhaskell.org/">Real World Haskell</a> book is a great start point. The Haskell community is simply fantastic. Much conversation happens on <a href="http://www.haskell.org/haskellwiki/IRC_channel">IRC</a>, where there is typically an armada of people just waiting to discuss the finer points of this wonderful language with you.</p>
Pattern matching in Python2009-05-11T00:00:00-07:00http://monkey.org/~marius//pattern-matching-in-python<p>One of my favorite things about <a href="http://www.erlang.org/">various</a> <a href="http://www.haskell.org/">functional</a> <a href="http://www.clojure.org/">programming</a> <a href="http://caml.inria.fr/ocaml/">languages</a> is pattern matching. It often allows for very succinct and elegant declarative expressions, and in the dynamic variants it allows for easy in-line lightweight type checking.</p>
<p>So, naturally I wanted the same in Python, the programming language we use at work.</p>
<p>Pattern matching is most powerful when it enjoys first-class support in a language. In addition to succinct syntax, this affords you the ability to integrate pattern matchers with control constructs, allowing conditional execution of code based on various patterns. It also may give you a degree of composability not possible otherwise. For example, <a href="http://www.erlang.org/">Erlang</a> has not only a <code>case</code> statement, but allows for <em>clauses</em> in functions, so that pattern matching is done on the arguments of functions with the same name and arity. For instance in Erlang, you could implement a simple REST-style HTTP request handlers like so:</p>
<div class="highlight"><pre><code class="language-erlang" data-lang="erlang"><span class="c">%% handle(Path, Method)</span>
<span class="nf">handle</span><span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="p">_)</span> <span class="o">-></span>
<span class="n">not_a_resource</span><span class="p">;</span>
<span class="nf">handle</span><span class="p">(</span><span class="nv">Path</span><span class="p">,</span> <span class="n">'PUT'</span><span class="p">)</span> <span class="o">-></span>
<span class="n">create_new_resource</span><span class="p">(</span><span class="nv">Path</span><span class="p">);</span>
<span class="nf">handle</span><span class="p">(</span><span class="nv">Path</span><span class="p">,</span> <span class="n">'POST'</span><span class="p">)</span> <span class="o">-></span>
<span class="n">update_resource</span><span class="p">(</span><span class="nv">Path</span><span class="p">);</span>
<span class="nf">handle</span><span class="p">(</span><span class="nv">Path</span><span class="p">,</span> <span class="n">'GET'</span><span class="p">)</span> <span class="o">-></span>
<span class="n">retrieve_resource</span><span class="p">(</span><span class="nv">Path</span><span class="p">);</span>
<span class="nf">handle</span><span class="p">(_,</span> <span class="p">_)</span> <span class="o">-></span>
<span class="n">invalid_request</span><span class="p">.</span></code></pre></div>
<p>Now I can simply call <code>handle(Path, Method)</code> to deal with my request. Notice also how powerful ordering is here: I exclude the resource <code>"/"</code> entirely by matching on it first. Note also that some arguments here are <em>binding</em>, while others are not. For example in the first clause, nothing is bound, in the second, we bind <code>Path</code> if the method matches <code>'PUT'</code>.</p>
<p>How do we graft something like this onto a language like Python? It’s especially tricky because we’d like to maintain the ever elusive “Pythonics.” While I’m quite sure Guido would never even touch this stuff, we can at least maintain the spirit! Start off by importing the match primitives:</p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="kn">from</span> <span class="nn">match</span> <span class="kn">import</span> <span class="n">M</span><span class="p">,</span> <span class="n">A</span><span class="p">,</span> <span class="n">A_</span></code></pre></div>
<p><code>M</code> creates a destructuring match expression (a “matcher”), <code>A</code> is the a binding argument, and <code>A_</code> is the “any” argument. Any <code>A</code> arguments need to have a positional specifier. This is achieved with the division operator. So <code>A/0</code> names the first value in the returned destructured tuple. With the help of a few operators, we compose a match expression:</p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="n">M</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span> <span class="p">(</span><span class="n">A_</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="n">A</span><span class="o">/</span><span class="mi">1</span><span class="p">,</span> <span class="n">A</span><span class="o">/</span><span class="mi">0</span><span class="p">))</span>
<span class="o"><</span><span class="n">match</span><span class="o">.</span><span class="n">M</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x726f0</span><span class="o">></span></code></pre></div>
<p>So, these expressions are entirely useless until they are <em>bound</em>. The <code>==</code> operator takes care of that:</p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="n">M</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span> <span class="p">(</span><span class="n">A_</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="n">A</span><span class="o">/</span><span class="mi">1</span><span class="p">,</span> <span class="n">A</span><span class="o">/</span><span class="mi">0</span><span class="p">))</span> <span class="o">==</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span></code></pre></div>
<p>If you precede the match-expression with the <code>~</code> operator, the expression becomes a “pure” matching expression, and does not destructure the second operand, it just returns <code>True</code> or <code>False</code> depending on whether it matched successfully.</p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="o">~</span><span class="n">M</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span> <span class="p">(</span><span class="n">A_</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="n">A</span><span class="o">/</span><span class="mi">1</span><span class="p">,</span> <span class="n">A</span><span class="o">/</span><span class="mi">0</span><span class="p">))</span> <span class="o">==</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="bp">True</span></code></pre></div>
<p>Now, to make things more interesting, match-expressions can be <code>or</code>-ed together, resulting in the first successful match. For example <code>M([A/0]) | M(A/0)</code> doesn’t care whether the value is a list of length 1 or a literal.</p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="n">M</span><span class="p">([</span><span class="n">A</span><span class="o">/</span><span class="mi">0</span><span class="p">])</span> <span class="o">|</span> <span class="n">M</span><span class="p">(</span><span class="n">A</span><span class="o">/</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="p">[</span><span class="mi">5</span><span class="p">]</span>
<span class="mi">5</span>
<span class="o">>>></span> <span class="n">M</span><span class="p">([</span><span class="n">A</span><span class="o">/</span><span class="mi">0</span><span class="p">])</span> <span class="o">|</span> <span class="n">M</span><span class="p">(</span><span class="n">A</span><span class="o">/</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="mi">5</span>
<span class="mi">5</span></code></pre></div>
<p>Finally, matchers can specify “default” values to be returned they match successfully. This is to help deal with polymorphic return values. For example, the following expression:</p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="n">M</span><span class="p">([</span><span class="n">A</span><span class="o">/</span><span class="mi">0</span><span class="p">])</span> <span class="o">|</span> <span class="n">M</span><span class="p">([],</span> <span class="n">d</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="o">==</span> <span class="n">arr</span><span class="p">[:</span><span class="mi">1</span><span class="p">]</span></code></pre></div>
<p>picks the first element if <code>arr</code> is nonempty, otherwise it returns <code>False</code>.</p>
<p>I’ve put the <a href="http://github.com/mariusaeriksen/match/tree/master">code up on GitHub</a>. I’m not super happy with the aesthetics of it but it’s interesting to experiment with. Among other things, it definitely needs list and dictionary destructuring.</p>