OCaml Planet

OCamlCore Forge News: Lacaml v6.0.0 released

Christophe Troestler — 2012-02-01T22:23:41+00:00

Lacaml is a binding to the celebrated linear algebra routines BLAS and LAPACK. This version builds with OASIS. Moreover the precision dependent code is available with Lacaml.D,... instead of the former Lacaml.Impl.D,... Enjoy.

Gerd Stolpmann: Plasma Map/Reduce Slightly Faster Than Hadoop

2012-02-01T22:00:28+00:00

A performance test

Last week I spent some time running map/reduce jobs on Amazon EC2. In particular, I compared the performance of Plasma, my own map/reduce implementation, with Hadoop. I just wanted to know how much my implementation was behind the most popular map/reduce framework. However, the suprise was that Plasma turned out as slightly faster in this setup.

I would not call this test a "benchmark". Amazon EC2 is not a controlled environment, as you always only get partial machines, and you don't know how much resources are consumed by other users on the same machines. Also, you cannot be sure how far the nodes are off from each other in the network. Finally, there are some special effects coming from the virtualization technology, especially the first write of a disk block is slower (roughly half the normal speed) than following writes. However, EC2 is good enough to get an impression of the speed, and one can hope that all the test runs get the same handicap on average.

The task was to sort 100G of data, given in 10 files. Each line has 100 bytes, divided into a key of 8 bytes, a TAB character, 90 random bytes as value, and an LF character. The key was randomly chosen from 65536 possible values. This means that there were lots of lines with the same key - a scenario where I think it is more typical of map/reduce than having unique keys. The output is partitioned into 80 sets.

I allocated 1 larger node (m1-xlarge) with 4 virtual cores and 15G of RAM acting as combined name- and datanode, and 9 smaller nodes (m1-large) with 2 virtual cores and 7.5G of RAM for the other datanodes. Each node had access to two virtual disks that were configured as RAID-0 array. The speed for sequential reading or writing was around 160 MB/s for the array (but only 80 MB/s for the first time blocks were written). Apparently, the nodes had Gigabit network cards (the maximum transfer speed was around 119MB/s).

During the tests, I monitored the system activity with the sar utility. I observed significant cycle stealing (meaning that a virtual core is blocked because there is no free real core), often reaching values of 25%. This could be interpreted as overdriving the available resources, but another explanation is that the hypervisor needed this time for itself. Anyway, this effect also questions the reliability of this test.

The contrahents

Hadoop is the top dog in the map/reduce scene. In this test, the version from Cloudera 0.20.2-cdh3u2 was used, which contains more than 1000 patches against the vanilla 0.20.2 version. Written in Java, it needs a JVM at runtime, which was here IcedTea 1.9.10 distributing OpenJDK 1.6.0_20. I did not do any tuning, hoping that the configuration would be ok for a small job. The HDFS block size was 64M, without replication.

The contender is Plasma Map/Reduce. I started this project two years ago in my spare time. It is not a clone of the Hadoop architecture, but includes many new ideas. In particular, a lot of work went into the distributed filesystem PlasmaFS which features an almost complete set of file operations, and controls the disk layout directly. The map/reduce algorithm uses a slightly different scheme which tries to delay the partitioning of the data to get larger intermediate files. Plasma is implemented in OCaml, which isn't VM-based but compiles the code directly to assembly language. In this test, the blocksize was 1M (Plasma is designed for smaller-sized blocks). The software version of Plasma is roughly 0.6 (a few svn revisions before the release of 0.6).

Results

The runtimes:

Hadoop:	2265 seconds (37 min, 45 s)
Plasma:	1975 seconds (32 min. 55 s)

Given the uncertainty of the environment, this is no big difference. But let's have a closer look at the system activity to get an idea why Plasma is a bit faster.

CPU

In the following I took simply one of the datanodes, and created diagrams (with kSar):

Note that kSar does not draw graphs for %iowait and %steal, although these data are recorded by sar. This is the explanation why the sum of user, system and idle is not 100%.

What we see here is that Hadoop consumes all CPU cycles, whereas Plasma leaves around 1/3 of the CPU capacity unused. Given the fact that this kind of job is normally I/O-bound, it just means that Hadoop is more CPU-hungry, and would have benefit from getting more cores in this test.

Network

In this diagram, reads are blue and red, whereas writes are green and black. The first curve shows packets per second, and the second bytes per second:

Summing reads and writes up, Hadoop uses only around 7MB/s on average whereas Plasma transmits around 25MB/s, more than three times as much. There could be two explanations:

Because Hadoop is CPU-underpowered, it remains below its possibilities
The Hadoop scheme is more optimized for keeping the network bandwidth as low as possible

The background for the second point is the following: Because Hadoop partitions the data immediately after mapping and sorting, the data has (ideally) only to cross the network once. This is different in Plasma - which generally partitions the data iteratively. In this setup, after mapping and sorting only 4 partitions are created, which are further refined in the following split-and-merge rounds. As we have here 80 partitions in total, there is at least one further step in which data partitioning is refined, meaning that the data has to cross the network roughly twice. This already explains 2/3 of the observed difference. (As a side note, one can configure how many partitions are initially created after mapping and sorting, and it would have been possible to mimick Hadoop's scheme by setting this value to 80.)

Disks

These diagrams depict the disk reads and writes in KB/second:

The average numbers are (directly taken from sar):

	Hadoop	Plasma
Read/s:	17.6 MB/s	31.2 MB/s
Write/s:	30.8 MB/s	33.9 MB/s

Obviously, Plasma reads data around twice as often from disk than Hadoop, whereas the write speed is about the same. Apart from this, it is interesting that the shape of the curves are quite different: Hadoop has a period of high disk activity at the end of the job (when it is busy merging data), whereas Plasma utilizes the disks better during the first third of the job.

Plausibility

Neither of the contenders utilized the I/O resources at all times best. Part of the difficulty of developing a map/reduce scheme is to achieve that the load put onto the disks and onto the network is balanced. It is not good when e,g, the disks are used to 100% at a certain point and the network is underutilized, but during the next period the network is at 100% and the disk not fully used. A balanced distribution of the load reaches higher throughput in total.

Let's analyze the Plasma scheme a bit more in detail. The data set of 100G (which does not change in volume during the processing) is copied four times in total: once in the map-and-sort phase, and three times in the reduce phase (for this volume Plasma needs three merging rounds). This means we have to transfer 4 * 100G of data in total, or 40G of data per node (remember we have 10 nodes). We ran 22 cores for 1975 seconds, which gives a capacity of 43450 CPU seconds. Plasma tells us in its reports that it used 3822 CPU seconds for in-RAM sorting, which we should subtract for analyzing the I/O throughput. Per core these are 173 seconds. This means each node had 1975-173 = 1802 seconds for handling the 40G of data. This makes around 22 MB per second on each node.

The Hadoop scheme differs mostly in that the data is only copied twice in the merge phase (because Hadoop by default merges more files in one round than Plasma). However, because of its design there is an extra copy at the end of the reduce phase (from disk to HDFS). This means Hadoop also solves the same job by transferring 4 * 100G of data. There is no counter for measuring the time spent for in-RAM sorting. Let's assume this time is also around 3800 seconds. This means each node had 2265 - 175 = 2090 seconds for handling 40G of data, or 19 MB per second on each node.

Conclusion

It looks very much as if both implementations are slowed down by specifics of the EC2 environment. Especially the disk I/O, probably the essential bottleneck here, is far below what one can expect. Plasma probably won because it uses the CPU more efficiently, whereas other aspects like network utilization are better handled by Hadoop.

For my project this result just means that it is on the right track. Especially, this small setup (only 10 nodes) is easily handled, giving prospect that Plasma is scalable at least to a small multitude of this. The bottleneck would be here the namenode, but there is still a lot of headroom.

Where to get Plasma

Plasma Map/Reduce and PlasmaFS are bundled together in one download. Here is the project page.

Gerd Stolpmann works as O'Caml consultant

Caml Weekly News: Caml Weekly News, 31 Jan 2012

2012-01-31T12:00:00+00:00

SQL engine in OCaml with client side cache / Vim plugin / Other Caml News

Ocsigen project: New js_of_ocaml example: Hyperbolic tree viewer

Ocsigen team — 2012-01-27T17:20:00+00:00

New js_of_ocaml example available: Hyperbolic tree viewer. Game: find human beings!

This program has been fully written in OCaml and compiled to JS using the js_of_ocaml compiler. You can find the source code in js_of_ocaml repository.

Opa: Sessions: handling state, communication & concurrency in Opa.

Adam Koprowski — 2012-01-27T13:33:23+00:00

As François-Régis mentioned in the previous post (hey, thanks for taking over and making the show go on :) I'm on holidays now. I rarely have access to the internet so this is the main reason why I don't answer comments and won't post much. Please bare with me; I'll be back on 13th of September and then it will again be business as usual. But since I got a chance I decided to make this one sun-charged post while still on holidays :). Ok, let's get started.

Sessions are a very central concept in Opa. And as things stand I don't think enough attention is devoted to them in the manual (we received a lot of compliments about the state of the docs and we're happy to hear you like them, but I think there are still things that can be improved). I've been meaning to write about sessions for a longer while and I think it's high time to do that.

Summary

You will learn:

About state encapsulation using sessions.
About event-driven style of programming with sessions.
How communication is accomplished in Opa (yes, you guessed it, with sessions).

Sessions

In case you haven't realized yet, Opa is a functional language (although not pure, as the database manipulations and many client-side operations are impure). That means that there are no variables. This can come as a shocker to some of you not used to this style of programming and the words: `‘How the heck can I program anything without variables??’' seem to be coming to one's lips. Well, it turns out you can. In fact: it turns out such a programming style has huge benefits; listing them is beyond the scope of this article, I'll only hint: think about debugging and testing (another hint: a pure function will give the same result for the same arguments. Always. Yes, that's a good thing).

Still, there are times when you simply need to have a state. And to modify it. Not to worry, sessions can do that for you in a safe and elegant manner. So what are they exactly? The description from the standard library is a good start:

`‘A session is a unit of state and concurrency. A session can be created on a server or on a client, and can be shared between several servers.’'

A session is just some value (session's state) and a message handler. Messages can be send to the sessions and upon receipt are handled by the handler, which in turn can change the session's state (or terminate it). Big part of the magic is that it doesn't matter where is the session and where are you sending the message from — client and server or two different servers; whatever the scenario things will just work (of course the location of the session relative to the location of the message sender can have a huge impact on performance, so making sure the session is where it should be is often a good place to start when tweaking your app's performance).

Let's see how can we create a session:

type Session.instruction('state) =
        {set: 'state} /** Carry on with a new value of the state value.*/
      / {unchanged}   /** Carry on with the same state.*/
      / {stop}        /** Stop this session. Any further message will be ignored */

Session.make : 'state, ('state, 'message -> Session.instruction('state)) -> channel('message)

A Session.instruction can either change the state of the session (set), leave it unchanged (unchanged) or terminate the session (stop). We can create a new session with Session.make that takes two arguments: initial state of the session and a message handler, i.e., a function that given the state and a message can do some work and then respond with Session.instruction.

Session.make after creating the session gives us a channel. We can then send messages to this channel with:

send : channel('message), 'message -> void

…and that's mostly it. There are more things that sessions can do and I encourage you to browse the Session API but those two functions are the backbone of Opa sessions.

Simple? Quite so, yes. But don't be fooled by the simplicity (actually something being simple is often an indication of a good design, not of lack of versatility) — sessions are mighty animals. They're powerful. They're very important. You better learn them if you want to become an Opa-ninja.

Example

Remember the simple watch example we developer a while back? For convenience let me put the source code here.

Id = {{ time = "time" }}

show_time() =
  time = Date.to_string(Date.now())
  Dom.transform([#{Id.time} <- <>{time}</>])

page() =
  <span id=#{Id.time} onready={_ -> Scheduler.timer(1000, show_time)} />

server = one_page_server("What's the time?", page)

Run

It's a very simple code but it has one annoying feature: we need to repeat the id of the <span> element; first we mention it when we create the element in the page() function and then when we modify the time in show_time(). In order to avoid easy mistakes that id was factored out as a constant. That's a good practice but let's see a different approach with sessions.

Let's try to implement a small abstraction for an interactive HTML fragment — we will call it just that, a fragment. According to Opa's taxonomy it is a component, so following our naming practice it will be put in a module CFragment.

Such a fragment should be able to render itself and then to possibly re-render itself upon a notification. To make it more interesting we will also equip it with some state, so that it will be able to render its new version based both on the incoming notification and its previous state and it will be able to subsequently modify this state.

Let's start defining a type for an answer to a notification send to a fragment.

package hands-on-opa.fragment

type CFragment.action('state) =
  { re_render : option(xhtml)
  ; change_state : option('state)
  }

such an answer can (optionally) request re-rendering of the HTML and (optionally) change fragment's state. Let me present the rest of the code of this fragment abstraction and then I'll try to explain what's happening.

@abstract type CFragment.fragment('msg) = channel('msg)

CFragment =
{{

  @private on_notification(id, handler, state, msg) =
    res = handler(state, msg)
    re_render(xhtml) = Dom.transform([#{id} <- xhtml])
    do Option.iter(re_render, res.re_render)
    match res.change_state with
    | {none} -> {unchanged}
    | {some=new_state} -> {set=new_state}

  create(init_state : 'state, init_xhtml : xhtml,
         handler : 'state, 'msg -> CFragment.action
        ) : (xhtml, CFragment.fragment('t)) =
    id = Dom.fresh_id()
    xhtml = <span id={id}>{init_xhtml}</>
    session = Session.make(init_state, on_notification(id, handler, _, _))
    (xhtml, session)

  notify(fragment : CFragment.fragment('msg), msg : 'msg) : void =
    Session.send(fragment, msg)

}}

First of all we will implement the fragment using a session, so an initialized fragment is just represented by a channel. However, due to the @abstract keyword this representation will not be visible from outside of the package hands-on-opa.fragment. That means that we will be able to manipulate fragments only using functions exposed by the CFragment module — a neat way to accomplishing abstraction and ensure invariants on data.

The create function takes 3 arguments:

init_state: the initial state of the fragment,
init_xhtml: the initial XHTML representation of the fragment,
handler: the handler for fragment notifications, which given the state and the incoming msg should return a CFragment.action indicating how to handle this event.

What happens within the function is that we obtain a fresh DOM id, wrap the init_xhtml with a <span> with that id, create a session with the init_state and finally return a pair of the XHTML for the fragment and the fragment itself (which as I mentioned we represent with a session).

The message handler for the session, on_notification, uses the handler given in the create function to decide what to do with the incoming notification. It optionally re-renders the XHTML (using the produced id) and optionally modifies the state of the session (i.e. fragment).

Finally we expose a notify function which let's us send notifications to the fragment. Remember that the CFragment.fragment type is abstract so anywhere outside of this package it is not visible as a session. So if we did not expose this functionality, users of this module would not be able to send messages using Session.send.

Ok, time to use our fragment component to implement the timer functionality. Below is the final code:

import hands-on-opa.fragment

show_time(t) = <>{Date.to_string(t)}</>

on_msg(_state, {update=date}) =
  {re_render=some(show_time(date)) change_state=none}

page() =
  (timer_xhtml, timer) = CFragment.create(void, show_time(Date.now()), on_msg)
  update_timer() = CFragment.notify(timer, {update=Date.now()})
  do Scheduler.timer(1000, update_timer)
  timer_xhtml

server = one_page_server("What's the time?", page)

package hands-on-opa.fragment

type CFragment.action('state) =
  { re_render : option(xhtml)
  ; change_state : option('state)
  }

@abstract type CFragment.fragment('msg) = channel('msg)

CFragment =
{{

  @private on_notification(id, handler, state, msg) =
    res = handler(state, msg)
    re_render(xhtml) = Dom.transform([#{id} <- xhtml])
    do Option.iter(re_render, res.re_render)
    match res.change_state with
    | {none} -> {unchanged}
    | {some=new_state} -> {set=new_state}

  create(init_state : 'state, init_xhtml : xhtml,
         handler : 'state, 'msg -> CFragment.action
        ) : (xhtml, CFragment.fragment('t)) =
    id = Dom.fresh_id()
    xhtml = <span id={id}>{init_xhtml}</>
    session = Session.make(init_state, on_notification(id, handler, _, _))
    (xhtml, session)

  notify(fragment : CFragment.fragment('msg), msg : 'msg) : void =
    Session.send(fragment, msg)

}}

Run

The show_time function converts a Date.date to its xhtml representation. The on_msg function reacts to notifications send to the timer fragment — we don't use the state, handle only one type of messages, namely requests to update the timer date and instruct the fragment to re-render itself with show_time(date).

In the page() function we first create a fragment with empty state (void), show_time(Date.now()) as the initial representation and on_msg as the notification handler — this gives us initial HTML (timer_xhtml) and the fragment itself (timer). The update_timer() function sends a notification to the timer asking it to update itself with Date.now(). Finally we schedule calls to update_timer every second (or 1000 milliseconds) and return the initial HTML, timer_xhtml.

Exercise

Note that in the above code the timer fragment lives on the server, which means that every timer update will make a round trip to the server. It's equivalent to the slow variant of this example that we developer in the original timer post. How would you fix that? (i.e. make the timer update fully in the browser without any communication with the server).

Uff. We're done. Only after writing this explanation I realized that this example can be quite heavy. Please take your time to slowly go through it and understand it (and if you're lost give me a shout in the comments… just please be patient if I don't react before coming back from holidays). If you really want to nail it down I'd suggest experimenting with this example, or developing something of your own.

Now, let's sit back and evaluate what we've achieved. The code is certainly not simpler than what we started with — on the contrary. However, the CFragment module is a useful abstraction that can be re-used in different contexts (maybe it should even make its way to Opa's stdlib?).

The code in watch.opa may not look simpler than what we had, but I'd argue it's cleaner. Imagine a more complex code with many fragments being updated in different places, by different triggers. Before we had dirty, direct DOM manipulations and all it took for things to go wrong was to make a typo in element's id or mistakenly use another id.

Now we replaced that with an event-based update system. To update a fragment we need to pass it to CFragment.notify so we cannot notify a non-existing fragment (though it can still happen that we forget to place the fragment within a page, resulting in the update having no effect). And fragments with different functionalities will likely respond to different type of notifications, so, to some extent, the type system will help us detect attempts at updating a wrong fragment.

Open a mental parenthesis for a side note. By the way, we received some complaints that in spite of all the fancy features, Opa is in some sense quite low-level because it work on the level of HTML. I'll be the first to agree with some criticism of Opa (it's a very new language after all), but I cannot agree with this one. To begin whatever framework/language you believe in ultimately pages are just HTML (unless what you believe in is Flash, but in that case I'd suggest you consider learning HTML, and fast). Abstracting it away, building something on top of HTML and exposing only this more high-level language to the developer is of course possible. But is that a good thing? I think at one time or another it'll come back biting your head; for instance when one wants to do something more fancy which is possible in HTML but was lost in the abstraction (and preserving all the features while simplifying the formalism is not an easy task).

But even more importantly: you can do that in Opa. No one stops you from building some higher-level abstraction over HTML — in fact Opa is quite well equipped to do so! In a sense, this is already slowly happening with extensions to our library of widgets and components which are higher-level elements to build pages with. Another interesting development in this direction is the Templating machinery, which allows, err, templating. Finally there is no reason not to develop a domain-specific language in Opa that could be used to replace (or complement) HTML markup, although I'm not aware of any efforts in that direction (and frankly personally I don't see a need for that).

Long story short: Opa maps to HTML in a very truthful manner. And that's good. If you want some higher-level framework on top, Opa is well equipped to handle that too. End of the side note.

Ok, time to wrap up this post. I'm not sure how successful I was in persuading you that sessions are important (and powerful) (and versatile). If I didn't, not to worry, we'll see them over and over again on this blog and I hope that eventually I'll prove my point.

OCamlCore Forge News: libsndfile-ocaml 0.8 available

Philippe STRAUSS — 2012-01-26T18:35:25+00:00

This is Erik de Castro Lopo own binding to it's libsndfile, maintained by Ph.Strauss. Usefull for reading/writing various audio file formats. Tarball and public git should be ok now.

The Caml Humps: Zermelo Proof Checker 1.0

2012-01-26T09:42:09+00:00

The Zermelo Proof Checker (ZPC) is a lightweight proof assistant based on standard set theory and Hindley-Milner type theory.

OCamlCore Forge Projects: libsndfile

2012-01-24T13:00:43+00:00

Binding to Libsndfile, audio files reading/writing

Caml Weekly News: Caml Weekly News, 24 Jan 2012

2012-01-24T12:00:00+00:00

otags reloaded 3.12.1 for OCaml 3.12.1 / ounit v1.1.1: Unit testing framework / polymorphic variants in match statements / Other Caml News

Victor Nicollet: OCaml Submodule Pattern

Victor Nicollet — 2012-01-23T16:55:59+00:00

My code is quite large for an OCaml project. The main RunOrg repository alone contains 46212 lines of OCaml code (plus an additional 5631 lines of OCaml mli files) — and then, there’s the web framework code and the independent plugins code.

It’s is Better™ to have many short files than a few long ones. One reason is incremental compiling with ocamlbuild : that the smaller your files are, the smaller the percentage of code to be compiled when you make a small change. Another reason is that files provide a natural delineation of code that makes it slightly easier to reason about.

The very process of splitting a large file into smaller files is also an excellent way to clean up the code. Every split is an opportunity to move some code to a more generic location — why have a CMember_importParser module when all of its functionality could fit into an OzCsv plugin module ? Even when no such generic solution exists, cutting through the jungle that a 2000-line module contains helps clean up dependencies, identify shared functionality and imagine better ways to design code.

Still, when cutting up code this way, the problem of encapsulation remains. If code that relates to pictures (an upload module, a transform module, a download module, an access rights module) is split across several files, it is desirable to let each file access functions and values from other values that would not otherwise be shown to modules not related to picture processing. For instance, a get_download_link function should be available throughout all picture-related modules, but the rest of the application should use the get_download_link_for_user function that checks whether the user is allowed to download the file.

In order to achieve several nested levels of encapsulation required to work with modules this way, I have come up with a convention :

A module name (and thus, a file name) is composed of segments written in camelCase and separated by underscores. For instance, CEntity_view_grid is a module name containing segments CEntity, view and grid.
Modules with only one segment are public. Any other module may include, open or otherwise reference them with no limitations beyond what the module signature says. So, CEntity may access MGroup freely.
Modules with N > 1 segments are private. They may only be accessed by modules which share the first N-1 segments. So, CEntity_view is available to modules CEntity and CEntity_edit but not CPicture.
A module with N segments may export any module with N+1 segments it can access, possibly under a more restrictive signature. For instance, CEntity_view is available to all other modules as CEntity.View.

To make these rules easier to respect, private module dependencies are made explicit by adding a list of module aliases at the top of each file. The top of my cEntity_view.ml file starts with :

module Sidebar     = CEntity_sidebar
module Unavailable = CEntity_unavailable
module Edit        = CEntity_edit
module Info        = CEntity_view_info
module Directory   = CEntity_view_directory
module Grid        = CEntity_view_grid
module Wall        = CEntity_view_wall

It is forbidden to use a private module without going through such an alias, and it is forbidden to define such an alias anywhere except at the top of the file. This makes it extremely easy to determine whether private access rules are respected.

The rule of thumb for splitting files (in my particular coding style) is :

Code for separate layers (model, view, controller…) go into separate public modules.
For complex code (such as complex rules in model or controller code), consider splitting files larger than 200 lines.
For simple code (such as HTML template or JSON serialization definitions), there is no splitting limit except for factoring out common behavior.