OCaml Planet

August 19, 2014

Dario Teixeira

Announcing Camlhighlight 3.0

I'm happy to announce release 3.0 of Camlhighlight, a library offering syntax highlighting and pretty-printing facilities for displaying code samples in Ocsigen applications. The library works by interfacing with the library portion of GNU Source-highlight, a popular application supporting the most common programming and markup languages.

This version features a smaller dependency set, now requiring Tyxml instead of forcing a dependency on Eliom. However, full compatibility with Eliom applications is maintained. The functorial interface used seems hairy at first glance, but it's actually not that complicated in practice. As an example, if your application uses only Tyxml and you wish to write a syntax-highlighted sample as a Html5.elt, you will first need to apply the Camlhighlight_write_html5.Make functor using Tyxml's Html5 and Svg modules as parameter:

module Html5_writer = Camlhighlight_write_html5.Make
(struct
include Html5.M
module Svg = Svg.M
end)

Similarly, if you intend the Html5_writer module to be Eliom compatible, then use Eliom's Html5.F.Raw and Svg.F.Raw modules as parameter:

module Html5_writer = Camlhighlight_write_html5.Make
(struct
include Eliom_content.Html5.F.Raw
module Svg = Eliom_content.Svg.F.Raw
end)

The package should be hitting the OPAM repository soon. Eliom users should beware that Camlhighlight requires Tyxml >= 3.2, whereas Eliom 4.0.0 requires Tyxml < 3.2. Therefore, should you want to use Camlhighlight in an Eliom application, you are advised to install the development version of Eliom (please see the Ocsigen site for instructions regarding Ocsigen's development repo for OPAM).

by Dario at August 19, 2014 03:48 PM

OCaml Platform

OPAM 1.2: Repository Pinning

Most package managers support some pin functionality to ensure that a given package remains at a particular version without being upgraded. The stable OPAM 1.1 already supported this by allowing any existing package to be pinned to a target, which could be a specific released version, a local filesystem path, or a remote version-controlled repository.

However, the OPAM 1.1 pinning workflow only lets you pin packages that already exist in your OPAM repositories. To declare a new package, you had to go through creating a local repository, registering it in OPAM, and adding your package definition there. That workflow, while reasonably clear, required the user to know about the repository format and the configuration of an internal repository in OPAM before actually getting to writing a package. Besides, you were on your own for writing the package definition, and the edit-test loop wasn't as friendly as it could have been.

A natural, simpler workflow emerged from allowing users to pin new package names that don't yet exist in an OPAM repository:

  1. choose a name for your new package
  2. opam pin add in the development source tree
  3. the package is created on-the-fly and registered locally.

To make it even easier, OPAM can now interactively help you write the package definition, and you can test your updates with a single command. This blog post explains this new OPAM 1.2 functionality in more detail; you may also want to check out the new Packaging tutorial relying on this workflow.

From source to package

For illustration purposes in this post I'll use a tiny tool that I wrote some time ago and never released: ocp-reloc. It's a simple binary that fixes up the headers of OCaml bytecode files to make them relocatable, which I'd like to release into the public OPAM repository.

"opam pin add"

The command opam pin add <name> <target> pins package <name> to <target>. We're interested in pinning the ocp-reloc package name to the project's source directory.

cd ocp-reloc
opam pin add ocp-reloc .

If ocp-reloc were an existing package, the metadata would be fetched from the package description in the OPAM repositories. Since the package doesn't yet exist, OPAM 1.2 will instead prompt for on-the-fly creation:

Package ocp-reloc does not exist, create as a NEW package ? [Y/n] y
ocp-reloc is now path-pinned to ~/src/ocp-reloc

NOTE: if you are using beta4, you may get a version-control-pin instead, because we added auto-detection of version-controlled repos. This turned out to be confusing (issue #1582), because your changes wouldn't be reflected until you commit, so this has been reverted in favor of a warning. Add the --kind path option to make sure that you get a path-pin.

OPAM Package Template

Now your package still needs some kind of definition for OPAM to acknowledge it; that's where templates kick in, the above triggering an editor with a pre-filled opam file that you just have to complete. This not only saves time in looking up the documentation, it also helps getting consistent package definitions, reduces errors, and promotes filling in optional but recommended fields (homepage, etc.).

opam-version: "1.2"
name: "ocp-reloc"
version: "0.1"
maintainer: "Louis Gesbert <louis.gesbert@ocamlpro.com>"
authors: "Louis Gesbert <louis.gesbert@ocamlpro.com>"
homepage: ""
bug-reports: ""
license: ""
build: [
  ["./configure" "--prefix=%{prefix}%"]
  [make]
]
install: [make "install"]
remove: ["ocamlfind" "remove" "ocp-reloc"]
depends: "ocamlfind" {build}

After adding some details (most importantly the dependencies and build instructions), I can just save and exit. Much like other system tools such as visudo, it checks for syntax errors immediately:

[ERROR] File "/home/lg/.opam/4.01.0/overlay/ocp-reloc/opam", line 13, character 35-36: '.' is not a valid token.
Errors in /home/lg/.opam/4.01.0/overlay/ocp-reloc/opam, retry editing ? [Y/n]

Installation

You probably want to try your brand new package right away, so OPAM's default action is to try and install it (unless you specified -n):

ocp-reloc needs to be installed.
The following actions will be performed:
 - install   cmdliner.0.9.5                        [required by ocp-reloc]
 - install   ocp-reloc.0.1*
=== 1 to install ===
Do you want to continue ? [Y/n]

I usually don't get it working the first time around, but opam pin edit ocp-reloc and opam install ocp-reloc -v can be used to edit and retry until it does.

Package Updates

How do you keep working on your project as you edit the source code, now that you are installing through OPAM? This is as simple as:

opam upgrade ocp-reloc

This will pick up changes from your source repository and reinstall any packages that are dependent on ocp-reloc as well, if any.

So far, we've been dealing with the metadata locally used by your OPAM installation, but you'll probably want to share this among developers of your project even if you're not releasing anything yet. OPAM takes care of this by prompting you to save the opam file back to your source tree, where you can commit it directly into your code repository.

cd ocp-reloc
git add opam
git commit -m 'Add OPAM metadata'
git push

Publishing your New Package

The above information is sufficient to use OPAM locally to integrate new code into an OPAM installation. Let's look at how other developers can share this metadata.

Picking up your development package

If another developer wants to pick up ocp-reloc, they can directly use your existing metadata by cloning a copy of your repository and issuing their own pin.

git clone git://github.com/OCamlPro/ocp-reloc.git
opam pin add ocp-reloc/

Even specifying the package name is optional since this is documented in ocp-reloc/opam. They can start hacking, and if needed use opam pin edit to amend the opam file too. No need for a repository, no need to share anything more than a versioned opam file within your project.

Cloning already existing packages

We have been focusing on an unreleased package, but the same functionality is also of great help in handling existing packages, whether you need to quickly hack into them or are just curious. Let's consider how to modify the `omd` Markdown library.

opam source omd --pin
cd omd.0.9.7
...patch...
opam upgrade omd

The new opam source command will clone the source code of the library you specify, and the --pin option will also pin it locally to ensure it is used in preference to all other versions. This will also take care of recompiling any installed packages that are dependent on omd using your patched version so that you notice any issues right away.

There's a new OPAM field available in 1.2 called dev-repo. If you specify this in your metadata, you can directly pin to the upstream repository via opam source --dev-repo --pin.

If the upstream repository for the package contains an opam file, that file will be picked up in preference to the one from the OPAM repository as soon as you pin the package. The idea is to have:

  • a development opam file that is versioned along with your source code (and thus accurately tracks the latest dependencies for your package).
  • a release opam file that is published on the OPAM repository and can be updated independently without making a new release of the source code.

How to get from the former to the latter will be the subject of another post! In the meantime, all users of the beta are welcome to share their experience and thoughts on the new workflow on the bug tracker.

by Louis Gesbert at August 19, 2014 12:00 AM

August 18, 2014

Github OCaml jobs

Full Time: Software Engineer for Complex Web Applications at Emcien in Atlanta, GA

Emcien develops four products based on a patented pattern-detection engine.

Using the discrete mathematical concept of sparse matrix graphs, these applications solve problems in manufacturing, retail, and law enforcement. This approach, plus big data sets, create challenging problems for our Software Engineers.

We are looking for more developers to join our team.

If you mastered the concepts behind web development and have experience dealing with the web stack, Emcien is willing to train you in Ruby on Rails. This position is an excellent opportunity for anyone who wants to make the transition from Python, PHP, Java, .NET or oCaml. If you are an experienced Rails engineer, Emcien presents some unique and challenging problems to solve.

Since we are a small team of developers, each team member dives into a variety of code. Although you do not have to be an expert in each of these areas, this position will expose you to CoffeeScript, MySql, AWS, D3.js, HAML, Backbone.js, and several other technologies. This means that your skills will advance with the latest ideas and innovations in web development.

Skills & Requirements

Must Have

  • Experience designing and building complex software systems
  • Experience with a dynamic language (Python or Ruby preferred)
  • Experience with a web framework such as Rails, Django, Pylons, CakePHP, Express.js
  • Experience with the unix command-line tools (beyond ls and cd)
  • Basic understanding of relational databases

Nice to Have

  • Experience with Ruby on Rails
  • Experience with structured Javascript (server-side or client-side)
  • Understanding of testing methodologies
  • Experience with Amazon AWS
  • Experience with (or interest in) Hadoop, Redis, and other data storage systems
  • Experience using git for source control
  • Familiarity with basic functional programming techniques

About Emcien, Inc. Emcien offers a suite of analytical applications that reveal patterns in data streams and actionable tasks to maximize profit. Emcien's analytics are based on innovative technology that leverages proprietary graph algorithms to eliminate the need for tedious data modeling and long implementation cycles.

  • Emcien does not sponsor any Visas.
  • All Web Developers must work out of our Atlanta Headquarters.
  • Compensation: based on experience
  • Principals only. Recruiters, please don't contact this job poster.
  • Please, no phone calls about this job!
  • Please do not contact job poster about other services, products or commercial interests.

August 18, 2014 05:46 PM

OCamlCore Forge News

Camlhighlight 3.0 released

Camlhighlight is a library offering syntax highlighting facilities for Ocsigen applications. This version features a smaller dependency set: it requires Tyxml but not Eliom.

by Dario Teixeira at August 18, 2014 03:41 PM

August 14, 2014

OCaml Platform

OPAM 1.2.0 public beta released

It has only been 18 months since the first release of OPAM, but it is already difficult to remember a time when we did OCaml development without it. OPAM has helped bring together much of the open-source code in the OCaml community under a single umbrella, making it easier to discover, depend on, and maintain OCaml applications and libraries. We have seen steady growth in the number of new packages, updates to existing code, and a diverse group of contributors.

OPAM has turned out to be more than just another package manager. It is also increasingly central to the demanding workflow of industrial OCaml development, since it supports multiple simultaneous (patched) compiler installations, sophisticated package version constraints that ensure statically-typed code can be recompiled without conflict, and a distributed workflow that integrates seamlessly with Git, Mercurial or Darcs version control. OPAM tracks multiple revisions of a single package, thereby letting packages rely on older interfaces if they need to for long-term support. It also supports multiple package repositories, letting users blend the global stable package set with their internal revisions, or building completely isolated package universes for closed-source products.

Since its initial release, we have been learning from the extensive feedback from our users about how they use these features as part of their day-to-day workflows. Larger projects like XenAPI, the Ocsigen web suite, and the Mirage OS publish OPAM remotes that build their particular software suites. Complex applications such as the Pfff static analysis tool and Hack language from Facebook, the Frenetic SDN language and the Arakoon distributed key store have all appeared alongside these libraries. Jane Street pushes regular releases of their production Core/Async suite every couple of weeks.

One pleasant side-effect of the growing package database has been the contribution of tools from the community that make the day-to-day use of OCaml easier. These include the utop interactive toplevel, the IOCaml browser notebook, and the Merlin IDE extension. While these tools are an essential first step, there's still some distance to go to make the OCaml development experience feel fully integrated and polished.

Today, we are kicking off the next phase of evolution of OPAM and starting the journey towards building an OCaml Platform that combines the OCaml compiler toolchain with a coherent workflow for build, documentation, testing and IDE integration. As always with OPAM, this effort has been a collaborative effort, coordinated by the OCaml Labs group in Cambridge and OCamlPro in France. The OCaml Platform builds heavily on OPAM, since it forms the substrate that pulls together the tools and facilitates a consistent development workflow. We've therefore created this blog on opam.ocaml.org to chart its progress, announce major milestones, and eventually become a community repository of all significant activity.

Major points:

  • OPAM 1.2 beta available: Firstly, we're announcing the availability of the OPAM 1.2 beta, which includes a number of new features, hundreds of bug fixes, and pretty new colours in the CLI. We really need your feedback to ensure a polished release, so please do read the release notes below.

  • In the coming weeks, we will provide an overview of what the OCaml Platform is (and is not), and describe an example workflow that the Platform can enable.

  • Feedback: If you have questions or comments as you read these posts, then please do join the platform@lists.ocaml.org and make them known to us.

Releasing the OPAM 1.2 beta4

We are proud to announce the latest beta of OPAM 1.2. It comes packed with new features, stability and usability improvements. Here the highlights.

Binary RPMs and DEBs!

We now have binary packages available for Fedora 19/20, CentOS 6/7, RHEL7, Debian Wheezy and Ubuntu! You can see the full set at the OpenSUSE Builder site and download instructions for your particular platform.

An OPAM binary installation doesn't need OCaml to be installed on the system, so you can initialize a fresh, modern version of OCaml on older systems without needing it to be packaged there. On CentOS 6 for example:

cd /etc/yum.repos.d/
wget http://download.opensuse.org/repositories/home:ocaml/CentOS_6/home:ocaml.repo
yum install opam
opam init --comp=4.01.0

Simpler user workflow

For this version, we focused on improving the user interface and workflow. OPAM is a complex piece of software that needs to handle complex development situations. This implies things might go wrong, which is precisely when good support and error messages are essential. OPAM 1.2 has much improved stability and error handling: fewer errors and more helpful messages plus better state backups when they happen.

In particular, a clear and meaningful explanation is extracted from the solver whenever you are attempting an impossible action (unavailable package, conflicts, etc.):

$ opam install mirage-www=0.3.0
The following dependencies couldn't be met:
  - mirage-www -> cstruct < 0.6.0
  - mirage-www -> mirage-fs >= 0.4.0 -> cstruct >= 0.6.0
Your request can't be satisfied:
  - Conflicting version constraints for cstruct

This sets OPAM ahead of many other package managers in terms of user-friendliness. Since this is made possible using the tools from irill (which are also used for Debian), we hope that this work will find its way into other package managers. The extra analyses in the package solver interface are used to improve the health of the central package repository, via the OPAM Weather service.

And in case stuff does go wrong, we added the opam upgrade --fixup command that will get you back to the closest clean state.

The command-line interface is also more detailed and convenient, polishing and documenting the rough areas. Just run opam <subcommand> --help to see the manual page for the below features.

  • More expressive queries based on dependencies.

    $ opam list --depends-on cow --rec
    # Available packages recursively depending on cow.0.10.0 for 4.01.0:
    cowabloga   0.0.7  Simple static blogging support.
    iocaml      0.4.4  A webserver for iocaml-kernel and iocamljs-kernel.
    mirage-www  1.2.0  Mirage website (written in Mirage)
    opam2web    1.3.1 (pinned)  A tool to generate a website from an OPAM repository
    opium       0.9.1  Sinatra like web toolkit based on Async + Cohttp
    stone       0.3.2  Simple static website generator, useful for a portfolio or documentation pages
  • Check on existing opam files to base new packages from.

    $ opam show cow --raw
    opam-version: "1"
    name: "cow"
    version: "0.10.0"
    [...]
  • Clone the source code for any OPAM package to modify or browse the interfaces.

    $ opam source cow
    Downloading archive of cow.0.10.0...
    [...]
    $ cd cow.0.10.0

We've also improved the general speed of the tool to cope with the much bigger size of the central repository, which will be of importance for people building on low-power ARM machines, and added a mechanism that will let you install newer releases of OPAM directly from OPAM if you choose so.

Yet more control for the packagers

Packaging new libraries has been made as straight-forward as possible. Here is a quick overview, you may also want to check the OPAM 1.2 pinning post.

opam pin add <name> <sourcedir>

will generate a new package on the fly by detecting the presence of an opam file within the source repository itself. We'll do a followup post next week with more details of this extended opam pin workflow.

The package description format has also been extended with some new fields:

  • bug-reports: and dev-repo: add useful URLs
  • install: allows build and install commands to be split,
  • flags: is an entry point for several extensions that can affect your package.

Packagers can limit dependencies in scope by adding one of the keywords build, test or doc in front of their constraints:

depends: [
  "ocamlfind" {build & >= 1.4.0}
  "ounit" {test}
]

Here you don't specifically require ocamlfind at runtime, so changing it won't trigger recompilation of your package. ounit is marked as only required for the package's build-test: target, i.e. when installing with opam install -t. This will reduce the amount of (re)compilation required in day-to-day use.

We've also made optional dependencies more consistent by removing version constraints from the depopts: field: their meaning was unclear and confusing. The conflicts field is used to indicate versions of the optional dependencies that are incompatible with your package to remove all ambiguity:

depopts: [ "async" {>= "109.15.00"} & "async_ssl" {>= "111.06.00"} ]

becomes:

depopts: [ "async" "async_ssl" ]
conflicts: [ "async" {< "109.15.00"}
             "async_ssl" {< "111.06.00"} ]

There is an upcoming `features` field that will give more flexibility in a clearer and consistent way for such complex cases.

Easier to package and install

Efforts were made on the build of OPAM itself as well to make it as easy as possible to compile, bootstrap or install. There is no more dependency on camlp4 (which has been moved out of the core distribution in OCaml 4.02.0), and the build process is more conventional (get the source, run ./configure, make lib-ext to get the few internal dependencies, make and make install). Packagers can use make cold to build OPAM with a locally compiled version of OCaml (useful for platforms where it isn't packaged), and also use make download-ext to store all the external archives within the source tree (for automated builds which forbid external net access).

The whole documentation has been rewritten as well, to be better focused and easier to browse. Please leave any feedback or changes on the documentation on the issue tracker.

Try it out !

The public beta of OPAM 1.2 is just out. You're welcome to give it a try and give us feedback before we roll out the release!

We'd be most interested on feedback on how easily you can work with the new pinning features, on how the new metadata works for you... and on any errors you may trigger that aren't followed by informative messages or clean behaviour.

If you are hosting a repository, the administration scripts may help you quickly update all your packages to benefit from the new features.

by OCaml Platform Team at August 14, 2014 12:00 AM

August 09, 2014

Shayne Fletcher

Digits

Digits

Counting is a common problem in computer programming. I recently had a need for an algorithm to compute the number of base 10 digits of an integer (for example, there are 3 digits in the number 569). Here's what I worked out.

module type S=sig
(**Count the number of base 10 digits of an integer*)
val digits : int -> int

(**Prepend with zeroes as necessary to get the 'right'
number of columns *)
val digits_10 : int -> int -> string

end

module A : S = struct
let digits n =
let rec loop acc n =
let k = n / 10 in
if k = 0 then acc else loop (acc + 1) k
in loop 1 n

let digits_10 i n =
let l = digits n in
let s = string_of_int (abs n) in
let maybe_sign = if n < 0 then "-" else "" in
if l >= i then (maybe_sign^s)
else
let lz=string_of_int 0 in
let rec aux acc j =
if j = 0 then acc
else
aux (lz^acc) (j - 1) in
maybe_sign ^ aux s (i - l)
end

module Test_io = struct

let test_0 () =
let print_line i =
Printf.printf "There are %d digits in %d\n" (A.digits i) i
in
let rec loop i n =
if i = n then ()
else
let x =
int_of_float (10. ** (float_of_int i)) in
print_line x; loop (i + 1) n
in (*Range over 0 to to 10^19 *) loop 0 19

let test_1 () =
let print_line i =
Printf.printf "%s\n" i
in
let rec loop i n =
if i = n then ()
else
let x = A.digits_10 3 i in
print_line x; loop (i + 1) n
in (*Range over 0 to 100, 3 columns*)loop 0 100

let run () = test_0 (); test_1 ()

end

let () = Test_io.run ()
The program outputs

There are 1 digits in 1
There are 2 digits in 10
There are 3 digits in 100
There are 4 digits in 1000
There are 5 digits in 10000
There are 6 digits in 100000
There are 7 digits in 1000000
There are 8 digits in 10000000
There are 9 digits in 100000000
There are 10 digits in 1000000000
There are 11 digits in 10000000000
There are 12 digits in 100000000000
There are 13 digits in 1000000000000
There are 14 digits in 10000000000000
There are 15 digits in 100000000000000
There are 16 digits in 1000000000000000
There are 17 digits in 10000000000000000
There are 18 digits in 100000000000000000
There are 19 digits in 1000000000000000000
000
001
002
003
004
005
006
.
.
.
097
098
099

Fun, fun, fun!

by Shayne Fletcher (noreply@blogger.com) at August 09, 2014 05:51 PM

August 07, 2014

Dario Teixeira

Announcing Bookaml 1.0

I'm happy to announce release 1.0 of Bookaml, a simple library for validating ISBNs, gathering information about a book given its ISBN, or to find any number of books matching given search criteria.

Bookaml is closely tied to the Amazon Product Advertising API, which it uses internally for retrieving book information. Therefore, if you intend to use the library's facilities beyond basic validation of ISBN numbers, you will need an Amazon Web Services account and associated access keys (these are freely obtainable, though registration is required).

(Note that though adding more features related to the Amazon Product Advertising API is an ever-present temptation, this is not the goal of Bookaml. You may find Bookaml's code a good starting point for that purpose, in which case I encourage you to fork the project. Bookaml itself should remain focused on its original goal.)

The core library depends on Batteries, Calendar, Cryptokit, and Ocamlnet's Netstring.

Bookaml is engine-agnostic on the matter of which XML parser and HTTP fetcher are actually used for contacting the Amazon servers and parsing the result. The intention was to be friendly towards monadic libraries such as Lwt or Async. Bookaml ships with two such engines, each having their own set of dependencies:

  • Bookaml_ocamlnet: Tyxml (for the XML parser) and Ocamlnet's Netclient (for HTTP fetching).
  • Bookaml_ocsigen: Tyxml (for the XML parser) and Ocsigenserver (for HTTP fetching). Note that this engine lives in Lwt monad.

The API documentation is available online. Particularly of interest is the Bookaml_ISBN module for handling and validating ISBNs (and which may be used even without an AWS account), and module type ENGINE, which documents the functions used for gathering book information and searching for books.

Another good place to get started is to peruse the small included example, which illustrates the use of the library from within an Ocsigen/Eliom server-side application (note that you will need actual AWS credentials if you intend to run this example).

by Dario at August 07, 2014 01:50 PM

August 06, 2014

OCamlCore Forge News

Release of Bookaml 1.0

Bookaml is a library providing a basic API to gather information about a book given its ISBN, or to find any number of books matching given search criteria. The library is closely tied to the Amazon Product Advertising API, which is used internally for retrieving book information.

by Dario Teixeira at August 06, 2014 04:30 PM