All Posts

Erlang ‘One Weird Trick’ Goodiebag

I’ve asked the Erlang twittersphere what they think every Erlang developer should know about. The result is this Erlang random goodiebag. I would love to hear your ‘One weird trick’ in the comments!

written in Erlang Read on →

5 Great TODO Apps for Mac

There are plenty of great TODO apps out there. If you want stop procrastinating and become super productive, you probably should be using one of these desktop Mac apps.

The following list is my selection of 5 great TODO apps, designed for your Mac desktop.

written in apps, productivity, todo Read on →

5 Great Writing Apps for Mac

There are plenty of great writing apps out there. If you want to focus on writing, you probably should be using one of these desktop Mac apps.

The following list is my selection of 5 great writing apps, designed for your Mac desktop that sync nicely with their iOS counterparts.

written in apps, productivity, writing Read on →

My 5 Tips for New Runners

Yours truly started running just over 7 months ago with the goal to get into shape. I dropped 17 KG since and now really enjoy running. I even finished my first 10 miles race! (1:21:05). Here are 5 things that I learned in the process;

written in running Read on →

Erlang/OTP 17.0 on OSX Mavericks With WX and a Working Observer

Update April 15 – 2014: Erlang/OTP 17.0 with a working WX library is now also available through Homebrew: brew install wxmac erlang. HT @dhc_

This post is an update of HOWTO: “Erlang R17-RC2 on OSX Mavericks with WX and a working Observer”.

The best Erlang yet

Today’s Erlang/OTP 17.0 release is ‘the best Erlang yet’ and contains two significant language changes: Maps and Named arguments in funs.

Erlang uses wxWidgets, a cross platform GUI library for it’s GUI tools. This build dependency was hard to get working pre-17, especially for 64-bit Erlang. However, 17.0 brings double rainbows and care bears for everyone that reads this HOWTO. So Enjoy!

nonode@nohost

Set correct Xcode path for compilation

As far as I know you need have Xcode install to compile Erlang from source. You can download Xcode via the Mac App Store

If you have multiple versions of Xcode installed (beta’s for example), make sure the Command Line Tools are installed and are pointing to the correct Xcode version.

Initiating an install of the Xcode Command Line Tools:

1
$ xcode-select —install

And verify that the CL-tools point to the correct Xcode install

1
$ xcode-select -s /Applications/Xcode.app/Contents/Developer

Install wxWidgets

wxWidgets is a Cross Platform GUI library that’s used by Erlang for applications like Observer.

Execute this line and get some coffee, walk the dog, take out the trash and/or play with your kids. Compilation takes a while.

1
2
3
4
5
6
$ curl -O http://optimate.dl.sourceforge.net/project/wxwindows/3.0.0/wxWidgets-3.0.0.tar.bz2
$ tar xvjf wxWidgets-3.0.0.tar.bz2
$ cd wxWidgets-3.0.0.tar.bz2
$ ./configure —with-cocoa —prefix=/usr/local
$ make && sudo make install
$ export PATH=/usr/local/bin:$PATH

Check that you got the correct wx-config

1
$ which wx-config

Install kerl

Kerl is a utility that helps you build and manage multiple instances of Erlang/OTP.

Create ~/.kerlrc. I use $ vim ~/.kerlrc.

Add these lines:

1
KERL_CONFIGURE_OPTIONS="—disable-debug —without-javac —enable-shared-zlib —enable-dynamic-ssl-lib —enable-hipe —enable-smp-support —enable-threads —enable-kernel-poll —with-wx"

Build & Install Erlang with kerl

1
2
$ kerl update releases
$ kerl build 17.0 17.0

For a 32-bit Erlang prefix kerl build with CPPFLAGS:

1
$ CPPFLAGS="-arch i386" kerl build 17.0 17.0

Install:

1
$ kerl install 17.0 ~/erlang_17_0

Activate:

1
$ . ~/erlang_17_0/activate

And bliss out on your new wx-enabled Erlang:

1
$ erl -s observer start

written in Erlang

Your First Erlang Program (in Style).

Always wanted to learn Erlang? Let’s create your first Erlang “Hello World” program in style!

In this HOWTO I’ll show you how to setup a bleeding edge Erlang development VPS and how to run you first Erlang program.

Main ingredient: Cores

Erlang’s main strength is it’s concurrency support. It likes cores, so for our ‘Hello World’ program we obviously need cores. Lot’s! Not 4, not 8, 20!

Create an account on Digital Ocean if you don’t have one yet (love them) and we’re going to boot up their biggest instance. It’s a steal at less than 1 dollar per hour. Just make sure you destroy it when done.

64GB and 20 cores will make our Hello World so snappy!

Fullscreen%2029/03/14%2021:36

  • Pick a datacenter location near you.
  • Select the latest version of Ubuntu: 13.10 x64.
  • Create the Droplet.
  • And ssh to your Droplet with the credentials received from Digital Ocean: ssh root@your_ip_address.

Bleeding Edge Erlang

We’re going to compile Erlang from it’s github repository master branch, At the time of writing it’s a few commits after R17 release candidate 2 which comes with a Hipe LLVM backend, maps and named funs. If that doesn’t make any sense, no worries, just remember it’s the fastest Erlang yet. And fast is good.

Install the required Ubuntu packages:

1
$ apt-get install tmux build-essential emacs24 git-core libncurses5-dev libssl-dev autconf htop

Fire up Tmux:

1
$ tmux

Install Kerl, a tool which makes building and switching Erlang versions easy.

Let’s add some good configuration options for our Erlang installation

1
$ emacs .kerlrc

And add

1
KERL_CONFIGURE_OPTIONS="—disable-debug —without-javac —enable-shared-zlib —enable-dynamic-ssl-lib —enable-hipe —enable-smp-support —enable-threads —enable-kernel-poll"

And because we can, we forge our Erlang installation on 20 cores. Muahahaha.

To see those cores sweat for you on compilation, create another tmux window CTRL-b c and run htop.

Fullscreen%2029/03/14%2021:52

Besides the eye candy, compilation finishes under 5 minutes on a 20 core Digital Ocean Droplet. Whoop!

To start compilation of Erlang:

1
$ export MAKEFLAGS=-j20 && ./kerl build git git://github.com/erlang/otp.git master erlang_llvm

After compilation we need to install the build and activate it:

1
2
$ kerl install erlang_llvm erlang_llvm
$ . ~/erlang_llvm/activate

Great! We are now ready for our Pièce de résistance.

Just say: Hello!

Real Erlang hacker use Emacs, so let’s setup Emacs for Erlang development.

Fetch a good base config for Emacs:

Start up Emacs emacs. It will complain that it can’t find projmake-mode. Let’s fix that:

1
[ESC]-x package-install [Enter] projmake-mode

Exit emacs:

1
[CTRL]-x [CTRL]-c

Start up Emacs again emacs. Great! We can finally start writing our “Hello World” program. Oh, not, wait. First, we create a projmake file. The file is needed by Projmake-mode, a Flymake inspired mode that compiles your program on every save and shows build errors and warnings inline. Really useful!

1
[CRTL]-x f projmake [Enter]

Add these line and save the file

1
2
3
(projmake
 :name  "Hello"
 :shell "erlc +native hello.erl")

Ok, now we can really start writing our “Hello World” program and put those 20 cores and 64GB RAM to good use.

1
[CTRL]-x f hello.erl [Enter]

And type or paste:

1
2
3
4
5
module(hello).
export([just_say/0]).

just_say() –>
    io_format("hello~n", []).

And save with [CTRL]-x [CTRL]-s.

Whoops. We made an error as projmake-mode shows:

Fullscreen%2029/03/14%2021:24

Replace io_format with io:format and save again. That fixes our error!.

Let’s run our program. Fire up the Erlang shell with:

1
[ESC]x erlangshell

Load the hello module with:

1
1> m(hello). [Enter]

And run you first Erlang function….

1
2> hello:just_say(). [Enter]

Bliss!

Congratulations. You now have a powerful Erlang development environment in your hands.

Check out A beginners guide to Erlang to continue your Erlang binge.

written in Erlang

Selenium Webdriver for Erlang Quick Start

Selenium is the industry standard for automated testing of web applications. Together with Webdriver, a ‘remote control’ API for all major browsers, it enables you to create robust integration test for the browser.

The great people of QuviQ, creators of the unique Quickcheck test framework, created an Erlang Webdriver client implementation (Github repository).

It’s trivial to get started with the following steps:

Step 1: Add webdrv to the rebar.config of your project

Open rebar.config in your favorite editor, and make sure webdrv is listed as dependency. I use a fork of the original repository that support rebar:

1
2
3
{deps, [
    {webdrv, "", {git, "https://github.com/ehedenst/webdrv.git", {branch, "master"}}},
       ]}.

Step 2: Get and compile webdrv

Go to the root of your Erlang project and execute:

1
$ rebar get-deps compile

Step 3: Get & start the Google chromedriver

For this quick start we will be using the Google Chromedriver. Get the right package for your environment here. I’m now on a Mac, so:

1
2
3
$ curl -O http://chromedriver.storage.googleapis.com/2.9/chromedriver_mac32.zip
$ unzip chromedriver_mac32.zip
$ ./chromedriver

The last line starts up the Chromedriver server and if all went well, you should get the following output:

1
Starting ChromeDriver (v2.9.248307) on port 9515

Important! This server needs to be running during test execution.

Step 4: Your first Erlang webdrvr test!

Save the following module in src/random_org_test.erl. In this test we open a page, fill in a form, submit the form, and check if an expected piece of text is indeed present in the response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
module(random_org_test).

compile(export_all).

include_lib("webdrv/include/webdrv.hrl").

define(CHROMEDRIVER, "http://localhost:9515/").

test() –>
    {ok, _Pid} = webdrv_session:start_session(test, ?CHROMEDRIVER,  webdrv_cap:default_chrome(), 10000),
    webdrv_session:set_url(test, "http://www.random.org/integers/"),
    {ok, Emin} = webdrv_session:find_element(test, "name", "min"),
    webdrv_session:clear_element(test, Emin),
    webdrv_session:send_value(test, Emin, "5"),
    {ok, Emax} = webdrv_session:find_element(test, "name", "max"),
    webdrv_session:clear_element(test, Emax),
    webdrv_session:send_value(test, Emax, "15"),
    webdrv_session:submit(test, Emax),
    {ok, PageSource} = webdrv_session:get_page_source(test),
    string:str(PageSource, "Here are your random numbers") > 0,
    webdrv_session:stop_session(test).

Step 5: Run the test

Run your test by opening up the Erlang shell..

1
$ erl -pa ebin deps/*/ebin

..and execute the test function

1
1> random_org_test:test().

You should see the Chrome browser opening in the background, quickly flashing some pages, closing, and on the Erlang shell the anticlamatic output ok.

Further information

written in Erlang

Erlang R17-rc1 on OSX Mavericks With WX and a Working Observer

Update 15 April 2014 Erlang/OTP 17.0 is released. Read the updated HOWTO here.

Update 11 Feb 2014 Dan GudMundsson pointed out that starting with R17 both 32 and 64 bit Erlang will work with wxWidgets. I’ve updated part of this blog post with the instructions found in the official Erlang/OTP installation HOWTO.

This post is based on HOWTO: Getting wx to work with Erlang r16b02 on OS X by Martin Schuerrer

Erlang R17-rc1 is ‘the best Erlang yet’ and contains two significant language changes: Maps and named arguments in funs.

Erlang uses wxWidgets, a cross platform GUI library for it’s GUI tools. This build dependency was hard to get working pre-R17, especially for 64-bit Erlang. However, R17 brings double rainbows and care bears for everyone that reads this HOWTO. So Enjoy!

nonode@nohost

Set correct Xcode path for compilation

1
$ xcode-select -s /Applications/Xcode.app/Contents/Developer

or if you installed a beta version (in my case 51-Beta5):

1
$ xcode-select -s /Applications/Xcode51-Beta5.app/Contents/Developer

Install wxWidgets

wxWidgets is a Cross Platform GUI library that’s used by Erlang for applications like Observer.

Execute this line and get some coffee, walk the dog, take out the trash and/or play with your kids. Compilation takes a while.

1
2
3
4
5
6
$ curl -O http://optimate.dl.sourceforge.net/project/wxwindows/3.0.0/wxWidgets-3.0.0.tar.bz2
$ tar xvjf wxWidgets-3.0.0.tar.bz2
$ cd wxWidgets-3.0.0.tar.bz2
$ ./configure —with-cocoa —prefix=/usr/local
$ make && sudo make install
$ export PATH=/usr/local/bin:$PATH

Check that you got the correct wx-config

1
$ which wx-config

Install kerl

Kerl is a utility that helps you build and manage multiple instances of Erlang/OTP.

Create ~/.kerlrc. I use $ vim ~/.kerlrc.

Add these lines:

1
KERL_CONFIGURE_OPTIONS="—disable-debug —without-javac —enable-shared-zlib —enable-dynamic-ssl-lib —enable-hipe —enable-smp-support —enable-threads —enable-kernel-poll —with-wx"

Build & Install Erlang with kerl

1
2
$ kerl update releases
$ kerl build 17.0-rc1 17rc1

For a 32-bit Erlang prefix kerl build with CPPFLAGS:

1
$ CPPFLAGS="-arch i386" kerl build 17.0-rc1 17rc1

Install:

1
$ kerl install 17rc1 ~/17rc1

Activate:

1
$ . ~/17rc1/activate

And bliss out on your new wx-enabled Erlang:

1
$ erl -s observer start

written in Erlang

How Apple Will Implement Mobile Payments

The hot new rumor is that Apple is going for the mobile payment business. Sources of Re/Code claim that Apple and Paypal are in talks about a partnership, perhaps Apple using a whitelabel version of Paypal as a backoffice for their new service.

If Paypal and Apple are going to partner, my best guess is that this will NOT be using a whitelabel, but a much more lightweight integration like this:

  • Apple will partner with Visa, Mastercard, Venmo, Paypal and such.
  • Apple will allows their users to hook up their phone’s with those payment providers in a similar way as now done with Facebook, Flickr, Vimeo etc. Like this:
    setting_example
  • Apple will provider app developers with a Mobile Payment framework, similar to the Social Framework that will provide developers an abstracted way to accept mobile payments. Likely with TouchID confirmation for easy of use. Possibly with a way to let the user to configure a standard shipping and/or billing address.
  • So, when the user is checking out a physical good, a dialog will be shown where the user will select the desired payment method. Like this:
    payment_selection

Some reasons why I think a deep level partnering will not happen:

  • Apple does not have a track record partnering with third parties on such a deep level. They want to be able to call the shots. Control is the keyword.
  • Apple probably doesn’t want to deal with the support effort of chargebacks for goods they don’t deliver. (see also the physical good restriction for the current in app purchase stuff). It’s a totally different business.

Upvote this story on Hacker News

written

Three Affordable 4K Retina Monitors Introduced at CES 2014

2014 just started and is already a great year for high pixel density lovers. Three affordable 4K desktop monitors where introduced at CES 2014. Release is expected to be in Q1 & Q2 of this year.

The 4k monitors:

Your laptop or PC probably needs to support DisplayPort 1.2 to take advantage of the high resolution. My work supplied Mid-2009 Macbook Pro certainly doesn’t. Time for an upgrade!

written in retina

Continuous Integration for Erlang With Travis-CI

Erlang/OTP is designed for building large, scalable, soft-realtime systems with high availability. Testing such systems is non-trivial, useful automated testing even more so. That’s why Erlang comes with some advanced testing libraries.

The three most important methods are explained here by a few simple examples:

  • Unit testing
  • Quickcheck
  • Common test

First clone the project from Github using the command:

1
$ git clone git@github.com:wardbekker/ci_quickstart.git

For compiling and executing the project we use Rebar, a sophisticated build-tool for Erlang projects that follows OTP principles. Steps to build rebar:

1
2
3
4
5
6
7
8
9
10
$ git clone git://github.com/basho/rebar.git
$ cd rebar
$ ./bootstrap
Recompile: src/getopt
Recompile: src/rebar_utils
==> rebar (compile)
Congratulations! You now have a self-contained script called "rebar" in
your current working directory. Place this script anywhere in your path
and you can use rebar to build OTP-compliant apps.

Unit testing with EUnit

Let’s start with the most simple test method; EUnit. It’s Erlang unit testing library. A unit test check if a function returns the expected result for a given input. In the example below the function addition is defined in the module ci_quickstart_math and two assertions are used:

  • ?assertEqual(expected, actual)
  • ?assertNotEqual(expected, actual).

Try it our yourself with executing EUnit from the command line: rebar get-deps compile eunit:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
module(ci_quickstart_math).
export([addition/2]).

ifdef(TEST).
include_lib("eunit/include/eunit.hrl").
endif.

addition(X, Y) –>
    X + Y.

ifdef(TEST).

simple_test() –>
    ?assertEqual(4, addition(2,2)),
    ?assertNotEqual(3, addition(1,1)).

endif.

Did all test pass? Excellent! Now the bad news. The actual value of this type of test if quite low. Are we sure the addition function works correct for all possible input? We are now only certain of these cases:

  • addition(2,2) == 4
  • addition(1,1) /= 3

And even then, when I change the body of the addition function in obviously something totally wrong:

1
2
 addition(X, Y) –>
    4.

The tests will still pass!

So, with unit tests our assertions may be correct, but the function body of addition can be a steaming pile of canis faeces.

It’s even worse; As in this case, the arguments of addition are 64-bit small integers, which have a range of -576460752303423489 – 576460752303423488. With two arguments, that is a humongous amount of inputs we should test to be really sure our function works correctly. In the example unit test we only check two? Even adding twenty more cases, the hard worker that you are, effectively have very little value.

Depressed already? On to the good stuff.

QuickCheck

Continuing with the addition example; what we actually want is a test method that generated all possible inputs, and checks the result. Erlang has this, and the method is called QuickCheck. Erlang even has multiple QuickCheck-style libraries available:

A QuickCheck test, also called a property for addition function looks like this:

1
2
3
4
5
6
prop_sum() –>
    ?FORALL(
        {X, Y},
        {int(), int()},
        addition(X,Y)  Y == X
    ).

Test this example from the command line by executing ./shell.sh. You will enter the Erlang shell. Then execute proper:quickcheck(ci_quickstart_math:prop_sum())..

If we look at the implementation of the QuickCheck test, notice that we are not testing specific numbers. We are testing a property of the addition function, namely when we add int X and Y, and subtract Y from the result of the addition, we should be left with X again.

The code {int(), int()} specifies that the QuickCheck should generate tuples with two random integers. Each generated tuple is bound to the pattern {X, Y} by Erlang pattern matching. Quickcheck will generate 100 combinations by default. With the numtests option we can increase this considerably: proper:quickcheck(ci_quickstart_math:prop_sum(),[{numtests,10000}])..

The challenge when using Quickcheck style testing, is to come up with good function properties. This is much harder than writing unit tests. It’s even more difficult to reason about function properties than actually writing the actual function. So why bother?

Reasons to use QuickCheck:

  • Test correct functioning for all inputs.
  • Compared to unit tests, QuickCheck test tend to find more bugs than unit tests.
  • You need to reason about your code on a deeper level which improves your understanding of the problem you are solving, which tends to result in better code.

Common Test

As you might know, Erlang is a very good fit for building concurrent, distributed and fault tolerant systems. Testing if what you build is actually has those properties, is quite complex.

For that, Erlang offers Common Test. This test frameworks can do the heavy lifting required for meaningful system tests. The inherent complexity of concurrent, distributed en fault tolerant systems is reflected in Common Test. So, in this introduction we only take a very quick glance. In this example we mimic the initial unit test using pattern matching for assertions.

1
2
3
4
5
6
7
8
9
10
11
12
13
module(basic_SUITE).
include_lib("common_test/include/ct.hrl").

export([all/0]).
export([test1/1, test2/1]).

all() –> [test1,test2].

test1(Config) –>
    3 = ci_quickstart_math:addition(1,2). %% validated using pattern matching

test2(Config) –>
    2 = ci_quickstart_math:addition(1,1).  %% validated using pattern matching

Continuous integration with Travis-CI

During development, you run your Erlang automated tests on your own workstation. But there comes a point where that’s no longer feasible because of the long duration or high load. Or you work in a team setting where it’s important that only working code is integrated. In those cases and for several other good reasons, you need to use a Continuous integration system.

There are several continuous integration systems that allow you to run automated tests for Erlang. In this example we use Travis-CI. It’s a free hosted continuous integration service for the open source community. Travis-CI integrates with the popular Github.

Let’s add our example project to Travis-CI.

Preparation

The build process of Travis-CI is configured with a .travis.yml file in the root of our repository:

1
2
3
4
5
6
7
language: erlang // This project is an Erlang project
notifications:
  email: you@example.org // Notify this e-mail address on build failures
otp_release: // The Erlang/OTP versions to run your test against.
  – R15B01
  – R15B
  – R14B04

Travis-CI Setup

This video shows how to register with Travis-CI:

  • Log in with your Github account.
  • Open the Travis-CI profile page.
  • Enable the repository that you want to test.

That’s it!

Travis-CI Success Run

This video shows how Travis-CI will report an successful integration build

Travis-CI Failure Run

This video shows how Travis-CI will report an failed integration build

If you entered your e-mail address in .travis.yml, you will receive a mail about which commit broke the build:

Broken build e-mail notification

When your colleague fixed the build (you would never commit broken code, right?), you will receive a mail that the build is fixed by a new commit.

Fixed build e-mail notification

written in Erlang

A Beginners Guide to Erlang

In the beginner’s mind there are many possibilities, but in the expert’s there are few

Shunryu Suzuki Zen Mind Beginner’s Mind

These starting points helped me getting up to speed with Erlang. I hope this information helps you too.

How install Erlang/OTP

You can install Erlang/OTP from source, but I like the ready-to-go packages provided by Erlang Solutions.

Writing Erlang

I use Aquamacs, An Emacs for mac users, with the Erlang mode provided by Erlang/OTP. Combined with Eric B Merritt’s projmake-mode and Mochiweb reloader this makes for a productive development environment.

Documentation

  • The official Erlang documentation is pretty good, but the writing style / structure takes a while to get used to. But the info is certainly there.
  • I keep a local copy of Erldocs on my development machine for quick access. Unfortunatly it doesn’t have an R16 copy and function signatures are not shown fully correct, but it works for me.
  • Learn you some erlang is a free online guide (and available as a dead tree version). It’s a very good intro for learning Erlang.

Best places to ask for help

The Erlang Questions Mailinglist is the best place to ask your Erlang questions. Don’t be surprised if you question is answered by Erlang inventors themselves!

As with other programming languages Stack overflow is also a great place to get answers to your pressing Erlang questions.

Erlang books

Compared to Java, the quantity of Erlang books is low. But the quality is pretty good! And a little birdy told me that some great new books will be published in the near future. Warning: affiliate links to Amazon ahead. You will be sponsoring my caffeine intake. Thank you.


Erlang Programming by Francesco Cesarini

Learn you some Erlang by Fred Hebert

Building Web Applications with Erlang: Working with REST and Web Sockets on Yaws

Programming Erlang: Software for a Concurrent World

Erlang and OTP in Action

Building Scalable Applications with Erlang (Developer’s Library)

Introducing Erlang

Erlang/OTP


Testing and Continuous integration

I recently gave a talk introducing the great automated testing facilities of Erlang. Check out the video and slides

With help from former collegue & friend Josh Kalderimis I’ve added the first iteration of Erlang support to Travis-CI. It’s a free continuous integration service for open source projects. It’s used by well known Erlang projects like Basho Riak, Elixir and Mochiweb. See the documentation how to add your project.

If you can read Dutch, you can also read my blog post about Geautomatiseerd testen met Erlang/OTP en Travis-CI which covers the same ground.

Who to follow on twitter

Two of the three Erlang inventors are active tweeps:

Want more? Bryan Hunter maintains a list of Erlang folks on twitter .

I’m also on twitter

Popular Erlang projects

Want to learn Erlang from the absolute best? Most popular Erlang projects are open source. Make some coffee, clone the repositories and learn from the code of the Erlang gods. Below a selection:

  • Erlang OTP – The full Erlang/OTP sourcecode.
  • Basho Riak – A decentralized datastore.
  • Cowboy – Small, fast, modular HTTP server written in Erlang.
  • Rebar – A sophisticated build-tool for Erlang projects that follows OTP principles.
  • Mochiweb – MochiWeb is an Erlang library for building lightweight HTTP servers.
  • Webmachine – A REST-based system for building web applications.

Explore more popular Erlang projects on Github’s Erlang page

Conferences & User groups

The Erlang Factory conferences are the best places to meet professional Erlang developers. I’ve attended a few of them, and I am always amazed by the quality of the speakers and the hallway discussions. Pro tip: make sure you have a substantial lunch and then stay for the drinks.

A few local active user groups that I’m aware of:

Explore more Erlang user groups on Meetup

Training

I attended the three day Erlang/OTP express course from Erlang Solutions and got a much better understanding of OTP. Erlang Solutions also provide E-learning training.

I also heard some good things from the Erlang Camp by Erlware. This is a intensive two day trainings for beginners and intermediate Erlang programmers.

written in Erlang

[Dutch] Scrum 2009 ≠ Scrum 2012

Tijdens de recent bezochte Scrum Product Owner cursus werd duidelijk dat Scrum is veranderd. Mijn begrip van Scrum, zoals geleerd tijdens de Scrum Master training van Jeff Sutherland in 2009, was aan herziening toe. Hieronder mijn bevindingen:

Scrum in 2009

Jeff Sutherland, één van de bedenkers van scrum, legde in de 2009 training de nadruk op de volgende concepten:

  • Hyperproductive teams. Dit zijn teams met een hoge mate van plichtsbesef, samenwerking en eigenaarschap (ownership), met een enorme productiviteit tot gevolg.
  • Velocity. Velocity druk uit hoeveel functionaliteit, eventueel in abstracte story points, er ontwikkeld is per sprint. Naast feedback voor voor het team werd velocity gebracht als een belangrijke metriek voor performance en planning rapportage voor het management.
  • Definition of done. Bij voorkeur gehanteerd als een Oost-Duitste grensovergang ten tijde van de DDR. Vanaf de eerste sprint dient de product increment production-ready opgeleverd te worden.

Scrum in 2012

In de recente Scrum Product Owner cursus door Roman Pichler, bleek dat in de afgelopen drie jaar het belang en toepassing van deze concepten sterk zijn veranderd:

  • Hyperproductivity is niet één maal genoemd in de Product Owner cursus. Productiviteit is niet (meer?) het doel van Scrum. Het doel van Scrum is de effectieve realisatie van nieuwe, succesvolle en innovatieve producten met enthousiaste eindgebruikers.
  • Velocity als rapportagemiddel wordt sterk afgeraden. Het is één van de hulpmiddel voor de product owner om de verwachte release planning bij te werken, maar niet meer. De echte waarde van product increments is het verifiëren van aannamens en het ontdekken van belangrijke functionaliteit. Negatieve feedback en sprint/story mislukkingen zijn net zo waardevol. Een sprint met zero velocity geeft derhalve zéér waardevolle feedback. Ik kan uit ervaring spreken dat het zeer verleidelijk is voor management om velocity te misbruiken als rapportagemetriek van de team prestaties.
  • Definition of Done (DoD). Uiteraard moet er in de loop van de ontwikkeling production-ready software worden opgeleverd, maar in de eerste fase van het project is experimenteren veel belangrijker. Uitvinden wat werkt, en wat niet werkt. Hierdoor is de mate van verandering hoog. Een veeleisende DoD werkt dan als een verlammende bureaucratie. Pas wanneer de basis functionaliteit van het product stabiliseert, is het aan te raden om de DoD aan te scherpen.

Niet efficient

In mijn optiek benadrukken de veranderingen de sweet spot van Scrum; ontwikkeling van nieuwe features of geheel nieuwe producten met een hoge mate van innovatie. Innovatie is een proces van vallen en opstaan. Ontdekken wat werkt, en vooral ook wat niet werkt. Dit leerproces is derhalve per definitie niet efficient. Scrum is wel een heel erg effectieve manier van innoveren. Veel ruimte voor waardevolle feedback en multidisciplinaire samenwerking waardoor het team snel leert. In deze blog post gaat Roman Pichler dieper in op de toepasbaarheid van Scrum in de verschillende fases van product innovatie

Waar is het changelog?

Gezien de grote veranderingen was ik benieuwd of er een officiële Scrum changelog is. Zijn de veranderingen een verschil in interpretatie tussen de verschillende trainers, of zijn de spelregels veranderd. Helaas, er is geen changelog. Wel de Scrum Guide, de officiële gids met de actuele spelregels van Scrum. De meest recente versie is uit october 2011. Ik ga deze uitpluizen of er nog andere belangrijke veranderingen zijn. Wie weet materiaal voor een volgende blog post. Renee Troughton van het Agile Forest blog heeft reeds voorwerk gedaan in zijn serie artikelen Scrum Evolution over Time.

written

Enable SPDY for Your Erlang Website

Google’s Strange Loop presentation Making the Web Faster inspired me to enable SPDY on one of my side projects Zin In Sushi. This simple sushi restaurant review site is build in Erlang using Basho’s Webmachine in combination with Erlydtl.

Below is a quick writeup of the steps needed.

Assumptions:

  • Your Erlang powered website is running on port 3000 (Substitute this with your own magic value) on HTTP.
  • You are running a recent version of Ubuntu (I’m running 12.04 LTS).
  • Your domain is www.example.org.

The steps in summary:

  • Reverse proxy setup with Apache 2
  • SSL Setup with Apache 2
  • Installation of mod_spdy
  • $ sudo service apache restart, and you’re done!

Reverse proxy setup with Apache 2

First install Apache 2 if not already present:

1
$ sudo apt-get install apache2

Enable the proxy_http Apache module to get reverse proxy support:

1
$ sudo a2enmod proxy_http

Add a new site to Apache’s available-sites

1
$ sudo vim /etc/apache2/sites-available/reverse_proxy

And past in the following contents (and replace the domain name and ports if needed):

1
2
3
4
5
6
7
8
<VirtualHost *:80>
     ServerName www.example.org

     ProxyPreserveHost On
     ProxyRequests Off
     ProxyPass / http://localhost:3000/
     ProxyPassReverse / http://localhost:3000/
</VirtualHost>

Enable the site by executing:

1
$ sudo a2ensite reverse_proxy

Next up; SSL support!

SSL Setup with Apache 2

Enable the ssl Apache module to enable SSL:

1
$ sudo a2enmod ssl

SSL certificates are not free, so in this article we generate a self signed SSL certificate to keep moving:

1
2
$ sudo mkdir /etc/apache2/ssl
$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/apache2/ssl/apache.key -out /etc/apache2/ssl/apache.crt

Important! When generating the key, make sure the FQDN matches your domain name:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
——–
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New York
Locality Name (eg, city) []:NYC
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Awesome Inc
Organizational Unit Name (eg, section) []:Dept of Merriment
Common Name (e.g. server FQDN or YOUR name) []:example.org
Email Address []:webmaster@example.org

After generating the required encryption keys, we re-edit the reverse_proxy site:

1
$ sudo vim /etc/apache2/sites-available/reverse_proxy

And append the following configuration code:

1
2
3
4
5
6
7
8
9
10
11
12
13
<VirtualHost *:443>
     ServerName www.example.org

     SSLEngine on
     SSLCertificateFile /etc/apache2/ssl/apache.crt
     SSLCertificateKeyFile /etc/apache2/ssl/apache.key

     SSLProxyEngine on
     ProxyPreserveHost On
     ProxyRequests Off
     ProxyPass / http://localhost:3000/
     ProxyPassReverse / http://localhost:3000/
</VirtualHost>

Important! We need to check if Apache is listening on port 443 (SSL) and virtualhost support is enabled for this. /etc/apache2/ports.conf should have the following lines enabled:

1
2
3
4
<IfModule mod_ssl.c>
    NameVirtualHost *:443
    Listen 443
</IfModule>

Now we’re in a good shape to add SPDY support

Installation of mod_spdy

You need to fetch the latest mod_spdy package from Google:

Check the architecture of your system by running uname -m.

Install the package by running the following commands:

1
2
3
$ sudo dpkg -i mod-spdy-*.deb
$ sudo apt-get -f install
$ sudo a2enmod spdy

Le moment suprême

Restart Apache to power up the new configuration:

1
$ sudo service apache2 restart

And browse to your website https://www.example.org in Google Chrome.

To verify mod_spdy is working, open chrome://net-internals/#spd just after the first page load. Your server’s domain should be listed in the table under the SPDY sessions heading.gs

written

[Dutch] Geautomatiseerd Testen Met Erlang/OTP en Travis-CI - Een Introductie.

Erlang/OTP is ontworpen voor het bouwen van grote, schaalbare, soft-realtime systemen met een hoge beschikbaarheid. Het testen van dergelijke systemen is niet eenvoudig, laat staan geautomatiseerd testen. Voor Erlang zijn er dan ook geavanceerde automatische test methoden beschikbaar.

De drie belangrijkste methoden worden hier kort besproken aan de hand van een test project. De methoden zijn:

Je cloned het project van Github met het volgende commando:

1
$ git clone git@github.com:wardbekker/ci_quickstart.git

Voor het compileren van het project en uitvoeren van de testen gebruik je Rebar, een sophisticated build-tool for Erlang projects that follows OTP principles. Je bouwt Rebar als volgt:

1
2
3
4
5
6
7
8
9
10
$ git clone git://github.com/basho/rebar.git
$ cd rebar
$ ./bootstrap
Recompile: src/getopt
Recompile: src/rebar_utils
==> rebar (compile)
Congratulations! You now have a self-contained script called "rebar" in
your current working directory. Place this script anywhere in your path
and you can use rebar to build OTP-compliant apps.

Unit testing met EUnit

Je begint met de eenvoudigste test methode; EUnit. Dit is een unit testing bibliotheek voor Erlang. In een unit test controleer je of de functie goed werkt bij bekende input en resultaat. In dit voorbeeld heb je de functie addition geimplementeerd in de module ci_quickstart_math en twee assertions. (Je voert deze test uit op de commandline met: rebar get-deps compile eunit):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
module(ci_quickstart_math).
export([addition/2]).

ifdef(TEST).
include_lib("eunit/include/eunit.hrl").
endif.

addition(X, Y) –>
    X + Y.

ifdef(TEST).

simple_test() –>
    ?assertEqual(4, addition(2,2)),
    ?assertNotEqual(3, addition(1,1)).

endif.

Het slechte nieuws is dat de waarde van deze test zeer laag is. Weet je nu zeker dat optelling goed gaat in alle gevallen? Het enige wat de test nu aantoont is dat:

  • addition(2,2) == 4
  • addition(1,1) /= 3

Stel, je verandert de implementatie van de function addition in:

1
2
 addition(X, Y) –>
    4.

De testen slagen in dit geval, maar dit betekend niet dat de implementatie van addition correct is.

Sterker nog; De argumenten zijn in dit geval 64-bit small integers, en die hebben een bereik van -576460752303423489 t/m 576460752303423488. Met twee argumenten, betekend dit dat er enorm veel verschillende input mogelijk is. En in de unit test controleer je er maar 3!?!? Ook al ben je een harde werker en test je wel 10! addities, in feite is de waarde van de unit test niet verbeterd en nog steeds erg laag.

Wat nu?

QuickCheck

Wat je eigenlijk wil is een test methode dat alle mogelijke input variaties genereerd en de bijbehorende output controleert. Deze methode heet QuickCheck. Voor Erlang zijn er een aantal QuickCheck frameworks beschikbaar:

Een Quickcheck test voor de addition functie:

1
2
3
4
5
6
prop_sum() –>
    ?FORALL(
        {X, Y},
        {int(), int()},
        addition(X,Y)  Y == X
    ).

Voer de test uit door de Erlang Shell op te starten met ./shell.sh en de volgende functie aanroep proper:quickcheck(ci_quickstart_math:prop_sum()).

Specifieke nummers worden niet getest. Je gaat nu controleren of de functie voldoet aan de eigenschap dat als je Y weer er afhaalt, je X overhoud.

{int(), int()} genereerd tuples met twee random integers. De tuple wordt gebonden aan {X, Y} door pattern matching. Standaard worden er 100 combinaties getest, en dit aantal voer je op met de numtests optie: proper:quickcheck(ci_quickstart_math:prop_sum(),[{numtests,10000}])..

De uitdaging bij het werken met QuickCheck is het bedenken van de eigenschappen van de functie. Dit is lastiger dan het maken van een unit test. Sterker nog, het schrijven van de functie is vaak nog eenvoudiger dan het redeneren over de eigenschappen. Het positieve effect van QuickCheck op de kwaliteit van je code, en de manier waarop je als developer over je code nadenkt maakt deze tool een zeer waardevol onderdeel van je test gereedschapskist.

Common Test

Zoals bekend is Erlang uitermate geschikt voor het bouwen van concurrent, distributed en fault tolerant systemen. Om te controleren of je systeem werkt zoals verwacht, is complex.

Hiervoor is Common Test in het leven geroepen. Dit krachtige test framework is uitermate geschikt voor de ontwikkeling van pittige systeem tests. De inherente complexiteit van concurrent, distributed en fault tolerant systemen maakt Common Test complex. Hoe je een serieuze OTP applicatie op de pijnbank legt met CT valt derhalve buiten de scope van deze blogpost. Hier onder wel een minimaal Comment Test waarin de EUnit testen worden nagebootst door gebruik van pattern matching.

1
2
3
4
5
6
7
8
9
10
11
12
13
module(basic_SUITE).
include_lib("common_test/include/ct.hrl").

export([all/0]).
export([test1/1, test2/1]).

all() –> [test1,test2].

test1(Config) –>
    3 = ci_quickstart_math:addition(1,2). %% validated using pattern matching

test2(Config) –>
    2 = ci_quickstart_math:addition(1,1).  %% validated using pattern matching

Continuous integration met Travis-CI

Stel, je hebt een flinke hoeveelheid Eunit, Common Test en Quickcheck testen geïmplementeerd. Het uitvoeren van alle geavanceerde testen duurt lang en belast je systeem fors. Om deze, en nog meer goede redenen, is Continuous integration aan te raden.

Er zijn legio systemen waarmee het mogelijk is om continuous integration voor Erlang op te zetten. In dit voorbeeld gebruik je het hosted systeem Travis-CI. Deze dienst ondersteunt Erlang, integreert met het populaire Github en zorgt voor een vliegende start. En het is gratis voor open source projecten.

Voorbereiding

Het build proces van Travis-CI configureer je via het .travis.yml-bestand in de root van je repository. Een voorbeeld:

1
2
3
4
5
6
7
language: erlang // De repository bevat een Erlang project
notifications:
  email: you@example.org // Build success en failures stuurt Travis-CI naar dit adres.
otp_release: // Travis-CI test/bouwt je project voor meerdere Erlang/OTP versies.
  – R15B01
  – R15B
  – R14B04

Travis-CI Setup

Deze video toont hoe je start met Travis-CI:

  • Log in met je Github account.
  • Ga naar de Travis-CI profile pagina.
  • Schakel de gewenste Github repository in.

That’s it!

Setup

Travis-CI Success Run

Deze video toont hoe Travis-CI een geslaagde integration build rapporteerd:

Success

Travis-CI Failure Run

Deze video toont hoe Travis-CI een mislukte integration build rapporteerd:

Fail

Als je e-mail adres in .travis.yml staat, krijg je ook een e-mail notificatie dat de laatste commit de build gebroken heeft:

Broken build e-mail notification

Als de fout verholpen is, krijg je de volgende e-mail als de build weer slaagt:

Fixed build e-mail notification

written in Erlang

Erlang Fprof Output Confusing? Try KCachegrind.

Your Erlang code is perfect, but to find out why other peoples code runs dog slow you probably profile the code with fprof like this:

1
2
3
(node@host)1> fprof:apply(module, function, [arguments]).
(node@host)1> fprof:profile().
(node@host)1> fprof:analyse({dest, “outfile.fprof”}).

The printout of fprof analyse is a text dump of the result, which can grow over 1000 lines and contains a lot of noise which makes it hard to locate the bottlenecks. Below a truncated sample of an actual fprof trace.

KCachegrind

KCachegrind to the rescue! With this tool you can visually inspect the fprof analyse result with sorting, a fancy call graph view, callee map and more.

images

As KCachegrind can’t read fprof analysis output directly, you need to convert it first to the callgrind format with the Erlgrind script by Isac Sacchi e Souza.

1
$ ./erlgrind outfile.fprof callgrind.001

KCachegrind & Erlgrind Installation

For installation of KCachegrind on my Mac I use Homebrew, a package manager for OSX. Notice that you install qcachegrind, the QT version of KCachegrind.

1
2
3
$ brew install qcachegrind
$ brew instal graphviz
$ sudo ln -s /usr/local/bin/dot /usr/bin/dot

Installing the Erlgrind (Github) script:

1
2
3
$ curl -O "https://raw.github.com/isacssouza/erlgrind/master/src/erlgrind&quot;
$ chmod +x erlgrind
$ mv erlgrind /usr/local/bin/

And open qcachegrind:

1
$ open ~/Applications/qcachegrind.app

Enjoy!

written in Erlang

Inverted Index Compression Techniques

Today I presented at at the Dutch Erlang Usergroup meeting, hosted by Spilgames.

In this presentation I explained:

  • How to build a inverted index that supports boolean AND queries
  • Elias Ɣ (gamma) encoding and Delta gap compression to reduce the size of inverted indexes

The Slides

written

A Basic Full Text Search Server in Erlang

In the beginner’s mind there are many possibilities, but in the expert’s there are few

Shunryu Suzuki Zen Mind Beginner’s Mind

This post explains how to build a basic full text search server in Erlang. The server has the following features:

  • indexing
  • stemming
  • ranking
  • faceting
  • asynchronous search results
  • web frontend using websockets

Familiarity with the OTP design principles is recommended.

The sample application (build with help from my colleague Michel Rijnders mies@tty.nl) uses the Creative Commons Data Dump from StackExchange as demo data.

We cover the following subjects:

Running the Sample Application

Clone the source from GitHub:

 git clone git://github.com/tty/async_search.git

And start the application:

$ rebar get-deps compile && erl -pa `pwd`/ebin `pwd`/deps/*/ebin +P 134217727
Eshell> application:start(async).
Eshell> stackoverflow_importer_ser:import().

Visit http://localhost:3000, you should see the following page:

http://localhost:3000/

Sample ranked search output for the query erlang armstrong:

http://localhost:3000/

Sample tags facets output for the query java:

http://localhost:3000/

OTP Supervision Tree

supervisor tree

Looking at the OTP application supervision tree is a good way to understand the architecture of an OTP application.

The application supervisor async_sup starts up the following supervisors:

  • keyword_sup. A keyword_ser process is created for every unique word in the StackExchange posts. This keyword_ser is linked to the keyword_sup supervisor (a simple_one_for_one supervisor). The keyword_ser child process maintains a list of document positions of a keyword (an inverted index).
  • facet_sup. A keyword_ser process is also created for every unique facet category in the StackExchange posts. This keyword_ser process is linked to the facet_sup supervisor (a simple_one_for_one supervisor as well). The keyword_ser child process maintains a list of facet values with the IDs of the documents the facets appear in.

The application supervisor also start the following gen_server singleton processes:

  • stackoverflow_importer_ser. This server imports the demo Stack Overflow data.
  • document_ser. This server holds a copy of all documents, so it can return the original title and body of matching Stack Overflow posts in the results.
  • query_ser. This server’s task is to run the actual query and return results.
  • websocket_ser. This server provides a HTTP frontend for the search engine.

No attention is given to fault tolerance (apart from the basic restart strategies), thus parts of the search index are lost if a keyword_ser process terminates.

Demo Data Import

The StackExchange data is provided as XML. Since some of the documents are quite large, it’s not recommended to load the full XML documents in memory. The solution is to use a SAX parser which treats a XML file as a stream, and triggers events when new elements are discovered. The search server uses the excellent SAX parser from the Erlsom library by Willem de Jong.

In the example below erlsom:parse_sax reads the XML file from FilePath and calls the function sax_event if an XML element is found.

When the element is a row element (i.e. a post element), attributes like Id, Title and Body are stored in a dictionary. For every post a copy of all the attributes in document_ser is saved. This is used for returning the actual posts for a query match. After that the add_attribute_tokens function is called:

The add_attribute_tokens function does two things. It calls add_facet (discussed later) and it creates a list of tuples with all the words and their position in the document. This process is called tokenization. Each token/position tuple is then submitted to the add_keyword_position function of the keyword_ser for indexing.

Indexing

Indexing of the tuples, or keywords, is handled by the keyword_ser. For every unique word a keyword_ser process is started if not already present. The state of a keyword_ser process is a dictionary with the document ID as key and a list of positions as value. The document ID corresponds to the ID of the Stack Overflow post.

The keyword_server_name function generates a unique name under which the keyword_ser process is registered, so the module can check if a keyword already has a process or a new process needs to be created.

Stemming

Stemming is the process for reducing inflected words to their base form. Computing and computer both are stemmed to comput. So when a user searches on computing, it also matches text that contains computer. This makes it possible to return results that are relevant, but do not exactly match the query.

In our sample application all keywords are stemmed using the popular Porter Algorithm. The Erlang implementation by Alden Dima is used in the application.

erlang:phash2 is used to transform the stemmed name to a hash, to make sure the registered process name is valid.

Faceting

Faceted search is an important navigation feature for search engines. A user can drill down the search results by filtering on pre-defined attributes, like in this example of a digital camera search on CNET:

Faceted search example

As mentioned above, the data import the function add_attribute_tokens also calls the add_facet function. Using pattern matching the Tags and the Creationdate attributes are selected for faceting. Tags is a so called multivalue facet, as a Stack Overflow post can have one or more tags assigned. For every tag and creation date the facet_ser:add_facet_value function is called.

facet_ser works very similar to keyword_ser. For every facet category, Tag or Creationdate in our case, a facet_ser processes is started. The state of a facet_ser is a dictionary with the Tag or Creationdate values as key and their document IDs as dictionary values.

Querying and Relevance Ranking

In previous sections is shown:

  • how the XML demo data is parsed.
  • how this data is stemmed and indexed by creating a keyword_ser process for every unique keyword.
  • how this data is indexed for faceted search by creating a facet_ser process for every facet category.

With the function stackoverflow_importer_ser:import() these steps are executed, and your Erlang node is now ready for querying. So how does that work?

Querying

Querying is handled by passing the user’s query terms to the function do_async_query of the singleton query_ser server. When calling this function you need to specify the module, function and optional reference attribute which will be called when query results are available.

In the handle_cast the following steps are executed:

  • keyword_ser:do_query return all document ids that contain one or more of the user’s query terms, including the relevance ranking score, which will be discussed below.
  • All original documents are stored during indexing in a document_ser process. All matching documents are collected.
  • The callback function is invoked with the matching documents and their ranking scores as arguments.
  • Facet results are retrieved for any FacetCategories that are specified by calling facet_ser:get_facets.
  • And the callback function is invoked a second time with the facet results as arguments.

Relevance Ranking

Relevance in this context denotes how well a retrieved document matches the user’s search query. Most fulltext search-engines use the BM25 algorithm to determine the ranking score of each document, so let’s use that too.

BM25 calculates a ranking score based on the query term frequency in each documents.

See the async_bm25.erl for the implementation.

Displaying the Search Results

As discussed, the query_ser:do_async_query can be called to query our full-text search engine. To allow users to send queries and see the result the websocket_ser module is created. This singleton gen_serverstarts up a Misultin HTTP server on Port 3000. If you browse to http://localhost:3000 you will see a search box. Communication with the search engine is done through websockets.

So, when a user posts a query, this message is received by the websockets_ser:handle_websocket receive block. The query_ser:do_async_query function is called and query results are expected on websockets_ser:query_results function.

The query_results function formats the results as HTML and sends this through the websocket. When received, the HTML is appended to the user’s page.

A similar process is executed when the facet results are received:

Improvements

Some obvious features that are lacking from this sample application:

  • The author of this post is an Erlang newbie. Corrections/suggestions to the code are most welcome. You can send them to <ward@tty.nl>
  • Pretty much no attention is given to performance / memory usage.
  • Fault tolerence for the index data. When a server containing index state dies, it will not be revived.
  • Tuple structures passed between modules are not specified. Would be nice to use record syntax for it.
  • No unit/quickcheck/common test added.
  • No function/type specifications.
  • etc..

So, that why it’s called a sample application ;-)

written in Erlang