Skip to content

Instantly share code, notes, and snippets.

@3rdcycle

3rdcycle/gist:2e01679876f92ecd0855

Last active August 29, 2015 14:06
Show Gist options
  • Save 3rdcycle/2e01679876f92ecd0855 to your computer and use it in GitHub Desktop.
Save 3rdcycle/2e01679876f92ecd0855 to your computer and use it in GitHub Desktop.
Download ZIP
Installing OpenQuake on Mac OS X 10.9
Raw
gistfile1.md

OpenQuake on Mac OS X 10.9

This HowTo describes how to install OpenQuake on Mac OS X 10.9 from source.

Note that even though the instructions below seem to work, the result has not been thoroughly tested. OS X is not officially supported by GEM. Unless you have a very good reason to run OpenQuake on OS X it is recommended that you stick to the official platform which is Ubuntu 12.04.

Also, it's fairly involved. You've been warned ;-)

Caveats

  • We don't build the geoutils speedups
  • We don't build support for distributed computing. You'll always have to run oq-engine with --no-distribute.

Prerequisites

  • The latest Xcode with command line tools installed

  • Homebrew. This HowTo assumes that homebrew is setup to install everything in /usr/local (the default)

  • Brewed versions of

    • python (2.7)
    • gcc (4.9)
    • lzlib (1.6)

    There's no need to explicitly specify versions here, since these are the latest and homebrew installs them by default as of this writing.

  • Virtualenv (pip install virtualenv)

  • No previous postgresql installation (we'll install our own)

  • It's recommended not to have any other 3rd party packages (e.g. from Macports, fink) since those could lead to conflicts.

Install geos

Openquake seems to use geos 3.2.2 in their VM appliances. This version has a known bug though, and won't compile on OS X. The newest releases on the other hand are incompatible with Django (< 1.5) which we install later. Version 3.3.2 is one that seems to work, but since brew has dropped the 'versions' command and since old geos libraries cause all sorts of troublew with unrelated packages such as GMT we're going to install the latest anyway. This means that we'll have to hack Django 1.3 later. For now just

brew install geos

Install and setup postresql and postgis

Openquake needs postgis 1.5 and postgresql 9.1. The default dependency for postgis 1.5 is postgresql 9.0 however, so we'll have to tweak postgis' dependencies to install 9.1 instead.

Postgis 1.5 is available (as postgis15 from homebrew/versions). To gain access to to homebrew/versions type:

brew tap homebrew/versions

Unfortunately, as of this writing the postgis15 formula does not install everything we need. A pull request has been submitted to fix this but you will have to apply these changes manually until it's accepted.

As mentioned above, we also need to change the postgresql dependency. To do that brew edit postgis15 and change all occurences (3) of postgresql9 to postgresql91.

Once that's taken care of, we can install postgis together with postgresql:

brew install postgis15 --with-comments

Next, we need to init the database.

initdb /usr/local/var/postgres

To make postgresql play nicely with the Django version we're going to install (1.3) we need to change a setting in /usr/local/var/postgres/postgresql.conf. Open the file with a text editor, uncomment the standard_conforming_strings line and set it to off.

Side note: according to GEM's technology stack page openquake should be compatible with Django 1.4 which would resolve the issue above. I couldn't get Django 1.4 to work with openquake though.

Finally we need to launch postgresql and make sure it auto-starts on reboots:

ln -sfv /usr/local/opt/postgresql91/homebrew.mxcl.postgresql91.plist ~/Library/LaunchAgents
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.postgresql91.plist

The first of these commands might fail with "FATAL: could not create shared memory segment". In that case you need to increase the maximum allowed amount of shared memory. Type the following two commands

sudo sysctl -w kern.sysv.shmall=65536
sudo sysctl -w kern.sysv.shmmax=16777216

To make those changes persistent across reboots, add them to /etc/sysctl.conf, like this:

kern.sysv.shmall=65536
kern.sysv.shmmax=16777216

Check if everything works by logging in

psql postgres <your username>

Create a virtualenv for openquake

A virtualenv allows us to keep all the (very) specific dependencies for openquake nicely separated from our regular python environment. Switch to a suitable directory (I use ~/.virtualenv) and type

virtualenv openquake

Now activate the environment with

source openquake/bin/activate

Install python dependencies

Most of openquakes python dependencies can be installed in one go (see below). Two are special however and need to be taken care of first.

  1. numpy needs to be there before we install scipy. This one is straight forward:

     pip install numpy==1.6.1

  2. lxml provides python bindings for libxml2. This one is more difficult as it won't work with the native libxml2 that comes with OS X. First get the source:

     git clone https://github.com/lxml/lxml.git

    We're also gonna need Cython to build lxml:

     pip install cython

    Then cd to the lxml directory an run

     CFLAGS="-I/usr/local/include/lzma" python setup.py build --static-deps --libxml2-version=2.8.0

    followed by

     python setup.py install

Now we're ready for the remaining dependencies. Copy the lines below into a requirements.txt file:

Django==1.3.1
Shapely==1.2.15
amqplib==1.0.0
anyjson==0.3.1
celery==2.4.6
futures==2.1.6
kombu==1.4.3
lxml==3.4.0
numpy==1.6.1
psutil==0.4.1
psycopg2==2.4.5
python-dateutil==1.5
pytz==2011k
scipy==0.14.0
wsgiref==0.1.2

and execute

pip install -r requirements.txt

Fix Django

This fix was first included in Django 1.5. Since OpenQuake requires 1.3 for now, we're going to have to apply the fix manually. It will enable Django to correctly parse the geos version string.

Head over to GitHub and apply the changes in this commit.

Install OpenQuake

Clone the main OpenQuake source code packages:

git clone https://github.com/gem/oq-engine.git
git clone https://github.com/gem/oq-hazardlib.git
git clone https://github.com/gem/oq-risklib.git
git clone https://github.com/gem/oq-commonlib.git

Go into the commonlib and install it with

python setup.py install

Since you're installing from source, you probably want to work on the code later. In that case it might be more convenient do to do a develop installation which links to the cloned repo instead of moving everything to site-packages. In the case of oq-engine, install won't copy over all the necessary files anyway, so you'll either have to do a develop install or copy the missing files manually (e.g. openquake.cfg).

We'll develop install the remaining packages with

python setup.py develop

Setup OpenQuake

Before you run the OpenQuake engine for the first time you might want to adjust some settings in openquake.cfg (I didn't, the defaults should be ok).

The last step is preparing the openquake database. The engine directory contains a script under /openquake/engine/bin/oq_create_db which does this. Since the script was written for ubuntu we need to make some adjustments first:

  1. Replace the table space path in tspace_path with /usr/local/var/postgres/pg_tblspc
  2. Replace the db_admin_user with your local user name
  3. At the bottom of the script correct the paths for the postgis sql files. Here's everything that needs to be changed (in git diff format):
@@ -172,13 +172,14 @@ fi
 echo ".. Creating database $db_name .."
 psql -p $db_port -d postgres -U $db_admin_user -c "CREATE DATABASE $db_name OWNER oq_admin" $psql_batch_options

-postgres_contrib="/usr/share/postgresql/9.1/contrib"
-postgis="$postgres_contrib/postgis-1.5"
+#postgres_contrib="/usr/share/postgresql/9.1/contrib"
+#postgis="$postgres_contrib/postgis-1.5"
+postgis="/usr/local/Cellar/postgis15/1.5.8/share/postgis"
 # Load the PostGIS stuff into the newly created OpenQuake database, must be run as postgres
 echo ".. Loading postgis functions/data into $db_name .."
 psql -p $db_port -U $db_admin_user $psql_batch_options -d $db_name -f $postgis/postgis.sql
 psql -p $db_port -U $db_admin_user $psql_batch_options -d $db_name -f $postgis/spatial_ref_sys.sql
-psql -p $db_port -U $db_admin_user $psql_batch_options -d $db_name -f $postgres_contrib/postgis_comments.sql
+psql -p $db_port -U $db_admin_user $psql_batch_options -d $db_name -f $postgis/postgis_comments.sql

# fix ownership of three public tables from postgres to oq_admin
psql -p $db_port -U $db_admin_user $psql_batch_options -d $db_name -c "ALTER TABLE spatial_ref_sys OWNER TO oq_admin"

Now execute the script. It will complain about an "illegal username" but you can ignore that.

Run OpenQuake

First we need to upgrade the db

oq-engine --upgrade-db

Next, try to run one of the demos (from the engine directory)

oq-engine --run-hazard=./demos/hazard/SimpleFaultSourceClassicalPSHA/job.ini --no-distribute

This might take a few minutes to finish, but should run without errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment