NBIC Galaxy Server: Maintenance Guideline

From BioAssist
Revision as of 16:22, 5 September 2011 by David.van.enckevort (Talk | contribs)

Jump to: navigation, search

Schedule

For the Galaxy server, a monthly maintenance is scheduled on first Monday afternoon of each month. Prior to each maintenance, the collected update requests from previous month are discussed within the maintenance team in the form of email or skype call. If a decision is made for updating the Galaxy server and pipelines, or adding tools and libraries, the actual update will be executed by a dedicate team member.

We also schedule a bi-yearly maintenance on tools and libraries. This is scheduled on each January and July. During this maintenance, we will freeze the current on line Galaxy server into an online VM. Then we will perform tool and libraries update on the online version @ Vancis.

There will be 3 groups of NBIC Galaxy servers:

  1. current version @Vancis on line
  2. VM of 2 previous versions@cheaper providers on line. (tentative)
  3. downloadable VM of all previous versions

Responsibility

  • Pieter monitors the release notes and bug reports of Proteomics tools and Galaxy
  • Leon monitors the release notes and bug reports of NGS tools
  • Jeroen coordinates the maintenance task and performs the update
  • David maintains the virtual machine package and assists Jeroen

Galaxy Server

General Policy

Galaxy at Penn State is updated very frequently. However we will not push ourselves to follow their pace. We will only update the server if there are:

  • New features or bug fixes we want to have on the NBIC Galaxy server

or

  • Security patches

Install/Update Instructions

To install a new version of Galaxy from Penn State, here are the steps we should follow as a maintenance team member. Jeroen is planning to automate the entire procedure.

  1. lock the job submission at the NBIC Galaxy server.
  2. log into galaxy.nbic.nl with your account. You only need to use "galaxy" account to install all the following. Never use root! (except for stopping and re-starting services)
  3. go to the directory of /opt/galaxy/prog/sources/
  4. Fetch the source
    • Check out the latest code from Penn State: hg clone http://www.bx.psu.edu/hg/galaxy galaxy-<year>-<month>-<date>
    • This is just a copy for reference. Do not modify this copy!
  5. Copy this folder to /opt/galaxy/prog/. This will become the active installation with local mods.
    • cp /opt/galaxy/prog/sources/galaxy-<year>-<month>-<date> /opt/galaxy/prog/ -r
  6. Stop the Galaxy daemon and enable the maintenance page
    • (as root) open /etc/httpd/conf.d/galaxy.conf in a text editor and switch the comments: turn the lines for the normal galaxy config into comments and uncomment the alias that points to a page with a maintenance notification.
    • (as root) /sbin/service httpd reload
    • (as root) /etc/init.d/galaxy stop
  7. Update the Galaxy symbolic link
    • cd /opt/galaxy/prog/
    • rm galaxy
    • ln -s galaxy-<year>-<month>-<date>/ galaxy
  8. Configure the new galaxy: we keep the all the files (datasets) associated with histories stored in the MySQL database in a separate database folder outside the galaxy root folder.
    • cd /opt/galaxy/prog/galaxy/
    • rmdir database (<- Make sure this is the EMPTY database folder of the new install and not a symlink pointing to our data in /opt/galaxy/data/database/
    • ln -s /opt/galaxy/data/database/ database
  9. Test starting up the Galaxy server (and fetch/check eggs if necessary)
    • You need to first make sure the entire Galaxy directory is owned by the user "galaxy"
    • "sudo /etc/init.d/galaxy start" to test the start up of new server.
    • If you are getting conflicting eggs warnings, try to fetch and check the eggs in separate steps:
      • sudo -u nbic python ./scripts/fetch_eggs.py
      • sudo -u nbic python ./scripts/check_eggs.py
  10. update /opt/galaxy/prog/nbic_gmr with the latest content from galaxytools trunk at trac.nbic.nl
  11. Customize the new Galaxy: We keep copies of modified (config) files in the conerice repository. You should diff our customized versions in the the conerice repo with the new plain vanilla versions. When necessary merge updates from the Galaxy developers with our local customizations and commit a new version to the conerice repo. If there are no changes you can simply copy the files below:
    • cp conerice/masterfiles/galaxy/sombrero/static/welcome.html /opt/galaxy/prog/galaxy/static/
    • cp conerice/masterfiles/galaxy/sombrero/config/universe_wsgi.ini /opt/galaxy/prog/galaxy/
    • cp conerice/masterfiles/galaxy/sombrero/config/tool_conf.xml /opt/galaxy/prog/galaxy/
    • cp conerice/masterfiles/galaxy/sombrero/config/datatypes_conf.xml /opt/galaxy/prog/galaxy/
    • cp conerice/masterfiles/galaxy/sombrero/lib/galaxy/datatypes/registry.py /opt/galaxy/prog/galaxy/lib/galaxy/datatypes/
    • cp conerice/masterfiles/galaxy/sombrero/tools/data_source/upload.py /opt/galaxy/prog/galaxy/tools/data_source/
  12. Create symlinks for our tools and datatypes from the NBIC Galaxy Module Repository
    • cd /opt/galaxy/prog/galaxy/tools
    • ln -s /opt/galaxy/prog/nbic_gmr nbic_gmr
    • cd /opt/galaxy/prog/galaxy/static/images
    • ln -s /opt/galaxy/prog/nbic_gmr/data/image-links/ nbic_gmr
    • cd /opt/galaxy/prog/galaxy/lib/galaxy/datatypes/
    • for item in $(ls /opt/galaxy/prog/nbic_gmr/data/datatypes/); do ln -s /opt/galaxy/prog/nbic_gmr/data/datatypes/$item; done
  13. Create symlinks for custom tool-data files in /nbic/prog/galaxy/tool-data/
    • cd /opt/galaxy/prog/galaxy/tool-data/
    • for item in $(ls /opt/galaxy/data/tool-data/); do ln -s /opt/galaxy/data/tool-data/$item; done
  14. Make sure you have the most recent version of the init.d script and that Galaxy is configured for automatic start and stop when switching to the various run levels:
    • (as root) cp conerice/masterfiles/galaxy/sombrero/init.d/galaxy /etc/init.d/
    • (as root) chkconfig --add galaxy
  15. Restart the Galaxy daemon. (To do this, you need to be root, but the "runuser" command makes sure that the Galaxy instance will be run as user "nbic")
    • (as root) service galaxy start (if this fails, see below)
  16. Check the log file /opt/galaxy/prog/galaxy/paster.log to see if Galaxy started (without errors). Sometimes, you are required to upgrade the database schema. If this is the case, Galaxy won't start and you receive a message to run manage_db.sh. Backup the galaxy DB before you run the upgrade script.
  17. If everything seems fine, edit /etc/httpd/conf.d/galaxy.conf back to production mode and reload the webserver config.
    • (as root) open /etc/httpd/conf.d/galaxy.conf in a text editor and switch the comments: remove comments for the normal galaxy config and turn the alias that points to a maintenance notification into a comment.
    • (as root) service httpd reload

Tools/libraries

On each distribution of the NBIC Galaxy server (e.g. online version or downloadable VMs), only one version of a tool/library exists. To guarantee a certain life span of each tool (which could be useful for users if they want to replicate their data analysis), we will support each version of every tool and library for at least 1 year. Since we plan updates the tools twice a year, we will clone the current online server into a downloadable VM before the update and make it available as the previous galaxy. The main VM can then be updated. When we additionally archive all the older VMs, an analysis can always be repeated.

A list of tools installed at the NBIC Galaxy server can be found at NBIC Galaxy Server: Current Tools

Linux tools/libs from a distro

If tools or libs are available from a distro repository these should be installed as root using a suitable package maneger depending on the repo. Currently we use RedHat/CentOS/Fedora flavored Linux OSses so you can use yum:

  • sudo su -
  • yum search [tool/lib]
  • yum install [package]

When you need to add or modify config files for the package you've just installed, please make sure to add a copy or commit your changes to the conerice repo.

Linux tools/libs not available from a distro repo and 3rd Party software

Software should never be installed in default paths, but in /usr/local/ instead. If you need a newer version of some package, don't overwrite the version installed as part of a distro! Before you start compiling/installing stuff, make sure to source /opt/galaxy/prog/nbic_compile_env.bashrc to setup additional environment variables.

Perl modules available from CPAN

Perl specific: additional Perl modules go into /usr/local/perl_plus/. If a module is available from CPAN you can install it as follows (using your own account; hence never as root):

  1. Start the CPAN shell. If this is the first time you use it you'll have to answer a bunch of questions to configure CPAN. The defaults should be mostly fine.
    • perl -MCPAN -eshell
  2. Change the location where new Perl modules will be installed:
    • o conf makepl_arg "PREFIX=/usr/local/perl_plus/"
    • o conf mbuildpl_arg "--install_base /usr/local/perl_plus/"
  3. To store the settings above, so you won't have to type them again next time:
    • o conf commit
  4. Try to install your module of interest:
    • install MY::Perl::Module

Python eggs available as tar.gz

You can try to install these with easy_install:

  1. Save the downloaded archive to /opt/galaxy/prog/sources/python/
  2. run easy_install
    • easy_install --prefix /usr/local/python_plus/ /nbic/prog/sources/python/downloaded-python-egg.tar.gz

Everything else

When you need to install something from source, please keep a copy of the source in /opt/galaxy/prog/sources/ for future reference. Perl modules go into /usr/local/perl_plus/, Python eggs into /usr/local/python_plus/ and everything else either in /usr/local/ or in /usr/local/some_package_i_installed/ in case of complicated installs you prefer to keep in a separate sub dir.

For All

If necessary update the environment by modifying the /opt/galaxy/prog/nbic_*_env.bashrc scripts and commit changes to these files to the conerice repo.

NBIC Tools

NBIC Galaxy server presents a number of tools developed tools by NBIC members. These tools should be of interest for a large number of users. We cannot support fancy tools that only address one's individual problem unless the developer of that tool joins the maintenance crew.

Installation

At the moment, we request developers to upload their tools to https://trac.nbic.nl/galaxytools/ and provide a README file to explain its installation.

For admins, here are the steps to add tool in the NBIC Galaxy server:

  1. Log into galaxy.nbic.nl with your account. You need to use galaxy user account (sudo su galaxy) to do all the following. Never use root!
  2. Go to the directory of /opt/galaxy/prog/nbic_gmr and check out the requested tools.
  3. Update "tool_conf.xml" and optionally "datatypes_conf.xml" accordingly.
  4. Commit the modifications to these *_conf.xml files to the conerice repo.
  5. Check the README at the root of the galaxytools repo and optionally for a README in the subdir for a tool to check if additional mods are required.

Data Files

Installation Location

  • Reference genomes data files should be installed at: /opt/galaxy/data/Genomes/

Linux tips

Creating new user accounts

To keep the server safe it is important to have a policy on the user accounts.

  • Create a new user with the following command (replace group1,group2 and user with the correct values of course, if no special group memberships are necessary, you can omit the -G ...)
sudo useradd -G sudo -m -U -b /home -s /bin/bash -c 'Full name of the user' username
  • Always use a strong password on user accounts (8 or more characters and use letters, numbers and specials). Set the initial password with
sudo passwd user
  • When you create an account and have to e-mail the password to the user enforce a password change on first use with the following command:
sudo chage -d 0 user
  • When you are giving temporary access to someone set an expiration date for the account with the following command:
sudo chage -E YYYY-MM-DD user
  • make sure the user is member of the right groups. Users who need to have access to galaxy need to be in the devs group, but only give group membership if it is really necessary. Anyone who is member of the devs group can potentially break stuff...
  • To change the group memberships later on you can use the following command:
sudo usermod -a -G devs user

Problem with missing eggs

You may sometimes get error messages on "Removed conflicting egg:" during the startup. This might be caused by the incorrect access rights. A quick and dirty solution is to "sudo chmod a+rwx galaxy -R". This way you are sure "nbic" can access all the necessary files.

Another complain might come from a missing decorator module (as the example below). Somehow this is not automatically resolved by galaxy during the startup. So the solution is that you need to download and add that egg manually to the "eggs" directory, e.g. "/nbic/prog/galaxy-2010-11-14/eggs/decorator-3.1.2-py2.6.egg".

[nbic@app-prod-vh02 galaxy-2010-11-14]$ sh run.sh 
Some eggs are out of date, attempting to fetch...
Warning: Beaker (a dependant egg of Mako) cannot be fetched
Fetched http://eggs.g2.bx.psu.edu/new/Tempita/Tempita-0.1-py2.6.egg
Removed conflicting egg: /nbic/prog/galaxy/eggs/Tempita-0.1-py2.6.egg
Fetched http://eggs.g2.bx.psu.edu/new/lrucache/lrucache-0.2-py2.6.egg
Removed conflicting egg: /nbic/prog/galaxy/eggs/lrucache-0.2-py2.6.egg
Warning: decorator (a dependant egg of sqlalchemy-migrate) cannot be fetched
Fetched http://eggs.g2.bx.psu.edu/new/NoseHTML/NoseHTML-0.3.1-py2.6.egg
Removed conflicting egg: /nbic/prog/galaxy/eggs/NoseHTML-0.3.1-py2.6.egg
Fetched http://eggs.g2.bx.psu.edu/new/numpy/numpy-1.3.0-py2.6-linux-x86_64-ucs4.egg
Removed conflicting egg: /nbic/prog/galaxy/eggs/numpy-1.3.0-py2.6-linux-x86_64-ucs4.egg
Fetched http://eggs.g2.bx.psu.edu/new/Beaker/Beaker-1.4-py2.6.egg
Removed conflicting egg: /nbic/prog/galaxy/eggs/Beaker-1.4-py2.6.egg
Fetched http://eggs.g2.bx.psu.edu/new/SVGFig/SVGFig-1.1.6-py2.6.egg
Removed conflicting egg: /nbic/prog/galaxy/eggs/SVGFig-1.1.6-py2.6.egg
Fetched http://eggs.g2.bx.psu.edu/new/WebError/WebError-0.8a-py2.6.egg
Removed conflicting egg: /nbic/prog/galaxy/eggs/WebError-0.8a-py2.6.egg
Fetched http://eggs.g2.bx.psu.edu/new/MySQL_python/MySQL_python-1.2.3c1_5.1.41_static-py2.6-linux-x86_64-ucs4.egg
Removed conflicting egg: /nbic/prog/galaxy/eggs/MySQL_python-1.2.3c1_5.1.41_static-py2.6-linux-x86_64-ucs4.egg
Fetched http://eggs.g2.bx.psu.edu/new/twill/twill-0.9-py2.6.egg
Removed conflicting egg: /nbic/prog/galaxy/eggs/twill-0.9-py2.6.egg
Fetched http://eggs.g2.bx.psu.edu/new/decorator/decorator-3.1.2-py2.6.egg
Removed conflicting egg: /nbic/prog/galaxy/eggs/decorator-3.1.2-py2.6.egg
Fetched http://eggs.g2.bx.psu.edu/new/GeneTrack/GeneTrack-2.0.0_beta_1_dev_48da9e998f0caf01c5be731e926f4b0481f658f0-py2.6.egg
Removed conflicting egg: /nbic/prog/galaxy/eggs/GeneTrack-2.0.0_beta_1_dev_48da9e998f0caf01c5be731e926f4b0481f658f0-py2.6.egg
Fetched http://eggs.g2.bx.psu.edu/new/nose/nose-0.11.1-py2.6.egg
Removed conflicting egg: /nbic/prog/galaxy/eggs/nose-0.11.1-py2.6.egg
Fetch successful.
python path is: /nbic/prog/galaxy-2010-11-14/eggs/Whoosh-0.3.18-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/pycrypto-2.0.1-py2.6-linux-x86_64-ucs4.egg, /nbic/prog/galaxy-2010-11-14/eggs/python_lzo-1.08_2.03_static-py2.6-linux-x86_64-ucs4.egg, /nbic/prog/galaxy-2010-11-14/eggs/bx_python-0.5.0_dev_f74aec067563-py2.6-linux-x86_64-ucs4.egg, /nbic/prog/galaxy-2010-11-14/eggs/amqplib-0.6.1-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/pexpect-2.4-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/Babel-0.9.4-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/Mako-0.2.5-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/WebHelpers-0.2-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs4.egg, /nbic/prog/galaxy-2010-11-14/eggs/wchartype-0.1-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/elementtree-1.2.6_20050316-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/docutils-0.4-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/WebOb-0.8.5-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/Routes-1.12.3-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg, /nbic/prog/galaxy-2010-11-14/eggs/PasteDeploy-1.3.3-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/PasteScript-1.7.3-py2.6.egg, /nbic/prog/galaxy-2010-11-14/eggs/Paste-1.6-py2.6.egg, /nbic/prog/galaxy-2010-11-14/lib, /usr/local/python_plus/lib64/python2.6/site-packages, /usr/local/python_plus/lib/python2.6/site-packages, /usr/local/lib/python2.6/site-packages, /nbic/prog/python_plus/lib64/python2.6/site-packages, /nbic/prog/python_plus/lib/python2.6/site-packages, /nbic/prog/lib/python2.6/site-packages, /nbic/prog/galaxy/lib, /nbic/prog/galaxy-2010-11-14, /usr/lib64/python26.zip, /usr/lib64/python2.6, /usr/lib64/python2.6/plat-linux2, /usr/lib64/python2.6/lib-tk, /usr/lib64/python2.6/lib-old, /usr/lib64/python2.6/lib-dynload, /usr/lib64/python2.6/site-packages, /usr/lib/python2.6/site-packages, /usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,085 Loaded sniffer for datatype: galaxy.datatypes.binary:Bam
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,085 Loaded sniffer for datatype: galaxy.datatypes.binary:Sff
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,086 Loaded sniffer for datatype: galaxy.datatypes.nbic:RAW
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,086 Loaded sniffer for datatype: galaxy.datatypes.xml:BlastXml
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,086 Loaded sniffer for datatype: galaxy.datatypes.nbic:Mzml
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,087 Loaded sniffer for datatype: galaxy.datatypes.nbic:Kml
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,087 Loaded sniffer for datatype: galaxy.datatypes.nbic:Xml
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,087 Loaded sniffer for datatype: galaxy.datatypes.sequence:Maf
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,088 Loaded sniffer for datatype: galaxy.datatypes.sequence:Lav
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,088 Loaded sniffer for datatype: galaxy.datatypes.sequence:csFasta
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,088 Loaded sniffer for datatype: galaxy.datatypes.qualityscore:QualityScoreSOLiD
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,089 Loaded sniffer for datatype: galaxy.datatypes.qualityscore:QualityScore454
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,089 Loaded sniffer for datatype: galaxy.datatypes.sequence:Fasta
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,089 Loaded sniffer for datatype: galaxy.datatypes.sequence:Fastq
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,090 Loaded sniffer for datatype: galaxy.datatypes.interval:Wiggle
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,090 Loaded sniffer for datatype: galaxy.datatypes.images:Html
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,090 Loaded sniffer for datatype: galaxy.datatypes.images:Pdf
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,091 Loaded sniffer for datatype: galaxy.datatypes.sequence:Axt
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,091 Loaded sniffer for datatype: galaxy.datatypes.interval:Bed
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,091 Loaded sniffer for datatype: galaxy.datatypes.interval:CustomTrack
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,092 Loaded sniffer for datatype: galaxy.datatypes.interval:Gff
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,092 Loaded sniffer for datatype: galaxy.datatypes.interval:Gff3
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,092 Loaded sniffer for datatype: galaxy.datatypes.tabular:Pileup
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,093 Loaded sniffer for datatype: galaxy.datatypes.interval:Interval
galaxy.datatypes.registry INFO 2010-11-14 15:02:36,093 Loaded sniffer for datatype: galaxy.datatypes.tabular:Sam
galaxy.eggs WARNING 2010-11-14 15:02:36,295 Warning: decorator (a dependant egg of sqlalchemy-migrate) cannot be fetched
Traceback (most recent call last):
  File "/nbic/prog/galaxy-2010-11-14/lib/galaxy/web/buildapp.py", line 82, in app_factory
    app = UniverseApplication( global_conf = global_conf, **kwargs )
  File "/nbic/prog/galaxy-2010-11-14/lib/galaxy/app.py", line 29, in __init__
    from galaxy.model.migrate.check import create_or_verify_database
  File "/nbic/prog/galaxy-2010-11-14/lib/galaxy/model/migrate/check.py", line 8, in <module>
    from migrate.versioning import repository, schema
  File "/nbic/prog/galaxy-2010-11-14/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/repository.py", line 9, in <module>
    from migrate.versioning import exceptions, script, version, pathed, cfgparse
  File "/nbic/prog/galaxy-2010-11-14/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/__init__.py", line 1, in <module>
    from py import PythonScript
  File "/nbic/prog/galaxy-2010-11-14/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py", line 10, in <module>
    from migrate.versioning.template import template
  File "/nbic/prog/galaxy-2010-11-14/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/template.py", line 10, in <module>
    from migrate.versioning import pathed
  File "/nbic/prog/galaxy-2010-11-14/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/pathed.py", line 10, in <module>
    from migrate.versioning.util import KeyedInstance
  File "/nbic/prog/galaxy-2010-11-14/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/util/__init__.py", line 5, in <module>
    from decorator import decorator
ImportError: No module named decorator