Difference between revisions of "TavernaGrid"

From BioAssist
Jump to: navigation, search
(2008-12-09)
(2008-12-11)
Line 66: Line 66:
 
* Implement single- and multiple-input file upload and multiple output file download for jobs.
 
* Implement single- and multiple-input file upload and multiple output file download for jobs.
 
* Investigate possibility of long-running proxy delegation.
 
* Investigate possibility of long-running proxy delegation.
 +
 +
== 2008-12-15 ==
 +
Code-in-progress has been committed to SVN.
 +
* Single- and multiple-input file and multiple-output file tested and working.
 +
* API documentation [https://wiki.nbic.nl/index.php/Service_Oriented_Architecture updated].
 +
Still to-do:
 +
* Finish planning and research for Taverna extension (mostly complete).
 +
* Long-running proxy delegation (optional).
  
 
= Timebox 1 (T1) Work Package =
 
= Timebox 1 (T1) Work Package =

Revision as of 13:02, 15 December 2008

TavernaGrid is a project by SARA and Eagle Genomics Ltd., funded by NBIC. The primary goal of the project is to allow users to compose Grid web services with Taverna.

News

2008-10-29

We just had the project kick-off meeting! Richard Holland was at SARA for three days. During many short sessions, we've managed to get a firm understanding of what needs to be done, and we agreed on the first timebox, which will run until Christmas. Some documents were drafted, which can be found in this Wiki.

2008-11-15

API documentation uploaded.

2008-12-05

Code-in-progress has been committed to SVN.

Done so far:

  • A working SOAP client that sends requests over HTTPS and includes custom headers.
  • A working SOAP server that receives reqeusts over HTTPS and parses custom headers.
  • An API that can retrieve a certificate from MyProxy, modify it with VOMS attributes, and delegate it to WMProxy.
  • The API can also submit a job, but this bit is currently broken (giving SSL errors from the WMProxy server, not due to communication with WMProxy, but due to communication between WMProxy and some other internal server or piece of code - to be investigated next week).

Next week:

  • Visiting Manchester on Tuesday+Wednesday to meet the team, find out more about Taverna, and discuss the best way of implementing the new SOAP client code as a Taverna plugin.
  • Try to work out what's going on internally in WMProxy as to why it's giving SSL errors when submitting jobs. Might need a hand here.

Before end of sub-contract:

  • Produce plan of how to integrate this all into a Taverna plugin.
  • Make API able to fully submit jobs. Hopefully to include Machiel's modifications to be able to include InputSandbox.
  • Simplify API and example code where possible and make sure fully internally documented.
  • Possibly add API code for delegating user certificates to MyProxy.

2008-12-08

Code-in-progress has been committed to SVN.

Jobs can now be submitted and executed and output (or errors) retrieved. No-input, single-output jobs are possible. The client and server side of the service both work. All this is correctly achieved via delegated certificates from MyProxy locally modified within the service to have the VOMS attributes added. Certificates need to be manually delegated to MyProxy before calling the service.

As a result of visiting the team in Manchester, I've produced a plan for the Taverna integration which Alex and Stuart have both reviewed and agreed is sensible. The plan has been placed here. I believe this is detailed enough to serve as a template for the next unit of work to be planned, but please let me know if you need more detail. In terms of timescale for this I can start some simpler parts of it in January, but Tom will be presenting a code tutorial in Manchester in February which I plan to attend, at which point I hope to learn enough to get the more complicated parts of the Taverna plugin code written.

Tasks remaining to complete before end of sub-contract, in order of priority:

  • Fix dependency issues in ws-client.
  • Remove temporary code inside the API/ws-server/ws-client implementations and make them into neat and tidy deliverables.
  • Produce detailed usage docs on the wiki.
  • Javadoc everything in detail.
  • Implement single- and multiple-input file upload and multiple output file download for jobs.
  • Simplify API and implement MyProxy delegation methods for client side to remove need to manually delegate certificate.
  • Investigate possibility of long-running proxy delegation.

2008-12-09

Code-in-progress has been committed to SVN. API documentation amended and updated.

  • Dependency issues in ws-client have been fixed and the SOAP call code simplified.
  • Temporary code has been removed throughout.
  • MyProxy delegation on the client side has now been implemented and is demonstrated by ws-client.

Remaining to do in this sub-contract:

  • Correct and expand all JavaDocs.
  • Fully document system on Wiki.
  • Implement single- and multiple-input file upload and multiple output file download for jobs.
  • Implement asynchronous web service methods and demonstration of their use.
  • Investigate possibility of long-running proxy delegation.

2008-12-11

Code-in-progress has been committed to SVN.

  • All JavaDocs done.
  • Asynchronous calls implemented and demonstrated in ws-client.
  • API documentation complete.

Remaining to do in this sub-contract:

  • Implement single- and multiple-input file upload and multiple output file download for jobs.
  • Investigate possibility of long-running proxy delegation.

2008-12-15

Code-in-progress has been committed to SVN.

  • Single- and multiple-input file and multiple-output file tested and working.
  • API documentation updated.

Still to-do:

  • Finish planning and research for Taverna extension (mostly complete).
  • Long-running proxy delegation (optional).

Timebox 1 (T1) Work Package

Timebox Parameters
Start date 2008-10-27
End date 2008-12-23
Eagle Genomics hours
Period Days Comments
2008-10-27 — 2008-10-29 3 Kick-off meeting
2008-11-01 — 2008-11-14 4 50% of 4 days/week
2008-11-15 — 2008-11-24 0 Richard unavailable
2008-11-26 — 2008-12-23 17 4 days/week
Total: 24 ≋ 192hrs ≋ € 9,792

MoSCoW Prioritized Actions and Acceptance Criteria

Do not edit the Acceptance Criteria without the consent of all team members!

Musts

  • An example WS-I web service (hereafter called TestWS) has been created and deployed.
  • The functionality of the TestWS must be testable. The trivial way to do this is to build a test client (which is listed as a SHOULD), but good alternatives are acceptable.
  • The TestWS accepts authentication and authorization information in SOAP headers:
    • MyProxy hostname, port, username, password;
    • VO-name, VOMS-server hostname and port, VOMS-server X.509 certificate DN.
  • Proof that the VOMS-proxy created by the TestWS will allow WMS job submission and SE data manipulation. This could be proven by having the TestWS actually perform these tasks (which is listed as a COULD), but good alternatives are acceptable. E.g., TestWS could create a PEM-formatted proxy file, that can then be tested for validity with command line utilities.
  • Investigate what exactly needs to be done to have Taverna properly call our TestWS, and produce a draft version of the Stage Plan for the next timebox.

Shoulds

  • Create an example client.
  • Proper JavaDoc documentation of all code.
  • Proper System documentation in this wiki. (What is where, how to deploy, etc.)
  • Code is in the GForge SCM.
  • Describe the extra credential XML in the SOAP header (SARA).
  • Draft a document that can be used for our meeting with the Taverna development team.
    • Things we don't understand (Richard)
    • Things we'd like to see in Taverna2 (All)
    • People we'd like to meet (Pieter, Marco)
  • Have a (two day) meeting in Manchester with the Taverna development team, somewhere at the end of this timebox.

Coulds

  • The TestWS can submit Grid jobs to a WMS and manipulate data on an SE.
  • Taverna extension/plugin development:
    • User interface for managing the configuration.
    • Storing the configuration in a proper configuration file.
    • Creating a Service Provider
      • First, create a "stub" Service Provider, that's just a copy of the (default?) WSDL/SOAP Service Provider, without any extra functionality.
      • Then, extend this Service Provider to submit Credential information in the SOAP header.
  • Create an XSD for the extra credential XML in the SOAP header (SARA).

Won'ts

  • Anonymous proxy delegation/retrieval to/from the MyProxy server for long lived jobs.

Reporting

  • Richard, Machiel and Pieter will have weekly Skype meetings. During these meetings:
    • all identified risks will be considered, and the Risk Log will be updated.
    • progress will be discussed on the basis of the prioritized Acceptance Criteria above.
  • The minutes of these meetings (first draft by SARA, corrections by Eagle Genomics) will serve as detailed progress reports. No further progress reports are expected from Eagle Genomics.

Risk Log

This risk log is incomplete.

  • We can't agree on a proper WS framework to start with.
  • SARA doesn't provide the necessary environment for Richard to deploy the TestWS.