Consortium for Management of Experimental Data in Structural Biology
Meetings

Third imgCIF workshop (new series) at BSR 2007 in Manchester and at Diamond:

The Management of Synchrotron Image Data:
Changes to the imgCIF dictionary and software, interaction with NeXus

Sponsored by DOE under grant ER64212-1027708-0011962, NSF under grant DBI-0610407 and NIH under grant 1R13RR023192-01A1

Workshop Report
Draft for Comments and Corrections

G. Winter 14 Aug 07	C. Broennimann 14 Aug 07	A. Ashton 17 Aug 07	M. Dougherty 17 Aug 07	M. Folk 17 Aug 07

The pace of data collection and the volume of data collected at synchrotron beam lines is increasing. The ACA Data, Standards, and Computing Committee spearheaded an effort to improve the efficiency of the handling and storage of these data by encouraging the adoption of common data formats and standard software interfaces. The goal of this was firstly to have the data be self defining, therefore equally accessible to data-reduction and -visualization codes. The second goal, for the purposes of secure archiving, was to provide robust internal documentation of the source of the data.

The current effort began in 2005, building on work started in the mid 1990's on a Crystallographic Binary Format (CBF) proposed by Andy Hammersley. This effort was the basis for the image-supporting Crystallographic Information Format/Crystallographic Binary Format (imgCIF/CBF). The first imgCIF/CBF workshop took place at Brookhaven National Laboratory in 1997 and proposed a format combining support for an efficient binary representation of images with a fully CIF-compliant ASCII equivalent. An imgCIF/CBF dictionary and software to support the format were created, are available on the web, and are described in Volume G of the IUCr International Tables for Crystallography. Now the community should adopt a consensus standard for management of data at synchrotron beam lines and to make it easier for users to process data taken from various beam lines. Also, as our science evolves, new concepts will be considered: possibilities include NeXus and XML.

The first workshop in the new series on "Management of Synchrotron Image Data: imgCIF File System and Beyond", was held on 22 July 2006 as part of the 2006 ACA meeting in Honolulu, Hawaii. That workshop concluded that that was "the right time for more widespread use of imgCIF ... [and that] SR sources should start writing imgCIF image files as soon as possible, employing the imgCIF dictionary already adopted by the IUCr Committee on the Maintenance of the CIF Standard (COMCIFS) and published on the web and in International Tables Volume G.] " [from the report of the workshop, see http://www.medsbio.org/meetings/ACA_2006_WK02_Report.html].

Subsequent to the Hawaii workshop intensive work was started in response to these recommendations. Both the imgCIF dictionary and the supporting software library were reviewed and, after meetings at SLS and ESRF, extended. See SLS_report.html and ESRF_report.html for more information on those meetings. The work continued in collaboration with members of the community (see the imgCIF mailing list http://www.iucr.org/iucr-top/cif/cbf/imgcif-l/.

In light of this activity a workshop on data formats for synchrotron image data was held after the NSLS/CFN meeting on 24 May 2007 at BNL in the Biology Department. Topics discussed included proposed extensions to imgCIF, the use of NeXus, progress on software and the status of imgCIF at Diamond and at SLS. That workshop concluded that work was needed on support for the handling of uncorrected images and true bitmaps, creation of a utility to "tidy" CIFS, creation of an agreed interface with XML, NeXus and HDF, clarification of the specification of the relationship between detector specification and the physical locations of pixels in the laboratory, tags for robotics and remotes access, and the creation of more cookbooks. See http://www.medsbio.org/mettings/BNL_May07_imgCIF_Workshop_Report.html.

In the short time between the second and third workshops, work on many of the items on this task list began. In addition, building on discussions with J. Steinbrener at the second workshop, discussions with Matt Dougherty on the image needs in the microscopy community, and, through Matt Dougherty, with Mike Folk of the HDF Group on techniques for integration between imgCIF and NeXus, HDF and XML were started. Work on CBFlib continued and version 0.7.8 was released.

The third imgCIF workshop was held in two sessions at BSR 2007 in Manchester and at Diamond. Herbert Bernstein and Alun Ashton organized this workshop. The purpose of this workshop was to provide a review of the status of imgCIF and CBFlib for the European user community and to discuss the integration of imgCIF with NeXus, HDF and XML. The Manchester session was used for an introduction and review of the status of imgCIF and for some discussion. The Diamond Light Source session was used to deal with more detailed technical issues and further discussion.

The first session was in Manchester on Tuesday, 14 August 2007. It provided an introduction to imgCIF and NeXus and a brief review of current progress. This session took place in the Barbirolli Room at Bridgewater Hall from 12.45 to 13.45. The session began with a review of the status of imgCIF by H. Bernstein, followed by a review of the status of use of imgCIF with the Pilatus 6M detector at the Swiss Light Source by C. Broennimann and a presentation of a developer's perspective on imgCIF by G. Winter. The major discussion at this session was about what information, in what format whould be in the header of an image presented as a CBF.

The second session was at the Diamond Light Source (DLS) on Friday, 17 August 2007, to discuss recent changes in the imgCIF dictionary and software and the interaction with NeXus. This session took place in room 1.17 of Diamond House. It began with short presentations and some discussion from 13:15 to 14:20. Then there was a break, and more substantive discussions continued from 14:30 until about 17:15. The presentations began with H. Bernstein giving a recap of the status of imgCIF and proposals on versioning and integration between imgCIF and NeXus. A. Ashton and C. Nielsen then presented various aspects of the use of imgCIF at DLS. M. Dougherty presented the status of data formats in cryo-electron microscopy. M. Folk reviewed the capabilities and status of HDF.

The Manchester session was attended by 20 people, 14 of whom were formally registered, and the DLS session was attended by 14 people, 11 of whom were able to remain for the full discussions. A total of 28 people participated in one or both sessions.

The discussions at both sessions were constructive and productive and, at times, passionate. While there were sharp disagreements, they were over how best to use imgCIF, not over whether to use imgCIF, and there seemed to be the reasonable propsect that eventual compromise and consensus will be achived after further discussions and email exchanges. The major points of discussion and their status were:

• miniCBF headers: The Pilatus 6m detector is now in production use at SLS and is being offered by Dectris to beamlines at other synchrotrons. The detector produces images in CBF format, but with the metadata gathered into one text block. These CBF images are processed by W. Kabsch's XDS and by A. Arvai's adxv. This first production use of CBF for a large number of images raised concerns about what should be called a "CBF". The major area of disagreement was over how much information in what format should be in an image header for it to be properly identified as a CBF or imgCIF file. There was strong agreement that all the essential metadata for an image should be in the header with the image. The disagreement arose over
  • what is essential: just the metatdata that changes image to image, or the metadata essential to a full description of the raw data, or all the metadata needed for the entire structural study, even if most of it has to presented as unknown at the time of the actual data collection.
  • how should it be presented: as a single consolidated text block as in the current SLS miniCBF files (see this sample of an SLS miniCBF header) or as the expanded full imgCIF header produced from the miniCBF files by the CBFlib example program convert_minicbf (see this sample of a full SLS CBF header)?
  These issues had been raised at the first workshop in Hawaii. Now production use of imgCIF at SLS makes resolution of these issues urgent.

  After some discussion, it was agreed that the important issue was not to specify what SLS might choose to use or produce but to specify what the major processing programs might require and to clearly identify each CBF file as to the completeness of the headers that they produce for purposes of processing by various programs. Alternatives considered included a special comment for incomplete headers, versus the special looped tag within each image data block specifying compliance with the needs of particular processing programs (e.g. _software_compliance.complies_with mosflm _software_compliance.complies_with_version "7.0.1" ) Harry Powell has provided a sample of the header items he proposes should be required (see this sample of header proposed by H. Powell). Discussions with SLS and Dectris representatives between the two workshop sessions and by email afterwards suggest that agreement on a full header to be offered to beamlines and users who want them is likely to be achievable.

• CBFlib performance issues: DLS and ADSC have completed tests of CBF images as produced by ADSC detectors and are now satisfied that they are being produced correctly for both A. Arvai's adxv and H. Powell's mosflm. However, production use has been delayed by performance issues with CBFlib version 0.7.6. It was noted that CBFlib 0.7.7 included performance improvements suggested by E. Eikenberry and that a memory leak identified by N. Sauter had been corrected in CBFlib 0.7.8. In any case the problem apprears to be an interaction between memory and disk use with symptoms of a progressive slowdown in repeated use of in a single process. Once this is resolved, production use of CBF at DLS should be feasible. H. Bernstein, G. Darakev and N. Darakev will work with C. Nielsen to provide detailed benchmarking on various algorithms in CBFlib and work to bring the performance issues under control.

• Versioning: In order to allow for corrections, adjustments and refinements of data values such a beam centers without loss of earlier versions, the addition of optional version tags to all categories was proposed. Those version tags would point to a new VERSION category which would provide timestamps and a tree structure to multiple versions. More details will be provided via the imgCIF list.

• Integration with NeXus, HDF and XML: M. Folk's presentation on HDF and A. Ashton's discussion on the use of NeXus at DLS help to make it clear that it is time to resolve the open issues on integration between imgCIF and NeXus, HDF and XML. In order to integrate imgCIF with NeXus, it was proposed that the entire set of CIF tags be imported into NeXus with the prefix "CBF_" to avoid namespace conflicts. It was generally agreed that this was reasonable and would work, but raised the issue of needing to remove duplicate presentations of the same information. A proposal to make use of DDLm-like methods and to allow that same information to be present in multiple locations in the NeXus tree along with the transforms needed to relate the duplicated information was not accepted at this time. It was agreed that users will need versions of the same information in NeXus and imgCIF files with duplicated information pruned. Further discussion on the process to deal with duplicates in the course of moving such files from one side to the other is needed. The use of binUTF to allow efficient presentation of binaries within the XML presentation of image data was discussed. No objection was raised. H. Bernstein will prepare a more detailed proposal on the combining of trees and the HDF and XML presentation of image data for discussion on the imgcif-l and medsbio-l lists.

• Microscopy Images: M. Dougherty's presentation made it clear that the microscopy community faces image representation issues similar to that of the protein crystallography community with even higher data volumes as well as a larger number of formats lacking performance, extensibility and archivability. It is clear that much can be gained by both communities through collaboration on image format design. Inter-operable datasets will allow downstream researchers greater modeling opportunities and unforeseeable methods of analysis. Near the end of the day, M. Dougherty and H. Bernstein agreed to the establishment of a wiki in the MEDSBIO.org web site to gather information and propose design requirements needed for structural biology image formats, building on work started by M. Dougherty and M. Folk, et al. When the wiki is ready for use, a journal article will be prepared to announce its availability.

(The pre-workshop agenda and charge is at http://www.medsbio.org/meetings/BSR_2007_imgCIF_Workshop.html).

The agenda for the workshop as held was

• Tuesday, 14 August 2007, 12:45 - 13:45
  Barbirolli Room at Bridgewater Hall, Manchester
  • 12:45 - 13:00 lunch
  • 13:00 - 13:15 Introduction and review of status imgCIF -- Herbert J. Bernstein
    handout, dictionary, CBFlib manual
  • 13:15 - 13:25 Experience with imgCIF for the Pilatus 6M detector at the SLS -- Christian Broennimann
  • 13:25 - 13:35 "imgCIF: a developers perspective" -- Graeme Winter
  • 13:35 - 13:45 Discussion
• Friday, 17 August 2007, 10:00 - 17:15
  room 1.17 of Diamond House, Diamond Light Source, Didcot, Chilton
  • 10:00 - 12:30 Preliminary informal discussions among early arrivals
  • 12:30 - 13:15 Lunch
  • 13:15 - 13:30 imgCIF and the interaction with NeXus -- Herbert J. Bernstein
    handout
  • 13:30 - 13:40 Experience with imgCIF at Diamond -- Alun Ashton
    handout
  • 13:40 - 13:45 Issues in use of imgCIF at DLS for ADSC detectors -- Chris Nielsen
  • 13:45 - 14:05 Status of Data Formats in Cryo-electron Microscopy -- Matt Dougherty
    talk
  • 14:05 - 14:20 HDF -- Mike Folk
    handout
  • 14:20 - 14:40 Break
  • 14:40 - 15:30 Technical discussion of open issues in the interaction among imgCIF, NeXus, HDF and XML
  • 15:30 - 17:00 Technical discussion of open issues in imgCIF (e.g. versioning of data, speed issues, miniCBFs, cylindrical detectors, etc.)
  • 17:00 - 17:15 Wrapsupand future plans.

Travel Information

The workshop is completed. The following information is to assist participants in completing their travel paperwork.

The 14 May 2007 workshop session was held at the BSR 2007 meeting venue in Manchester. For travel information for that session, see http://www.srs.ac.uk/bsr2007/pages/travel.htm.

The 17 May 2007 workshop session was held as the Diamond Light Source (DLS). Registered participants at BSR 2007 were able to avail themselves of the transportation to DLS by making arrangements with the meeting organizers (anita at ya-ya dot co dot uk). Alternatively DLS travel information was available at http://www.diamond.ac.uk/AboutDiamond/Location

For those for whom travel reimbursement had been arranged, forms were provided at the workshops. A copy is available as a PDF BSR_Aug07_expense.pdf, which is to be filled out, signed and returned with receipts to Herbert J. Bernstein, Kramer Science Center KSC-121, Dowling College, Oakdale, NY 11769-1999, USA.

Participants