Consortium for Management of Experimental Data in Structural Biology
First imgCIF workshop (new series) at ACA 2006 Meeting

The pace of data collection and the volume of data collected at synchrotron beam lines is increasing. The ACA Data, Standards, and Computing Committee is spearheading an effort to improve the efficiency of the handling and storage of these data by encouraging the adoption of common data formats and standard software interfaces. The goal of this is firstly to have the data be self defining, therefore equally accessible to data-reduction and -visualization codes. The second goal, for the purposes of secure archiving, is to provide robust internal documentation of the source of the data.

The first imgCIF/CBF workshop took place at Brookhaven National Laboratory in 1997 and proposed a format combining support for an efficient binary representation of images with a fully CIF-compliant ASCII equivalent. An imgCIF/CBF dictionary and software to support the format were created, are available on the web, and are described in Volume G of the IUCr International Tables for Crystallography. Now the community should adopt a consensus standard for management of data at synchrotron beam lines, and to make it easier for users to process data taken from various beam lines. Also, as our science evolves, new concepts will be considered: possibilities include NeXus and XML.

The first workshop in the new series, on "Management of Synchrotron Image Data: imgCIF File System and Beyond", was held on 22 July 2006 as part of the 2006 ACA meeting in Honolulu, Hawaii. The workshop was attended by 39 participants and began with brief introductory talks in the morning by R. Sweet, H. Bernstein, G. Bricogne, C. Nielsen, W. Minor, J. Pflugrath, R. Sweet, A. Ashton, J. Westbrook, B. McMahon and T. Proffen. The rest of the workshop was devoted to lively discussion lead in alternation by Sweet and Bernstein and which produced a surprising degree of consensus. Productive discussion continued, even after formal conclusion of the workshop, late into the afternoon, despite the lure of the beach just outside the meeting room.

The charge to the workshop was to "Organize the adoption of a standard, self-documenting format for diffraction image files." Rather immediately, the focus was on adoption of the current imgCIF/CBF standard that has been approved by the IUCr COMCIFS. The format of the meeting included eleven short talks, presenting a little history and background, and then descriptions by investigators, with various backgrounds and interests, of their own goals or apprehensions about adoption of the imgCIF standard.

Talks in the imgCIF Workshop (WK.02) Saturday, 22 July 2006, 8:30 am -- 5:00 pm, Oahu Room
(The pre-workshop agenda and charge is at http://www.medsbio.org/meetings/ACA_2006_WK02.html)

8:30	Definition of the problem

        What are the objectives of this workshop?    R. Sweet                 [handout]

8:50	Brief refresher on the structure and flexibility of imgCIF and available supporting 
software and mechanisms for making changes to both.  H. Bernstein             [handout,
                                                                               dictionary,
                                                                               CBFlib]

...
10:00-12:20  Short (10-15 minutes) prepared statements (based primarily on handouts, rather 
than visual aids) to help frame the discussion, tentatively:

        What is the problem we should be solving?               G. Bricogne
        Issues for equipment vendors.                           C. Nielsen    [handout]
        Issues for data-red'n software developers.              W. Minor      [handout]
                                                                J. Pflugrath  [handout]
        Issues for beam-line software developers.               R. Sweet
        Issues for other software developers.                   A. Ashton     [handout]
        Issues for data archivists.                             J. Westbrook  [handout]
        Issues for CIF wonks.                                   B. McMahon    [handout,
                                                                               ITVG imgCIF Extract]
        Issues for relative to NeXus, HDF and XML.              T. Proffen    [handout,
                                                                               State of NeXus paper]

The second segment of the meeting was open discussion on a range of topics:

• Consider the whole flow of information/knowledge in the experiment; where does imgCIF fit?
• How does one deal with corrected vs uncorrected data?
• What are the impediments to immediate adoption: can we define a time line?
• How should the intellectual property issues relating to compression algorithms be handled?
• What will be the mechanism for an open-source protocol for continuing development of imgCIF?
• And finally, we should talk about the dictionary to understand whether to add new items now.

The conclusions and recommendations of the workshop were:

• 1. This is the right time for more widespread use of imgCIF.
• 2. SR sources should start writing imgCIF image files as soon as possible, employing the imgCIF dictionary already adopted by the IUCr Committee on the Maintenance of the CIF Standard (COMCIFS) and published on the web and in International Tables Volume G. Such image files will provide at least as complete and accurate information as the various file headers written now, and will have the added benefit of stimulating further conversion of programs that feed the experimental parameters and of programs that reduce the data. Small teams have been created to start the process by working through one detector/beamline combination at a time.
• 3. An important step in achieving this goal will be the creations of certified stand-alone conversion programs ("jiffies") that will convert from imgCIF to the existing standard native file format used for each detector system in the field and from standard native file formats to imgCIF. This will provide software developers with benchmarks against which to work and allow work with imgCIF to start while conversions of major packages are in progress.
• 4. The files should include both the corrected and the raw images, with the option of putting both together in the same imgCIF file.
• 5. Those interested in imgCIF should look for a way to assure continuity in development. Maintenance and development of the dictionary and software library need to be taken up by a consortium where continued interest and, especially, continued funding are assured. There are several possibilities, and some study will be done in the coming year.
• 6. The items above are the primary "to do list" for the coming year. In addition, it is clear that work is also needed on:
  • 6.1. Identification of a the critical subset of parameters from the imgCIF and mmCIF dictionaries that define "just the image."
  • 6.2. Additions to the dictionary to support derived parameters that may be recalculated and re-refined over time (e.g. beam centers), each variant of which needs to be recorded with a time stamp and other details to identify the history of these parameters.
  • 6.3. The interaction with NeXus to facilitate data management at facilities that will be using NeXus for the overall data management but which will be producing imgCIF for PX beamline users to take home.
  • 6.4. Storage of additional data defining the environment within which an experiment has been done such as images of pixels that are blocked by beam stops or otherwise "dead".

While work on these issues is important and helps to explain the need for continued funding discussed in item 5, above, it is our strong belief that our primary focus must be on items 1 through 5, or the base for this other work will not be as firm as it should be.

Partial support was provided by the U.S. Department of Energy under grant ER64212-1027708-0011962, NSF under grant DBI-0610407 and Area Detector Systems Corporation (ADSC). The workshop was run thanks to the help and cooperation of the ACA, the ACA Continuing Education Committee and the ACA Data, Standards, and Computing Committee. The workshop would not have been a success without the hard work of the ACA Buffalo Office and the Hawaii meeting organizing committee. The support, cooperation and encouragement from all these sources are gratefully acknowledged.