MX raw image data formats, metadata and validation

IUCr XXV for Prague, CZ has been postponed from August 2020 until August 2021. The Day-0 Workshop on MX raw image data formats, metadata and validation is in two editions:

As an e-meeting via BlueJeans on Saturday, 22 August 2020, from 9 am until 3 pm, Prague Time (GMT+2) which is 1 hour later than London time (8 am until 2 pm), 6 hours later than New York City time (3 am until 9 am), which has been help and is reported on below, and

As a face-to-face Day-0 workshop to be held in Prague on Saturday, 14 August 2021.

This is a report for the 22 August 2020 e-workshop on MX raw image data formats, metadata and validation. The registered participants were:

Registered Participants

name	affilitation	email
Jun Aishima	Brookhaven National Lab	jaishima at bnl dot gov
Alun Ashton	Paul Scherrer Institut	alun dot ashtonat psi dot ch
Oskar Aurelius	MAX IV Laboratory	oskar dot aureliusat maxiv dot lu dot se
Frances C. Bernstein	Bernstein + Sons	fcb at bernstein-plus-sons dot com
Herbert J. Bernstein	Ronin Institute for Independent Scholarship,c/o NSLS-II BNL	yayahjb at gmail dot com
Gleb Bourenkov	EMBL Hamburg	gleb at embl-hamburg dot de
Aaron Brewster	LBL	asbrewster at lbl dot gov
Gerard Bricogne	Global Phasing Ltd.	gb10 at globalphasing dot com
Sandor Brockhauser	European XFEL	sandor dot brockhauser at xfel dot eu
Tom Caradoc-Davies	Australian Synchrotron -- ANSTO	thomasc at ansto dot gov dot au
Scott Classen	ALS -- Berkeley National Laboratory	sclassen at lbl dot gov
Ian Clifton	Oxford University	ian dot clifton at chem dot ox dot ac dot uk
Gregg Crichlow	RCSB Protein Data Bank	gc663@iqb.rutgers.edu at iqb dot rutgers dot edu
Vikram Dalal	Indian Institute of Technology Roorkee	vikram dot dalal dot 37 at gmail dot com
Fabio Dall'Antonia	European XFEL	fabio dot dall dot antonia at xfel dot eu
Poonam Dhankhar	Department of Biotechnology, Indian Institute of Technology Roorkee	dhankharp16 at gmail dot com
Daniel Eriksson	Australian Synchrotron -- ANSTO	daniele dot eriksson at ansto dot gov dot au
Andreas Förster	DECTRIS Ltd	andreas dot foerster at dectris dot com
Luca Gelisio,	Center for Free-Electron Laser Science, DESY	luca dot gelisio at cfel dot de
Carlotta Giacobbe	ESRF	carlotta dot giacobbe at gmail dot com
Andy Götz	ESRF	andy dot gotz at esrf dot fr
David Hall	Diamond Light Source	david dot hall at diamond dot ac dot uk
Paul Hathaway	Diamond Light Source	paul dot hathaway at diamond dot ac dot uk
John Helliwell	School of Chemistry, The University of Manchester, UK	john dot helliwell at manchester dot ac dot uk
Natalie Johnson	CCDC	njohnson at ccdc dot cam dot ac dot uk
Wonchull Kang	University of California Irvine	w dot kang at uci dot edu
Wei-Chun Kao	University of Freiburg	wei-chun dot kao at biochemie dot uni-freiburg dot de
Rajnandani Kashyap	CSIR- Indian Institute of Chemical Technology, Hyderabad, India	kashyap dot rajnandani at gmail dot com
Kong Wai-Po	The Hong Kong Polytechnic University	wai-po-ball dot kong at connect dot polyu dot hk
Dale Kreitler	CBMS, NSLS-II, BNL	dkreitler at bnl dot gov
Loes M. J. Kroon-Batenburg	Bijvoet Center for Biomolecular Research, Utrecht University	l dot m dot j dot kroon-batenburg at uu dot nl
Filip Leonarski	PSI	filip dot leonarski at psi dot ch
Filipe Maia	Uppsala University	filipe at xray dot bmc dot uu dot se
Brian McMahon	IUCr	bm at iucr dot org
Valerio Mariani	SLAC National Accelerator Laboratory	valmar at slac dot stanford dot edu
Jie Nan	MAX IV Laboratory	jie dot nan at maxiv dot lu dot se
Marina Nikolova	EMBL, Hamburg	nikolova at embl-hamburg dot de
Majid Ounsy	SOLEIL Synchrotron	majid dot ounsy at synchrotron-soleil dot fr
Evgenii Osipov	KU Leuven	e dot m dot osipov at gmail dot com
Jason Price	ANSTO -- Melbourne	jasonp at ansto dot gov dot au
Daniel Salvat	ALBA Synchrotron (Barcelona)	dsalvat at cells dot es
Gianluca Santoni	ESRF	gianluca dot santoni at esrf dot fr
Thomas Schneider	EMBL Hamburg and DESY	thomas dot schneider at embl-hamburg dot de
Irwin Selvam	University of Manchester	irwin dot selvam at postgrad dot manchester dot ac dot uk
Kate Mary Louise Smith	PSI	kate dot smith at psi dot ch
S. Sriram	Indian Institute of Science	hypowergravity at gmail dot com
Amy Thompson	School of Chemistry and Molecular Biosciences, The University of Queensland	amy dot thompson2 at uq dot net dot au
Andrew Thompson	SOLEIL Synchrotron	andrew dot thompson at synchrotron-soleil dot fr
Alexandra Tolstikova	DESY	alexandra dot tolstikova at desy dot de
Sameer Velankar	PDB-e	sameer at ebi dot ac dot uk
Clemens Vonrhein	Global Phasing Ltd., Cambridge, UK	vonrhein at globalphasing dot com
Suzanna Ward	Cambridge Crystallographic Data Centre (CCDC)	ward at ccdc dot cam dot ac dot uk
John Westbrook	RCSB, Protein Data Bank	john dot westbrook at rcsb dot org
Ben Williams	Diamond Light Source	ben dot williams at diamond dot ac dot uk
Jon Wright	ESRF	wright at esrf dot fr
Smita Yadav	Regional Centre for biotechnology, Haryana-121001.	smita at rcb dot res dot in

The most important background information for the 2020 edition is the Gold Standard paper, [Herbert J. Bernstein, Andreas Förster, Asmit Bhowmick, Aaron S. Brewster, Sandor Brockhauser, Luca Gelisio, David R. Hall, Filip Leonarski, Valeria Mariani, Gianluca Santoni, Clemens Vonrhein, Graeme Winter, "Gold Standard for macromolecular crystallography diffraction data," IUCrJ 7, no. 5 (2020)] https://doi.org/10.1107/S2052252520008672

The talks linked to this report and this edited chat provide a great deal of information on the the state of work and issues related to using this standard. The following discussion points seem worth highlighting. Our thanks to John Helliwell in providing notes on these subjects.

Linking between the PDB and raw data repositories:
The WorldWide PDB (wwPDB) encourages depositors to deposit raw data of their structure and then provide the DOI in the "Related experimental data sets" user interface during structure deposition in wwPDB OneDep. Depositors are strongly encouraged by the wwPDB to deposit related raw exprimental data (e.g. diffraction image data) in one of the curated archives, but if the raw data is in one of the non-curated archives, such as Zenodo, instead, the depositor should list that DOI. The details of the information that may be provided are given in the PDBx dictionary in the PDBX_RELATED_EXP_DATA_SET category. The wwPDB deposition FAQ at https://www.wwpdb.org/deposition/faq has information for depositors on how to link to raw data during their PDB deposition.
Both macromolecular crystallography and and chemical crystallography require the recording of data such as sequences for MX or crystal morphology for CX, information that may not be needed for data collection, but which may be available at the time of data collection and which should be preserved with the raw data for eventual use in paper writing and structure deposition.
Preserving this data in CIF format is simple, but classic NeXus/HDF5 has not in the past provided all the necessary equivalent tags. In order to allow Gold Standard NeXus/HDF5 files to be complete, the NeXus NXpdb base class allows for the incorporation of all necessary mmCIF/PDBx dictionary tags within NeXus files, e.g. within NXsample. The entire core CIF dictionary is included within this set of tags, so all the crystal morphology tags are available.
Event synchronization is not considered in the Gold Standard as it now stands.
For XFEL data, pump-probe experiments and other time-resolved experiments, time-stampled event logging is important. There are tags for time stamping in NeXus, and some of them are already used, but more work is needed to resolve performance and optimal representation issues. We will not delay implementation of the Gold Standard as it now stands while these are discussed, but will give a high priority to improving documentation of time stamps and to exploring alternatives during the Fall of 2020.
The Gold Standard does not speak to the issue of data precision.
It is important that data be reported to a precision appropriate to the experiment. Both CIF and NeXus provide tags to carry the uncertainties in data. When information to fill in these fields is available, it should be recorded. We will add reminders about these tags to the documentation of the Gold Standard.
Different detectors require different calibration and range information.
The necessary tags are available in both CIF and NeXus. We will add more examples in Zenodo as they are developed.

It is clear from the talks and the discussion that the Gold Standard as it now exists is useful, but additional documentation and examples of more use cases are needed. We plan on more discussions in Fall 2020 and Spring 2021 and to have much to report in Prague at IUCr XXV in summer 2021.

Agenda

Saturday, 22 August 2020
Prague London New York	topic		min	conf
9:00 -- 9:15 8:00 -- 8:15 3:00 -- 3:15	Set up for BlueJeans, test connections		15
Session 1	Introduction to the Gold Standard
9:15 -- 9:45 8:15 -- 8:45 3:15 -- 3:45	Loes Kroon-Batenburg	FAIR and MX Raw Data pdf	30	yes
9:45 -- 10:15 8:45 -- 9:15 3:45 -- 4:15	Herbert J Bernstein (Ronin Institute)	MX raw data formats and the Gold Standard pdf	30	yes
10:15 -- 10:30 9:15 -- 9:30 4:15 -- 4:30	Coffee break (bring your own coffee, tea or other refreshments)		15
Session 2	Stream to Gold Standard - Working with FAIR data
10:30 -- 10:45 9:30 -- 9:45 4:30 -- 4:45	Andreas Förster (DECTRIS)	Stream2 and FileWriter2 pdf	15	yes
10:55 -- 11:10 9:55 -- 10:10 4:55 -- 5:10	Paul Hathaway (DLS) Ben Williams (DLS)	The Gold Standard at DLS pdf	15	yes
11:10 -- 11:25 10:10 -- 10:25 5:10 -- 5:25	Filip Leonarski (PSI)	The Gold Standard at PSI pdf	15	yes
11:25 -- 11:35 10:25 -- 10:35 5:25 -- 5:35	break (bring your own coffee, tea or other refreshments)		10
11:35 -- 11:50 10:35 -- 10:50 5:35 -- 5:50	Dale Kreitler (NSLS-II)	The Gold Standard at NSLS-II pdf	15	yes
11:50 -- 12:05 10:50 -- 11:05 5:50 -- 6:05	Jie Nan (MAX IV)	The MX Data at Max IV pdf	15	yes
12:05 -- 12:30 11:05 -- 11:30 6:05 -- 6:30	Clemens Vonrhein (Global Phasing)	Requirements of automatic processing pdf	25	yes
12:30 -- 12:55 11:30 -- 11:55 6:30 -- 6:55	Sameer Velankar (PDB-e)	How can PDB submission be made more automatic pdf	25	yes
12:55 -- 13:30 11:55 -- 12:30 6:55 -- 7:30	Lunch break (bring your own lunch or breakfast ..., as appropriate to your time zone)		35
Session 3	The future of the Gold Standard
13:30 -- 13:50 12:30 -- 12:50 7:30 -- 7:50	Aaron Brewster (LBL)	The Gold Standard at XFELs pdf	20	yes
13:50 -- 14:10 12:50 -- 13:10 7:50 -- 8:10	Jason Price (ANSTO -Melbourne (Australian Synchrotron)) and Natalie Johnson (CCDC)	HDR Detectors and Chemical Crystallography at the Australian Synchrotron MX Beamlines pdf, Synchrotron Data in the CSD pdf	20	yes
14:10 -- 15:00 13:15 -- 14:00 8:10 -- 9:00	Open discussion What metadata are lacking at the moment? What problems are we envisioning?		50