Write out scaling/merging statistics in mmCIF format by default in xia2 and DUI

[dagewa]

Following #281 we can write out mmCIF formatted reflections files from DIALS programs. As a subset of this, we include data scaling statistics in this file (see 007bea5). The PDB deposition pipeline can now accept the mmCIF-formatted scaling statistics files to merge with the coordinate mmCIF file, as noted by @berrisfordjohn. The developers of CCP4 interfaces would like it if this file was written out by default. In particular, if xia2 and DUI produced these files on every run to completion, then it would make deposition easier down the line.

Should this file be output by default whenever dials.scale is run?

What issues are there? I can see one - how do we ensure that the statistics match with a particular MTZ used for structure determination?

[graeme-winter]

I humbly note #1438 (comment) that you explicitly rejected this idea three weeks ago on the grounds that "derived data go stale"

I offer no objection to adding the additional dials.export step to xia2 and you can do precisely the same in DUI but I would think always outputting mmCIF format reflections from dials.scale every time it is run is excessive, particularly as by design you may want to run it again?

[jbeilstenedmands]

With regards to ensuring the matching of (unmerged?) MTZ and mmcif, these should contain the same data (or directly derived in a merged MTZ case), one should therefore be able to run the same merging code on both (using the audited dials version perhaps?) to verify that both give the same stats as given in the mmcif.

Also I know the mmcif has several auditing fields, we should be able to add detailed auditing to MTZ (in MTZ language I think this is referred to as 'history') so that it's obvious if the two were created at a different point. Although this might require allowing dials.export to export multiple formats at once, this should be easy to add.

[dagewa]

I humbly note #1438 (comment) that you explicitly rejected this idea three weeks ago on the grounds that "derived data go stale"

I offer no objection to adding the additional dials.export step to xia2 and you can do precisely the same in DUI but I would think always outputting mmCIF format reflections from dials.scale every time it is run is excessive, particularly as by design you may want to run it again?

Do note that I reopened your issue, which you closed in haste, because my only objection was that you wanted to dump this stuff in the experiments file, not in the "proper place" 🙄

[dagewa]

Agree that adding to xia2 and DUI sounds reasonable. What I meant with dials.scale was to only output the merging statistics portion, but I recognise the potential headaches therein. If we don't mind exporting the whole reflections mmCIF, including the stats, then it is certainly easier to track provenance

[graeme-winter]

Agree that adding to xia2 and DUI sounds reasonable. What I meant with dials.scale was to only output the merging statistics portion, but I recognise the potential headaches therein. If we don't mind exporting the whole reflections mmCIF, including the stats, then it is certainly easier to track provenance

Right - in most cases ${USER} only cares about what xia2 or whatever does - if in the export step for mmCIF reflections they include stats etc. then we are all good. But I think having this as part of dials.export once you are done scaling is probably in line with our expectations of what we do, else we could be reasonably asked to output scaled data in .sca, .XDS, .... formats as well 🙂

[dagewa]

This is currently not doing what I would expect

dials.export /tmp/xia2_test/DEFAULT/scale/20_refined_cell.expt /tmp/xia2_test/DEFAULT/scale/18_scaled.refl intensity=scale  mmcif.hklout=foo.mmcif
DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235
DIALS 3.dev.239-g7417272c0
The following parameters have been modified:

intensity = auto profile sum *scale
mmcif {
  hklout = "foo.mmcif"
}
input {
  experiments = /tmp/xia2_test/DEFAULT/scale/20_refined_cell.expt
  reflections = /tmp/xia2_test/DEFAULT/scale/18_scaled.refl
}

Data appears to be scaled, setting mtz.hklout = 'scaled.mtz'

That is, I ask for mmcif, but get an mtz (and no mmcif gets written)

[rjgildea]

You also need format=mmcif, i.e.:

$ dials.export $(dials.data get -q x4wide_processed)/AUTOMATIC_DEFAULT_scaled.{expt,refl} mmcif.hklout=foo.mmcif format=mmcif
DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235
DIALS 3.dev.243-g91234cfc9
The following parameters have been modified:

format = mtz sadabs nxs *mmcif mosflm xds xds_ascii json
mmcif {
  hklout = "foo.mmcif"
}
input {
  experiments = /Users/rjgildea/software/cctbx_py3/build/dials_data/x4wide_processed/AUTOMATIC_DEFAULT_scaled.expt
  reflections = /Users/rjgildea/software/cctbx_py3/build/dials_data/x4wide_processed/AUTOMATIC_DEFAULT_scaled.refl
}

Data appears to be scaled, setting intensity = scale
Read 78483 predicted reflections
Selected 66080 scaled reflections
Wrote reflections to foo.mmcif

Or you can leave out the mmcif.hklout and just set format=mmcif:

$ dials.export $(dials.data get -q x4wide_processed)/AUTOMATIC_DEFAULT_scaled.{expt,refl} format=mmcif
DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235
DIALS 3.dev.243-g91234cfc9
The following parameters have been modified:

format = mtz sadabs nxs *mmcif mosflm xds xds_ascii json
input {
  experiments = /Users/rjgildea/software/cctbx_py3/build/dials_data/x4wide_processed/AUTOMATIC_DEFAULT_scaled.expt
  reflections = /Users/rjgildea/software/cctbx_py3/build/dials_data/x4wide_processed/AUTOMATIC_DEFAULT_scaled.refl
}

Data appears to be scaled, setting intensity = scale
Data appears to be scaled, setting mmcif.hklout = 'scaled.cif'
Read 78483 predicted reflections
Selected 66080 scaled reflections
Wrote reflections to scaled.cif

I agree that this is potentially confusing, but the default format is mtz unless you explicitly set format= to some other allowable value.

[dagewa]

I can vouch that the confusion potential was realised. Thanks.

[rjgildea]

The default values for mtz.hklout and mmcif.hklout are auto, where their values are automatically defined based on whether integrated or scaled data are passed as input. Perhaps if one of these are explicitly set then this should override the format= parameter?

(as an aside, I'm not a big fan of the hklout parameter name)

[dagewa]

Ok, that gets me where I wanted to be. I'm not sure about adding this by default to every dials.export call on scaled data. The resultant mmcif file is twice the size of the scaled.mtz (for toy example of x4wide) and takes considerably longer to write out. The PDB is not currently accepting unmerged reflections in mmCIF format AFAIK. Possibly it would be more useful to write out a merged mmcif whenever we do the merge step?

[dagewa]

Having said that, a bzip2-compressed unmerged reflections mmcif is smaller than the unmerged MTZ (in fact, just smaller than the bzip2-compressed MTZ in the case I looked at). Writing this file compressed makes sense to me, as it is meant to simply exist without change until such time that it is uploaded to the PDB.

[graeme-winter]

Having said that, a bzip2-compressed unmerged reflections mmcif is smaller than the unmerged MTZ (in fact, just smaller than the bzip2-compressed MTZ in the case I looked at). Writing this file compressed makes sense to me, as it is meant to simply exist without change until such time that it is uploaded to the PDB.

Still not convinced that this should be written out by default? Unless there is some specific pressure which means that the output via dials.export is not appropriate e.g. inside xia2? I note that dials.scale does not - by default - output MTZ files even.

[graeme-winter]

Having said that, a bzip2-compressed unmerged reflections mmcif is smaller than the unmerged MTZ (in fact, just smaller than the bzip2-compressed MTZ in the case I looked at). Writing this file compressed makes sense to me, as it is meant to simply exist without change until such time that it is uploaded to the PDB.

Still not convinced that this should be written out by default? Unless there is some specific pressure which means that the output via dials.export is not appropriate e.g. inside xia2? I note that dials.scale does not - by default - output MTZ files even.

Oh wait, I see conversation has moved on - defaults in dials.export - same holds I feel though (MTZ is "special" in export, but maybe it should not be)

[dagewa]

I'm keeping this open here at the moment to uncover the issues (point of confusion above, file size etc.) but I think we've agreed that the behaviour change should actually be in xia2 and DUI. In DIALS the user can get mmcif if they want.

[dagewa]

I guess if xia2 is going to write a compressed mmcif it would be make sense if DIALS writes the file as present, and xia2 does the compression?

[rjgildea]

I guess if xia2 is going to write a compressed mmcif it would be make sense if DIALS writes the file as present, and xia2 does the compression?

Ideally, dials.export could compress on-the-fly depending on the file extension - e.g. if you pass mtz.hklout=foo.mmcif.gz
or mtz.hklout=foo.mmcif.bz2 then it would output a compressed file as appropriate.

[rjgildea]

Would it be a performance advantage if dials.export was able to output multiple formats at once? I.e. the .type of the format= phil parameter is currently choice which only allows a single value to be chosen. This could be changed to choice(multi=True) and then (along with some code modifications) it could output multiple formats in one job, without having to re-load the experiments/reflections from disk once per output filetype.

dials/command_line/export.py

Lines 77 to 78 in efe8a19

	format = *mtz sadabs nxs mmcif mosflm xds xds_ascii json
	.type = choice

[dagewa]

That would be very useful imo. Do we even need the format= if the user provides output filenames? MTZ could be default if no filename is provided. If mmcif.hklout is set then you get only an mmcif. If mmcif.hklout and mtz.hklout are set then you get both. Would that work?

[dagewa]

Or a single output.hklout parameter where the format is recognised by extension. This could be multi, so output.hklout=foo.mmcif output.hklout=foo.mtz gives both types. Providing neither defaults to mtz

[berrisfordjohn]

Hi I guess I should comment on the experience that Phenix had with outputting mmCIF. They found that they had to output both PDB and mmCIF by default after every refinement run – the PDB was used by users for viewing in Coot / further model building and the mmCIF only used when it came to deposition. They found if they put another barrier in the way of producing mmCIF (i.e. users must select an extra option) then users never used it and then complained that they didn’t have an mmCIF file to deposit to the PDB. HTH John

[dagewa]

Thanks @berrisfordjohn, we recognise the problem. We assume that most users (especially those using a CCP4 interface like i2 or the cloud) would be driving xia2 or DUI, so always writing out the mmcif "file of record" at the end of these jobs makes sense. For DIALS command-line use we tend towards getting nothing for free. The user at that level should specify exactly what they want.

[berrisfordjohn]

Yes, this makes perfect sense.

[berrisfordjohn]

Hi Did this discussion get concluded? If so, are there now new / existing options for Dials / Xia2 to output an mmCIF file which contains data processing statistics? I would like to pass this information onto Eugene for inclusion in CCP4 so that users can use the statistics from Dials for deposition. I would also like to pass this onto Frank’s group at Diamond as we are working with them to improve automatic deposition for fragment screening. Thanks John From: John Berrisford <jmb@ebi.ac.uk> Sent: 23 October 2020 10:17 To: 'dials/dials' <reply@reply.github.com>; 'dials/dials' <dials@noreply.github.com> Cc: 'Mention' <mention@noreply.github.com> Subject: RE: [dials/dials] Write out scaling/merging statistics in mmCIF format by default in xia2 and DUI (#1457) Yes, this makes perfect sense. John From: David Waterman <notifications@github.com <mailto:notifications@github.com> > Sent: 23 October 2020 09:52 To: dials/dials <dials@noreply.github.com <mailto:dials@noreply.github.com> > Cc: John Berrisford <jmb@ebi.ac.uk <mailto:jmb@ebi.ac.uk> >; Mention <mention@noreply.github.com <mailto:mention@noreply.github.com> > Subject: Re: [dials/dials] Write out scaling/merging statistics in mmCIF format by default in xia2 and DUI (#1457) Thanks @berrisfordjohn <https://github.com/berrisfordjohn> , we recognise the problem. We assume that most users (especially those using a CCP4 interface like i2 or the cloud) would be driving xia2 or DUI, so always writing out the mmcif "file of record" at the end of these jobs makes sense. For DIALS command-line use we tend towards getting nothing for free. The user at that level should specify exactly what they want. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/dials/dials/issues/1457#issuecomment-715200246> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADCP7N7NS4TQMXFGST6J5FLSME73FANCNFSM4SZ3ESXA> . <https://github.com/notifications/beacon/ADCP7N6ESHSKZS2R6KCAIQ3SME73FA5CNFSM4SZ3ESXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFKQRN5Q.gif>

[dagewa]

Hi @berrisfordjohn

Did this discussion get concluded?

Unless there are objections I sense tacit agreement that DUI and xia2 should run dials.export format=mmcif automatically. I will create issues in the relevant repositories.

However, the discussion has also identified areas in which dials.export could be improved. Of these, I consider the issue of on-the-fly compression to be important to address before automatically writing mmcif, given the very large file size that unmerged mmcifs can take.

An alternative would be to write out merged data only, along with the statistics, to keep file size down, dials.export does not currently make that type of file though, plus we want unmerged mmcif eventually anyway, so we might as well jump right there.

[wojdyr]

Hi everyone,

I came here from David's comment from project-gemmi/gemmi#116 (comment)

IIUC two separate issues are discussed here:

• as per the title, writing statistics that are to be included in coordinate mmCIF file,
• writing reflection mmCIF file with scaled unmerged data.

The first thing is important for the deposition task and the file with statistics only would be small, so the size is not a concern.

MmCIF file with scaled unmerged reflections could be useful if it contained information that's not in MTZ.
But not directly for deposition, at least for now. For deposition (1) it'd need to be appended to reflection mmCIF file with merged data – which is easy, (2) special magic is required to deposit unmerged data with more than I/sigI only. This magic is a temporary solution. To allow depositing of scaled unmerged data with frame numbers etc ASAP some temporary short-cuts have been added to OneDep. In practice, if you'd want to deposit mmCIF from dials.scale you'd need to work with the PDB developers to make sure that your files use these special short-cuts. Or just wait for a while, until these things are sorted out in OneDep.