-
Notifications
You must be signed in to change notification settings - Fork 0
/
quality_questions.qmd
1271 lines (1231 loc) · 86.9 KB
/
quality_questions.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
---
title: "Quality Questions"
format:
html:
embed-resources: true
css: apd_style.css
html-table-processing: none
---
::: {.callout-note collapse="false"}
This guidance is an ALPHA draft. It is in development and we are still working to ensure that it meets user needs.
Please get in touch with feedback to support the guidance by creating a [GitHub Issue](https://github.com/best-practice-and-impact/analysis_project_documentation/issues) or [emailing us](mailto:ASAP@ons.gov.uk).
:::
To get the most out of the template, we strongly recommend that teams identify who will take the key quality assurance roles of Commissioner, Senior Responsible Owner and Analytical Assurer and name the analytical team at the start of the analytical cycle. This is crucial because together these roles help to make sure that the analysis you do is fit-for-purpose.
# I. Scoping
::: {.panel-tabset}
## Quality questions and why they matter
```{=html}
<table class="simpleTable">
<TR VALIGN=TOP>
<th scope = "col"> </th>
<th scope = "col"> Quality Question</th>
<th scope = "col"> Why do I need to know the answer to this?</th>
</tr>
<tr>
<td>Q1</td>
<td>What question does the analysis try to answer?</td>
<td>A clear understanding of the analysis question is critical. It helps
your team to scope out requirements, understand the strengths and limitations of the analysis and make sure it is fit for purpose.<br><br>
If the question is not clear, you risk designing and delivering analysis which does not meet user needs.</td>
</tr>
<tr>
<td>Q2</td>
<td>Why do you need to answer this analysis question?</td>
<td>Knowing why you need the analysis, what it is for and how it will be used will help you to understand the importance and impact of your work and how it supports decision making.<br><br>It will also help you to make sure the analysis is fit for purpose and correctly answers the question.</td>
</tr>
<tr>
<td>Q3</td>
<td>Which organisational priorities does this analysis address?</td>
<td>Knowing how the work aligns with organisational priorities shows how it will fit with wider strategic objectives and why you should do the analysis now.<br><br>It informs the level of assurance needed to confirm the work is fit for purpose.</td>
</tr>
<tr>
<td>Q4</td>
<td>If you use a model, is it business critical?</td>
<td>Identifying if the work is business critical determines the assurance needed to ensure it is fit for purpose.</td>
</tr>
<tr>
<td>Q5</td>
<td>Who needs the answer to the analysis question?</td>
<td>Knowing what your outputs will be used for can ensure they meet user needs. A good understanding of uses is essential for making sure that your analysis is fit for purpose.</td>
</tr>
<tr>
<td>Q6</td>
<td>Who do you need to consult to make sure you meet the right user
needs?</td>
<td>Analysis must be well understood by relevant users, or else
risks scope creep and misspecification.<br><br>Identify relevant stakeholders and users, consult them before designing the analysis, and consider their views.<br><br>Consulting the right stakeholders helps you and your users agree on how to answer the question and what the output should look like. This means you can check the users' understanding of the process and its quality, and that the final ouput meets user needs.</td>
</tr>
<tr>
<td>Q7</td>
<td>How will you know you have answered the analysis question correctly?</td>
<td>Being clear about the outputs required and acceptable uncertainty
is essential for producing accurate and reliable outputs where users understand the work's limitations and uncertainty.<br><br>It is also essential when designing verification and validation activities to check the robustness of results under a range of plausible assumptions about methods and data.</td>
</tr>
<tr>
<td>Q8</td>
<td>What is the estimated time and resource required to answer the analysis question (in months and FTE)?</td>
<td>Without a clear understanding of time and resources available,
you may overcommit. It is important to push back against unrealistic demands if there is not enough time to effectively quality assure the analysis.<br><br>The users and commissioner should fully understand and accept increased risks to quality when time and resource pressures are unavoidable.</td>
</tr>
<tr>
<td>Q9</td>
<td>What is the impact if the analysis is not done now?</td>
<td>Understanding why the work needs to happen now will help you to
prioritise and use limited resources for the right activities.</td>
</tr>
<tr>
<td>Q10</td>
<td>What is the impact if the analysis is not done correctly?</td>
<td>Understanding the possible legal, financial or reputational consequences if the analysis is not carried out correctly helps you to design proportionate assurance activities.<br><br>You should consider how these consequences line up with the risk appetite of your organisation when you design your mitigation.</td>
</tr>
<tr>
<td>Q11</td>
<td>Name the commissioner, senior responsible owner and analytical assurer of this analysis?</td>
<td>Clear accountability makes sure that important decisions are
signed off by the right people. There should be a clear understanding of who is responsible for managing, producing and quality assuring the analysis in the team.</td>
</tr>
<tr>
<td>Q12</td>
<td>What tools and resources will you use in production? Are they the best for the job?</td>
<td>Before starting, identify all the skills and resources needed to produce the final output in a sustainable and reproducible way and quality assure it at every step.<br><br>The platform used to host the analysis and the software to build and run the analysis should be appropriate and risks considered. For example, producing critical outputs in Excel does not comply with best practice and is unlikely to be robust or verifiable.</td>
</tr>
<tr>
<td>Q13</td>
<td>Do you have the right internal and external resources and
capability to deliver the analysis?</td>
<td>If the team lacks capability, resource, or time, this increases the risk that the analysis will not be fit for purpose or sufficiently assured.</td>
</tr>
<tr>
<td>Q14</td>
<td>What are the anticipated risks of the analysis?<br><br>Have you
discussed these risks with customers and stakeholders?</td>
<td>You must identify potential risks and their impact well in advance to enable effective mitigation and quicker, confident decision-making.<br><br>
It is important that users understand risks to ensure that their expectations and requirements are met. You should document how you have identified and are monitoring and mitigating risks.</td>
</tr>
<tr>
<td>Q15</td>
<td>Is there a contingency plan prepared if your mitigation plans fail?</td>
<td >You should account for risks that you can mitigate. Include risks that have a high impact on the analysis but a low probability happening. Without well designed contingencies, we put quality at risk.</td>
</tr>
<tr>
<td>Q16</td>
<td>Do the data and analysis comply with ethical requirements?</td>
<td >Analysis must comply with ethics standards to ensure public confidence. You must consider the ethical implications of the analysis when you create the workflow and report your findings.</td>
</tr>
<tr>
<td>Q17</td>
<td>What relevant questions are outside the scope of the analysis?</td>
<td >Limiting the scope of analysis shapes the quality of outputs and what can be done with them. By being clear about the limitations of the
analysis we can mitigate or accept them. Limitations must be documented so everybody using the analysis is aware of them.</td>
</tr>
<tr>
<td>Q18</td>
<td>How will you peer review and assure the analysis?</td>
<td>Internal audit and peer review are critical for monitoring and
assuring that the analysis is performed appropriately and meets the
required aims and objectives. The outcomes of peer review should be
documented and relevant actions and recommendations should be prioritised, addressed, and taken forward.</td>
</tr>
<tr>
<td>Q19</td>
<td>Will external experts be involved in development and scrutiny of
analysis?</td>
<td >You should commission external specialists to peer review or
audit the analysis in proportion to risks around use. They should be able to draw on expertise and experience across government and beyond to get feedback, exchange experience and suggest best practice to improve the analysis.</td>
</tr>
</table>
```
## The questions and the Code of Practice
<table class="simpleTable" cellspacing="0" cellpadding="0">
<TR VALIGN=TOP>
<th scope = "col"> </th>
<th scope = "col"> Quality Question</th>
<th scope = "col"> Which Code pillar and practice are most relevant here?
*Trustworthiness (T), Quality (Q), Value (V)</th>
</tr>
<tr>
<td>Q1</td>
<td>What question does the analysis try to answer?</td>
<td>V1.1 Statistics producers should maintain and refresh their understanding of the use and potential use of the statistics and data. They should consider the ways in which the statistics might be used and the nature of the decisions that are or could be informed by them.</td>
</tr>
<tr>
<td>Q2</td>
<td>Why do you need to answer this analysis question?</td>
<td>V1.1 Statistics producers should maintain and refresh their
understanding of the use and potential use of the statistics and data. They should consider the ways in which the statistics might be used and the nature of the decisions that are or could be informed by them.</td>
</tr>
<tr>
<td>Q3</td>
<td>Which organisational priorities does this analysis address?</td>
<td>T2.1 The Chief Statistician/Head of Profession for Statistics
should have sole authority for deciding on methods, standards and procedures, and on the content and timing of the release of regular and ad hoc official statistics. This should include: determining the need for new official statistics, ceasing the release of official statistics, and the development of experimental statistics.</td>
</tr>
<tr>
<td>Q4</td>
<td>If you use a model, is it business critical?</td>
<td>Q3.2 Quality assurance arrangements should be proportionate to
the nature of the quality issues and the importance of the statistics in serving the public good. Statistics producers should be transparent about the quality assurance approach taken throughout the preparation of the statistics. The risk and impact of quality issues on statistics and data should be minimised to an acceptable level for the intended uses.</td>
</tr>
<tr>
<td>Q5</td>
<td>Who needs the answer to the analysis question?</td>
<td>V1 Users of statistics and data should be at the centre of statistical production; their needs should be understood, their views sought and acted upon, and their use of statistics supported.</td>
</tr>
<tr>
<td>Q6</td>
<td>Who do you need to consult to make sure you meet the right user
needs?</td>
<td>V1 Users of statistics and data should be at the centre of statistical production; their needs should be understood, their views sought and acted upon, and their use of statistics supported.<br><br>V1.2 Statistics producers should use appropriate ways to increase awareness of the statistics and data, communicate effectively with the widest possible audience, and support users and potential users in identifying relevant statistics to meet their needs.</td>
</tr>
<tr>
<td>Q7</td>
<td>How will you know you have answered the analysis question correctly?</td>
<td>Q3 Producers of statistics and data should explain clearly how
they assure themselves that statistics and data are accurate, reliable,
coherent and timely.</td>
</tr>
<tr>
<td>Q8</td>
<td>What is the estimated time and resource required to answer the analysis question (in months and FTE)?</td>
<td>T5 People producing statistics should be appropriately skilled,
trained and supported in their roles and professional development.<br><br>T4.3 Sufficient human, financial and technological resources should be provided to deliver statistical services that serve the public good.<br><br>T3.5 Statistics and data should be released on a timely basis and at intervals that meet the needs of users as far as practicable. The statistics should be released as soon as they are considered ready, under the guidance of the Chief Statistician or Head of Profession for Statistics.</td>
</tr>
<tr>
<td>Q9</td>
<td>What is the impact if the analysis is not done now?</td>
<td>T2.1 The Chief Statistician/Head of Profession for Statistics
should have sole authority for deciding on methods, standards and procedures, and on the content and timing of the release of regular and ad hoc official statistics. This should include: determining the need for new official statistics, ceasing the release of official statistics, and the development of experimental statistics.</td>
</tr>
<tr>
<td>Q10</td>
<td>What is the impact if the analysis is not done correctly?</td>
<td>Q3 Producers of statistics and data should explain clearly how
they assure themselves that statistics and data are accurate, reliable,
coherent and timely.</td>
</tr>
<tr>
<td>Q11</td>
<td>Name the commissioner, senior responsible owner and analytical assurer of the analysis?</td>
<td>T5.2 The roles and responsibilities of those involved in the
production of statistics and data should be clearly defined with supporting guidance provided to help staff carry out their roles.</td>
</tr>
<tr>
<td>Q12</td>
<td>What tools and resources will you use in production? Are they the best for the job?</td>
<td>T5 People producing statistics should be appropriately skilled,
trained and supported in their roles and professional development.<br>T5.5 Staff should be provided with the time and resources required to develop their skills, knowledge and competencies.<br><br>T4.3 Sufficient human, financial and technological resources should be provided to deliver statistical services that serve the public good.</td>
</tr>
<tr>
<td>Q13</td>
<td>Do you have the right internal and external resources and
capability to deliver the analysis?</td>
<td>Q3 Producers of statistics and data should explain clearly how
they assure themselves that statistics and data are accurate, reliable,
coherent and timely.<br><br>T4.3 Sufficient human, financial and technological resources should be provided to deliver statistical services that serve the public good.</td>
</tr>
<tr>
<td>Q14</td>
<td>What are the anticipated risks of the analysis?<br><br>Have you
discussed these risks with customers and stakeholders?</td>
<td>Users of statistics and data should be at the centre of statistical production; their needs should be understood, their views sought and acted upon, and their use of statistics supported.<br><br>V4.3 Users should be involved in the ongoing development of statistics and data, exploring and testing statistical innovations, so that the statistics remain relevant and useful.<br><br>Q3.2 Quality assurance arrangements should be proportionate to the nature of the quality issues and the importance of the statistics in serving the public good. Statistics producers should be transparent about the quality assurance approach taken throughout the preparation of the statistics. The risk and impact of quality issues on statistics and data should be minimised to an acceptable level for the intended uses.</td>
</tr>
<tr>
<td>Q15</td>
<td>Is there a contingency plan prepared if your
mitigation plans fail?</td>
<td>T4.5 Organisations should be open about their commitment to
quality and make clear their approach to quality management. They should
ensure that the organisational structure and tools are in place to manage
quality effectively, and promote and adopt appropriate quality standards.</td>
</tr>
<tr>
<td>Q16</td>
<td>Do the data and analysis comply with ethical requirements?</td>
<td>T6 Organisations should look after people’s information securely
and manage data in ways that are consistent with relevant legislation and
serve the public good.<br><br>T1.1 Everyone that works in organisations producing official statistics should handle and use statistics and data with honesty and integrity, guided by established principles of appropriate behaviour in public life.</td>
</tr>
<tr>
<td>Q17</td>
<td>What relevant questions are outside the scope of the analysis?</td>
<td>Q1.6 The causes of limitations in data sources should be identified and addressed where possible. Statistics producers should be open
about the extent to which limitations can be overcome and the impact on the statistics.<br><br>Q2.4 Relevant limitations arising from the methods and their application, including bias and uncertainty, should be identified and explained to users. An indication of their likely scale and the steps taken to reduce their impact on the statistics should be included in the explanation.</td>
</tr>
<tr>
<td>Q18</td>
<td>How will you peer review and assure the work?</td>
<td>Q3 Producers of statistics and data should explain clearly how
they assure themselves that statistics and data are accurate, reliable,
coherent and timely.<br><br>T4.6 Independent measures, such as internal and external audit, peer review and National Statistics Quality Reviews, should be used to evaluate the effectiveness of statistical processes. Statistics producers should be open about identified areas for improvement.</td>
</tr>
<tr>
<td>Q19</td>
<td>Will external experts be involved in development and scrutiny of
analysis?</td>
<td>Q3 Assured quality Producers of statistics and data should
explain clearly how they assure themselves that statistics and data are
accurate, reliable, coherent and timely.<br><br>T4.6 Independent measures, such as internal and external audit, peer review and National Statistics Quality Reviews, should be used to evaluate the effectiveness of statistical processes. Statistics producers should be open about identified areas for improvement.</td>
</tr>
</table>
## Linking the questions to AQuA Roles
<table class="simpleTable" cellspacing="0" cellpadding="0">
<TR VALIGN=TOP>
<th> </th>
<th scope = "col"> Quality Question</th>
<th scope = "col"> Which AQuA role(s) would normally answer this?</th>
<th scope = "col"> Why are these AQuA roles involved?</th>
</tr>
<tr>
<td>Q1</td>
<td >What question does the analysis try to answer?</td>
<td>Commissioner, analyst</td>
<td>The commissioner sets out the commission. They work with the analyst team to ensure that everyone has a common understanding of the problem.
</td>
</tr>
<tr>
<td>Q2</td>
<td>Why do we need to answer this analysis question?</td>
<td>Commissioner, analyst, analytical assurer</td>
<td>The analyst must document the purpose of the analysis and the levels of quality and certainty that are needed to meet user requirements. The commissioner and analytical assurer make sure the analysis aligns with the stated purpose.</td>
</tr>
<tr>
<td>Q3</td>
<td>Which organisational priorities does this analysis address?</td>
<td>Commissioner</td>
<td>The commissioner makes sure that key aspects of the problem, scope and complexities, including programme constraints, are captured and clearly communicated. They also make sure there is proportionate governance in place to support the analysis and its role in the wider project or programme</td>
</tr>
<tr>
<td>Q4</td>
<td>If you use a model, is it business critical?</td>
<td>Commissioner, Senior Responsible Owner</td>
<td>Business critical models must be managed appropriately so that the right specialists are responsible for developing, using and assuring them.<br><br>Decision-makers need sufficient assurance from an appropriate level in the organisation that the model is fit for purpose before using it to inform a decision. For business critical analysis and modelling, the commissioner should be satisfied with the seniority of the analytical assurer.</td>
</tr>
<tr>
<td>Q5</td>
<td>Who needs the answer to the analysis question?</td>
<td>Commissioner,analyst,analytical assurer</td>
<td>The commissioner makes sure that the right stakeholders have been identified so that the scope and boundaries of the analysis can be appropriately explored. The analyst team and analytical assurer should also contribute.</td>
</tr>
<tr>
<td>Q6</td>
<td>Who do you need to consult to make sure you meet the right user
needs?</td>
<td>Analyst, commissioner</td>
<td>Analysts should explore the analysis requirements and scope with all relevant stakeholders to make sure a wide range of perspectives are sought. The commissioner should be aware and briefed.</td>
</tr>
<tr>
<td>Q7</td>
<td>How will you know you have answered the analysis question correctly?</td>
<td>Commissioner, analyst</td>
<td>During the design and conduct of analysis, the commissioner should set out details like the level of precision, accuracy and uncertainty that are needed.</td>
</tr>
<tr>
<td>Q8</td>
<td>What is the estimated time and resource required to answer this
analysis question (in months and FTE)?</td>
<td>Commissioner, senior responsible owner, analyst team</td>
<td>During commissioning and scoping, the commissioner and analyst will need to make trade-offs between time, resources and quality. They should work together to agree and document the right balance across these constraints.</td>
</tr>
<tr>
<td>Q9</td>
<td>What is the impact if the analysis is not done now?</td>
<td>Commissioner</td>
<td>The commissioner makes sure that there is sufficient time and resource for the required level of assurance to be delivered and that they understand the risks when time and resource pressures
are unavoidable.</td>
</tr>
<tr>
<td>Q10</td>
<td>What is the impact if the analysis is not done correctly?</td>
<td>Commissioner, analyst</td>
<td>The commissioner makes sure the analyst team understands the context for the analysis question. This helps the analyst to understand and assess likely risks and determine the right analytical and quality assurance response.</td>
</tr>
<tr>
<td>Q11</td>
<td>Name the commissioner, senior responsible owner and analytical assurer of this analysis?</td>
<td>Commissioner</td>
<td>During scoping of the analysis, the commissioner makes sure there is proportionate governance in place to support the analysis and its role in the wider project or programme.</td>
</tr>
<tr>
<td>Q12</td>
<td>What tools and resources will you use in production? Are they the best for the job?</td>
<td>Commissioner</td>
<td>The commissioner makes sure that there is enough time and resource for the required level of assurance to be delivered. They must be confident that they understand the risks when time and resource pressures are unavoidable.</td>
</tr>
<tr>
<td>Q13</td>
<td>Do you have the right internal and external resources and
capability to deliver the analysis?</td>
<td>Commissioner</td>
<td>The commissioner makes sure that there is enough time and resource for the required level of assurance to be delivered. They must be confident that they understand the risks when time and resource pressures are unavoidable.</td>
</tr>
<tr>
<td>Q14</td>
<td>What are the anticipated risks of the analysis?<br><br>Have you
discussed these risks with customers and stakeholders?</td>
<td>Analytical assurer, commissioner, analyst</td>
<td>The analystical assurer should challenge and test the understanding of the problem. The commissioner and analyst work with the analytical assurer to make sure that all share a common understanding.
</td>
</tr>
<tr>
<td>Q15</td>
<td>Is there a contingency plan prepared if your mitigation plans fail?</td>
<td>Commissioner, analytical assurer</td>
<td>The commissioner makes sure that there is enough time and resource for the required level of assurance to be delivered. They must be confident that they understand the risks when time and resource pressures are unavoidable.<br><br>If there is a need for urgent action, such as mitigation of unacceptable but uncertain risks, the commissioner may ask for further analysis. They might also commission extra evidence-gathering in parallel to inform the policy response when uncertainty is reduced. The analytical assurer must advise the commissioner on whether sufficient analytical quality assurance has happened and inform them about any outstanding risks.</td>
</tr>
<tr>
<td>Q16</td>
<td>Do the data and analysis comply with ethical requirements?</td>
<td>Analyst, commissioner, analytical assurer</td>
<td>The analyst should make sure that there is appropriate ethical approval for the analysis. The commissioner and analytical assurer should be informed.</td>
</tr>
<tr>
<td>Q17</td>
<td>What relevant questions are outside the scope of the analysis?</td>
<td>Commissioner, analyst</td>
<td>The commissioner and the analyst work together at the scoping stage to get a clear understanding of analytical requirements. During scoping, the commissioner makes sure that the right aspects of the problem, scope and complexities, including programme constraints, are captured and clearly communicated.</td>
</tr>
<tr>
<td>Q18</td>
<td>How will you peer review and assure the analysis?</td>
<td>Analytical assurer, commissioner, analyst</td>
<td>The analytical assurer makes sure quality assurance plans for the analysis are appropriate for the decision it supports. All analysis requires some level of quality assurance. Analyst and commissioner should be involved.</td>
</tr>
<tr>
<td>Q19</td>
<td>Will external experts be involved in development and scrutiny of
analysis?</td>
<td>Analytical assurer, commissioner, analyst</td>
<td>The analytical assurer should challenge the proposed approach. Check
that it delivers as intended and meets customer needs. It is good
practice to engage subject matter experts in this review. Analyst and
commissioner should be involved.</td>
</tr>
</table>
:::
# II. Design
::: {.panel-tabset}
## Quality questions and why they matter
<table class="simpleTable" cellspacing="0" cellpadding="0">
<TR VALIGN=TOP>
<th scope = "col"> </th>
<th scope = "col"> Quality Question</th>
<th scope = "col"> Why do I need to know the answer to this?</th>
</tr>
<tr>
<td> Q20 </td>
<td>Is there a simple, plain English description of what the analysis is for and what it does?</td>
<td>Writing a plain English description of what the analysis is for and how it works means that everybody in the team, including new starters, and others with no subject or technical expertise can understand the purpose of the analysis and what it does.
</td>
</tr>
<tr>
<td>Q21</td>
<td>Does the analysis have a logic flowchart which explains the
end-to-end steps in the workflow?</td>
<td>Setting out a clear summary of the analysis process in a diagram
helps the team, users and customers to understand at a glance what the
analysis does, where inputs come from, how they are processed and how it generates outputs.</td>
</tr>
<tr>
<td>Q22</td>
<td> When do you expect to start and finish each stage of analysis: data collection, processing, quality assurance, analysis and dissemination?
</td>
<td>Clearly setting out the time you need to perform each stage of analysis helps you to evaluate if the time allocated to each stage is right and plan mitigation if plans look too ambitious.</td>
</tr>
<tr>
<td>Q23</td>
<td>Does any part of the analysis rely on manual processing? Have
you considered the cost and benefits of fully automating the process?</td>
<td>Manual processes are inefficient and more risky than well-designed automated ones. Analysis with manual steps like copying and pasting data between files, manually updating cells in tables, or moving data between software packages is harder to assure and carries extra quality risks.</td>
</tr>
<tr>
<td>Q24</td>
<td>What happens if team members, reviewers or users find a mistake in the analysis?<br><br>Do you have a clear and efficient process for addressing issues and preventing them from happening again?</td>
<td>Mistakes happen in analysis. You should have a clear and efficient process for reporting, documenting and addressing errors. Being open and honest about problems and working together to solve them in a supportive way creates an atmosphere of openness in the team and is critical for upholding users' trust in your output.<br><br>Teams should take reasonable steps to understand and document how and why errors came about, and mitigate the risk of them happening again. Your mitigation approach should be consistent with your department's revision policies.</td>
<tr>
<td>Q25</td>
<td>Have you assessed uncertainty?</td>
<td>Consider how uncertainty impacts on all stages of the analysis.
Think about quantifying and measuring uncertainty as early as possible.
Identify and review sources of uncertainty regularly.<br><br>If you delay the assessment of uncertainty until late in the process, it is often difficult and costly to mitigate risks. In some cases, you may need to revise methods or alter commissioning decisions. If a potential source of uncertainty is overlooked, this can limit the usefulness and impact of the analysis.</td>
</tr>
</tr>
</table>
## The questions and the Code of Practice
<table class="simpleTable" cellspacing="0" cellpadding="0">
<TR VALIGN=TOP>
<th scope = "col"> </th>
<th scope = "col"> Quality Question</th>
<th scope = "col"> Which pillar and principle of the Code of Practice are most relevant here?
*Trustworthiness (T), Quality (Q), Value (V)</th>
</tr>
<tr>
<td>Q20</td>
<td>Is there a simple description in plain English of what the analysis is for and what it does?</td>
<td>V3 Statistics and data should be presented clearly, explained meaningfully and provide authoritative insights that serve the public good.</td>
</tr>
<tr>
<td>Q21</td>
<td>Does the analysis have a logic flowchart which explains the
end-to-end conceptual steps in the work flow?</td>
<td>V3 Statistics and data should be presented clearly, explained
meaningfully and provide authoritative insights that serve the public good.</td>
</tr>
<tr>
<td>Q22</td>
<td>When do you expect to start and finish each stage of analysis:
data collection, processing, quality assurance, analysis and
dissemination?</td>
<td>T3.5 Statistics and data should be released on a timely basis and at intervals that meet the needs of users as far as practicable. The statistics should be released as soon as they are considered ready, under the guidance of the Chief Statistician/Head of Profession for Statistics.<br><br>Q3.3 The quality of the statistics and data, including their accuracy and reliability, coherence and comparability, and timeliness and punctuality, should be monitored and reported regularly.</td>
</tr>
<tr>
<td>Q23</td>
<td>Does any part of the analysis rely on manual processing? Have
you considered the cost and benefits of fully automating the process?
</td>
<td>V4 Statistics producers should be creative and motivated to
improve statistics and data, recognising the potential to harness
technological advances for the development of all parts of the production and dissemination process.<br><br>T4.3 Sufficient human, financial and technological resources should be provided to deliver statistical services that serve the public good.</td>
</tr>
<tr>
<td>Q24</td>
<td>What happens if team members, reviewers or users find a mistake in the analysis?<br><br>Do you have a clear and efficient process for addressing issues and preventing them from happening again?</td>
<td>Q3.4 Scheduled revisions, or unscheduled corrections that result
from errors, should be explained alongside the statistics, being clear on
the scale, nature, cause and impact.<br><br>V5 Statistics and data should be published in forms that enable their reuse. Producers should use existing data wherever possible and only ask for more where justified.</td>
</tr>
<tr>
<td>Q25</td>
<td>Have you assessed uncertainty?</td>
<td>Q2.4 Relevant limitations arising from the methods and their
application, including bias and uncertainty, should be identified and
explained to users. An indication of their likely scale and the steps taken to reduce their impact on the statistics should be included in the
explanation.</td>
</tr>
</table>
## Linking the questions to AQuA Roles
<table class="simpleTable" cellspacing="0" cellpadding="0">
<TR VALIGN=TOP>
<th scope = "col"> </th>
<th scope = "col"> Quality Question</th>
<th scope = "col"> Which AQuA role(s) would normally answer this?</th>
<th> Why are these AQuA roles involved?</th>
</tr>
<tr>
<td>Q20</td>
<td>Is there a simple, plain English description of what the analysis is for and what it does?</td>
<td>Analyst, analytical assurer</td>
<td>The analyst should produce sufficient design documentation. Best practice can include a description of the analysis, user requirements, design specification, functional specification, data dictionary, and test plan. The analytical assurer should give feedback to make sure documentation is fit for purpose.</td>
</tr>
<tr>
<td>Q21</td>
<td>Does the analysis have a logic flowchart which explains the
end-to-end steps in the workflow?</td>
<td>Analyst, analytical assurer</td>
<td>The analyst should produce sufficient design documentation. Best practice can include a description of the analysis, user requirements, design specification, functional specification, data dictionary, and test plan. The analytical assurer should give feedback to make sure documentation is fit for purpose.</td>
</tr>
<tr>
<td>Q22</td>
<td>When do you expect to start and finish each stage of analysis:
data collection, processing, quality assurance, analysis and
dissemination?</td>
<td>Commissioner, analyst</td>
<td>The analyst and commissioner should work together during scoping
to set out and agree trade-offs between time, resources and quality and establish the optimal balance of these constraints.</td>
</tr>
<tr>
<td>Q23</td>
<td>Does any part of the analysis rely on manual processing? Have
you considered the cost and benefits of fully automating the process?</td>
<td>Commissioner, analyst</td>
<td>The analyst should use a risk-based approach to understand the
areas of greatest potential error and focus assurance efforts on these areas. The analyst should brief the commissioner so they understand the impact of any reduction in the thoroughness of analytical quality assurance activities.</td>
</tr>
<tr>
<td>Q24</td>
<td>What happens if team members, reviewers or users find a mistake in the analysis?<br><br>Do you have a clear and efficient process for addressing issues and preventing them from happening again?</td>
<td>Analyst, analytical assurer</td>
<td>The analyst should use a risk-based approach to highlight the
areas of greatest potential error and focus assurance efforts on these areas.</td>
</tr>
<tr>
<td>Q25</td>
<td>Have you assessed uncertainty?</td>
<td>Commissioner, analyst</td>
<td>Commissioners should expect and require information about
uncertainty from analysts. They should challenge them when it is absent, inadequate or ambiguous.<br><br>Commissioners may have identified sources of uncertainty as part of their wider considerations and should share them with the analyst. If the commissioner can explain in advance the impact on decision-making of different degrees of uncertainty, this can help the analyst to design and carry out the analysis at a proportionate level.</td>
</tr>
</table>
:::
# III. Doing and Checking Analysis
::: {.panel-tabset}
## Quality questions and why they matter
<table class="simpleTable" cellspacing="0" cellpadding="0">
<TR VALIGN=TOP>
<th scope = "col"> </th>
<th scope = "col"> Quality Question</th>
<th scope = "col"> Why do I need to know the answer to this?</th>
</tr>
<tr>
<td>Q26</td>
<td>How will the data in the analysis be processed before and during use?</td>
<td>Processing the data inputs will impact methods and outputs. A clear understanding of how these processes affect the workflow is essential for understanding quality.</td>
</tr>
<tr>
<td>Q27</td>
<td>Is the data appropriate given the methods selected?</td>
<td>A comprehensive understanding of data inputs is a prerequisite
for meeting user needs.</td>
</tr>
<tr>
<td>Q28</td>
<td>What are the strengths and limitations of the data you use?</td>
<td>Data requirements for analysis vary. Formats, coverage, time scales and granularity must all be appropriate for the research question.<br><br>A comprehensive understanding of data inputs is a prerequisite for meeting user needs. Without understanding the strengths and weaknesses of the data, it is impossible to make meaningful improvements to the analysis or the inputs to manage these limitations.</td>
</tr>
<tr>
<td>Q29</td>
<td>Is there a robust relationship between your team and data providers?<br><br>Do data providers understand how and why you use their data?</td>
<td>A good relationship with data suppliers helps to make sure that their data meet your requirements. Lack of communication can mean you are not aware of quality risks or changes in collection or processing steps that can affect your results.<br><br>You should communicate with your suppliers sufficiently to manage input quality. Data providers should have a good understanding of how and why you are using their data. This helps them to improve data quality and value and find and address gaps or issues that are relevant for your analysis.</td>
</tr>
<tr>
<td>Q30</td>
<td>Do you understand how data providers collect, process and quality assure the data you use?</td>
<td>Never assume that datasets are of sufficient quality. Make sure that suppliers give you the metadata and other supporting information you need to assure the quality of the data. Validate the information provided by suppliers using your own checks and confirmation if appropriate.</td>
</tr>
<tr>
<td>Q31</td>
<td>Is there a formal agreement to set out data content, when and how you will get the data? If not, why not?</td>
<td>A formal service level agreement with data providers makes sure that everybody understands what will be delivered, when and how. This is useful for setting out the division of responsibilities between data providers and your team for getting and sharing the data. It might specify formats, delivery, timescale, legal framework, accompanying metadata and quality checks.</td>
</tr>
<tr>
<td>Q32</td>
<td>Do you know what quality checks are carried out on the data before you receive them?</td>
<td>Data suppliers should be able to show that their data is sufficiently assured to meet your needs. You should be able to demonstrate that the data meet your needs and that reported quality matches what you observe in practice. Simply having a quality report is not enough.</td>
</tr>
<tr>
<td>Q33</td>
<td>How will you work with your data provider when your data
requirements change?</td>
<td>Review your data requirements regularly to ensure they are still relevant and feasible. Changes to requirements should be communicated to data providers well in advance and agreed by all stakeholders. If there is a formal agreement, it may need to be revised as requirements change.</td>
</tr>
<tr>
<td>Q34</td>
<td>How do you know if your data provider changes their systems or processes in a way that could impact the data you receive or the analysis you produce?</td>
<td>Data suppliers make changes to their definitions, methods and
systems. This may not affect the quality of their data but can affect how you process the data and what you can infer from it. Tailor communication with data suppliers so it is sufficiently frequent, effective and ongoing to get timely information about changes.</td>
</tr>
<tr>
<td>Q35</td>
<td>How did you choose the methods for the analysis? How do you know
the methods you use are appropriate?</td>
<td>You should be able to explain why you chose your methods. For
each method, document the underlying assumptions, why the method is suitable for answering the analysis question, why it is applicable to the type and distribution of data you are using, and how these decisions were signed off.</td>
</tr>
<tr >
<td>Q36</td>
<td>Have reasonable alternative methods been explored and rejected for good reasons?</td>
<td>There is often more than one way to answer a question with data.
When you have made choices about methods and approaches, explain how and why you considered and rejected other options. Unless there is evidence underpinning your choice, users cannot be sure that you have chosen the most suitable methods.</td>
</tr>
<tr>
<td>Q37</td>
<td>How do you know that your analysis is working correctly?</td>
<td>You need to be sure that your analysis produces the outputs you think it should and the processes run as expected. If you cannot demonstrate that scripts and processes work correctly, you cannot confirm the quality of the results.</td>
</tr>
<tr>
<td>Q38</td>
<td>Can you describe the assumptions of your analysis, when they were made and who made them and signed them off?</td>
<td>You must understand the assumptions your analysis makes. Assumptions set out how the analysis simplifies the world and mitigates uncertainty. If assumptions are inadequately set out or absent, important characteristics of the analysis and its inputs will be unclear, greatly increasing risk.<br><br>Without a comprehensive log of assumptions made by the analysis, an audit trail signed off by assumption owners, a version control log reflecting when assumptions were last updated, and evidence showing internal and external validation of assumptions, uncertainties may go unacknowledged and could drastically impact outputs.</td>
</tr>
<tr>
<td>Q39</td>
<td>How are assumptions validated and assured before you apply them?</td>
<td>A clear understanding of how assumptions have been externally and internally validated and signed off gives us confidence that they are reasonable.</td>
</tr>
<tr>
<td>Q40</td>
<td>How do you measure and report uncertainty in your analysis?</td>
<td>All analysis contains uncertainty. Quantifying and reporting uncertainty means we can inform users how precise reported values are and how much confidence they can have in the analysis. It will also help you to determine where the analysis can be improved.<br><br>There are many ways to quantify uncertainty in input data, assumptions, processes and outputs. Choose appropriate ones for your situation. For instance, uncertainties can be understood and quantified by comparing against similar or historical data, Monte Carlo simulation, break-even analysis or using expert judgement.</td>
</tr>
<tr>
<td>Q41</td>
<td>Have you considered the implications of relevant, unquantified uncertainties?</td>
<td>A good understanding of the uncertainties in the analysis workflow is critical to ensure the analysis and its outputs are fit for purpose.</td>
</tr>
<tr>
<td>Q42</td>
<td>Can you explain the impact of your analysis on downstream processes? Are there any risks around these dependencies?</td>
<td>Understanding how your analysis might be used would help ensure
that the right quality and assurance levels are in place. It would help you assess the risks around the use of analysis and if there are other stakeholders or users you need to consult.</td>
</tr>
<tr>
<td>Q43</td>
<td>Is all or part of the analysis reliant on a single person?</td>
<td>Single points of failure carry significant business risk. If only one person understands how to carry out all or part of the analysis or maintain the code then the process is extremely vulnerable.</td>
</tr>
<tr>
<td>Q44</td>
<td>Is it clear why important decisions about the analysis were made, who made them and when?</td>
<td>All analysis involves decisions. A comprehensive record of the decisions made in specifying and conducting the analysis ensures a full audit trail of why decisions were made, who made them and signed them off.</td>
</tr>
<tr>
<td>Q45</td>
<td>If changes need to be made to code or datasets, is it easy to track who made the changes and when and why they were made?</td>
<td>Good version control ensures a full understanding of when, why, and how changes were made to your analysis process. If it is hard to track changes, it will be hard to retrace steps if there is a problem and means you do not fully understand the process.</td>
</tr>
<tr>
<td>Q46</td>
<td>Would another analyst be able to reproduce your analysis output or continue the work without talking to you first?</td>
<td>Your analysis must be well documented and repeatable so that
somebody new can understand it, use it, and produce the same output with
the same inputs. Poor documentation can lead to errors.</td>
</tr>
<tr>
<td>Q47</td>
<td>Do you use internal peer review to check scripts and code, documentation, implementation of methods, processes and outputs?</td>
<td>You should independently review and validate the logical integrity of you analysis as well as the structure and functionality of the code against the research question.<br><br>A record of validation and verification activities undertaken, outstanding tasks and remedial actions helps to confirm that the correct analysis has been performed for the required purpose and the chosen approach minimises risk.</td>
</tr>
<tr>
<td>Q48</td>
<td>Is your code and analysis ever peer reviewed by someone outside your team or organisation?</td>
<td>External peer review is one of the best ways to ensure that the analysis and code are well made and fit for purpose. Without it, teams can reinforce their own biases and may not notice there is anything
wrong.</td>
</tr>
<tr>
<td>Q49</td>
<td>What is your assessment of the quality of your analytical
outputs?</td>
<td>Understanding and reporting on the quality of your analysis is critical to ensure fitness for purpose and maintain trust and reliability. This ensures that analysis can appropriately inform decision-making. Quality assessments are key information to share with users and a requirement of the Code of Practice for Statistics.</td>
</tr>
<tr>
<td>Q50</td>
<td>How do you assure yourselves that analysis you do is correct?</td>
<td>If you check that results fit with your expectations and you can explain discrepancies, this makes it easier to mitigate risk.<br><br>There are many ways to check if analysis is carried out correctly. For example, sensitivity analysis can help you understand which inputs have the greatest effect on the outputs.You can also compare figures from the analysis with similar data from other sources or from historical series.</td>
</tr>
<tr>
<td>Q51</td>
<td>Do the outputs of your analysis align with similar findings from
elsewhere? If not, can you explain why?</td>
<td> If you can, check that your outputs align with findings from previous runs of the analysis, alternate data sources, and comparable
studies. This gives you confidence that the analysis works as expected. You should be able to explain any inconsistencies that you see.</td>
</tr>
<tr>
<td>Q52</td>
<td>If you find outliers or unusual trends in the data, what steps do you take to investigate them?</td>
<td>It is crucial to check unusual trends and values in the data and
understand why they are there. Not all outliers are the same. Some have a strong influence, some not at all. Some are valid and important data values. Others might be errors.<br><br>Investigate outliers and unusual patterns thoroughly and take reasonable steps to check their impact on your final output. If you choose to exclude unusual values, you should explain why this is acceptable.</td>
</tr>
</table>
## The questions and the Code of Practice
<table class="simpleTable" cellspacing="0" cellpadding="0" >
<TR VALIGN=TOP>
<th scope = "col"> </th>
<th scope = "col"> Quality Question</th>
<th scope = "col"> Which pillar and principle of Code of Practice are relevant here?
*Trustworthiness (T), Quality (Q), Value (V)</th>
</tr>
<tr>
<td>Q26</td>
<td>How will the data used in the analysis be processed before and during use?</td>
<td>Q1 Statistics should be based on the most appropriate data to meet intended uses. The impact of any data limitations for use should be assessed, minimised and explained.</td>
</tr>
<tr>
<td>Q27</td>
<td>Is the data appropriate given the methods selected?</td>
<td>Q1 Statistics should be based on the most appropriate data to
meet intended uses. The impact of any data limitations for use should be
assessed, minimised and explained.</td>
</tr>
<tr>
<td>Q28</td>
<td>What are the strengths and limitations of the data you use?</td>
<td>Q1 Statistics should be based on the most appropriate data to meet intended uses. The impact of any data limitations for use should be assessed, minimised and explained.
<br><br>Q1.5 Potential bias, uncertainty and possible distortive effects in the source data should be identified and the extent of any impact on the statistics should be clearly reported.</td>
</tr>
<tr>
<td>Q29</td>
<td>Is there a robust relationship between your team and data providers?<br><br>Do data providers understand how and why you use their data?</td>
<td>Q1.2 Statistics producers should establish and maintain constructive relationships with those involved in the collection, recording, supply, linking and quality assurance of data, wherever possible.</td>
</tr>
<tr>
<td>Q30</td>
<td>Do you understand how data providers collect, process and quality assure the data you use?</td>
<td>Q1.1 Statistics should be based on data sources that are appropriate for the intended uses. The data sources should be based on definitions and concepts that are suitable approximations of what the statistics aim to measure, or that can be processed to become suitable for producing the statistics.<br><br>
Q1.6 The causes of limitations in data sources should be identified and addressed where possible. Statistics producers should be open about the extent to which limitations can be overcome and the impact on the statistics.</td>
</tr>
<tr>
<td>Q31</td>
<td>Is there a formal agreement to set out data content, when and how you will get the data? If not, why not?</td>
<td>Q1.3 A clear statement of data requirements should be shared with the organisations that provide that data, setting out decisions on timing, definitions and format of data supply, and explaining how and why the data will be used.</td>
</tr>
<tr>
<td>Q32</td>
<td>Do you know what quality checks are carried out on the data before you receive them?</td>
<td>Q1 Statistics should be based on the most appropriate data to meet intended uses. The impact of any data limitations for use should be assessed, minimised and explained.</td>
</tr>
<tr>
<td>Q33</td>
<td>How will you work with your data provider when your data requirements change?</td>
<td>Q1.2 Statistics producers should establish and maintain
constructive relationships with those involved in the collection, recording, supply, linking and quality assurance of data, wherever possible.</td>
</tr>
<tr>
<td>Q34</td>
<td>How do you know if your data provider changes their systems or processes in a way that could impact the data you receive or the analysis you produce?</td>
<td>Q1.2 Statistics producers should establish and maintain
constructive relationships with those involved in the collection, recording, supply, linking and quality assurance of data, wherever possible.<br><br>Q1.7 The impact of changes in the circumstances and context of a data source on the statistics over time should be evaluated. Reasons for any lack of consistency and related implications for use should be clearly explained to users.</td>
</tr>
<tr>
<td>Q35</td>
<td>How did you choose the methods for the analysis? How do you know the methods you use are appropriate?</td>
<td>Q2 Producers of statistics and data should use the best available methods and recognised standards, and be open about their decisions.</td>
</tr>
<tr>
<td>Q36</td>
<td>Have reasonable alternative methods been explored and rejected for good reasons?</td>
<td>Q2 Producers of statistics and data should use the best available methods and recognised standards, and be open about their decisions.</td>
</tr>
<tr>
<td>Q37</td>
<td>How do you know that your analysis is working correctly?</td>
<td>Q3 Producers of statistics and data should explain clearly how
they assure themselves that statistics and data are accurate, reliable,
coherent and timely.</td>
</tr>
<tr>
<td>Q38</td>
<td>Can you describe the assumptions of your analysis, when they
were made and who made them and signed them off?</td>
<td>Transparency means being clear and open about the choices you make and not holding back or being opaque about decisions.<br><br>Q1.6 The causes of limitations in data sources should be identified and addressed where possible. Statistics producers should be open about the extent to which limitations can be overcome and the impact on the statistics.<br><br>Q2.4 Relevant limitations arising from the methods and their application, including bias and uncertainty, should be identified and explained to users. An indication of their likely scale and the steps taken to reduce their impact on the statistics should be included in the explanation.</td>
</tr>
<tr>
<td>Q39</td>
<td>How are assumptions validated and assured before you apply them?</td>
<td>Q1.6 The causes of limitations in data sources should be identified and addressed where possible. Statistics producers should be open about the extent to which limitations can be overcome and the impact on the statistics.<br><br>Q2.4 Relevant limitations arising from the methods and their application, including bias and uncertainty, should be identified and explained to users. An indication of their likely scale and the steps taken to reduce their impact on the statistics should be included in the explanation.<br><br>Q3.5 Systematic and periodic reviews on the strengths and limitations in the data and methods should be undertaken. Statistics producers should be open in addressing the issues identified and be transparent about their decisions on whether to act.</td>
</tr>
<tr>
<td>Q40</td>
<td>How do you measure and report uncertainty in your analysis?</td>
<td>Q2.4 Relevant limitations arising from the methods and their application, including bias and uncertainty, should be identified and explained to users. An indication of their likely scale and the steps taken to reduce their impact on the statistics should be included in the explanation.</td>
</tr>
<tr>
<td>Q41</td>
<td>Have you considered the implications of relevant, unquantified uncertainties?</td>
<td>Q2.4 Relevant limitations arising from the methods and their application, including bias and uncertainty, should be identified and explained to users. An indication of their likely scale and the steps taken to reduce their impact on the statistics should be included in the explanation.<br><br>Q3.3 The extent and nature of any uncertainty in the estimates should be clearly explained.</td>
</tr>
<tr>
<td>Q42</td>
<td>Can you explain the impact of your work/output on downstream
processes? Are there any risks around these dependencies?</td>
<td>V1.1 Statistics producers should maintain and refresh their understanding of the use and potential use of the statistics and data. They should consider the ways in which the statistics might be used and the nature of the decisions that are or could be informed by them.</td>
</tr>
<tr>
<td>Q43</td>
<td>All or part of the analysis is reliant on a single person?</td>
<td>V4 Statistics producers should be creative and motivated to
improve statistics and data, recognising the potential to harness technological advances for the development of all parts of the production and dissemination process.<br><br>T4.3 Sufficient human, financial and technological resources should be provided to deliver statistical services that serve the public good.</td>
</tr>
<tr>
<td>Q44</td>
<td>Is it clear why important decisions were made and who made them?</td>
<td>Q2 Producers of statistics and data should use the best available methods and recognised standards, and be open about their decisions.</td>
</tr>
<tr>
<td>Q45</td>
<td>If changes need to be made to code or datasets, is it easy to track who made the changes and when and why they were made?</td>
<td>V4 Statistics producers should be creative and motivated to improve statistics and data, recognising the potential to harness technological advances for the development of all parts of the production and dissemination process.</td>
</tr>
<tr>
<td>Q46</td>
<td>Would another analyst be able to reproduce your analysis output or
continue the work without talking to you first?</td>
<td>V5 Statistics and data should be published in forms that enable their reuse. Producers should use existing data wherever possible and only ask for more where justified.</td>
</tr>
<tr>
<td>Q47</td>
<td>Do you use internal peer review to check scripts and code, documentation, implementation of methods, processes and outputs?</td>
<td>Q3 Producers of statistics and data should explain clearly how they assure themselves that statistics and data are accurate, reliable, coherent and timely.<br><br>T4.6 Independent measures, such as internal and external audit, peer review and National Statistics Quality Reviews, should be used to evaluate the effectiveness of statistical processes. Statistics producers should be open about identified areas for improvement.</td>
</tr>
<tr>
<td>Q48</td>
<td>Is your code and analysis ever peer reviewed by someone outside
your team or organisation?</td>
<td>Q3 Producers of statistics and data should explain clearly how they assure themselves that statistics and data are accurate, reliable, coherent and timely.<br><br>T4.6 Independent measures, such as internal and external audit, peer review and National Statistics Quality Reviews, should be used to evaluate the effectiveness of statistical processes. Statistics producers should be open about identified areas for improvement.</td>
</tr>
<tr>
<td>Q49</td>
<td>What is your assessment of the quality of your analytical outputs?</td>
<td>Q3 Producers of statistics and data should explain clearly how they assure themselves that statistics and data are accurate, reliable, coherent and timely.<br><br>Q3.2 Quality assurance arrangements should be
proportionate to the nature of the quality issues and the importance of the statistics in serving the public good. Statistics producers should be transparent about the quality assurance approach taken throughout the preparation of the statistics. The risk and impact of quality issues on statistics and data should be minimised to an acceptable level for the intended uses.</td>
</tr>
<tr>
<td>Q50</td>
<td>How do you assure yourself that the analysis you do is correct?</td>
<td>Q3 Producers of statistics and data should explain clearly how they assure themselves that statistics and data are accurate, reliable, coherent and timely.<br><br>Q3.2 Quality assurance arrangements should be
proportionate to the nature of the quality issues and the importance of the statistics in serving the public good. Statistics producers should be transparent about the quality assurance approach taken throughout the preparation of the statistics. The risk and impact of quality issues on statistics and data should be minimised to an acceptable level for the intended uses.</td>
</tr>
<tr>
<td>Q51</td>
<td>Do the outputs of your analysis align with similar findings from elsewhere? If not, can you explain why?</td>
<td>Q3 Producers of statistics and data should explain clearly how they assure themselves that statistics and data are accurate, reliable, coherent and timely.</td>
</tr>
<tr>
<td>Q52</td>
<td>If you find outliers or unusual trends in the data, what steps do you take to investigate them?</td>
<td>Q3 Producers of statistics and data should explain clearly how they assure themselves that statistics and data are accurate, reliable, coherent and timely.<br><br>Q3.3 The quality of the statistics and data, including their accuracy and reliability, coherence and comparability, and timeliness and punctuality, should be monitored and reported regularly. Statistics should be validated through comparison with other relevant statistics and data sources. The extent and nature of any uncertainty in the estimates should be clearly explained.</td>
</tr>
</table>
## Linking the questions to AQuA roles
<table class="simpleTable" cellspacing="0" cellpadding="0">
<TR VALIGN=TOP>
<th scope = "col"> </th>
<th scope = "col"> Quality Question</th>
<th scope = "col"> Which AQuA role(s) would normally answer this?</th>
<th scope = "col"> Why are these AQuA roles involved?</th>
</tr>
<tr>
<td>Q26</td>
<td>How will the data used in the analysis be processed before and during use?</td>
<td>Analyst, analytical assurer</td>
<td>The analyst should collect and manage the data. They must understand data accuracy and uncertainties and capture, manage and understand assumptions. Analytical assurer should check that data processing is sufficient to ensure fitness for purpose.</td>
</tr>
<tr >
<td>Q27</td>
<td>Is the data appropriate given the methods selected?</td>
<td>Analyst, analytical assurer, commissioner</td>
<td>The analyst should understand data accuracy and uncertainties and capture, manage and understand assumptions made. The analyst should engage appropriate subject matter experts at the appropriate time. The commissioner may be a subject matter expert. The analytical assurer should check that there is sufficient assurance around the choice of data.</td>
</tr>
<tr>
<td>Q28</td>
<td>What are the strengths and limitations of the data you use?</td>
<td>Analyst, analytical assurer</td>
<td>If applicable, analyst should undertake parametric analysis to understand the consequences of missing or uncertain data and assumptions. Analytical assurer should make sure there is sufficient consideration of strengths and limitations of data.</td>
</tr>
<tr>
<td>Q29</td>
<td>Is there a robust relationship between your team and data providers?<br><br>Do data providers understand how and why you use their data?<br><br>Do you understand how data providers collect, process and quality assure the data you use?<br>
</td>
<td>Analyst, analytical assurer</td>
<td>The analytical assurer should expect to see evidence that there has been sufficient dialogue between analysts and the providers of data and other evidence sources.</td>
</tr>
<tr>
<td>Q30</td>
<td>Do you understand how data providers collect, process and quality assure the data you use?</td>
<td>Analyst, analytical assurer</td>
<td>The analyst should ensure data formats, units, and context are properly understood and handled. They should design and implement quality checks to validate data inputs as required. Analytical assurer should verify that the right assurance is in place.</td>
</tr>
<tr>
<td>Q31</td>
<td>Is there a formal agreement to set out data content, when and how you will get the data? If not, why not?</td>
<td>Commissioner, analyst, analytical assurer</td>
<td>The commissioner may need to provide the analyst with agreement to use specific data. The analyst should ensure data formats, units, and context are properly understood and handled. Analytical assurer should verify that assurance is in place.</td>
</tr>
<tr>
<td>Q32</td>
<td>Do you know what quality checks are carried out on the data before you receive them?</td>
<td>Analyst</td>
<td>The analyst should understand data accuracy and uncertainties and capture, manage and understand implicit assumptions made. The analytical assurer should assess whether assurance is sufficient.</td>
</tr>
<tr>
<td>Q33</td>
<td>How will you work with your data providers when your data requirements change?</td>
<td>Commissioner, analyst, analytical assurer</td>
<td>During the design and conduct of analysis, the commissioner may need to provide the analyst with information, agreement to use resources or confirmation of assumptions or approach. The analyst should understand data accuracy and uncertainties and capture, manage and understand implicit assumptions made. The analytical assurer checks that assurance and mitigation are sufficient.</td>
</tr>
<tr>
<td>Q34</td>
<td>How do you know if your data provider changes their systems or processes in a way that could impact the data you receive or the analysis you produce?</td>
<td>Commissioner, analyst, analytical assurer</td>
<td>During the design and conduct of analysis, the commissioner provides the analyst with the information they need for the analysis to proceed. This could include agreement to use datasets, setting out of key assumptions and signing off assumptions developed during the project.
Analyst should understand data accuracy and uncertainties and capture, manage and understand assumptions made. The analytical assurer checks that assurance and mitigation are sufficient.
</td>
</tr>
<tr>
<td>Q35</td>
<td>How did you choose the methods for the analysis? How do you know the methods you use are appropriate?</td>
<td>Analyst, analytical assurer</td>
<td>During the design phase, the analyst will convert the commission into an analytical plan and will consider inputs, analytical methods and processes, and expected outputs. The analytical assurer should check that the proposed design meets the commissioner’s requirements and is sufficiently assured.</td>
</tr>
<tr>
<td>Q36</td>
<td>Have reasonable alternative methods been explored and rejected for good reasons?</td>
<td>Analyst, analytical assurer</td>
<td>The analyst should review the analysis as a whole and consider carefully whether there are other, better ways in which it could be done. The analytical assurer should check that the investigation of methods was sufficiently thorough and proportionate.</td>
</tr>
<tr>
<td>Q37</td>
<td>How do you know that your analysis is working correctly?</td>
<td>Analyst, analytical assurer</td>
<td>The analyst should validate that the analysis as set up to answer the specification of the commissioner. The analytical assurer checks that assurance and mitigation are sufficient so the analysis is fit for purpose.</td>
</tr>
<tr>
<td>Q38</td>
<td>Can you describe the assumptions of your analysis, when they were made and who made them and signed them off?</td>
<td>Analyst, analytical assurer, commissioner</td>
<td>The analyst should capture, manage and understand explicit and implicit assumptions made. The analytical assurer should assess whether these are sufficient. The commissioner should be made aware of key assumptions and confirm that they are happy that the assumptions are applied.</td>
</tr>
<tr>
<td>Q39</td>
<td>How are assumptions validated and assured before you apply them?</td>
<td>Analyst, analytical assurer</td>