data_quality.Rmd

---
title: "bupaR Docs | Data Quality"
---


```{r include = F}
library(bslib)
```

```{r echo = F, out.width="25%", fig.align = "right"}
knitr::include_graphics("images/icons/create.PNG")
```

***

# Data Quality 

```{r setup, include=FALSE}
library(daqapo)
library(dplyr)
data("hospital_actlog")
hospital_actlog <- activitylog(hospital_actlog)
data("hospital_events")
```


Despite the extensive opportunities that process mining techniques provide, the garbage in - garbage out principle still applies. Data quality issues are widespread in real-life data and can generate misleading results when used for analysis purposes. `daqapo` - Data Quality Assessment for Process-Oriented data - provides a set of assessment functions to identify a wide array of quality issues.


## Getting started

In the examples below, we use the dataset `hospital_actlog`, which is an artificial event log with data quality issues provided by `daqapo`.

```{r eval = F}
library(daqapo)
library(dplyr)
data("hospital_actlog")
data("hospital_events")
hospital_actlog <- activitylog(hospital_actlog)

```



```{r eval = F, echo = F}


## Activity Frequency Violations

# ```{r}
# hospital_actlog %>%
  # detect_activity_frequency_violations("Registration" = 1,
                                       "Clinical exam" = 1)
<!-- ``` -->


<!-- ## Activity Order Violations -->

# ```{r}
# hospital_actlog %>%
  # detect_activity_order_violations(activity_order = c("Registration", "Triage", "Clinical exam",
                                                      # "Treatment", "Treatment evaluation"))
<!-- ``` -->
```


## Attribute Dependencies

Detect violations of dependencies between attributes (i.e. condition(s) that should hold when (an)other condition(s) hold(s)).

Example: when the activity is "Registration", the originator should start with "Clerk". 

```{r}
hospital_actlog %>% 
  detect_attribute_dependencies(antecedent = activity == "Registration",
                                consequent = startsWith(originator,"Clerk"))
```

## Case ID Sequence Gaps

Detect gaps in the sequence of case identifiers.

```{r}
hospital_actlog %>%
  detect_case_id_sequence_gaps()
```

## Conditional Activity Presence

Check whether certain activities are present when a specific condition is satisfied. 

For example, if specialization is "TRAU", then the activity "Clinical exam" must take place. 

```{r}
hospital_actlog %>%
  detect_conditional_activity_presence(condition = specialization == "TRAU",
                                       activities = "Clinical exam")
```


## Duration Outliers

Detect duration outliers for particular activities. 

For example, the duration of "Treatment" should be within 1 standard deviation of its mean duration. 

```{r}
hospital_actlog %>%
  detect_duration_outliers(Treatment = duration_within(bound_sd = 1))
```

Or, the duration of "Treatment" should be within 0 to 15 minutes. 

```{r}
hospital_actlog %>%
  detect_duration_outliers(Treatment = duration_within(lower_bound = 0, upper_bound = 15))
```

## Inactive Periods

Detect periods of time in which no activity executions are recorded, using a threshold specified in minutes. 

For example, detect whether there are periods of more than 30 minutes without any activity executions. 

```{r}
hospital_actlog %>%
  detect_inactive_periods(threshold = 30)
```

## Incomplete Cases

Check whether there are cases that miss a specific activity. 

For example, does any of the cases miss the 5 listed activities?

```{r}
hospital_actlog %>%
  detect_incomplete_cases(activities = c("Registration","Triage","Clinical exam","Treatment","Treatment evaluation"))
```

## Incorrect Activity Names

Given a set of allowed activities, are there any activities that are incorrect?

```{r}
hospital_actlog %>%
  detect_incorrect_activity_names(allowed_activities = c("Registration","Triage","Clinical exam","Treatment","Treatment evaluation"))
```

## Missing Values 

Analyse the missing values of the log. This can be done in general, or at the level of activities or specific columns. 

```{r}
hospital_actlog %>%
  detect_missing_values()
```

```{r}
hospital_actlog %>% 
  detect_missing_values(level_of_aggregation = "activity")
```

```{r}
hospital_actlog %>% 
  detect_missing_values(
  level_of_aggregation = "column",
  column = "triagecode")
```

## Multiregistration

Detect whether there are multiple activity executions registered by the same resource (or for the same case), in a short period of time. This period of time can be specified with a threshold in seconds. 

```{r}
hospital_actlog %>%
  detect_multiregistration(threshold_in_seconds = 10)
```

## Overlaps

Check if a resource has performed two or more activities in parallel. 

```{r}
hospital_actlog %>%
  detect_overlaps()
```

## Related Activities

Check responded existence between two activities. If the antecedent activity is executed for a case, the consequent activity should be executed as well. Take a look at [Rule-Based Conformance Checking](https://bupaverse.github.io/docs/control_flow_analysis.html) for more control-flow rules that you can check.

```{r}
hospital_actlog %>%
  detect_related_activities(antecedent = "Treatment evaluation", 
                            consequent = "Treatment")
```

## Similar Labels

Check for similar labels in a specific column. Both the column and the maximum allowed edit distance for two labels to consider similar can be configured.

```{r}
hospital_actlog %>%
  detect_similar_labels(column_labels = "activity", max_edit_distance = 3)
```

## Time Anomalies

Detect activity executions with negative or zero duration.

```{r}
hospital_actlog %>%
  detect_time_anomalies()
```

## Unique Values

List all unique combinations of the specified columns. 

```{r}
hospital_actlog %>%
  detect_unique_values(column_labels = "activity")
```
```{r}
hospital_actlog %>%
  detect_unique_values(column_labels = c("activity", "originator"))
```

## Value Range Violations

Detect value range violation.

```{r}
hospital_actlog %>%
  detect_value_range_violations(triagecode = domain_numeric(from = 0, to = 5))
```


```{r footer, results = "asis", echo = F}
CURRENT_PAGE <-  stringr::str_replace(knitr::current_input(), ".Rmd",".html")
res <- knitr::knit_expand("_button_footer.Rmd", quiet = TRUE)
res <- knitr::knit_child(text = unlist(res), quiet = TRUE)
cat(res, sep = '\n')
```