/
README.Rmd
191 lines (139 loc) · 6.2 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
[](https://www.gnu.org/licenses/gpl-3.0)
[](https://travis-ci.org/PolMine/trickypdf)
[](https://codecov.io/gh/PolMine/trickypdf/branch/master)
```{r, echo = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "README-"
)
```
# trickypdf
## Purpose
At a basic level, R users have efficient tools to extract text from pdf documents.
The [pdftools](https://CRAN.R-project.org/package=pdftools), and the [Rpoppler](https://CRAN.R-project.org/package=Rpoppler) packages are extremely useful and handy.
There are couple of issues that may arise when processing pdf that remain tricky. The *trickypdf*-package offers a class *PDF* to handle problems that reoccurringly cause headaches when processing pdf
documents. The class offers methods to...
* remove stuff outside the main text region (page headers, page numbers etc) as a preprocessing step.
* handle multi-column layouts;
* reconstruct lines of text, if the (OCRed) document has been scanned in tilted fashion;
* reconstruct paragraphs.
The output will be a valid XML document, with optional document metadata. XML
is generated to serve as the input to a Natural Language Processing (NLP) pipeline. A method to
create browsable html from the xmlified pdf document is meant to assist quality checking in
corpus preparation. The package thus makes handling tricky problems with processing pdf
documents much easier!
## Installation
The package relies on the pdftohtml command line utility that is part of the poppler library.
On Linux, installation is easy:
```{sh, eval = FALSE}
sudo apt-get install -y poppler-utils
```
On MacOS, it is recommended to use Homebrew as a package manager. To install brew, open a terminal
and install Homebrew.
```{sh, eval = FALSE}
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
```
The snippet is taken from the [Homebrew website](https://brew.sh). You can then install poppler as
follows,
```{sh, eval = FALSE}
brew install poppler
```
To check that poppler is installed, check the availability of the pdftohtml command line utility.
```{sh, eval = FALSE}
pdftohtml -v
```
The trickypdf-package imports from a set of standard R packages that are available from CRAN using
`install.packages`.
```{r, eval = FALSE}
install.packages(
pkgs = c("methods", "xml2", "pbapply", "Rpoppler", "htmltools", "markdown", "stringi", "plyr")
)
```
Finally, trickypdf can be installed from GitHub using devtools.
```{r, eval = FALSE}
install.packages("devtools") # if you have not yet installed devtools
devtools::install_github("PolMine/trickypdf")
```
## Usage
```{r}
library(trickypdf)
```
### Scenario 1: PDF with column titles, page numbers, footnotes
As a sample document, the package includes the United Nations Millenium Declaration (pdf version).
We create an instance of the PDF class by supplying the filename.
```{r}
doc <- system.file(package = "trickypdf", "extdata", "pdf", "UN_Millenium_Declaration.pdf")
Decl <- PDF$new(filename_pdf = doc)
```
Let us have a look at the document. The *show_pdf*-method will open a browser window with the document.
```{r}
Decl$show_pdf()
```
For the pages of the document, we define boxes that define the area of the text that we want to
retain.
```{r}
Decl$add_box(page = 1, box = c(left = 110, top = 290, width = 400, height = 415))
Decl$add_box(page = 2, box = c(left = 110, top = 80, width = 400, height = 644))
Decl$add_box(page = 3, box = c(left = 110, top = 80, width = 400, height = 580))
Decl$add_box(page = 4, box = c(left = 110, top = 80, width = 400, height = 570))
Decl$add_box(page = 5, box = c(left = 110, top = 80, width = 400, height = 550))
Decl$add_box(page = 6, box = c(left = 110, top = 80, width = 400, height = 515))
Decl$add_box(page = 7, box = c(left = 110, top = 80, width = 400, height = 550))
Decl$add_box(page = 8, box = c(left = 110, top = 80, width = 400, height = 580))
Decl$add_box(page = 9, box = c(left = 110, top = 80, width = 400, height = 290))
```
We now drop anything outside the boxes (i.e page numbers, headers and column titles, footnotes). We
get the remaining text, do some postprocessing and xmlify things. By generating a html document,
we perform a quality check.
```{r}
Decl$remove_unboxed_text_from_all_pages()
Decl$get_text_from_pages()
Decl$purge()
Decl$xmlify()
Decl$xml2html()
Decl$browse()
```
Looks ok? Let is save the file to disk.
```{r}
output <- tempfile(fileext = ".xml")
Decl$write(filename = output)
```
### Scenario 2: PDF with two columns.
The second scenario may appear somewhat more difficult: It is a protocol of a meeting of
the UN General Assembly, a document with a two column layout.
```{r, eval = TRUE}
doc <- system.file(package = "trickypdf", "extdata", "pdf", "UN_GeneralAssembly_2016.pdf")
UN <- PDF$new(filename_pdf = doc)
UN$show_pdf()
```
As in the previous example, we define boxes. But this time, we define a second box for each
page that does not replace the previously defined box.
```{r}
UN$add_box(page = 1, box = c(top = 380, height = 250, left = 52, width = 255))
UN$add_box(page = 1, box = c(top = 232, height = 400, left = 303, width = 255), replace = FALSE)
UN$add_box(page = 2, box = c(top = 80, height = 595, left = 52, width = 255))
UN$add_box(page = 2, box = c(top = 80, height = 595, left = 303, width = 255), replace = FALSE)
```
In the first scenario, we dropped anything outside the boxes and got everything that was still on the
page. Now we explicitly grab the text in the boxes.
```{r}
UN$get_text_from_boxes(paragraphs = TRUE)
UN$xmlify()
```
Let us inspect the html of the resulting xml.
```{r}
UN$xml2html()
UN$browse()
```
Does it look ok? Let's save it to disk.
```{r}
output <- tempfile(fileext = ".xml")
UN$write(filename = output)
```
## Contributing to package development
Please note that this project is released with a [Contributor Code of Conduct](CONDUCT.md). By participating in this project you agree to abide by its terms.