Skip to content

daimrod/csa

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Quickstart

You will need Maven 3 and Java 8 to build the project. The other dependencies such as UIMA, uimaFIT, StanfordNLP and OpenNLP are handled by maven. You can find a list of the dependencies in the pom.xml.

  1. Clone the repository. git clone https://github.com/daimrod/csa.git
  2. Retrieve PubMed Open Access Corpus and CSV information. (more information)
    • ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/articles.A-B.tar.gz
    • ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/articles.C-H.tar.gz
    • ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/articles.I-N.tar.gz
    • ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/articles.O-Z.tar.gz
    • ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv
  3. Extract the corpus in the directory of your choice, we will suppose it’s in \~/corpus.
  4. Go to project directory and build it using maven. mvn -Ddev=true package
  5. Create a file annotator.conf with the following information:
    inputDirectory
    the directory containing the PubMed Corpus
    outputDirectory
    the directory used to store the results
    listArticlesFilename
    a file containing the name of the articles to read
    mappingFilename
    a file describing the patterns used (more on this later)
    windowSize
    the size of the citation context window
  6. Run the Annotator. java -cp target/csa-1.0-SNAPSHOT.jar jgreg.internship.nii.WF.AnnotatorWF -config annotator.conf
  7. Create a file statistic.conf with the following information:
    inputDirectory
    the directory containing the results previously computed
    mappingFilename
    same as before
    outputFile
    the file containing the extracted statistics
    infoFile
    the file containing some additional information
  8. Run the Statistic module. java -cp target/csa-1.0-SNAPSHOT.jar jgreg.internship.nii.WF.StatisticsWF -config statistic.conf

Files format

Configuration files

Here is a example of the annotator.conf file:

inputDirectory = ~/corpus/
outputDirectory = ~/workspace/output/
listArticlesFilename = ~/workspace/mylist.txt
mappingFilename = ~/workspace/hs-mapping.lst
windowSize = 1

The statistic.conf file has the exact same syntax:

inputDirectory = ~/workspace/output/
mappingFilename = ~/workspace/hs-mapping.lst
outputFile = ~/workspace/output/all-out.dat
infoFile = ~/workspace/output/info.dat

Mapping file

The mapping file is used to describe the order of the annotation in the results and where to find the cue phrases for each annotation.

order = negative neutral positive

# Sentiment cues phrases
negative = ~/workspace/negative.pat
neutral = ~/workspace/neutral.pat
positive = ~/workspace/positive.pat

Patterns files

The Annotator module uses the Stanford NLP Token Sequence Matcher to match cues pharses. You can find a description of the accepted syntax here.

The pattern files must have one pattern per line, here are some examples of the accepted patterns:

good
/state-of-the-art/
{ tag:"NN" } achieve

How to run it in parallel?

You can dispatch the processing on N processes by splitting the list of articles in N chunks (e.g. using the split(1) command) and using the GNU Parallel tool.

For example, to use 20 cores:

split -n l/20 path/to/listArticlesFilename list-
ls list-* | parallel --halt 2 \
                     java -cp target/csa-1.0-SNAPSHOT.jar \
                     jgreg.internship.nii.WF.AnnotatorWF \
                     -config annotator.conf \
                     -listArticlesFilename {}

The split(1) command will split the file listArticlesFilename in 20 files prefixed by “list-“. We then use the parallel(1) command to run as many java processes as input file and overriding the listArticlesFilename parameter from the configuration file using a command line parameter.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published