Got my hands on a lot of plain text books for training ML from bookcorpus and didn't know what all to do with them.
I hadn't experimented much with grep and quickly wore that out on the files. Then, I eventually moved on to NLTK for something more to do.
Feel free to add more plain text books to the \epubtxt\ directory and change the path variable within main.py to the desired book.
- Tokenize plain text books into words, cleans the words, then you can create frequency distribution plots and/or word clouds.
- Launch XLaunch (you got this when you installed VcXsrv), choose multiple windows, start no client, select Disable access control, add the additional parameters "-ac" (no quotes)
- run
git clone https://github.com/LewkyB/NLTK_Tools.git cd NLTK_Tools- run
docker build -t lewkb/nltktool:1.0 . - run
docker run -ti --rm -e DISPLAY=$DISPLAY --name nl lewkb/nltktool:1.0
If after running you received the error similiar to "_tkinter.TclError: couldn't connect to display "192.168.1.1:0.0"" you will you need changed your $DISPLAY variable in your ~/.bashrc.
I solved this by running ipconfig /all in cmd.exe and trying all the listed IPv4 addresses. I initially tried using export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}'):0 in my ~/.bashrc, but for some reason with my WSL1 the only nameserver I pull is 8.8.8.8. You can quickly test differnt IPv4 addresses by just entering export DISPLAY="your IPv4 address here":0.0 (ex. export DISPLAY=192.168.0.0.1:0.0). Run docker run -ti --rm -e DISPLAY=$DISPLAY --name nl lewkb/nltktool:1.0 each time you change the $DISPLAY.
Downloads are all very fast.
- download for 2gb zip that unpacks to 6gb, about 19000 books
- download the 37gb monster with 197000 books
- links are courtesy of this comment on Github