Skip to content

Latest commit

 

History

History
241 lines (185 loc) · 12.6 KB

README_EN.md

File metadata and controls

241 lines (185 loc) · 12.6 KB
Raw

BallonTranslator

简体中文 | English | Русский | 日本語 | Indonesia

Yet another computer-aided comic/manga translation tool powered by deep learning.

preview

Features

  • Fully automated translation

    • Support automatic text-detection, recognition, removal, and translation. Overall performance is dependent upon these modules.
    • Lettering is based on the formatting estimation of the original text.
    • Works decently with manga and comics.
    • Improved manga->English, English->Chinese typesetting (based on the extraction of balloon regions.).

  • Image editing

    • Support mask editing & inpainting (something like spot healing brush tool in PS)
    • Adapted to images with extreme aspect ratio such as webtoons

  • Text editing

    • Support rich text formatting and text style presets, translated texts can be edited interactively.
    • Support search & replace
    • Support export/import to/from word documents

Installation

On Windows

If you don't want to install Python and Git by yourself and have access to the Internet:
Download BallonsTranslator_dev_src_with_gitpython.7z fromMEGA or Google Drive, unzip it and run launch_win.bat.
Run scripts/local_gitpull.bat to get the latest update.

Run the source code

Install Python < 3.12 (dont use the one installed from microsoft store) and Git.

# Clone this repo
$ git clone https://github.com/dmMaze/BallonsTranslator.git ; cd BallonsTranslator

# Launch the app
$ python3 launch.py

Note the first time you launch it will install the required libraries and download models automatically. If the downloads fail, you will need to download the data folder (or missing files mentioned in the terminal) from MEGA or Google Drive and save it to the corresponding path in source code folder.

Build macOS application (compatible with both intel and apple silicon chips)

Note macOS can also run the source code if it didn't work.

录屏2023-09-11 14 26 49

1. Preparation

截屏2023-09-08 13 44 55_7g32SMgxIf
  • Put all the downloaded resources into a folder called data, the final directory tree structure should look like:
data
├── libs
│   └── patchmatch_inpaint.dll
└── models
    ├── aot_inpainter.ckpt
    ├── comictextdetector.pt
    ├── comictextdetector.pt.onnx
    ├── lama_mpe.ckpt
    ├── manga-ocr-base
    │   ├── README.md
    │   ├── config.json
    │   ├── preprocessor_config.json
    │   ├── pytorch_model.bin
    │   ├── special_tokens_map.json
    │   ├── tokenizer_config.json
    │   └── vocab.txt
    ├── mit32px_ocr.ckpt
    ├── mit48pxctc_ocr.ckpt
    └── pkuseg
        ├── postag
        │   ├── features.pkl
        │   └── weights.npz
        ├── postag.zip
        └── spacy_ontonotes
            ├── features.msgpack
            └── weights.npz

7 directories, 23 files
  • Install pyenv command line tool for managing Python versions. Recommend installing via Homebrew.
# Install via Homebrew
brew install pyenv

# Install via official script
curl https://pyenv.run | bash

# Set shell environment after install
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc

2、Build the application

# Enter the `data` working directory
cd data

# Clone the `dev` branch of the repo
git clone -b dev https://github.com/dmMaze/BallonsTranslator.git

# Enter the `BallonsTranslator` working directory
cd BallonsTranslator

# Run the build script, will ask for password at pyinstaller step, enter password and press enter
sh scripts/build-macos-app.sh

📌The packaged app is at ./data/BallonsTranslator/dist/BallonsTranslator.app, drag the app to macOS application folder to install. Ready to use out of box without extra Python config.

To use Sugoi translator(Japanese-English only), download offline model, move "sugoi_translator" into the BallonsTranslator/ballontranslator/data/models.

Usage

It is recommended to run the program in a terminal in case it crashed and left no information, see the following gif.

  • The first time you run the application, please select the translator and set the source and target languages by clicking the settings icon.
  • Open a folder containing images of a comic (manga/manhua/manhwa) that need translation by clicking the folder icon.
  • Click the Run button and wait for the process to complete.

The font formats such as font size and color are determined by the program automatically in this process, you can predetermine those formats by change corresponding options from "decide by program" to "use global setting" in the config panel->Lettering. (global settings are those formats shown by the right font format panel when you are not editing any textblock in the scene)

Image Editing

Inpaint Tool

Image Editing Mode, Inpainting Tool

rect tool

Rect Tool

To 'erase' unwanted inpainted results, use the inpainting tool or rect tool with your right button pressed.
The result depends on how accurately the algorithm ("method 1" and "method 2" in the gif) extracts the text mask. It could perform worse on complex text & background.

Text editing

Text Editing Mode

Batch Text Formatting & Auto Layout

OCR & Translate Selected Area

Shortcuts

  • A/D or pageUp/Down to turn the page
  • Ctrl+Z, Ctrl+Shift+Z to undo/redo most operations. (note the undo stack will be cleared after you turn the page)
  • T to text-editting mode (or the "T" button on the bottom toolbar).
  • W to activate text block creating mode, then drag the mouse on the canvas with the right button clicked to add a new text block. (see the text editing gif)
  • P to image-editting mode.
  • In the image editing mode, use the slider on the right bottom to control the original image transparency.
  • Disable or enable any automatic modules via titlebar->run, run with all modules disabled will re-letter and re-render all text according to corresponding settings.
  • Set parameters of automatic modules in the config panel.
  • Ctrl++/Ctrl+- (Also Ctrl+Shift+=) to resize image.
  • Ctrl+G/Ctrl+F to search globally/in current page.
  • 0-9 to adjust opacity of lettering layer
  • For text editing: bold - Ctrl+B, underline - Ctrl+U, Italics - Ctrl+I
  • Set text shadow and transparency in the text style panel -> Effect.

Headless mode (Run without GUI)

python launch.py --headless --exec_dirs "[DIR_1],[DIR_2]..."

Note the configuration (source language, target language, inpaint model, etc) will load from config/config.json.
If the rendered font size is not right, specify logical DPI manually via --ldpi , typical values are 96 and 72.

Automation modules

This project is heavily dependent upon manga-image-translator, online service and model training is not cheap, please consider to donate the project:

Sugoi translator is created by mingshiba.

Text detection

OCR

  • All mit* models are from manga-image-translator, support English, Japanese and Korean recognition and text color extraction.
  • manga_ocr is from kha-white, text recognition for Japanese, with the main focus being Japanese manga.
  • Support using OCR from Starriver Cloud (Tuanzi Manga OCR). Username and password need to be filled in, and automatic login will be performed each time the program is launched.
    • The current implementation uses OCR on each textblock individually, resulting in slower speed and no significant improvement in accuracy. It is not recommended. If needed, please use the Tuanzi Detector instead.
    • When using the Tuanzi Detector for text detection, it is recommended to set OCR to none_ocr to directly read the text, saving time and reducing the number of requests.
    • For detailed instructions, see Tuanzi OCR Instructions (Chinese only)

Inpainting

Translators

Available translators: Google, DeepL, ChatGPT, Sugoi, Caiyun, Baidu. Papago, and Yandex.

  • Google shuts down translate service in China, please set corresponding 'url' in config panel to *.com.
  • Caiyun, ChatGPT, Yandex, Baidu, and DeepL translators needs to require a token or api key.
  • DeepL & Sugoi translator (and it's CT2 Translation conversion) thanks to Snowad14.
  • Sugoi translates Japanese to English completely offline.
  • Sakura-13B-Galgame

To add a new translator, please reference how_to_add_new_translator, it is simple as subclass a BaseClass and implementing two interfaces, then you can use it in the application, you are welcome to contribute to the project.

FAQ & Misc

  • If your computer has an Nvidia GPU or Apple silicon, the program will enable hardware acceleration.
  • Add support for saladict (All-in-one professional pop-up dictionary and page translator) in the mini menu on text selection. Installation guide
  • Accelarate performance if you have a NVIDIA's CUDA or AMD's ROCm device as most modules uses PyTorch.
  • Fonts are from your system's fonts.
  • Thanks to bropines for the Russian localization.
  • Added Export to photoshop JSX script by bropines.
    To read the instructions, improve the code and just poke around to see how it works, you can go to scripts/export to photoshop -> install_manual.md.