Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V1 for Automation Website translations Issues Generation Ref: Issue 325 #407

Open
wants to merge 60 commits into
base: main
Choose a base branch
from

Conversation

sergioarmgpl
Copy link
Contributor

Hi Dear TAG Env Team, this is the final the work that we are doing with @Dianmz.

This includes a workflow called Check outdated content, that generates new Issues for Languages for ES & ZH languages at the moment. Currently the issues are assigned to Diana and I, but should be modified to the new groups that you suggested or new ones.

The issues will look like this:
Screenshot 2024-04-30 at 22 05 41

And inside the issue will look like this:
Screenshot 2024-04-30 at 22 05 53

cc: @leonardpahlke @guidemetothemoon

Sergio Méndez and others added 30 commits April 3, 2024 22:45
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Dianmz <dcmc2297@gmail.com>
Signed-off-by: Dianmz <dcmc2297@gmail.com>
Signed-off-by: Dianmz <dcmc2297@gmail.com>
Signed-off-by: Dianmz <dcmc2297@gmail.com>
Signed-off-by: Dianmz <dcmc2297@gmail.com>
Signed-off-by: Dianmz <dcmc2297@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>

feat: added branches main to check outdated content workflow
Signed-off-by: leonardpahlke <leonard.pahlke@googlemail.com>
Signed-off-by: leonardpahlke <leonard.pahlke@googlemail.com>
Also includes translation of search.md file

Signed-off-by: thelooter <evekolb2204@gmail.com>
guidemetothemoon and others added 18 commits April 30, 2024 21:54
Signed-off-by: Kristina Devochko <guidemetothemoon@gmail.com>
Signed-off-by: leonardpahlke <leonard.pahlke@googlemail.com>
…cncf#376)

* Add German translation for _index.md

Signed-off-by: Rishikesh <rishikeshjagadale19@gmail.com>
Signed-off-by: Kristina Devochko <kristina.devochko@tietoevry.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>

feat: updated check outdated content
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>

feat: added v2 for create issue

feat: changes on workflow
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>

feat: change on es
…nguages (#40)

* change file

* adding base

* changes

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* feat: Adding  Assigneeses

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* labels

* test

* test2

* test3

* change to en

* fix: comment step

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* feat: testing various languages

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* test: comparing languages

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* test: testing changes

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* feat: creating issue for zh language

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* test: testing all languages

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* feat: testing a new step

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* feat: testinbg more languages

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* fix: iteration

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* feat: changes

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* fix: iteration

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* text: changes

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* test: new call

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* fix: calling a new yml

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* feat: adding create issue file

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* feat: calling the file

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* feat: creating issues

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* fix:CURL

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* fix:adding credentials

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* feat:testing

Signed-off-by: Dianmz <dcmc2297@gmail.com>

* Update check-outdated-content.yml

Signed-off-by: Diana Menendez <61333059+Dianmz@users.noreply.github.com>

* feat:final details per language

Signed-off-by: Dianmz <dcmc2297@gmail.com>

---------

Signed-off-by: Dianmz <dcmc2297@gmail.com>
Signed-off-by: Diana Menendez <61333059+Dianmz@users.noreply.github.com>
…hanges

Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Copy link

netlify bot commented May 1, 2024

Deploy Preview for tag-env-sustainability ready!

Name Link
🔨 Latest commit 26b422a
🔍 Latest deploy log https://app.netlify.com/sites/tag-env-sustainability/deploys/6631cea29d8e6c000828172c
😎 Deploy Preview https://deploy-preview-407--tag-env-sustainability.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Sergio Méndez added 8 commits April 30, 2024 22:23
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Signed-off-by: Sergio Méndez <sergioarm.gpl@gmail.com>
Copy link
Member

@leonardpahlke leonardpahlke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much! I added a couple of comments.

cc @nate-double-u you may be interested in this work too. It about automatically detecting if translations to the TAG website need to happen & opens issues for translators to do translations.

Comment on lines +4 to +5
---
name: Check outdated content
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all the work! Awesome that it works! Three comments from my side:

  1. We need documentation
  2. We may need to revise some parts of the script

Documentation

Could we add some documentation about this. We do not have any development guide for the website so lets create a new document WEBSITE_DEVELOPMENT.md which has a section about how we do translations. The document can be extended with additional content (website dev related stuff).

About the translations, it would be good to explain:

  1. How does it work / What happens if you make a change to the english version of the website
  2. maintenance steps
    2.1. Adding a new translation. (for example, French - which steps do we need to take to support translations)
    2.2. Which people are notified to facilitate translations and how to edit this information
  3. Explain what happens if the workflow is not followed; like:
    3.1. Changes are made to the Spanish side initially (our workflow does not work, right - this is ok! But it would be good to write a sentence about that)
    3.2. The Translation issue is closed, and no translation was made. In this case, the translation will stay out of sync, right. (this is fine too 😄 - but would be good to document it)
    3.3. ...
  4. How to work with the pipeline - how to test it if you make edits (step 1, step 2)

Adding this would be fantastic! (feel free to add to this if you feel it is relevant information)

Script / Code

For my taste, the script is too long to be inline. It's not just a few commands which we can understand quickly, it's a bit more complicated; so we need to move it. We can stick to shell scripts.

example:

    - name: Run script file
      run: |
         chmod +x ./public/scripts/test.sh
         ./public/scripts/test.sh
      shell: bash

After these changes were made, I will have a closer look at the logic. I added a few more comments to the code already.

🙌

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks true, I will add this to the contributors document.

- name: Detecting pending translations
shell: bash
run: |
##### DEBUG section, this will be removed later ###########
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says "remove later". It would be good to keep this logging functionality and enable these logs over a --debug flag.

Example code by an LLM (needs to be tested)

# Default debug level
DEBUG_LEVEL="info"

# Parse command-line options
while [[ "$1" != "" ]]; do
    case $1 in
        --debug)
            DEBUG_LEVEL="debug"
            ;;
        *)
            # Unknown option
            echo "Unknown option: $1"
            exit 1
            ;;
    esac
    shift
done

# Log function
_log() {
    local message="$1"
    local level="$2"

    # Determine if the message should be logged based on DEBUG_LEVEL
    case "$DEBUG_LEVEL" in
        debug)
            # Log all messages
            echo "$level: $message"
            ;;
        info)
            # Log only info and error messages
            if [[ "$level" == "info" || "$level" == "error" ]]; then
                echo "$level: $message"
            fi
            ;;
        error)
            # Log only error messages
            if [[ "$level" == "error" ]]; then
                echo "$level: $message"
            fi
            ;;
    esac
}

# Example usage:
_log "This is an info message" "info"
_log "This is a debug message" "debug"
_log "This is an error message" "error"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it would be spectacular if there is an --dry-run option. So we can see which issues would be opened without opening any. (but we dont need to add this - would be great tho 😄, very helpful debugging, i suppose). Example execution: translation.sh <ORG> <REPO> <PR NUMBER> --dry-run --debug

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, We were thinking about to have a script instead of put this in the pipeline, is a good idea.


OUTPUT_DIR="./outdated"
CONTENT_DIR="website/content"
languages=("es" "zh")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering how we make sure that we have minimal maintenance, adding new translations in the future. We could for example look at the repository and check the folder names https://github.com/cncf/tag-env-sustainability/tree/main/website/content - we have a folder en & zh & es &de, we establish a base with en (we dont need to open a PR for en since its already merged). This could eliminate referencing languages in the script.

Assigning people over groups could work this way too. We assign tag-env-translators-<LANGUAGE REFERENCE> so for example tag-env-translators-es.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we could work on it.

Comment on lines +46 to +48
for lang in ${languages[@]}; do
touch $lang.txt
done
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mind explaining this code? We create a txt file per translation language. So we end up with zh.txt, es.txt ... why do we need these files? Probably we add this info to the issue descriptions, right.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure we have to document that part.

Comment on lines +67 to +100
compare () {
FILE_PATH=$1
# Actually compare between the old and latest English content and log diff in the file
if [[ -f "${FILE_PATH}" ]]; then
# File exists
# Check changes
git diff ${OLD_BRANCH}..${LATEST_BRANCH} -- ${FILE_PATH} > temp.diff
if [[ -s "temp.diff" ]]; then
echo "(DEBUG) ${FILE_PATH} is outdated."
mkdir -p ${OUTPUT_DIR}/${FILE_PATH%/*}
mv temp.diff ${OUTPUT_DIR}/${FILE_PATH}
else
echo "check if ${FILE_PATH} exist in other languages"
for lang in ${languages[@]}; do
NEW_FILE_PATH=$(echo "${FILE_PATH}" | sed -e "s/${base_lang}\//${lang}\//g")
if [[ ! -e "${NEW_FILE_PATH}" ]]; then
echo "The file ${FILE_PATH} needs to be translated to $lang"
echo ${NEW_FILE_PATH} >> $lang.txt
fi
done
fi
else
echo "(DEBUG) ${FILE_PATH} does not exist."
fi
}

find $base_lang -iname "*.md" > files.txt
sort files.txt > files_temp.txt;mv files_temp.txt files.txt
INFILE=files.txt

while IFS= read -r line
do
compare "$line"
done < "$INFILE"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part makes me think if we may want to move the logic to smth like Go or Python, which handles more complex logic better IMO. Thoughts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that it could be less lines with bash, but let me document the script.

@thelooter
Copy link
Contributor

It looks like there was some commit mess-up, since there are some commits in here, that are already in master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants