Feat/emotions multimodal (video) #564

moon-strider · 2023-08-30T20:10:11Z

Bringing emotion detection annotator to dream.

WARNING: DO NOT DELETE BRANCH AFTER MERGING, THE LINKS TO feat/emotions FILES ARE IN SOME REPORT PAPERS AS OF 30.08.2023

dilyararimovna · 2023-08-31T09:13:38Z

annotators/emotion_detection/requirements.txt

+yarl==1.8.2
+zipp==3.15.0
+flask==2.2.2
+blinker==1.5.0


@anna-a-m эти зависимости все нужны?)

Не уверена, что нужны

redundant requirements removed

dilyararimovna · 2023-08-31T09:14:06Z

annotators/emotion_detection/Dockerfile

+
+RUN pip install gdown
+
+RUN git clone https://github.com/anna-a-m/MultimodalERC repo


pls pick up some particular commit because now it is not fixed at all

particular commit picked

dilyararimovna · 2023-08-31T09:14:14Z

annotators/emotion_detection/Dockerfile

+
+RUN apt install -y ffmpeg libsm6 libxext6
+
+RUN pip install gdown


versions added

dilyararimovna · 2023-08-31T09:14:18Z

annotators/emotion_detection/Dockerfile

+COPY . /src
+RUN pip install -r requirements.txt
+
+RUN apt install -y ffmpeg libsm6 libxext6


versions added

dilyararimovna · 2023-08-31T09:15:25Z

annotators/emotion_detection/service_configs/emotion_detection/service.yml

+    args:
+      SERVICE_PORT: 8025
+      SERVICE_NAME: emotion_detection
+      CUDA_VISIBLE_DEVICES: 0


if the model needs GPU, then CUDA_VISIBLE_DEVICES: 0 should be given to the environment not to build args

Removed from args

dilyararimovna · 2023-08-31T09:15:37Z

annotators/emotion_detection/service_configs/emotion_detection/service.yml

+  volumes:
+  - ./annotators/emotion_detection:/src/drm
+  ports:
+  - 8025:8025


occupied port. select another one

dilyararimovna · 2023-08-31T09:16:01Z

assistant_dists/dream_ocean/dev.yml

+  emotion-detection:
+    volumes:
+      - "./annotators/emotion_detection:/src/drm"
+      - "/home/dikanskiy/emotion_test:/src/datafiles"


nope this is illegal. Remove this line

removed (but one would still need to add some mapping to /src/datafiles or have a videofile in this directory inside the container in order for the annotator to work)

why can't you get a file as images (path from attirbutes of the current utterance and the file itself from FILES container)?

the input to this service is a video + text so I used a test mounted folder as we discussed with @oserikov

dilyararimovna · 2023-08-31T09:17:09Z

annotators/emotion_detection/test.py

+#     response = requests.post("http://0.0.0.0:8026/model", json={"personality": ["Hello world"]})
+#     assert response.status_code == 200
+#     assert response.json() == [{"EXT": 0, "NEU": 1, "AGR": 1, "CON": 0, "OPN": 1}]
+#     print("SUCCESS")


why tests are not added?

They are copied from personality_detection. Removed

dilyararimovna · 2023-08-31T09:17:58Z

annotators/emotion_detection/server.py

+
+@app.route("/model", methods=["POST"])
+def infer():
+    msg_text = request.json["last_human_utterances"][-1]["text"]


nope. You are getting a BATCH of samples. SO, you have to process them as a batch. And return a batch of responses.
Sometimes it would accept a batch of several elements (depends on agent). So, pls fix it.

redone like the personality_detection

dilyararimovna · 2023-08-31T14:03:40Z

@moon-strider you did not push actually.

annotators/emotion_detection/service_configs/emotion_detection/service.yml

dilyararimovna · 2023-09-04T07:32:49Z

assistant_dists/dream_ocean/dev.yml

@@ -144,4 +144,9 @@ services:
      - "./annotators/personality_detection:/src"
    ports:
      - 8026:8026
+  emotion-detection:
+    volumes:
+      - "./annotators/emotion_detection:/src/drm"


почему есть подпапка с названием drm? зачем она вообще

в dockerfile такой вообще папки не создается

Удалил

dilyararimovna · 2023-09-04T07:34:42Z

components/OQWoiqfnqwfnoqwib.yml

+  protocol: http
+  timeout: 15.0
+  url: http://emotion-detection:8045/model
+dialog_formatter: state_formatters.dp_formatters:last_human_bot_annotated_utterance


это другой форматтер.

Исправил

dilyararimovna · 2023-09-04T07:36:47Z

annotators/emotion_detection/server.py

+prefix_len = len(prefix)
+
+
+def jsonify_data(data: Any) -> Any:


а зачем, если есть полноценная функция jsonify, которую надо использовать

Делал как в personality_detection. Убрал эту функцию, нашёл аналог jsonify из fastapi

dilyararimovna · 2023-09-04T07:38:23Z

annotators/emotion_detection/server.py

+def process_video(video_path: str):
+    video_frames = get_frames(video_path)
+
+    video_model_name = "microsoft/xclip-base-patch32"


мне кажется, было бы круто названия используемых моделей вынести в build args

и параметры, типа длин и пр тоже в build args

Сделано

annotators/emotion_detection/server.py

assistant_dists/dream_ocean/dev.yml

dilyararimovna · 2023-09-04T10:37:06Z

annotators/emotion_detection/server.py

+        try:
+            text = msg_text[prefix_len:]
+            logger.info(f"Emotion Detection: {text}")
+            emotion = predict_emotion(text, "/src/datafiles/vid.mp4")


!!!!!!!!!!!! What is it?? what is the file?
If you are going to detect emotions using the corresponding file with voice, you are supposed to use FILES 3000 server (see multimodal dist), put the voice file to server, pass the fname or link to the file in human utterance attributes for every utterance

…port added

dilyararimovna · 2023-12-19T09:03:10Z

annotators/emotion_detection/service_configs/emotion_detection/environment.yml

@@ -0,0 +1,3 @@
+SERVICE_PORT: 8045
+SERVICE_NAME: emotion_detection
+CUDA_VISIBLE_DEVICES: 0


список аргументов не полный. См билд арги в соседнем файле

dilyararimovna · 2023-12-19T09:06:07Z

assistant_dists/dream_embodied/docker-compose.override.yml

@@ -10,6 +10,9 @@ services:
      ALWAYS_TURN_ON_ALL_SKILLS: 0
      LANGUAGE: EN

+  files:
+    image: julienmeerschart/simple-file-upload-download-server


зачем файл сервер добавляется в dream_embodied дистрибутив? елси в нем не используется он

Убрала
done

assistant_dists/dream_ocean/docker-compose.override.yml

dilyararimovna · 2023-12-19T09:08:26Z

assistant_dists/dream_ocean/pipeline_conf.json

+                "is_enabled": true,
+                "source": {
+                    "component": "components/OQWoiqfnqwfnoqwib.yml",
+                    "service": "annotators/emotion_detection/service_configs/emotion_detection"


тут должно быть через тире название (см по аналогии с другими компонентами). Папку также надо переназвать

В personality_detection тоже?

dilyararimovna · 2023-12-19T09:09:25Z

state_formatters/dp_formatters.py

    return [
        {
            "personality": [
                dialog["human_utterances"][-1]["annotations"].get(
-                    "spelling_preprocessing", dialog["human_utterances"][-1]["text"]
+                    dialog["human_utterances"][-1]["text"]


не верно написано, вы из аннотации делаете get по ключу =реплика пользователя

Полностью переделано
done

dilyararimovna · 2023-12-19T09:09:36Z

state_formatters/dp_formatters.py

+        {
+            "personality": [
+                dialog["human_utterances"][-1]["annotations"].get(
+                    dialog["human_utterances"][-1]["text"]


тоже самое

Полностью переделано
done

…ed erroes fixed

…nto feat/emotions

dilyararimovna · 2023-12-28T10:52:47Z

annotators/emotion_detection/Dockerfile

+RUN gdown 'https://drive.google.com/uc?id=1ZHw2MEbjLwU5A7EaNysgVHZlaWq4FJZG' -O '/data/bert-large-uncased_none_seed-42.pt'
+RUN gdown 'https://drive.google.com/uc?id=1XZVPI2ZvE0PbvhFMinsAp2z9SPPEsSjn' -O '/data/final_model.pt'
+RUN gdown 'https://drive.google.com/uc?id=1DMUHqbwqn-j9XIhoqnfSwqyqrlPMkVJe' -O '/data/XCLIP_Augmented.pt'
+RUN gdown 'https://drive.google.com/uc?id=1VxEQcso_Bsrzix1TykriJ2BLfUm9Gtx5' -O '/data/redundant_features.txt'


почему вы не хотите положить это в files.deeppavlov, как я предлагала?

Тут после Нового Года откроем новый пр и будут другие модели

annotators/personality_detection/Dockerfile

dilyararimovna · 2023-12-28T10:57:29Z

state_formatters/dp_formatters.py

@@ -509,6 +511,7 @@ def simple_formatter_service(payload: List):
    sent_rewrite_formatter, sent_segm_formatter, base_skill_selector_formatter
    """
    logging.info(f"answer {payload}")
+    logging.warning(f"answer {payload}")


удалите отладочный лог, пожалуйста)

oserikov · 2024-02-06T07:56:49Z

@dilyararimovna @rock-n-shrimproll why does emotional distribution not have files service in any of the docker-compose files? It simply won't work without it, am I wrong?

moon-strider added 6 commits August 28, 2023 16:01

emotion_detection initial commit

719cb90

many small fixes

c889829

gdown fix

a37bb53

major update to configs, small change in data receiving

bbebaa0

small typo fixed, logging added

b1449db

everything prepared for a pr

1518453

moon-strider requested review from dilyararimovna and oserikov August 30, 2023 20:10

moon-strider self-assigned this Aug 30, 2023

moon-strider added 4 commits August 30, 2023 23:15

conflict resolved, port for emotion_detection changed, codestyle fixed

2d0c946

codestyle fixed again

4e22e5a

yet another style fix

17603d1

go agane

43f918f

dilyararimovna requested changes Aug 31, 2023

View reviewed changes

moon-strider added 4 commits August 31, 2023 18:11

major changes for pr

db0d04b

flask -> fastapi, batch fix

59489ec

codestyle fixed

b76bdd5

redundant requirements removed

3514ad6

dilyararimovna requested changes Sep 4, 2023

View reviewed changes

another pr fix

6adffb0

dilyararimovna requested changes Sep 4, 2023

View reviewed changes

moon-strider added 8 commits September 21, 2023 16:15

added files:3000 to dream_ocean, emotion_detection external video sup…

7459812

…port added

fixed emotion_detection annotator

61b636d

emotion detection videofile support added

642646c

minor refactoring

b3ca3a4

style fix

4219a17

more pr fixes, more style fixed

2428782

even more pr fixes

d1dff5f

argument dockerfile fixes

1fddb31

another one

54d99c9

dilyararimovna changed the title ~~Feat/emotions~~ [WIP] Feat/emotions multimodal (video) Nov 14, 2023

moon-strider assigned rock-n-shrimproll Dec 9, 2023

Alexandra Dolidze and others added 4 commits December 11, 2023 19:44

new models added

348395b

Merge remote-tracking branch 'origin/dev' into feat/emotions

3ccd51e

minor changes

0d11584

formatters slightly changed

93a4947

dilyararimovna requested changes Dec 19, 2023

View reviewed changes

Alexandra Dolidze and others added 11 commits December 22, 2023 13:39

dp_formatter for emotion-detection and personality-detection + review…

ba19cf7

…ed erroes fixed

nano error fixed

9caeefb

port changed, redundant features file added to docker

58e5e31

port fixed(2)

d8ecfb7

port fixed(3)

437a551

Merge branch 'dev' into feat/emotions

9a41e4c

codestyle fixed

5347bc5

Merge branch 'feat/emotions' of https://github.com/deeppavlov/dream i…

70b2088

…nto feat/emotions

codestyle fixed 2

9150b11

personality-detection models added

890d7f8

minor changes

db14239

dilyararimovna requested changes Dec 28, 2023

View reviewed changes

Alexandra Dolidze and others added 3 commits December 29, 2023 15:32

personality detection models added to files.deeppavlov

9c2d20c

gdown -> wget from files

bbe438b

emotion detection files fixed

e5fc23e

dilyararimovna changed the title ~~[WIP] Feat/emotions multimodal (video)~~ Feat/emotions multimodal (video) Jan 15, 2024

dilyararimovna and others added 3 commits January 16, 2024 00:24

fix: tests

fec171b

lost file added

f053a18

feat: use dream persona in dream ocean

5f3dfdb

dilyararimovna approved these changes Jan 16, 2024

View reviewed changes

dilyararimovna merged commit bc61e2e into dev Jan 16, 2024
2 checks passed


		RUN pip install gdown

		RUN git clone https://github.com/anna-a-m/MultimodalERC repo


		RUN apt install -y ffmpeg libsm6 libxext6

		RUN pip install gdown

Feat/emotions multimodal (video) #564

Feat/emotions multimodal (video) #564

Conversation

moon-strider commented Aug 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dilyararimovna commented Aug 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oserikov commented Feb 6, 2024