[weverse] add extractors #4768

bradenhilton · 2023-11-04T18:24:38Z

Closes #3995.

Very WIP at the moment. I wanted to make sure I'm heading down the right path before I spend too much time on an incorrect solution.

Weverse requires an account and its API is not publicly documented.

At the moment, the Post extractor successfully downloads posts with images/videos, and skips text-only posts.

TODO:

Tests (as much as possible, maybe all we can do is test that extractors properly match their pattern?).
Update supportedsites.md.
Implement an extractor for a profile which downloads all posts from a community member using pagination, this would support both official artist posts and also fan posts.
Implement an extractor for the artist tab (/<community>/artist), which features all posts from the official artist(s) of a community in reverse chronological order.
Implement an extractor for the media tab (/<community>/media) which downloads all VOD content. Some of these are embeds from other sites such as YouTube, some are original content.
Implement an extractor for Moments. A Moment is a bit like a story from other social media sites, but the post itself does not expire as far as I can tell, only the ability to interact with it (like/comment etc.).
Implement an extractor for all Moments posted by an artist using pagination.

The API code is inspired by the yt-dlp implementation.

Speaking of yt-dlp, we could defer to it for media posts, as it supports VOD and live broadcasts as far as I can tell, but I haven't done this yet.

Some miscellaneous questions:

I noticed that you recently set Referer for requests by default, but it is absolutely mandatory for Weverse. Is it worth setting in the API headers?

The API responses use camelCase. Currently I am adding custom values to the metadata using snake_case. Should I stick to one style instead of mixing them?

bradenhilton · 2023-11-05T13:46:01Z

Example API responses:

Post (image)

{
  "extension": {},
  "body": "드디어 우리 노래 나왔잖아요!!\uD83D\uDC40\uD83E\uDEF6 \n피어나 어때요~~~??\n<w:attachment type=\"photo\" id=\"4-274880925\" />",
  "tags": [],
  "postId": "1-128752990",
  "postType": "NORMAL",
  "emotionCount": 30344,
  "commentCount": 3812,
  "publishedAt": 1698416087773,
  "membershipOnly": false,
  "availableActions": [
    "WRITE_COMMENT",
    "BOOKMARK",
    "WRITE_EMOTION",
    "READ_POST"
  ],
  "sectionType": "ARTIST",
  "hideFromArtist": false,
  "plainBody": "드디어 우리 노래 나왔잖아요!!\uD83D\uDC40\uD83E\uDEF6 \n피어나 어때요~~~??",
  "attachment": {
    "photo": {
      "4-274880925": {
        "url": "https://phinf.wevpstatic.net/MjAyMzEwMjdfMTMy/MDAxNjk4NDE2MDg3NDg0.uCJ26Yp7LLUBOJkCzex48s0BeG9Z8A4lgXZwWAqw64gg.c2KZ81NcEcFF4oJmGqpcuSH7MGAwwq9YkVIq3feLq04g.JPEG/image.jpg",
        "width": 2316,
        "height": 3088,
        "photoId": "4-274880925"
      }
    }
  },
  "shareUrl": "https://weverse.io/lesserafim/artist/1-128752990",
  "bookmarked": false,
  "viewerEmotionId": "3-1441778405",
  "locked": false,
  "hasProduct": false,
  "author": {
    "memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
    "communityId": 47,
    "joined": true,
    "profileName": "쥬하\uD83C\uDF3F",
    "profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
    "memberJoinStatus": "JOIN_COMMUNITY",
    "profileType": "ARTIST",
    "artistOfficialProfile": {
      "officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
      "officialName": "KAZUHA",
      "birthday": {
        "date": 1060387200000,
        "dateString": "2003.08.09",
        "timezone": "Asia/Seoul",
        "bday": false
      }
    },
    "hasOfficialMark": true,
    "profileSpaceStatus": "ACCESSIBLE",
    "myProfile": false
  },
  "community": {
    "communityId": 47,
    "communityName": "LE SSERAFIM",
    "logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
    "artistCode": "LESSERAFIM",
    "homeGradationColor": {
      "upLeftColorCode": "#7887c3",
      "upRightColorCode": "#455698",
      "downLeftColorCode": "#c0cbf5",
      "downRightColorCode": "#8e9edf",
      "menuTabLeftColorCode": "#465388",
      "menuTabRightColorCode": "#1b2e79"
    }
  }
}

Post (video)

{
  "extension": {},
  "body": "너무 귀엽죠??\uD83E\uDEF6\n오늘부터 잘 부탁해요~~\uD83D\uDE09 <w:attachment type=\"video\" id=\"4-864785\" />",
  "attachment": {
    "video": {
      "4-864785": {
        "videoId": "4-864785",
        "uploadInfo": {
          "width": 720,
          "videoId": "FC3D5A7233E610269DD0BB16F34AC9175D92",
          "playTime": 3,
          "height": 720,
          "imageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMDVfMjU5/MDAxNjk2NDg5MTg0NTg3.MqPrC4culB2vLsB1wjdKLSgzHuIrwNyfbZ2NFVxIXZAg.q5HU37Askl9whC3wZ9lMOymXanLgm68UypqZRYKH8Kgg.JPEG/bf494c1d-634c-11ee-9671-80615f0bcd6e_01.jpg",
          "uploadInfoVersion": 1
        }
      }
    }
  },
  "emotionCount": 28853,
  "commentCount": 2379,
  "postId": "2-127465849",
  "tags": [],
  "publishedAt": 1696489184386,
  "sectionType": "ARTIST",
  "membershipOnly": false,
  "availableActions": [
    "BOOKMARK",
    "WRITE_COMMENT",
    "WRITE_EMOTION",
    "READ_POST"
  ],
  "hideFromArtist": false,
  "postType": "NORMAL",
  "plainBody": "너무 귀엽죠??\uD83E\uDEF6\n오늘부터 잘 부탁해요~~\uD83D\uDE09",
  "shareUrl": "https://weverse.io/lesserafim/artist/2-127465849",
  "bookmarked": false,
  "locked": false,
  "hasProduct": false,
  "author": {
    "memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
    "communityId": 47,
    "joined": true,
    "profileName": "쥬하\uD83C\uDF3F",
    "profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
    "memberJoinStatus": "JOIN_COMMUNITY",
    "profileType": "ARTIST",
    "artistOfficialProfile": {
      "officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
      "officialName": "KAZUHA",
      "birthday": {
        "date": 1060387200000,
        "dateString": "2003.08.09",
        "timezone": "Asia/Seoul",
        "bday": false
      }
    },
    "hasOfficialMark": true,
    "profileSpaceStatus": "ACCESSIBLE",
    "myProfile": false
  },
  "community": {
    "communityId": 47,
    "communityName": "LE SSERAFIM",
    "logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
    "artistCode": "LESSERAFIM",
    "homeGradationColor": {
      "upLeftColorCode": "#7887c3",
      "upRightColorCode": "#455698",
      "downLeftColorCode": "#c0cbf5",
      "downRightColorCode": "#8e9edf",
      "menuTabLeftColorCode": "#465388",
      "menuTabRightColorCode": "#1b2e79"
    }
  }
}

Post (text)

{
  "extension": {},
  "tags": [],
  "publishedAt": 1696763700370,
  "emotionCount": 27997,
  "commentCount": 25772,
  "body": "피어나 뭐해요\uD83E\uDD79\uD83E\uDEF6",
  "availableActions": [
    "READ_POST",
    "BOOKMARK",
    "WRITE_COMMENT",
    "WRITE_EMOTION"
  ],
  "membershipOnly": false,
  "sectionType": "ARTIST",
  "plainBody": "피어나 뭐해요\uD83E\uDD79\uD83E\uDEF6",
  "postId": "3-135964325",
  "hideFromArtist": false,
  "attachment": {},
  "postType": "NORMAL",
  "shareUrl": "https://weverse.io/lesserafim/artist/3-135964325",
  "bookmarked": false,
  "locked": false,
  "hasProduct": false,
  "author": {
    "memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
    "communityId": 47,
    "joined": true,
    "profileName": "쥬하\uD83C\uDF3F",
    "profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
    "memberJoinStatus": "JOIN_COMMUNITY",
    "profileType": "ARTIST",
    "artistOfficialProfile": {
      "officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
      "officialName": "KAZUHA",
      "birthday": {
        "date": 1060387200000,
        "dateString": "2003.08.09",
        "timezone": "Asia/Seoul",
        "bday": false
      }
    },
    "hasOfficialMark": true,
    "profileSpaceStatus": "ACCESSIBLE",
    "myProfile": false
  },
  "artistReactions": [
    {
      "reactionType": "comment",
      "reactionCount": 19,
      "member": {
        "memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
        "communityId": 47,
        "joined": true,
        "profileName": "쥬하\uD83C\uDF3F",
        "profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
        "memberJoinStatus": "JOIN_COMMUNITY",
        "profileType": "ARTIST",
        "artistOfficialProfile": {
          "officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
          "officialName": "KAZUHA",
          "birthday": {
            "date": 1060387200000,
            "dateString": "2003.08.09",
            "timezone": "Asia/Seoul",
            "bday": false
          }
        },
        "hasOfficialMark": true,
        "profileSpaceStatus": "ACCESSIBLE",
        "myProfile": false
      }
    }
  ],
  "community": {
    "communityId": 47,
    "communityName": "LE SSERAFIM",
    "logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
    "artistCode": "LESSERAFIM",
    "homeGradationColor": {
      "upLeftColorCode": "#7887c3",
      "upRightColorCode": "#455698",
      "downLeftColorCode": "#c0cbf5",
      "downRightColorCode": "#8e9edf",
      "menuTabLeftColorCode": "#465388",
      "menuTabRightColorCode": "#1b2e79"
    }
  }
}

Moment

{
  "extension": {
    "moment": {
      "momentAsJson": "{\"description\":\"\",\"style\":\"\",\"version\":\"1.1.3\",\"clips\":[{\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA5MDlfNTIg\\/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG\\/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg\",\"events\":[],\"title\":\"\",\"videoId\":\"940264D33932D0CD690A57AA820458275CD7\",\"duration\":10,\"metadata\":{\"texts\":[{\"range\":{\"endTime\":10000,\"startTime\":0},\"text\":\"한국에서 처음으로 보낸 추석\uD83E\uDD0D\\n피어나 여러분 뭐해요~~??\"}]},\"resolution\":{\"width\":1080,\"height\":1920}}],\"tags\":[],\"isReviewedByCustomer\":\"N\",\"title\":\"\",\"meta\":{\"stickerCnt\":{\"giphy\":0},\"mediaSrcCnt\":{\"video\":0,\"image\":1}},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA5MDlfNTIg\\/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG\\/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg\",\"duration\":10,\"indexes\":[{\"title\":\"\",\"range\":{\"endTime\":10000,\"startTime\":0},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA5MDlfNTIg\\/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG\\/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg\"}],\"resolution\":{\"width\":1080,\"height\":1920}}",
      "body": "한국에서 처음으로 보낸 추석\uD83E\uDD0D\n피어나 여러분 뭐해요~~??",
      "expireAt": 1662784496362,
      "video": {
        "videoId": "4-83503",
        "uploadInfo": {
          "width": 1080,
          "videoId": "940264D33932D0CD690A57AA820458275CD7",
          "playTime": 10,
          "height": 1920,
          "imageUrl": "https://phinf.wevpstatic.net/MjAyMjA5MDlfNTIg/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg",
          "uploadInfoVersion": 1
        }
      }
    }
  },
  "tags": [],
  "publishedAt": 1662698096362,
  "postType": "MOMENT",
  "sectionType": "MOMENT",
  "membershipOnly": false,
  "body": "한국에서 처음으로 보낸 추석\uD83E\uDD0D\n피어나 여러분 뭐해요~~??",
  "hideFromArtist": false,
  "emotionCount": 4490,
  "commentCount": 315,
  "availableActions": ["READ_POST"],
  "postId": "3-104040928",
  "attachment": {},
  "plainBody": "한국에서 처음으로 보낸 추석\uD83E\uDD0D\n피어나 여러분 뭐해요~~??",
  "shareUrl": "https://weverse.io/lesserafim/moment/b60d95bc3b71f4d97b28ac1b971cc641/post/3-104040928",
  "bookmarked": false,
  "locked": false,
  "hasProduct": false,
  "author": {
    "memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
    "communityId": 47,
    "joined": true,
    "profileName": "쥬하\uD83C\uDF3F",
    "profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
    "memberJoinStatus": "JOIN_COMMUNITY",
    "profileType": "ARTIST",
    "artistOfficialProfile": {
      "officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
      "officialName": "KAZUHA",
      "birthday": {
        "date": 1060387200000,
        "dateString": "2003.08.09",
        "timezone": "Asia/Seoul",
        "bday": false
      }
    },
    "hasOfficialMark": true,
    "profileSpaceStatus": "ACCESSIBLE",
    "myProfile": false
  },
  "authorMomentPosts": {
    "paging": {},
    "data": [
      {
        "extension": {
          "moment": {
            "momentAsJson": "{\"description\":\"\",\"style\":\"\",\"version\":\"1.1.3\",\"clips\":[{\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA5MDlfNTIg\\/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG\\/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg\",\"events\":[],\"title\":\"\",\"videoId\":\"940264D33932D0CD690A57AA820458275CD7\",\"duration\":10,\"metadata\":{\"texts\":[{\"range\":{\"endTime\":10000,\"startTime\":0},\"text\":\"한국에서 처음으로 보낸 추석\uD83E\uDD0D\\n피어나 여러분 뭐해요~~??\"}]},\"resolution\":{\"width\":1080,\"height\":1920}}],\"tags\":[],\"isReviewedByCustomer\":\"N\",\"title\":\"\",\"meta\":{\"stickerCnt\":{\"giphy\":0},\"mediaSrcCnt\":{\"video\":0,\"image\":1}},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA5MDlfNTIg\\/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG\\/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg\",\"duration\":10,\"indexes\":[{\"title\":\"\",\"range\":{\"endTime\":10000,\"startTime\":0},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA5MDlfNTIg\\/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG\\/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg\"}],\"resolution\":{\"width\":1080,\"height\":1920}}",
            "body": "한국에서 처음으로 보낸 추석\uD83E\uDD0D\n피어나 여러분 뭐해요~~??",
            "expireAt": 1662784496362,
            "video": {
              "videoId": "4-83503",
              "uploadInfo": {
                "width": 1080,
                "videoId": "940264D33932D0CD690A57AA820458275CD7",
                "playTime": 10,
                "height": 1920,
                "imageUrl": "https://phinf.wevpstatic.net/MjAyMjA5MDlfNTIg/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg",
                "uploadInfoVersion": 1
              }
            }
          }
        },
        "tags": [],
        "publishedAt": 1662698096362,
        "postType": "MOMENT",
        "sectionType": "MOMENT",
        "membershipOnly": false,
        "hideFromArtist": false,
        "emotionCount": 4490,
        "commentCount": 315,
        "availableActions": ["READ_POST"],
        "postId": "3-104040928",
        "summary": {
          "videoCount": 1,
          "thumbnails": [
            {
              "type": "VIDEO",
              "url": "https://phinf.wevpstatic.net/MjAyMjA5MDlfNTIg/MDAxNjYyNjk4MDg5MzI3.2G5v-YnreeHKRK0qU9T_5L0PtxKnjO-UxF5Iir4S2dQg.--uDITfIRkc2SJ5e9uDGB1udmfhRBoXVK13hhXcMT8Qg.JPEG/C467F406-294F-4D4B-891F-04FDB7F1E6AB.jpg",
              "playTime": 10,
              "cVideoId": "4-83503"
            }
          ],
          "photoCount": 0,
          "snippets": []
        },
        "plainBody": "한국에서 처음으로 보낸 추석\uD83E\uDD0D\n피어나 여러분 뭐해요~~??",
        "shareUrl": "https://weverse.io/lesserafim/moment/b60d95bc3b71f4d97b28ac1b971cc641/post/3-104040928",
        "bookmarked": false,
        "locked": false,
        "hasProduct": false,
        "author": {
          "memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
          "communityId": 47,
          "joined": true,
          "profileName": "쥬하\uD83C\uDF3F",
          "profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
          "memberJoinStatus": "JOIN_COMMUNITY",
          "profileType": "ARTIST",
          "artistOfficialProfile": {
            "officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
            "officialName": "KAZUHA",
            "birthday": {
              "date": 1060387200000,
              "dateString": "2003.08.09",
              "timezone": "Asia/Seoul",
              "bday": false
            }
          },
          "hasOfficialMark": true,
          "profileSpaceStatus": "ACCESSIBLE",
          "myProfile": false
        },
        "community": {
          "communityId": 47,
          "communityName": "LE SSERAFIM",
          "logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
          "artistCode": "LESSERAFIM"
        }
      },
      {
        "extension": {
          "moment": {
            "momentAsJson": "{\"description\":\"\",\"style\":\"\",\"version\":\"1.1.3\",\"clips\":[{\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA3MjVfNzgg\\/MDAxNjU4NzM4OTg1MDUx.a5ffKA4rCn8iauE6VSqaC5o5QK0EoNTInA4KJKrcj-cg.S7nApdPG43lKfNGUEXA5kmPLQ-RN86D3J3u3mekW-u4g.JPEG\\/E48A02C9-A2FE-40E1-9C05-5AC5441A4EF7.jpg\",\"events\":[],\"title\":\"\",\"videoId\":\"45D6EC7EF41379A783252970FFAC04104648\",\"duration\":10,\"metadata\":{\"texts\":[{\"range\":{\"endTime\":10000,\"startTime\":0},\"text\":\"진짜 오늘 날씨 좋다~~~\uD83E\uDD0D\\n\\n저녁에도 밝은 여름 최고\uD83E\uDD1F\\n\\n좋은 하루 보내세요~~!!\"}]},\"resolution\":{\"width\":1080,\"height\":1920}}],\"tags\":[],\"isReviewedByCustomer\":\"N\",\"title\":\"\",\"meta\":{\"stickerCnt\":{\"giphy\":1},\"mediaSrcCnt\":{\"video\":0,\"image\":1}},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA3MjVfNzgg\\/MDAxNjU4NzM4OTg1MDUx.a5ffKA4rCn8iauE6VSqaC5o5QK0EoNTInA4KJKrcj-cg.S7nApdPG43lKfNGUEXA5kmPLQ-RN86D3J3u3mekW-u4g.JPEG\\/E48A02C9-A2FE-40E1-9C05-5AC5441A4EF7.jpg\",\"duration\":10,\"indexes\":[{\"title\":\"\",\"range\":{\"endTime\":10000,\"startTime\":0},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA3MjVfNzgg\\/MDAxNjU4NzM4OTg1MDUx.a5ffKA4rCn8iauE6VSqaC5o5QK0EoNTInA4KJKrcj-cg.S7nApdPG43lKfNGUEXA5kmPLQ-RN86D3J3u3mekW-u4g.JPEG\\/E48A02C9-A2FE-40E1-9C05-5AC5441A4EF7.jpg\"}],\"resolution\":{\"width\":1080,\"height\":1920}}",
            "body": "진짜 오늘 날씨 좋다~~~\uD83E\uDD0D\n\n저녁에도 밝은 여름 최고\uD83E\uDD1F\n\n좋은 하루 보내세요~~!!",
            "expireAt": 1658825391456,
            "video": {
              "videoId": "3-15501",
              "uploadInfo": {
                "width": 1080,
                "videoId": "45D6EC7EF41379A783252970FFAC04104648",
                "playTime": 10,
                "height": 1920,
                "imageUrl": "https://phinf.wevpstatic.net/MjAyMjA3MjVfNzgg/MDAxNjU4NzM4OTg1MDUx.a5ffKA4rCn8iauE6VSqaC5o5QK0EoNTInA4KJKrcj-cg.S7nApdPG43lKfNGUEXA5kmPLQ-RN86D3J3u3mekW-u4g.JPEG/E48A02C9-A2FE-40E1-9C05-5AC5441A4EF7.jpg",
                "uploadInfoVersion": 1
              }
            }
          }
        },
        "tags": [],
        "publishedAt": 1658738991456,
        "postType": "MOMENT",
        "sectionType": "MOMENT",
        "membershipOnly": false,
        "hideFromArtist": false,
        "emotionCount": 6976,
        "commentCount": 468,
        "availableActions": ["READ_POST"],
        "postId": "4-101115099",
        "summary": {
          "videoCount": 1,
          "thumbnails": [
            {
              "type": "VIDEO",
              "url": "https://phinf.wevpstatic.net/MjAyMjA3MjVfNzgg/MDAxNjU4NzM4OTg1MDUx.a5ffKA4rCn8iauE6VSqaC5o5QK0EoNTInA4KJKrcj-cg.S7nApdPG43lKfNGUEXA5kmPLQ-RN86D3J3u3mekW-u4g.JPEG/E48A02C9-A2FE-40E1-9C05-5AC5441A4EF7.jpg",
              "playTime": 10,
              "cVideoId": "3-15501"
            }
          ],
          "photoCount": 0,
          "snippets": []
        },
        "plainBody": "진짜 오늘 날씨 좋다~~~\uD83E\uDD0D\n저녁에도 밝은 여름 최고\uD83E\uDD1F\n좋은 하루 보내세요~~!!",
        "shareUrl": "https://weverse.io/lesserafim/moment/b60d95bc3b71f4d97b28ac1b971cc641/post/4-101115099",
        "bookmarked": false,
        "locked": false,
        "hasProduct": false,
        "author": {
          "memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
          "communityId": 47,
          "joined": true,
          "profileName": "쥬하\uD83C\uDF3F",
          "profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
          "memberJoinStatus": "JOIN_COMMUNITY",
          "profileType": "ARTIST",
          "artistOfficialProfile": {
            "officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
            "officialName": "KAZUHA",
            "birthday": {
              "date": 1060387200000,
              "dateString": "2003.08.09",
              "timezone": "Asia/Seoul",
              "bday": false
            }
          },
          "hasOfficialMark": true,
          "profileSpaceStatus": "ACCESSIBLE",
          "myProfile": false
        },
        "community": {
          "communityId": 47,
          "communityName": "LE SSERAFIM",
          "logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
          "artistCode": "LESSERAFIM"
        }
      },
      {
        "extension": {
          "moment": {
            "momentAsJson": "{\"description\":\"\",\"style\":\"\",\"version\":\"1.1.3\",\"clips\":[{\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA3MTlfMjg1\\/MDAxNjU4MTkwNjA1ODg3.qtMr3ltfmVDFt-XLFzXLLIjrJXu_On12_zsNPi6GYi4g.uAvNvpkpOJS6wwo1ckrIJeHSEM-shrNOMWW5WopHqXgg.JPEG\\/95CE3178-E10B-4298-B877-396268E6798F.jpg\",\"events\":[],\"title\":\"\",\"videoId\":\"233B24713027EC3B674EDDE9FB61461CF404\",\"duration\":10,\"metadata\":{\"texts\":[{\"range\":{\"endTime\":10000,\"startTime\":0},\"text\":\"우와우\\n이런 것 있었는지 몰랐어!!ㅋㅋㅋㅋ \\n더 생기면 좋겠다\uD83E\uDEF6ㅋㅋㅋ\"}]},\"resolution\":{\"width\":1080,\"height\":1920}}],\"tags\":[],\"isReviewedByCustomer\":\"N\",\"title\":\"\",\"meta\":{\"stickerCnt\":{\"giphy\":3},\"mediaSrcCnt\":{\"video\":0,\"image\":1}},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA3MTlfMjg1\\/MDAxNjU4MTkwNjA1ODg3.qtMr3ltfmVDFt-XLFzXLLIjrJXu_On12_zsNPi6GYi4g.uAvNvpkpOJS6wwo1ckrIJeHSEM-shrNOMWW5WopHqXgg.JPEG\\/95CE3178-E10B-4298-B877-396268E6798F.jpg\",\"duration\":10,\"indexes\":[{\"title\":\"\",\"range\":{\"endTime\":10000,\"startTime\":0},\"thumbnail\":\"https:\\/\\/weverse-phinf.pstatic.net\\/MjAyMjA3MTlfMjg1\\/MDAxNjU4MTkwNjA1ODg3.qtMr3ltfmVDFt-XLFzXLLIjrJXu_On12_zsNPi6GYi4g.uAvNvpkpOJS6wwo1ckrIJeHSEM-shrNOMWW5WopHqXgg.JPEG\\/95CE3178-E10B-4298-B877-396268E6798F.jpg\"}],\"resolution\":{\"width\":1080,\"height\":1920}}",
            "body": "우와우\n이런 것 있었는지 몰랐어!!ㅋㅋㅋㅋ \n더 생기면 좋겠다\uD83E\uDEF6ㅋㅋㅋ",
            "expireAt": 1658277011392,
            "video": {
              "videoId": "3-4450",
              "uploadInfo": {
                "width": 1080,
                "videoId": "233B24713027EC3B674EDDE9FB61461CF404",
                "playTime": 10,
                "height": 1920,
                "imageUrl": "https://phinf.wevpstatic.net/MjAyMjA3MTlfMjg1/MDAxNjU4MTkwNjA1ODg3.qtMr3ltfmVDFt-XLFzXLLIjrJXu_On12_zsNPi6GYi4g.uAvNvpkpOJS6wwo1ckrIJeHSEM-shrNOMWW5WopHqXgg.JPEG/95CE3178-E10B-4298-B877-396268E6798F.jpg",
                "uploadInfoVersion": 1
              }
            }
          }
        },
        "tags": [],
        "publishedAt": 1658190611392,
        "postType": "MOMENT",
        "sectionType": "MOMENT",
        "membershipOnly": false,
        "hideFromArtist": false,
        "emotionCount": 4157,
        "commentCount": 252,
        "availableActions": ["READ_POST"],
        "postId": "4-100700103",
        "summary": {
          "videoCount": 1,
          "thumbnails": [
            {
              "type": "VIDEO",
              "url": "https://phinf.wevpstatic.net/MjAyMjA3MTlfMjg1/MDAxNjU4MTkwNjA1ODg3.qtMr3ltfmVDFt-XLFzXLLIjrJXu_On12_zsNPi6GYi4g.uAvNvpkpOJS6wwo1ckrIJeHSEM-shrNOMWW5WopHqXgg.JPEG/95CE3178-E10B-4298-B877-396268E6798F.jpg",
              "playTime": 10,
              "cVideoId": "3-4450"
            }
          ],
          "photoCount": 0,
          "snippets": []
        },
        "plainBody": "우와우\n이런 것 있었는지 몰랐어!!ㅋㅋㅋㅋ \n더 생기면 좋겠다\uD83E\uDEF6ㅋㅋㅋ",
        "shareUrl": "https://weverse.io/lesserafim/moment/b60d95bc3b71f4d97b28ac1b971cc641/post/4-100700103",
        "bookmarked": false,
        "locked": false,
        "hasProduct": false,
        "author": {
          "memberId": "b60d95bc3b71f4d97b28ac1b971cc641",
          "communityId": 47,
          "joined": true,
          "profileName": "쥬하\uD83C\uDF3F",
          "profileImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjQ2/MDAxNjk3NTU0OTg0MTcz.Ox-UsVCCMxdDfO5eTlOtS3rXjqoY0Rd1RxdN3rt3iTkg.PSneXamakKLjQuB8JR5Fs1R0WxUl5iXsGBS0V_6pzowg.JPEG/c88356d1-8f41-46ed-9f06-2d3810713011.jpeg",
          "memberJoinStatus": "JOIN_COMMUNITY",
          "profileType": "ARTIST",
          "artistOfficialProfile": {
            "officialImageUrl": "https://phinf.wevpstatic.net/MjAyMzEwMThfMjEz/MDAxNjk3NTU0OTc3NjAw.D9zosDQRuK28hOBJsF0JzIeXRn3rkSsXXKa2Japn4XEg.2dYdFv1-y1kjxEqXYtPunIzzGPl7syCSxcbN-Ni0mT4g.JPEG/6faffa29-8c53-4211-b025-69ca813c9ad3.jpeg",
            "officialName": "KAZUHA",
            "birthday": {
              "date": 1060387200000,
              "dateString": "2003.08.09",
              "timezone": "Asia/Seoul",
              "bday": false
            }
          },
          "hasOfficialMark": true,
          "profileSpaceStatus": "ACCESSIBLE",
          "myProfile": false
        },
        "community": {
          "communityId": 47,
          "communityName": "LE SSERAFIM",
          "logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
          "artistCode": "LESSERAFIM"
        }
      }
    ]
  },
  "community": {
    "communityId": 47,
    "communityName": "LE SSERAFIM",
    "logoImage": "https://phinf.wevpstatic.net/MjAyMzEwMTJfMTI4/MDAxNjk3MDc5MTA3Njg4.tWmeZYTw5wXcgDYW1CdqRI9Sn81B7Fd2V2CF7Qt7bXkg.hOVkhjZpWx8q4YS0LHinzGDsaptQFkM1CvgKEu9aLJgg.PNG/6a41660d-68a1-40a7-8e90-0687914cbc77.png",
    "artistCode": "LESSERAFIM",
    "homeGradationColor": {
      "upLeftColorCode": "#7887c3",
      "upRightColorCode": "#455698",
      "downLeftColorCode": "#c0cbf5",
      "downRightColorCode": "#8e9edf",
      "menuTabLeftColorCode": "#465388",
      "menuTabRightColorCode": "#1b2e79"
    }
  }
}

For endpoints with pagination support, their response includes a paging.nextParams.after value that can be sent as an after param in the next request.

Some of these values could probably be removed from the final metadata (or the metadata could be created from scratch with cherry picked values), though I'm hesitant to remove too much for versatility with naming/filtering.

bradenhilton · 2023-11-05T22:56:50Z

I found myself needing to pull an all-nighter, so I went ahead and just implemented most of the TODOs to keep myself busy.

Some of the extractors probably aren't necessary. WeverseMomentsExtractor in particular could probably be merged into WeverseMomentExtractor, and an optional capture group could be used for single Moments instead. I wanted to keep things separate for now as I'm still learning how everything works.

bradenhilton · 2023-11-07T17:17:54Z

@mikf Feel free to tear this to pieces, I want it to work well.

I'm decently happy with this feature-wise. There are some remaining issues:

The code could be much DRY-er. I'm currently working on extracting some logic to functions.
The code should better adhere to current conventions.
I'm not a fan of how I'm currently handling login, as self.login() is executed every time an extractor is called from Message.Queue. Is this just a consequence of how I've structured the code (extractors deferring to other extractors)?
- Somewhat related, I'm seeing a lot of logs about resetting dropped connections when making requests. Do I need to init and reuse a session somewhere to prevent this?
If possible, for embeds (which are currently just YouTube videos as far as I can tell, I haven't found an embed that isn't a YouTube video yet), I would like to be able to set the directory to directory_fmt, but allow ytdl to set the filename instead of using filename_fmt. I have download.ytdl.outtmpl set to null in my test config. Is it possible to set download options per extractor (e.g. extractor.weverse.download.ytdl.outtmpl)? Or are there any code changes I can make to achieve this?

So far, I've manually tested (with --config-ignore --config ./config.json --verbose -- <URL>) the following:

Task	Defaults	`skip-embeds` = `true`	`videos` = `"ytdl"`	`videos` = `false`
Login (cookies from browser)	✅
Login (`access-token` config set)	✅
Login (`username` and `password` config set)	✅
Artist `Post` (image)	✅
Artist `Post` (video)	✅		✅	✅
Artist `Post` (text)	✅
Fan `Post` (image)	✅
Artist `Profile`	✅		✅	✅
`Artists Tab`	✅		✅	✅
`Moment`	✅		✅	✅
Artist `Moments`	✅		✅	✅
`Media` (image)	✅
`Media` (video)	✅		✅	✅
`Media` (embed)	✅	✅	✅	✅
`Media Tab`	✅	✅	✅	✅
`Media Category`	✅	✅	✅	✅

with this base config.json:

{
  "downloader": {
    "mtime": false,
    "ytdl": {
      "module": "yt_dlp",
      "outtmpl": null
    }
  },
  "extractor": {
    "path-restrict": "windows",
    "postprocessors": [
      {
        "key": "date",
        "name": "mtime"
      }
    ],
    "weverse": {
      "postprocessors": [
        {
          "async": false,
          "command": [
            "exiftool",
            "-quiet",
            "-overwrite_original",
            "-preserve",
            "-charset",
            "utf8",
            "-AllComments={shareUrl}/",
            "{_path[4:]}"
          ],
          "event": "after",
          "name": "exec"
        }
      ],
    },
  }
}

I don't have any active paid memberships, so I don't know if their API endpoints/responses differ at all. I can look into it if/when these initial extractors are merged. Some VOD content also appears to be completely DRM protected.

bradenhilton · 2023-11-12T01:07:39Z

I found a new endpoint which was the missing piece to allow all non-paginating extractors to work without authentication. I have now added some more robust tests.

bradenhilton · 2024-02-11T19:41:50Z

@mikf I believe this is pretty much finished now (pending a review at your convenience, of course).

I attempted to rewrite it in the style of some other extractors such as twitter and bluesky (e.g. using something like posts() instead of items() where possible), but I'm unsure if there are enough similarities between Weverse and those sites for it to be of any use, so I reverted to delegating for all extractors that use pagination.

ytdl support has been removed. Only yt-dlp has Weverse extractors (for Media videos and live streams/VODs, not Post videos), I am not sure if ytdl support adds much value here.

I also have most if not all possible relevant API responses saved if you'd like them for context (or to create the extractors yourself if you prefer).

bradenhilton force-pushed the weverse branch from 4b512b6 to 3b60dab Compare November 4, 2023 18:40

bradenhilton force-pushed the weverse branch from 3b60dab to 3dfc3ce Compare November 5, 2023 22:55

bradenhilton force-pushed the weverse branch 2 times, most recently from cbb6bc7 to 6f16e1a Compare November 7, 2023 10:13

bradenhilton marked this pull request as ready for review November 7, 2023 10:14

bradenhilton force-pushed the weverse branch 3 times, most recently from 55c8167 to 7954d84 Compare November 12, 2023 01:02

bradenhilton force-pushed the weverse branch from 7954d84 to fff03a4 Compare November 13, 2023 01:47

bradenhilton force-pushed the weverse branch 5 times, most recently from 27344dc to a2ad565 Compare February 11, 2024 02:52

[weverse] add extractors

963d311

bradenhilton force-pushed the weverse branch from a2ad565 to 963d311 Compare February 11, 2024 02:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[weverse] add extractors #4768

[weverse] add extractors #4768

bradenhilton commented Nov 4, 2023 •

edited

bradenhilton commented Nov 5, 2023

bradenhilton commented Nov 5, 2023

bradenhilton commented Nov 7, 2023

bradenhilton commented Nov 12, 2023

bradenhilton commented Feb 11, 2024

[weverse] add extractors #4768

Are you sure you want to change the base?

[weverse] add extractors #4768

Conversation

bradenhilton commented Nov 4, 2023 • edited

bradenhilton commented Nov 5, 2023

bradenhilton commented Nov 5, 2023

bradenhilton commented Nov 7, 2023

bradenhilton commented Nov 12, 2023

bradenhilton commented Feb 11, 2024

bradenhilton commented Nov 4, 2023 •

edited