Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazily load images on paint, not layout #3950

Open
wants to merge 18 commits into
base: master
Choose a base branch
from

Conversation

dnsge
Copy link
Contributor

@dnsge dnsge commented Sep 5, 2022

Pull request checklist:

  • CHANGELOG.md was updated, if applicable

Description

Instead of loading image pixmap data on layout, we try to do it when the image is first painted. This PR will increase the efficacy of #3915.

To demonstrate the behavior, you can do the following:

  1. Open debug popup
  2. Open a channel
  3. Switch to a different channel (or an empty tab)
  4. Send a message in the first channel (from webchat or another instance) that contains an emote
  5. Observe that the "Loaded images" value in the debug popup doesn't increase until you swap back to the channel in which the message was sent
Example Screenshots 1a810b4 on left, this PR on right.

Before sending message:
Before
After sending PogChamp message:
Message Sent

Note that "Loaded images" increases by 2 on the left as the broadcaster badge was also then loaded.

After refocusing channel in which message was sent:
Focused

Todo:

  • Fix/verify emote scaling selection

Fixes #3929

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

src/messages/Image.cpp Show resolved Hide resolved
src/messages/ImageSet.cpp Outdated Show resolved Hide resolved
src/messages/ImageSet.cpp Outdated Show resolved Hide resolved
src/messages/ImageSet.cpp Outdated Show resolved Hide resolved
src/messages/ImageSet.cpp Outdated Show resolved Hide resolved
src/messages/layouts/MessageLayoutElement.cpp Show resolved Hide resolved
src/messages/layouts/MessageLayoutElement.cpp Show resolved Hide resolved
src/messages/layouts/MessageLayoutElement.cpp Show resolved Hide resolved
src/messages/layouts/MessageLayoutElement.hpp Outdated Show resolved Hide resolved
src/messages/layouts/MessageLayoutElement.hpp Outdated Show resolved Hide resolved
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

src/messages/ImagePriorityOrder.cpp Show resolved Hide resolved
src/messages/ImagePriorityOrder.cpp Outdated Show resolved Hide resolved
src/messages/ImagePriorityOrder.hpp Outdated Show resolved Hide resolved
src/messages/ImageSet.cpp Show resolved Hide resolved
src/messages/layouts/MessageLayoutElement.cpp Show resolved Hide resolved
src/messages/layouts/MessageLayoutElement.hpp Show resolved Hide resolved
@dnsge dnsge marked this pull request as ready for review September 7, 2022 04:09
@Felanbird Felanbird requested a review from pajlada October 3, 2022 21:06
@pajlada
Copy link
Member

pajlada commented Oct 3, 2022

I'm going to run this PR as my daily driver for a while, and I'd urge others to do the same

To test a PR build, you can browse to the linked PR, click the "Checks" tab, then select the "Build" job on the left side, then scroll down to find the build that fits your platform. Note that you need to be signed into GitHub to download from that page.

@Felanbird
Copy link
Collaborator

Felanbird commented Oct 3, 2022

FFZ badges are missing their background on Windows
image
image

Interestingly does NOT apply to custom VIP badges, they continue to work fine.
image

@pajlada
Copy link
Member

pajlada commented Oct 9, 2022

I've run this for a week now, only thing I've noticed is what Felanbird mentioned, some special badges don't render their background properly

src/messages/Image.cpp Outdated Show resolved Hide resolved
src/messages/Image.cpp Outdated Show resolved Hide resolved
Copy link
Member

@pajlada pajlada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current issues:
FFZ badges don't use their background
The emoteScale doesn't work in normal messages

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

auto image =
emote->images.getImageOrLoaded(container.getScale());
if (!image->isEmpty())
float overallScale =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: variable 'overallScale' of type 'float' can be declared 'const' [misc-const-correctness]

Suggested change
float overallScale =
float const overallScale =

}

int currentWidth = metrics.horizontalAdvance(currentText);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: variable 'currentWidth' of type 'int' can be declared 'const' [misc-const-correctness]

Suggested change
int currentWidth = metrics.horizontalAdvance(currentText);
int const currentWidth = metrics.horizontalAdvance(currentText);

}

void PriorityImageLayoutElement::addCopyTextToString(QString &str,
uint32_t from,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: parameter 'from' is unused [misc-unused-parameters]

Suggested change
uint32_t from,
uint32_t /*from*/,


void PriorityImageLayoutElement::addCopyTextToString(QString &str,
uint32_t from,
uint32_t to) const
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: parameter 'to' is unused [misc-unused-parameters]

Suggested change
uint32_t to) const
uint32_t /*to*/) const

}

void PriorityImageLayoutElement::paint(QPainter &painter,
const MessageColors &messageColors)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: parameter 'messageColors' is unused [misc-unused-parameters]

Suggested change
const MessageColors &messageColors)
const MessageColors & /*messageColors*/)

return this->getRect().left();
}
else if (index == 1)
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: repeated branch in conditional chain [bugprone-branch-clone]

    {
    ^
Additional context

src/messages/layouts/MessageLayoutElement.cpp:410: end of the original

    }
     ^

src/messages/layouts/MessageLayoutElement.cpp:412: clone 1 starts here

    {
    ^

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

auto image =
emote->images.getImageOrLoaded(container.getScale());
if (!image->isEmpty())
float overallScale = getSettings()->emoteScale.getValue() *
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: variable 'overallScale' of type 'float' can be declared 'const' [misc-const-correctness]

Suggested change
float overallScale = getSettings()->emoteScale.getValue() *
float const overallScale = getSettings()->emoteScale.getValue() *

}

int currentWidth = metrics.horizontalAdvance(currentText);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: variable 'currentWidth' of type 'int' can be declared 'const' [misc-const-correctness]

Suggested change
int currentWidth = metrics.horizontalAdvance(currentText);
int const currentWidth = metrics.horizontalAdvance(currentText);

@kornes
Copy link
Contributor

kornes commented Oct 26, 2023

emote scale passed to addImageSetToContainer was ignored

diff --git a/src/messages/MessageElement.cpp b/src/messages/MessageElement.cpp
index 49d49aacb..ec9bd5c5a 100644
--- a/src/messages/MessageElement.cpp
+++ b/src/messages/MessageElement.cpp
@@ -27,7 +27,7 @@ namespace {
             return;
         }
 
-        auto size = priority->firstLoadedImageSize() * container.getScale();
+        auto size = priority->firstLoadedImageSize() * scale;
 
         container.addElement((new PriorityImageLayoutElement(
                                   element, std::move(*priority), size))

background is missing for FFZ badges. because makeImageLayoutElement() methods are not called anymore for elements, where ImageWithBackgroundLayoutElement happened. Diff dropping now dead code and passing background color down to PriorityImageLayoutElement

diff --git a/src/messages/MessageElement.cpp b/src/messages/MessageElement.cpp
index 49d49aacb..f3cbb6950 100644
--- a/src/messages/MessageElement.cpp
+++ b/src/messages/MessageElement.cpp
@@ -19,7 +19,8 @@ namespace chatterino {
 namespace {
     void addImageSetToContainer(MessageLayoutContainer &container,
                                 MessageElement &element,
-                                const ImageSet &imageSet, float scale)
+                                const ImageSet &imageSet, float scale,
+                                std::optional<QColor> bgColor = std::nullopt)
     {
         auto priority = imageSet.getPriority(scale);
         if (!priority)
@@ -30,7 +31,7 @@ namespace {
         auto size = priority->firstLoadedImageSize() * container.getScale();
 
         container.addElement((new PriorityImageLayoutElement(
-                                  element, std::move(*priority), size))
+                                  element, std::move(*priority), size, bgColor))
                                  ->setLink(element.getLink()));
     }
 
@@ -241,12 +242,6 @@ void EmoteElement::addToContainer(MessageLayoutContainer &container,
     }
 }
 
-MessageLayoutElement *EmoteElement::makeImageLayoutElement(
-    const ImagePtr &image, const QSize &size)
-{
-    return new ImageLayoutElement(*this, image, size);
-}
-
 LayeredEmoteElement::LayeredEmoteElement(
     std::vector<LayeredEmoteElement::Emote> &&emotes, MessageElementFlags flags,
     const MessageColor &textElementColor)
@@ -287,10 +282,9 @@ void LayeredEmoteElement::addToContainer(MessageLayoutContainer &container,
                 individualSizes.push_back(QSize(img->width(), img->height()) *
                                           overallScale);
             }
-
-            container.addElement(this->makeImageLayoutElement(
-                                         images, individualSizes, largestSize)
-                                     ->setLink(this->getLink()));
+            auto *newElement = new LayeredImageLayoutElement(
+                *this, images, individualSizes, largestSize);
+            container.addElement(newElement->setLink(this->getLink()));
         }
         else
         {
@@ -320,13 +314,6 @@ std::vector<ImagePtr> LayeredEmoteElement::getLoadedImages(float scale)
     return res;
 }
 
-MessageLayoutElement *LayeredEmoteElement::makeImageLayoutElement(
-    const std::vector<ImagePtr> &images, const std::vector<QSize> &sizes,
-    QSize largestSize)
-{
-    return new LayeredImageLayoutElement(*this, images, sizes, largestSize);
-}
-
 void LayeredEmoteElement::updateTooltips()
 {
     if (!this->emotes_.empty())
@@ -412,9 +399,11 @@ std::vector<LayeredEmoteElement::Emote> LayeredEmoteElement::getUniqueEmotes()
 }
 
 // BADGE
-BadgeElement::BadgeElement(const EmotePtr &emote, MessageElementFlags flags)
+BadgeElement::BadgeElement(const EmotePtr &emote, MessageElementFlags flags,
+                           std::optional<QColor> bgColor)
     : MessageElement(flags)
     , emote_(emote)
+    , bgColor_(bgColor)
 {
     this->setTooltip(emote->tooltip.string);
 }
@@ -425,7 +414,7 @@ void BadgeElement::addToContainer(MessageLayoutContainer &container,
     if (flags.hasAny(this->getFlags()))
     {
         addImageSetToContainer(container, *this, this->emote_->images,
-                               container.getScale());
+                               container.getScale(), this->bgColor_);
     }
 }
 
@@ -434,34 +423,13 @@ EmotePtr BadgeElement::getEmote() const
     return this->emote_;
 }
 
-MessageLayoutElement *BadgeElement::makeImageLayoutElement(
-    const ImagePtr &image, const QSize &size)
-{
-    auto element =
-        (new ImageLayoutElement(*this, image, size))->setLink(this->getLink());
-
-    return element;
-}
-
 // MOD BADGE
 ModBadgeElement::ModBadgeElement(const EmotePtr &data,
                                  MessageElementFlags flags_)
-    : BadgeElement(data, flags_)
+    : BadgeElement(data, flags_, {"#34AE0A"})
 {
 }
 
-MessageLayoutElement *ModBadgeElement::makeImageLayoutElement(
-    const ImagePtr &image, const QSize &size)
-{
-    static const QColor modBadgeBackgroundColor("#34AE0A");
-
-    auto element = (new ImageWithBackgroundLayoutElement(
-                        *this, image, size, modBadgeBackgroundColor))
-                       ->setLink(this->getLink());
-
-    return element;
-}
-
 // VIP BADGE
 VipBadgeElement::VipBadgeElement(const EmotePtr &data,
                                  MessageElementFlags flags_)
@@ -469,31 +437,11 @@ VipBadgeElement::VipBadgeElement(const EmotePtr &data,
 {
 }
 
-MessageLayoutElement *VipBadgeElement::makeImageLayoutElement(
-    const ImagePtr &image, const QSize &size)
-{
-    auto element =
-        (new ImageLayoutElement(*this, image, size))->setLink(this->getLink());
-
-    return element;
-}
-
 // FFZ Badge
 FfzBadgeElement::FfzBadgeElement(const EmotePtr &data,
                                  MessageElementFlags flags_, QColor color_)
-    : BadgeElement(data, flags_)
-    , color(std::move(color_))
-{
-}
-
-MessageLayoutElement *FfzBadgeElement::makeImageLayoutElement(
-    const ImagePtr &image, const QSize &size)
+    : BadgeElement(data, flags_, color_)
 {
-    auto element =
-        (new ImageWithBackgroundLayoutElement(*this, image, size, this->color))
-            ->setLink(this->getLink());
-
-    return element;
 }
 
 // TEXT
diff --git a/src/messages/MessageElement.hpp b/src/messages/MessageElement.hpp
index c35f9616e..3cbb965e9 100644
--- a/src/messages/MessageElement.hpp
+++ b/src/messages/MessageElement.hpp
@@ -316,10 +316,6 @@ public:
                         MessageElementFlags flags_) override;
     EmotePtr getEmote() const;
 
-protected:
-    virtual MessageLayoutElement *makeImageLayoutElement(const ImagePtr &image,
-                                                         const QSize &size);
-
 private:
     std::unique_ptr<TextElement> textElement_;
     EmotePtr emote_;
@@ -352,10 +348,6 @@ public:
     const std::vector<QString> &getEmoteTooltips() const;
 
 private:
-    MessageLayoutElement *makeImageLayoutElement(
-        const std::vector<ImagePtr> &image, const std::vector<QSize> &sizes,
-        QSize largestSize);
-
     QString getCopyString() const;
     void updateTooltips();
     std::vector<ImagePtr> getLoadedImages(float scale);
@@ -370,39 +362,29 @@ private:
 class BadgeElement : public MessageElement
 {
 public:
-    BadgeElement(const EmotePtr &data, MessageElementFlags flags_);
+    BadgeElement(const EmotePtr &emote, MessageElementFlags flags,
+                 std::optional<QColor> bgColor = std::nullopt);
 
     void addToContainer(MessageLayoutContainer &container,
                         MessageElementFlags flags_) override;
 
     EmotePtr getEmote() const;
 
-protected:
-    virtual MessageLayoutElement *makeImageLayoutElement(const ImagePtr &image,
-                                                         const QSize &size);
-
 private:
     EmotePtr emote_;
+    std::optional<QColor> bgColor_;
 };
 
 class ModBadgeElement : public BadgeElement
 {
 public:
     ModBadgeElement(const EmotePtr &data, MessageElementFlags flags_);
-
-protected:
-    MessageLayoutElement *makeImageLayoutElement(const ImagePtr &image,
-                                                 const QSize &size) override;
 };
 
 class VipBadgeElement : public BadgeElement
 {
 public:
     VipBadgeElement(const EmotePtr &data, MessageElementFlags flags_);
-
-protected:
-    MessageLayoutElement *makeImageLayoutElement(const ImagePtr &image,
-                                                 const QSize &size) override;
 };
 
 class FfzBadgeElement : public BadgeElement
@@ -410,11 +392,6 @@ class FfzBadgeElement : public BadgeElement
 public:
     FfzBadgeElement(const EmotePtr &data, MessageElementFlags flags_,
                     QColor color_);
-
-protected:
-    MessageLayoutElement *makeImageLayoutElement(const ImagePtr &image,
-                                                 const QSize &size) override;
-    const QColor color;
 };
 
 // contains a text, formated depending on the preferences
diff --git a/src/messages/layouts/MessageLayoutElement.cpp b/src/messages/layouts/MessageLayoutElement.cpp
index bf8aedf2c..e2619282e 100644
--- a/src/messages/layouts/MessageLayoutElement.cpp
+++ b/src/messages/layouts/MessageLayoutElement.cpp
@@ -327,9 +327,11 @@ int LayeredImageLayoutElement::getXFromIndex(size_t index)
 }
 
 PriorityImageLayoutElement::PriorityImageLayoutElement(
-    MessageElement &creator, ImagePriorityOrder &&order, const QSize &size)
+    MessageElement &creator, ImagePriorityOrder &&order, const QSize &size,
+    std::optional<QColor> bgColor)
     : MessageLayoutElement(creator, size)
     , order_(std::move(order))
+    , bgColor_(bgColor)
 {
     this->trailingSpace = creator.hasTrailingSpace();
 }
@@ -366,6 +368,11 @@ void PriorityImageLayoutElement::paint(QPainter &painter,
     {
         if (auto pixmap = imageToUse->pixmapOrLoad())
         {
+            if (this->bgColor_ && this->bgColor_.value().isValid())
+            {
+                painter.fillRect(QRectF(this->getRect()),
+                                 this->bgColor_.value());
+            }
             painter.drawPixmap(QRectF(this->getRect()), *pixmap, QRectF());
         }
     }
@@ -415,34 +422,6 @@ int PriorityImageLayoutElement::getXFromIndex(size_t index)
     }
 }
 
-//
-// IMAGE WITH BACKGROUND
-//
-ImageWithBackgroundLayoutElement::ImageWithBackgroundLayoutElement(
-    MessageElement &creator, ImagePtr image, const QSize &size, QColor color)
-    : ImageLayoutElement(creator, image, size)
-    , color_(color)
-{
-}
-
-void ImageWithBackgroundLayoutElement::paint(
-    QPainter &painter, const MessageColors & /*messageColors*/)
-{
-    if (this->image_ == nullptr)
-    {
-        return;
-    }
-
-    auto pixmap = this->image_->pixmapOrLoad();
-    if (pixmap && !this->image_->animated())
-    {
-        painter.fillRect(QRectF(this->getRect()), this->color_);
-
-        // fourtf: make it use qreal values
-        painter.drawPixmap(QRectF(this->getRect()), *pixmap, QRectF());
-    }
-}
-
 //
 // IMAGE WITH CIRCLE BACKGROUND
 //
diff --git a/src/messages/layouts/MessageLayoutElement.hpp b/src/messages/layouts/MessageLayoutElement.hpp
index 606161646..11b933a3e 100644
--- a/src/messages/layouts/MessageLayoutElement.hpp
+++ b/src/messages/layouts/MessageLayoutElement.hpp
@@ -104,7 +104,8 @@ class PriorityImageLayoutElement : public MessageLayoutElement
 {
 public:
     PriorityImageLayoutElement(MessageElement &creator,
-                               ImagePriorityOrder &&order, const QSize &size);
+                               ImagePriorityOrder &&order, const QSize &size,
+                               std::optional<QColor> bgColor = std::nullopt);
 
 protected:
     void addCopyTextToString(QString &str, uint32_t from = 0,
@@ -116,6 +117,7 @@ protected:
     int getXFromIndex(size_t index) override;
 
     const ImagePriorityOrder order_;
+    std::optional<QColor> bgColor_;
 };
 
 class LayeredImageLayoutElement : public MessageLayoutElement
@@ -138,19 +140,6 @@ protected:
     std::vector<QSize> sizes_;
 };
 
-class ImageWithBackgroundLayoutElement : public ImageLayoutElement
-{
-public:
-    ImageWithBackgroundLayoutElement(MessageElement &creator, ImagePtr image,
-                                     const QSize &size, QColor color);
-
-protected:
-    void paint(QPainter &painter, const MessageColors &messageColors) override;
-
-private:
-    QColor color_;
-};
-
 class ImageWithCircleBackgroundLayoutElement : public ImageLayoutElement
 {
 public:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Images for emotes, badges, etc. are not lazily loaded as intended
6 participants