Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Album artwork via Ueberzug #437

Open
wants to merge 43 commits into
base: master
Choose a base branch
from
Open

Album artwork via Ueberzug #437

wants to merge 43 commits into from

Conversation

evanc577
Copy link
Contributor

@evanc577 evanc577 commented Jan 3, 2021

Display album art in terminal with Ueberzug.

Added a new "artwork" screen that can be resized and positioned using existing ncmpcpp functions.

  • Looks for an image in the currently playing song's directory relative to the mpd_music_dir config value.
    • Checks cover.png, cover.jpg, cover.tiff, cover.bmp
    • Falls back to albumart_default_path if it can't find a suitable image.
  • The artwork is updated and drawn on if the artwork screen is visible and the song changes, or if the screen is resized.
  • The "artwork" screen is bound to "9" by default.
  • ueberzug executable must be visible in $PATH, ncmpcpp spawns it as a background process.
  • Album art size and position can be somewhat configured with the albumart_scaler option. See Ueberzug documentation.

Use ./configure --enable-artwork flag to compile artwork support.

PR draft thoughts

Some current limitations:

  • No support for embedded artwork.
  • Doesn't use mpd's native albumart, libmpdclient has not implemented this functionality yet anyways.
  • Only searches for "cover.jpg" in directory.
    • Now searches a few formats, as well as a configurable fallback image. Searches cover.png, cover.jpg, cover.tiff, and cover.bmp.
    • Fallback artwork is specified by albumart_default_path config option.
  • Artwork is aligned at the top-left of screen. Centering it probably requires parsing the image with imagemagick. As far as I can tell, Ueberzug doesn't make this possible by itself.
    • Doesn't seem like a great idea to pull in yet another dependency, and would be hacky. Ueberzug only supports character units, not pixel units.
    • Maybe try to get this supported in Ueberzug upstream?
    • Partial workaround: use the albumart_scaler option to customize look, values corresponding to Ueberzug's scaler options.
  • If using split screens and a popup appears (such as when adding a song to a playlist), the artwork disappears and need to switch/reload screens to make it appear again.
    • Fixed
  • Configure script doesn't check whether Ueberzug is installed or not.
    • N/A, runtime dependency

Dirty hacks that probably need to go away:

  • Uses posix_spawn() to start ueberzug, kills it on exit with std::atexit() and kill(). If ncmpcpp crashes and terminates abnormally, the spawned process will not be terminated.
    • Solved by fork-exec ueberzug directly and manually creating a pipe for ueberzug stdin.
    • Use boost::process which abstracts away a lot of the hard stuff...
  • Ueberzug is hardcoded, should refactor so other viable backends can be easily added.
    • Fixed

Screenshots

mpd_music_dir = "/path/to/mpd/library"
startup_screen = artwork
startup_slave_screen = playlist
startup_slave_screen_focus = yes
locked_screen_width_part = 25

Screenshot_20210103_154138

Screenshot_20210103_154214

Screenshot_20210103_154719

@evanc577 evanc577 changed the title Proof of concept: album artwork via Ueberzug Album artwork via Ueberzug Jan 6, 2021
@evanc577 evanc577 marked this pull request as ready for review January 6, 2021 02:52
@evanc577
Copy link
Contributor Author

evanc577 commented Jan 6, 2021

I think this PR is clean and feature-complete enough for review now....

@mvrozanti
Copy link
Contributor

Damn that's cool

@evanc577
Copy link
Contributor Author

a0bc7be updates the configure script to check for boost::process, which requires boost 1.64, released April 2017, seems safe enough.

e31a33e updates the configure script to check for boost::json in order to use json serialization and ueberzug's json parser. From ueberzug readme:

simple:
Key-value pairs seperated by a tab,
pairs are also seperated by a tab
warning ONLY FOR TESTING!
Simple was never intended for the usage in production!
It doesn't support paths containing tabs or line breaks
which makes it error prone.

boost::json requires boost 1.75, released December 2020. I'm not comfortable escaping json strings manually, given all the unicode stuff; I'd rather give it to a library to deal with. If that's too new then we can revert the last commit and use the simple ueberzug parser. If you use tabs or newlines, then you probably have bigger issues than ncmppcpp not working. MPD itself doesn't see paths with newlines on my system.

Not to mention json has its own problems. Linux doesn't require that filenames be valid unicode. json doesn't support non-unicode strings. We can construct a path that cannot be passed successfully to either ueberzug's simple or json parsers by creating a filename with both tabs and non-unicode characters (please don't do this).

$ cp img.jpg "$(printf "\t\xbd\xb2.jpg")"

@lenerd
Copy link
Contributor

lenerd commented Jan 10, 2021

boost::json requires boost 1.75, released December 2020. I'm not comfortable escaping json strings manually, given all the unicode stuff; I'd rather give it to a library to deal with. If that's too new then we can revert the last commit and use the simple ueberzug parser.

There is also the possibility of using Boost.PropertyTree for reading/writing JSON. I would also prefer to work with Boost.JSON, but Boost.PropertyTree has the advantage to be around since Boost 1.41.

Edit: I need to try out this PR. :)

@arybczak
Copy link
Collaborator

arybczak commented Jan 20, 2021

@evanc577 Thanks!

Would it be a lot of work to use PropertyTree instead? Considering that boost::json is very new.

@evanc577
Copy link
Contributor Author

Easy enough, switched to property_tree and lowered min boost version to 1.64

virtual void removeArtwork() = 0;
};

class UeberzugBackend : public ArtworkBackend
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What other backends could be there? I'm wondering whether this abstraction is needed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure? There are a lot of ways to draw images onto a terminal, but I haven't found any other that plays nicely with ncurses. But is the extra layer of abstraction that big of a deal?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would love libsixel is that's possible so i can use it in OSX as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ueberzug depends on X11, so Wayland and MacOS (probably? I don't own a mac) won't work. I think I will probably end up implementing sixel and kitty backends.

I did a quick and dirty kitty backend proof of concept with the kitty icat program. It works, but the experience isn't great (images flickering when switching screens). The kitty graphics protocol exposes more options that icat doesn't, and should let me address that issue, but requires me to use ImageMagick.

I've only briefly looked at sixel, but I think it should be possible too. The main problem is that I haven't found a good reference for how to actually write the escape sequences. Do I need to use libsixel? Or can I write the escape sequences myself? I think this also requires ImageMagick.

Copy link

@pslijkhuis pslijkhuis Mar 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Libsixel would be awesome. I know Iterm2 on Mac supports it and there's a PR waiting for Alacritty to support it as well.

alacritty/alacritty#4763

Thanks for your awesome work!

src/ncmpcpp.cpp Outdated Show resolved Hide resolved
src/screens/artwork.cpp Outdated Show resolved Hide resolved
@arybczak
Copy link
Collaborator

arybczak commented Jan 24, 2021

Do you think it's worth adding support for fetching covers from the Internet (and storing them in ~/.cache)? AFAIK MPDroid (android mpd client) does that and it's quite neat.

I don't know where it fetches them from though.

@evanc577
Copy link
Contributor Author

evanc577 commented Jan 24, 2021

Fetching art from the internet might be for another PR, this one's scope is already getting pretty big.

I've been playing around with libmpdclient, and even though it doesn't have direct support for the albumart API, I can manually call it and receive the data. Maybe this is a better idea than finding the art within ncmpcpp?

Pros:

  • Works with remote MPD servers.
  • Support for some embedded album art. Since there are a million different standards for embedded artwork, better to have MPD do it for us to avoid duplicating efforts.

Cons:

  • Since the artwork data is coming over the network now, it can potentially block the main thread for a long time.
    • Spawn a dedicated worker thread for reading the artwork data from MPD and displaying it. Need to open another connection to MPD to avoid races. Copy host, port, and password from main MPD connection.
  • MPD sends binary data in 8192B chunks by default. This can result thousands of network round-trips for large images and unacceptable performance.
    • Worked around by setting MPD binarylimit added in 0.22.4 to a larger value, such as 1MB. MPD seems to only send a max chuck size of around 200KB. A 10MB image requires around 40 round-trips, which is acceptable.

I've created a new branch https://github.com/evanc577/ncmpcpp/tree/artwork-mpd with a working implementation. It's only been tested with a local MPD server and no password.

@Barbaross93
Copy link

Barbaross93 commented May 19, 2021

Thanks for the clarification! It's unfortunate that mopidy doesn't have the support (yet, at least).

I suppose as an alternative, it could be possible to extract albumart URLs through mpris (which is exposed through mopidy-mpris), but I have no idea how feasible that would be to include here.

#include <map>
#include <boost/regex.hpp>

#include "charset.h"
#include "mpdpp.h"

MPD::Connection Mpd;
MPD::Connection Mpd_artwork;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since I am using a worker thread to retrieve artwork and MPD connections aren't thread safe, I need another MPD connection. The other option is to put synchronization everywhere. I think using a separate connection is better.

Alignment of album artwork within window. One of "N", "NE", "E", "SE", "S", "SW", "W", "NW", "center".
.TP
.B font_width = pixels
Manually set font width in pixels. Set to 0 to try auto detection, support depends on terminal emulator. Check error log if no image appears.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the font width/height?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the size of an individual character as displayed by the terminal. Basically take a screenshot of the cursor and count how many pixels large it is.

Some terminals don't have a way to detect how large the window actually is in pixels, so this is a manual workaround.


Artwork* myArtwork;

Magick::Blob Artwork::art_buffer;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why so many global variables?

Also, can you briefly describe how that works? I see there's a worker thread, but I'd like to have a high level description of what's going on.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are static member initializations for the Artwork class.

The basic idea of the architecture is:

  1. (main thread) When the artwork needs to be updated, send a command to the worker thread. The commands include updating the artwork or removing it. The main thread is not blocked, continues execution.
  2. (worker thread) Wakes up, processes the command it just received.
    • If an update command is received, fetch the correct image and resize if needed, then call one of the backends (Ueberzug or Kitty) to display it.
    • If remove command, call backend to remove the image from screen.
  3. (worker thread) Goes to sleep until a new command is available.

Artwork fetching is performed by the worker thread, which will either check the local filesystem, or query MPD. Once the raw image data is available, ImageMagick is used to resize and align the image according to the user's configuration.

The Ueberzug backend is implemented fairly easily. On startup, Ueberzug is spawned as a child process with a pipe to its stdin. When the image needs to be displayed or updated, the worker thread writes JSON to Ueberzug's stdin, and it handles the rest. On shutdown, the pipe is closed, which stops Ueberzug, and the process is reaped.

The Kitty backed is a bit more complex. For reference, the Kitty graphics protocol is described here. Since this protocol relies on writing to stdout, there can easily be race conditions if both the main and worker threads both try to write output at the same time. The solution is to only write to stdout from the main thread. On startup, a self pipe is created, with the write end given to the worker thread and the read end polled by the main select() loop. When the worker thread is finished processing a command, it will write some data to the pipe, which will wake up the main thread. The main thread can now write the needed data to stdout without a race.

}

// save current cursor position
std::cout << "\0337";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't using ncurses for moving the cursor work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I did try that before and it was extremely buggy / didn't work. Same with writing terminal control sequences via ncurses.

winsize Artwork::getWinSize()
{
struct winsize sz;
ioctl(STDOUT_FILENO, TIOCGWINSZ, &sz);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is for getting the current window size in pixels. The main issue here is that the terminal operates on characters, but images are measured in pixels, with no direct correlation between them. By knowing the width/height of the terminal window in pixels, and the number of character rows/columns, I can determine the size of each character in pixels and resize the output image correctly.

@vide0hanz
Copy link

Is there a branch for this I could try out?

@evanc577
Copy link
Contributor Author

Is there a branch for this I could try out?

The source branch for this PR is https://github.com/evanc577/ncmpcpp/tree/artwork but review progress seems to have stalled

@ruwey
Copy link

ruwey commented May 22, 2022

Just checked out your fork (amazing by the way) and was wondering: Does local have any point now that mpd_albumart does essentially the same thing? If the difference is in back end, then should there be an albumart_backend option?

@Barbaross93
Copy link

@evanc577 Is it possible to use an alternative backend other than ueberzug or kitty? For example, chafa is a fantastic tool for displaying images directly in the terminal either using ascii, blocks, or sixel format. An ability to use something like chafa, imgcat, or caca, etc. is a viable choice for folks that are running wayland (therefore can't use ueberzug) or any other terminal besides kitty (e.g. foot, alacritty).

@evanc577
Copy link
Contributor Author

@evanc577 Is it possible to use an alternative backend other than ueberzug or kitty? For example, chafa is a fantastic tool for displaying images directly in the terminal either using ascii, blocks, or sixel format. An ability to use something like chafa, imgcat, or caca, etc. is a viable choice for folks that are running wayland (therefore can't use ueberzug) or any other terminal besides kitty (e.g. foot, alacritty).

There aren't any alternatives implemented, but they shouldn't be difficult to implement. Assuming they play well with ncurses.

@arybczak thoughts about merging this PR?

@spvkgn
Copy link

spvkgn commented Jun 23, 2022

@evanc577 Can't build this PR on Ubuntu 20.04.

./configure --enable-artwork PATH=$PATH:/usr/lib/x86_64-linux-gnu/ImageMagick-6.9.10/bin-q16
…
make
…
g++ -DHAVE_CONFIG_H -I. -I..   -DU_USING_ICU_NAMESPACE=0  -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=600 -I/usr/include/x86_64-linux-gnu -I/usr/include/taglib -fopenmp -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16 -fopenmp -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16 -fopenmp -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16 -I/usr/include/x86_64-linux-gnu//ImageMagick-6 -I/usr/include/ImageMagick-6 -I/usr/include/x86_64-linux-gnu//ImageMagick-6 -I/usr/include/ImageMagick-6 -I/usr/include/x86_64-linux-gnu//ImageMagick-6 -I/usr/include/ImageMagick-6  -g -O2 -flto -ftree-vectorize -ffast-math -std=c++14 -MT screens/artwork.o -MD -MP -MF $depbase.Tpo -c -o screens/artwork.o screens/artwork.cpp &&\
mv -f $depbase.Tpo $depbase.Po
screens/artwork.cpp: In member function 'std::vector<unsigned char> KittyBackend::writeChunked(std::map<std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char> >, const Magick::Blob&)':
screens/artwork.cpp:695:42: error: passing 'const Magick::Blob' as 'this' argument discards qualifiers [-fpermissive]
  695 |   std::string b64_data_str = data.base64();
      |                                          ^
In file included from /usr/include/ImageMagick-6/Magick++/Image.h:15,
                 from /usr/include/ImageMagick-6/Magick++.h:12,
                 from ./screens/artwork.h:28,
                 from screens/artwork.cpp:21:
/usr/include/ImageMagick-6/Magick++/Blob.h:48:17: note:   in call to 'std::string Magick::Blob::base64()'
   48 |     std::string base64(void);
      |                 ^~~~~~
screens/artwork.cpp: In constructor 'Artwork::Artwork()':
screens/artwork.cpp:133:6: warning: ignoring return value of 'int pipe(int*)', declared with attribute warn_unused_result [-Wunused-result]
  133 |  pipe(pipefd);
      |  ~~~~^~~~~~~~
screens/artwork.cpp: In member function 'void Artwork::drawToScreen()':
screens/artwork.cpp:169:6: warning: ignoring return value of 'ssize_t read(int, void*, size_t)', declared with attribute warn_unused_result [-Wunused-result]
  169 |  read(pipefd_read, buf, BUF_SIZE);
      |  ~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
screens/artwork.cpp: In member function 'virtual void KittyBackend::setOutput(std::vector<unsigned char>, int, int)':
screens/artwork.cpp:732:7: warning: ignoring return value of 'ssize_t write(int, const void*, size_t)', declared with attribute warn_unused_result [-Wunused-result]
  732 |  write(pipefd_write, dummy, 1);
      |  ~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [Makefile:768: screens/artwork.o] Error 1
make[2]: Leaving directory '/tmp/build/ncmpcpp/src'
make[1]: *** [Makefile:499: all-recursive] Error 1
make[1]: Leaving directory '/tmp/build/ncmpcpp'
make: *** [Makefile:410: all] Error 2

@evanc577
Copy link
Contributor Author

evanc577 commented Jun 27, 2022

@spvkgn The imagemagick error was fixed sometime during imagemagick 7. In IM 6, blob::base64() doesn't mutate itself but it was not marked as const, so the error can probably safely be const_casted away.

I'll try to fix those syscall warnings, though if they ever return with an error something's probably pretty messed up and I'm not sure that there's much that can be done other than giving up.

@evanc577
Copy link
Contributor Author

@spvkgn Try building again. I worked around the IM6 issue making a copy of the variable before calling base64(), the cost should be relatively low because Blob is reference counted...

@camden99cdt
Copy link

This is such an awesome PR! I just compiled it at home and am in love with it. I've used a number of other album art scripts but this seems like the best solution.

Since most of my album art files are named folder.png, I suggest adding a line in the ncmpcpp config where one can set the filenames to search for album art, e.g.:

albumart_filenames = folder.png, cover.png, album.png

It would be also awesome if this variable allowed unknown values like *.jpg, *.png, etc. to accommodate music libraries with a variety of album art filenames.

In the meantime, if I wanted to change where ncmpcpp looks for artwork, would all I have to change is lines 435-438 in src/screens/artwork.cpp ?

@vide0hanz
Copy link

vide0hanz commented Feb 21, 2023

@evanc577 This is looking pretty good. One thing that'd be useful would be to point to a fallback image in the event artwork is missing from the directory. Currently it throws an Unexpected error: No file exsts message, and the artwork from the last track to have data be displayed remains which can result in confusion.

EDIT: Found it buried in the documentation :)

I don't know how feasible something like this would be, or even if its possible within the limitations of ueberzug, curses, etc. but I've always wished for a preview artwork feature to go alongside the library view. Such as, if you highlight an artist then it would automatically recursively search for any cover.jpg/png/tiff/etc on the first track it finds and update the image displayed.

Perhaps a more reasonable feature could be to implement dynamic color switching which utilizes something like Python's ColorThief to update ncmpcpp's colors based on the album artwork being shown. There is a neat little program out there called miniplayer which does this, for example. Funny enough, it also has a nifty little bug which actually sort of already does this to ncmpcpp, ref GuardKenzie/miniplayer#34.

@evanc577
Copy link
Contributor Author

Since most of my album art files are named folder.png, I suggest adding a line in the ncmpcpp config where one can set the filenames to search for album art, e.g.:

albumart_filenames = folder.png, cover.png, album.png

It would be also awesome if this variable allowed unknown values like *.jpg, *.png, etc. to accommodate music libraries with a variety of album art filenames.

Added

@camden99cdt
Copy link

Added

Hell yes thank you! Hope this badboy gets merged some day

@spvkgn
Copy link

spvkgn commented May 31, 2023

@evanc577 There is some build failure on Ubuntu 22.04, please see buildlog

src/screens/artwork.h Outdated Show resolved Hide resolved
@spvkgn
Copy link

spvkgn commented Jun 7, 2023

Ueberzug is not maintained anymore, so it would be useful to add support for Überzug++ replacement.

@ewof
Copy link

ewof commented Oct 31, 2023

ueberzug is being maintained again btw

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet