feat: add hls support #2794

mp3butcher · 2023-12-01T10:05:10Z

Description

implement HLS .
rely on nginx to serve files as liquidsoap hls.harbor is not recommended for direct audience.
The cons is we loose listeners stats if hls.
a mount point have been created in nginx libretime.conf:

 location  /hls { 
     alias @@HLS_PLAYOUT_OUTPUT@@;
  }

Note

Player not upgraded: To test use feat(legacy): upgrade player #2797 or vlc open url http://localhost/hls/main
Will not work when moving to ubuntu 22.04 (liquidsoap 2.0 have differents hls stream definition -with %audio syntax-)

Testing

Tested on ubuntu 20.04 with vagrant
docker bullseye

codecov · 2023-12-01T10:06:32Z

Codecov Report

Attention: 9 lines in your changes are missing coverage. Please review.

Comparison is base (2a91320) 70.35% compared to head (c7e95fd) 70.44%.
Report is 4 commits behind head on main.

Files	Patch %	Lines
shared/libretime_shared/cli.py	0.00%	5 Missing ⚠️
playout/libretime_playout/liquidsoap/main.py	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2794      +/-   ##
==========================================
+ Coverage   70.35%   70.44%   +0.09%     
==========================================
  Files         149      149              
  Lines        4034     4067      +33     
==========================================
+ Hits         2838     2865      +27     
- Misses       1196     1202       +6

Flag	Coverage Δ
analyzer	`46.91% <ø> (ø)`
api	`93.72% <ø> (ø)`
api-client	`64.40% <ø> (ø)`
playout	`47.79% <73.33%> (+0.39%)`	⬆️
shared	`88.48% <76.19%> (-0.62%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mp3butcher · 2023-12-01T12:12:52Z

i don't understand ci message (sorry I have versions problems with prettier...can't use git formatting)
https://github.com/libretime/libretime/actions/runs/7059733283/job/19217904794?pr=2794
if someone does, he'd be welcome...

paddatrapper · 2023-12-12T12:37:30Z

I hope to properly test this in the coming weeks. Could you also please update the docs describing this feature and how to configure/use it?

mp3butcher · 2023-12-12T13:39:05Z

I hope to properly test this in the coming weeks. Could you also please update the docs describing this feature and how to configure/use it?

The config file template is pretty self explanatory...HLS is composed of several streams with different quality embed in container with a format so you have to describe the container format and streams via a tuple (streamname,codec,bitrate)...
In the template i enable a hls output with 3 mp3 streams of differents quality....

…ern have to be modded

mp3butcher · 2023-12-12T15:37:54Z

playout/libretime_playout/liquidsoap/1.4/ls_lib.liq

make a function because output.file.hls varies accross liquidsoap versions

installer/nginx/libretime.conf

shared/libretime_shared/config/_models.py

paddatrapper · 2023-12-13T12:27:27Z

Is there a way to stream HLS to nginx instead of doing it via a file?

mp3butcher · 2023-12-13T15:08:57Z

Is there a way to stream HLS to nginx instead of doing it via a file?

http://nginx-rtmp.blogspot.com/2017/07/introducing-nginx-ts-module-for-hls-and.html can be a lead
It may be possible to stream with unsafe hls.harbor on publish and serve on play

legacy/application/models/StreamSetting.php

paddatrapper · 2023-12-13T21:30:39Z

After discussion, I think we should continue with the file-based solution instead - it is cleaner and induces less delay

jooola

Hi, I made first pass.

I think we should implement a simple working HLS feature, and iterate to add more options. Supporting too many formats/codecs is also risky, let's do not diverge from the spec.

I already started working on HLS a while back, did you take a look at my PR: #2234

jooola · 2023-12-18T16:58:53Z

legacy/application/configs/conf.php

+            $values['stream']['outputs']['shoutcast'],
+            $values['stream']['outputs']['hls']


Let's not merge HLS streams in the icecast/shoutcast streams. We can benefit from a fresh/clean settings schema, and I don't see the benefit from merging this in the "merged" streams.

It's mandatory to do that, it doesn't work if i remove this line (merged is the array parsed in StreamSettings.php)

Why is it mandatory? "It doesn't work" doesn't seem like a valid argument sorry 😉

Streams settings data (url,codec..) presented in the player are retrieved from this merged outputs array

shared/libretime_shared/config/_models.py

legacy/application/models/StreamSetting.php

jooola · 2023-12-18T17:12:54Z

installer/config.yml

+        # > must be one of ('mpegts', 'mp3', 'adts', 'mp4')
+        # > this field is REQUIRED


Let's reduce the amount of format for now, extra format can be added and tested in future PRs.

AC3/ADTS is in the spec, further it work as it should with hls.js so i'll let it .

I'll remove mp4 because i haven't achieve anything with liquidsoap 1.4 even if in spec...:/

ok to remove mp3

Also, I'll will move format property as stream property to fit liquidsoap stream model (it was a mistake to put it as hls property)

edit mp4 seems to require movflags=+dash+skip_sidx+skip_trailer+frag_custom+global_header

installer/config.yml

jooola · 2023-12-18T17:18:13Z

install

@@ -628,6 +628,7 @@ link_python_app libretime-playout-notify

 info "creating libretime-playout working directory"
 mkdir_and_chown "$LIBRETIME_USER" "$WORKING_DIR/playout"
+mkdir_and_chown "$LIBRETIME_USER" "$WORKING_DIR/playout/hls"


I prefer to have playout create the "$WORKING_DIR/playout/hls" directory, the "$WORKING_DIR/playout" is writable by libretime.

the hls directory must be created before nginx starts so we can't let playout create this directory...no?

Why ? We only read in this directory during a request, and response with 404 if no files are found. Sound doable to me.

if you think it's safe to launch nginx and serve a not-yet-created directory, so i'm good with removing this line but it will require testing (with docker it's required -liquidsoap can't create output directory-)

jooola · 2023-12-18T17:20:13Z

legacy/application/models/StreamSetting.php

+                $result = array_merge($result, [
+                    $prefix . 'port' => $_SERVER['SERVER_PORT'],
+                    $prefix . 'host' => $_SERVER['SERVER_NAME'],
+                    $prefix . 'type' => 'hls',
+                    $prefix . 'bitrate' => '',
+                ]);


Hmm, I think we prefer to use the config public url and not the webserver configs.

in the present configuration, you can use both as with other outputs:

default is webserver

if public url is provided it's use instead

The rest of the project doesn't rely on the web server configuration, this would diverge from the current expectation.

so you tell me to add settings for web server host/port?

We use the Config::get('general.public_url') (with the _raw key you get access to the URL object (League\Uri\Uri;).

libretime/legacy/application/models/StreamSetting.php

Lines 70 to 81 in e2cfbf4

if (!$result[$prefix . 'public_url']) {

$host = Config::get('general.public_url_raw')->getHost();

$port = $result[$prefix . 'port'];

$mount = $result[$prefix . 'mount'];

$result[$prefix . 'public_url'] = "http://{$host}:{$port}/{$mount}";

if ($result[$prefix . 'output'] == 'shoutcast') {

// The semi-colon is important to make Shoutcast stream URLs play instead turn into a page.

$result[$prefix . 'public_url'] .= ';';

}

}

But I think we create an new function and not use the backward compatible and complex Application_Model_StreamConfig::getOutput function.

mp3butcher · 2023-12-26T21:41:37Z

playout/libretime_playout/liquidsoap/templates/outputs.liq.j2

+
+{% for stream in output.streams -%}
+
+    streamout = %ffmpeg(format="{{output.format}}" ,


note that %ffmpeg (as well as %audio for liquidsoap>2.0) arguments can't be dynamically passed through a function we can customize for different lq versions (as i did for output.file.hls).....So we can't maintain hls for both lq 1.4 and 2.0

For the futur, if you really need to handle the %functions, you may use jinja templating and check against the version during the startup of playout.

mp3butcher and others added 2 commits December 1, 2023 10:08

hls support first commit

7efe718

Update config.yml for hls config

08b9d0e

mp3butcher changed the title ~~Hls support~~ feat(legacy) Hls support Dec 1, 2023

mp3butcher added 2 commits December 1, 2023 11:33

remove useless fields

ed56081

CI compliance

17f47ca

mp3butcher changed the title ~~feat(legacy) Hls support~~ feat(legacy): Hls support Dec 1, 2023

mp3butcher changed the title ~~feat(legacy): Hls support~~ feat(legacy): hls support Dec 1, 2023

mp3butcher and others added 2 commits December 1, 2023 13:23

Update install and config

bc5050c

make compatible with liquidsoap 1.4 output.file.hls prototype

3119454

mp3butcher force-pushed the hls branch from 5016431 to 3119454 Compare December 2, 2023 17:10

paddatrapper changed the title ~~feat(legacy): hls support~~ feat(playout): hls support Dec 3, 2023

mp3butcher added 2 commits December 12, 2023 16:19

liquidsoap 2.2 removed streaminfos parameter so crossversion hls patt…

c62cb8f

…ern have to be modded

add an alias to playout hls output

8f3bde7

mp3butcher commented Dec 12, 2023

View reviewed changes

use SERVER_PORT for hls stream

61aa1fb

paddatrapper reviewed Dec 12, 2023

View reviewed changes

liquidsoap 1.4 require differents streams bitrates for player to select

76010bc

mp3butcher force-pushed the hls branch 2 times, most recently from 5f95254 to bd4a99d Compare December 13, 2023 12:35

remove useless host port settings

45af9ed

mp3butcher force-pushed the hls branch from bd4a99d to 45af9ed Compare December 13, 2023 12:37

document hls output

faa3f86

add an alias to playout hls output

7da16a1

mp3butcher commented Dec 13, 2023

View reviewed changes

legacy/application/models/StreamSetting.php Show resolved Hide resolved

mp3butcher added 2 commits December 13, 2023 17:24

simplification

a439af8

add HlsStream to imports

4872cae

mp3butcher force-pushed the hls branch from f0a9d0b to 4872cae Compare December 13, 2023 16:35

Update configuration.md

5ee1e84

mp3butcher force-pushed the hls branch from a4cfd46 to 50980cc Compare December 18, 2023 07:15

jooola changed the title ~~feat(playout): hls support~~ feat: add hls support Dec 18, 2023

add missing changes

28e83ad

mp3butcher force-pushed the hls branch from 86f3e4b to 629f5c1 Compare December 18, 2023 16:58

add hls testing

2723f2b

mp3butcher force-pushed the hls branch from 629f5c1 to 2723f2b Compare December 18, 2023 17:10

jooola requested changes Dec 18, 2023

View reviewed changes

mp3butcher added 2 commits December 20, 2023 21:05

change hls configuration

2d62117

make hls output path an install setting

0833239

mp3butcher force-pushed the hls branch 3 times, most recently from 8b9dea9 to 29db68a Compare December 20, 2023 20:22

mp3butcher commented Dec 26, 2023

View reviewed changes

mp3butcher added 3 commits January 19, 2024 17:18

update test

44f2dae

update configuration files and manual

95316d6

setup docker files

07d9448

mp3butcher force-pushed the hls branch from 9b1af65 to 07d9448 Compare January 19, 2024 16:36

mp3butcher added 2 commits January 19, 2024 17:55

Merge remote-tracking branch 'upstream/main' into hls

3a6aefd

merge misses

6f76da0

mp3butcher force-pushed the hls branch from b242784 to 6f76da0 Compare January 19, 2024 17:22

test miss

06cf701

mp3butcher force-pushed the hls branch from 09bf3d5 to 06cf701 Compare January 19, 2024 17:53

update pytest snapshot

c7e95fd

mp3butcher force-pushed the hls branch from fd7e8ff to c7e95fd Compare January 19, 2024 22:13

mp3butcher requested a review from jooola January 20, 2024 17:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add hls support #2794

feat: add hls support #2794

mp3butcher commented Dec 1, 2023 •

edited

codecov bot commented Dec 1, 2023 •

edited

mp3butcher commented Dec 1, 2023 •

edited

paddatrapper commented Dec 12, 2023

mp3butcher commented Dec 12, 2023 •

edited

mp3butcher Dec 12, 2023 •

edited

paddatrapper commented Dec 13, 2023

mp3butcher commented Dec 13, 2023 •

edited

paddatrapper commented Dec 13, 2023

jooola left a comment

jooola Dec 18, 2023

mp3butcher Dec 20, 2023 •

edited

jooola Jan 17, 2024

mp3butcher Jan 18, 2024 •

edited

jooola Dec 18, 2023

mp3butcher Dec 20, 2023 •

edited

jooola Dec 18, 2023

mp3butcher Dec 20, 2023

jooola Jan 17, 2024

mp3butcher Jan 19, 2024 •

edited

jooola Dec 18, 2023

mp3butcher Dec 18, 2023

jooola Dec 18, 2023

mp3butcher Dec 18, 2023

jooola Dec 18, 2023 •

edited

jooola Dec 18, 2023

mp3butcher Dec 26, 2023 •

edited

jooola Jan 17, 2024

		$values['stream']['outputs']['shoutcast'],
		$values['stream']['outputs']['hls']

		# > must be one of ('mpegts', 'mp3', 'adts', 'mp4')
		# > this field is REQUIRED

	if (!$result[$prefix . 'public_url']) {
	$host = Config::get('general.public_url_raw')->getHost();
	$port = $result[$prefix . 'port'];
	$mount = $result[$prefix . 'mount'];

	$result[$prefix . 'public_url'] = "http://{$host}:{$port}/{$mount}";

	if ($result[$prefix . 'output'] == 'shoutcast') {
	// The semi-colon is important to make Shoutcast stream URLs play instead turn into a page.
	$result[$prefix . 'public_url'] .= ';';
	}
	}


		{% for stream in output.streams -%}

		streamout = %ffmpeg(format="{{output.format}}" ,

feat: add hls support #2794

Are you sure you want to change the base?

feat: add hls support #2794

Conversation

mp3butcher commented Dec 1, 2023 • edited

Description

Note

Testing

codecov bot commented Dec 1, 2023 • edited

Codecov Report

mp3butcher commented Dec 1, 2023 • edited

paddatrapper commented Dec 12, 2023

mp3butcher commented Dec 12, 2023 • edited

mp3butcher Dec 12, 2023 • edited

Choose a reason for hiding this comment

paddatrapper commented Dec 13, 2023

mp3butcher commented Dec 13, 2023 • edited

paddatrapper commented Dec 13, 2023

jooola left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mp3butcher Dec 20, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mp3butcher Jan 18, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mp3butcher Dec 20, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mp3butcher Jan 19, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jooola Dec 18, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mp3butcher Dec 26, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mp3butcher commented Dec 1, 2023 •

edited

codecov bot commented Dec 1, 2023 •

edited

mp3butcher commented Dec 1, 2023 •

edited

mp3butcher commented Dec 12, 2023 •

edited

mp3butcher Dec 12, 2023 •

edited

mp3butcher commented Dec 13, 2023 •

edited

mp3butcher Dec 20, 2023 •

edited

mp3butcher Jan 18, 2024 •

edited

mp3butcher Dec 20, 2023 •

edited

mp3butcher Jan 19, 2024 •

edited

jooola Dec 18, 2023 •

edited

mp3butcher Dec 26, 2023 •

edited