Use m3u-filter as m3u proxy

[adocampo]

Currently, I'm using xteve+plex, but I want to test TVHeadend + Antennas + Plex, as xteve + plex is not perfect. But if I set the huge playlist on TVhheadend it becomes completely irresponsive... I think if instead of passing a playlist of 150.000 channels, I set a playlist with 150 channels, it would work quite good. That's why I'm trying your software.

The thing is, I don't know really if m3u-filter can serve this filtered playlist or if it can update the source.... it is quite difficult to set up, as I don't find very intuitive. I just have an URL, I don't have or use xtream, nor I want anything related, just grab the source m3u, filter it, and serve the filtered m3u.

Do you think it's possible?

For example, my URL is something like:

http://iptv.example.com:80/get.php?username=aa123456&password=supersecretpassword&type=m3u_plus&output=mpegts

I've configured m3u-filter with this config.yml

 api: {host: 0.0.0.0, port: 8901, web_root: ./web}
 working_dir: ./data
 update_on_boot: true
  web_ui_enabled: true

and source.yml

 templates:
 - name: ALL_CHAN
   value: 'Group ~ ".*"'
 sources:
 - inputs:
     - type: xtream
       #url: 'http://iptv.example.com:80'
       url: 'http://iptv.example.com:80/get.php?username=aa123456&password=supersecretpassword&type=m3u_plus&output=mpegts'
       epg_url: 'http://iptv.example.com:80/xmltv.php?username=aa123456&password=supersecretpassword'
       username: aa123456
       password: supersecretpassword
       options: {xtream_info_cache: true}
   targets:
     - name: all_channels
       output:
         - type: xtream
       filter: "!ALL_CHAN!"
       options: {ignore_logo: false, xtream_skip_live_direct_source: true, xtream_skip_video_direct_source: true}
       sort:
         match_as_ascii: true
         groups:
           order: asc

And docker starts, but it has some error on start.

m3u-filter  | [2024-05-11T23:24:30Z INFO  m3u_filter::processing::playlist_processor] source is empty http://iptv.example.com:80/get.php?username=aa123456&password=supersecretpassword&type=m3u_plus&output=mpegts
m3u-filter  | [2024-05-11T23:24:30Z INFO  m3u_filter::processing::playlist_processor] {"stats": {"name": http://iptv.example.com:80/get.php?username=aa123456&password=supersecretpassword&type=m3u_plus&output=mpegts, "type": xtream, "errors": 0, "raw": {"groups": 0, "channels": 0}, "processed": {"groups": 0, "channels": 0}}}
m3u-filter  | [2024-05-11T23:24:30Z ERROR m3u_filter::processing::playlist_processor] cant download input url: http://iptv.example.com:80/get.php?username=aa123456&password=supersecretpassword&type=m3u_plus&output=mpegts/player_api.php?username=aa123456&password=supersecretpassword&action=get_live_categories  => Request failed: 404 Not Found
m3u-filter  | [2024-05-11T23:24:30Z ERROR m3u_filter::processing::playlist_processor] cant download input url: http://iptv.example.com:80/get.php?username=aa123456&password=supersecretpassword&type=m3u_plus&output=mpegts/player_api.php?username=aa123456&password=supersecretpassword&action=get_vod_categories  => Request failed: 404 Not Found
m3u-filter  | [2024-05-11T23:24:30Z ERROR m3u_filter::processing::playlist_processor] cant download input url: http://iptv.example.com:80/get.php?username=aa123456&password=supersecretpassword&type=m3u_plus&output=mpegts/player_api.php?username=aa123456&password=supersecretpassword&action=get_series_categories  => Request failed: 404 Not Found
m3u-filter  | [2024-05-11T23:24:30Z ERROR m3u_filter::processing::playlist_processor] source is empty http://iptv.example.com:80/get.php?username=aa123456&password=supersecretpassword&type=m3u_plus&output=mpegts
m3u-filter  | [2024-05-11T23:24:30Z ERROR m3u_filter::processing::playlist_processor] Source at 0 input is empty

But and I can see this after a manual update

The arrow point to the group I want to filter, but if I put this

it shows the one I want and another named "Switzerland"... besides, I want to ged rid of most (not all) SD and HD channels inside MOVISTAR + group

Can you please point how to accomplish it?

On the other hand, my proxy doesn't seems to work either, I don't know if it is because I have issues on the source list, or why. My api-proxy.yml is like this:

 server:
 - name: default
   protocol: http
   host: 192.168.1.100
   http_port: '8901'
   https_port: '443'
   rtmp_port: '1935'
   timezone: Europe/Madrid
   message: Welcome to m3u-filter
 user:
 - target: all_channels
   credentials:
   - username: username
     password: password
     proxy: redirect
     server: null

but opening a browser to http://192.168.1.100:8901/get.php?username=username&password=password

m3u-filter  | [2024-05-11T23:35:29Z INFO  actix_web::middleware::logger] 192.168.1.10 "GET /get.php?username=username&password=password HTTP/1.1" 204 0 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0" 0.000151
m3u-filter  | [2024-05-11T23:35:29Z INFO  actix_web::middleware::logger] 192.168.1.10 "GET /robots.txt HTTP/1.1" 404 0 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0" 0.000045
m3u-filter  | [2024-05-11T23:35:36Z ERROR m3u_filter::repository::m3u_repository] Target has no filename all_channels

Any idea what am I doing wrong?

Thank you!

[euzu]

@adocampo this is exactly what m3u-filter does. It fetches playlists from different sources and can create different outputs. The access to this newly created playlists can.be controlled by credentials.

Yes the configuration is a little bit complex.
Your config is wrong. If you have no xtream account and only a m3u url then do not use xtream type. The xtream url ends with the port and does not have any path section.

I will create a simple example config for you.

[euzu]

@adocampo I have created an example config for you: config.zip

Adocamo Example config

config.yml

This is a basic config.yml file. The Web-Authentication is disabled.

schedule: 0 0 8,18 * * * * Means it will update the playlist twice at 08:00 and 18:00.
But this depends on your docker time settings, it can be that your docker container has a different time.

If you dont want web-authentication because you use it only in your private network then skip
to next section source.yml. If you want to enable, you need to set

web_auth:
  enabled: true

And you need to generate a password with

./m3u-filter --genpwd

The created password should be set in the user.txt.

The format is username: password

adocampo: $argon2id$v=19$m=19456,t=2,p=1$ZnY1NUNYU215Q1dFQ3V0MG54Wlh2T1ZRZ3UzQ0JzMTN6Tzc4cmdFSHQ3ZnY1YzRyOEMxempqblZsanJlODZZaA$JvvXhUlHDaptb86xiPNjTo1Yt/IcjLfOwlaa/uQIkT0

source.yml

First you need to define your filter templates. The current example filters everything what is not what you want.

templates:
- name: ALL_CHANNEL
  value: 'Group ~ ".*"'

You need to set the right m3u url

- inputs:
  - type: m3u
    url: 'http://iptv.example.com:80/get.php?username=aa123456&password=supersecretpassword&type=m3u_plus&output=mpegts'

In the example the target name is set to all_cahnnels You can give the target a more meaningful name.
If you change the name you have to change it in api_proxy.yml too, because a user has access to a target.

   targets:
  - name: all_channels

I have added a exampe mapping which is optional, you can remove it.
What it does is: it looks for channels with the ending FHD, SD, HD and groups them.

    mapping:
      - group_quality

The name group_quality has to match the id setting in the mapping.yml.

mapping.yml

Is optional, if you dont need it, then remove the mapping section in source.yml
What it does is: it looks for channels with the ending FHD, SD, HD and groups them.

api_proxy.yml

This is for accessing the filtered playlists.

The default server configuration has to match your m3u-filter server.
Please set host and port to the right values.

server:
- name: default
  protocol: http
  host: 192.168.0.100
  http_port: 8901

Ignore the external server configuration if you only access your playlist in your local network.

!!!! The target name has to match the target name in source.yml !!!!

api_proxy.yml

user:
- target: all_channels  <--- !! THIS !!!

source.yml

targets:
  - name: all_channels   <--- !! THIS !!!

You can define a user per target. Token is optional and can be used instead of username/password combi

user:
- target: all_channels
  credentials:
  - username: adocampo
    password: adocampo.secret
    token: E8qtpNy3txZy
    proxy: redirect
    server: default

The defined user can choose from the following m3u urls to access the playlist. (You need only one not all)

• http://192.168.0.100:8901/get.php?username=adocampo&password=adocampo.secret
• http://192.168.0.100:8901/get.php?token=E8qtpNy3txZy
• http://192.168.0.100:8901/m3u?username=adocampo&password=adocampo.secret
• http://192.168.0.100:8901/m3u?token=E8qtpNy3txZy

[euzu]

@adocampo with the filter you can cherry pick everything you want.

This not.possible with ui.

If you need help with fikter, then provide me
group, name or title of the channels and describe what you expect as result.

But i am currently not at home. It could take a while to respond.

[adocampo]

No, I'm not getting the group I want, on the log I can see:

[2024-05-12T16:07:32Z INFO  m3u_filter::processing::playlist_processor] {"stats": {"name": http://iptv.example.com:80/get.php?username=aa123456&password=supersecretpassword&type=m3u_plus&output=mpegts, "type": m3u, "errors": 0, "raw": {"groups": 105, "channels": 27764}, "processed": {"groups": 0, "channels": 0}}}

It seems it can see the groups (105), but the filter seems to be not filtering.
I've tried with

 templates:
 - name: ALL_CHANNEL
   value: 'Group ~ "ES MOVISTAR+ [FHD]"'

but also changing the last line to

   value: 'Group ~ "ES MOVISTAR\+"'
   value: 'Group ~ "ES\ MOVISTAR\+"'
   value: 'Group ~ "ES MOVISTAR\+"'
   value: 'Group ~ "^ES\sMOVISTAR\+"'
   value: 'Group ~ "^ES\sMOVISTAR\s.*"'
   value: 'Group ~ "^ES MOVISTAR\s.*"
   value: Group ~ "^ES\sMOVISTAR\+.*"

none of the works, the only one which seems to work is

value: Group ~ "^ES\s.*"

but it takes all the groups starting with "ES", 30 in total, and I just want one...

Anyway, it went down from 275600 channels to 1160 channels, quite a lot less, and now TVHeadend in scanning, now it's just very slow, but the interface is responsive, at least :)

Obviously, the quality inside are FHD, HD, and SD, but I think tvheadend will be able to cherrypick the channels and publish them to plex in a more graphical way, but I would need to get the MOVISTAR filter work to reduce from 1160 channels to 160 or so...

[euzu]

@adocampo the regular expression is wrong.

The Filter has chars with special meanings
Group ~ "ES MOVISTAR+ [FHD]"

• is one or more, in your case it matches one or more R

[FHD] means one of F, H or D not FHD

Your expression.matches for example:

• ES MOVISTARRRR F
• ES MOVISTAR H
• ....

I think you should learn about regular expressions.

There is a page where you can test your regexps. [regex101(https://regex101.com/)

Dont forget to set Rust on the left side.

If you need help then explain me what you want to achieve.

[adocampo]

The group is not "ES MOVISTAR+ [FHD]", but "ES MOVISTAR+".
I'm not an expert on regexp, just basic usage, but shouldn't ^ES\ MOVISTAR.* or ^ES\sMOVISTAR.* match the group name? I mean I don't care the special characters like the plus symbol, just take any match of "ES MOVISTAR"... is what ^ES\s.* does and works, take anything starting with "ES".

[euzu]

Please look at my last comment at the end of this discussion.

[euzu]

@adocampo i forgot, you can use update_on_boot: true in config.yml if you want the playlist updated each time m3u_filter starts.

[euzu]

@adocampo here is an example filter

templates:
- name: moviestar_group
  value: 'Group ~ "ES\sMOVIESTAR\+"'
- name: fhd_channels
  value: 'Name ~ ".*FHD$"'
- name: not_working
  value: >-
        Name ~ "ES\:\sAMC\sFHD$"
        OR 
        Name "ES\:\sLA\s2\sFHD$"
- name: working_hd
  value: >-
        Name ~ "ES\:\sAMC\sHD$"
        OR 
        Name "ES\:\sLA\s2\sHD$"
- name: moviestar
  value: >-
        !moviestar_group!
        AND 
        ((!fhd_channels! 
            AND Not(!not_working!)) 
          OR !working_hd!)

sources:
- inputs:
  - type: m3u
    url: 'http://iptv.example.com:80/get.php?username=aa123456&password=supersecretpassword&type=m3u_plus&output=mpegts'
    name: iptv_example
  targets:
  - name: all_channels
    output:
      - type: m3u
        filename: playlist.m3u
    filter: "!moviestar!"
    options: {ignore_logo: false}

I wrote this on my phone, it can contain errors

[adocampo]

that's supposed to be sources.yml, right?

I don't really see any difference, well, now it seems to filter 27.788 channels instead of 1.160, so the group filter is ignored...

m3u-filter  | [2024-05-12T22:26:20Z INFO  m3u_filter::processing::playlist_processor] {"stats": {"name": iptv_example, "type": m3u, "errors": 0, "raw": {"groups": 105, "channels": 27788}, "processed": {"groups": 0, "channels": 0}}}

[euzu]

How do you check the result?
You need to call the url in the browser.
I wrote you some url's as examples.

If you can provide me an example m3u file i can set up the filter for you.

[adocampo]

I'm checking the result in the m3u-filter docker log output when it says it filters groups, and importing the list in tvheadend or in an iptv player when it says it filters zero groups.
Yes, the m3u that I import to those players is the one downloaded via http from m3u-filter.

The original mu3 playlist is this one: ...

And this is the m3u-filter generated m3u playlist downloaded right now from my m3u-filter instance with the following settings:
...

source.yml

templates:
- name: movistar_group
  value: 'Group ~ "ES\sMOVISTAR\+"'
- name: fhd_channels
  value: 'Name ~ ".*FHD$"'
- name: not_working
  value: >-
        Name ~ "ES\:\sAMC\sFHD$"
        OR 
        Name "ES\:\sLA\s2\sFHD$"
- name: working_hd
  value: >-
        Name ~ "ES\:\sAMC\sHD$"
        OR 
        Name "ES\:\sLA\s2\sHD$"
- name: moviestar
  value: >-
        !movistar_group!
        AND 
        ((!fhd_channels! 
            AND Not(!not_working!)) 
          OR !working_hd!)

sources:
- inputs:
  - type: m3u
    url: 'http://iptv.example.com:80/get.php?username=aa123456password=123456&type=m3u_plus&output=mpegts'
    name: iptv_example
  targets:
  - name: all_channels
    output:
      - type: m3u
        filename: playlist.m3u
    filter: "!movistar_group!"
    options: {ignore_logo: false}

mapping.yml

---
 mappings:
   templates:
     - name: quality_channel
       value: 'Name ~ ".*\s(?P<quality>FHD)"'
 
   tags:
     - name: quality
       captures:
         - quality
       concat: "|"
       prefix: " ["
       suffix: "]"
   mapping:
     - id: group_quality
       match_as_ascii: true
       mapper:
         - filter: "!quality_channel!"
           pattern: "!quality_channel!"
           suffix:
             group: '<tag:quality>'

api-proxy.yml

server:
- name: default
  protocol: http
  host: 192.168.1.100
  http_port: '8901'
  https_port: '443'
  rtmp_port: '1935'
  timezone: Europe/Madrid
  message: Welcome to m3u-filter
user:
- target: all_channels
  credentials:
  - username: malevolent
    password: supersecretpassword
    token: E8qtpNy3txZy
    proxy: redirect
    server: null

user.txt

adocampo: $argon2id$v=19$m=19456,t=2,p=1$ZnY1NUNYU215Q1dFQ3V0MG54Wlh2T1ZRZ3UzQ0JzMTN6Tzc4cmdFSHQ3ZnY1YzRyOEMxempqblZsanJlODZZaA$JvvXhUlHDaptb86xiPNjTo1Yt/IcjLfOwlaa/uQIkT

config.yml

api: {host: 0.0.0.0, port: 8901, web_root: ./web}
working_dir: ./data
update_on_boot: true
schedule: 0  0  8,18  *  *  *  * 
web_ui_enabled: true
web_auth:
  enabled: false
  secret: 77ac1ca497ba7153ea11a4d5695f7024
  issuer: m3u_filter
  userfile: user.txt

[euzu]

@adocampo you gave me wrong information. In the provided file is no group named ES: MOVIESTAR+

You can filter Group, Name and Title. If you look in the provided file then for example
#EXTINF:-1 tvg-id="moviestar.bg" tvg-name="BG: MOVIESTAR" tvg-logo="" group-title="Bulgaria",BG: MOVIESTAR

• Name => tvg-name="BG: MOVIESTAR"
• Group => group-title="Bulgaria"
• Title => BG: MOVIESTAR

Based on this file i will explain how to filter.

As an example we take the "ES MOVIESTART+" group. We assume that ES: TELECINCO FHD and ES: ANTENA 3 FHD are not working and want to replace them with their HDQuality.

Based on the file you provided, your source.yml should look like: (Dont forget to replace url: '<YOUR PROVIDER URL>')

templates:
- name: moviestar_group
  value: 'Group ~ "^ES\s+MOVISTAR\+$"'
- name: fhd_channels
  value: 'Name ~ ".*FHD$"'
- name: not_working
  value: >-
        (Name ~ "ES\:\sANTENA\s3\sFHD$"
        OR 
        Name ~ "ES\:\sTELECINCO\sFHD$")
- name: working_hd
  value: >-
        (Name ~ "ES\:\sANTENA\s3\sHD$"
        OR 
        Name ~ "ES\:\sTELECINCO\sHD$")
- name: moviestar
  value: >-
        !moviestar_group!
        AND 
        ((!fhd_channels! AND Not(!not_working!)) OR !working_hd!)

sources:
- inputs:
  - type: m3u
    url: '<YOUR PROVIDER URL>'
    name: iptv_example
  targets:
  - name: all_channels
    output:
      - type: m3u
        filename: playlist.m3u
    filter: "!moviestar!"
    options: {ignore_logo: false}

When i run it, and look at http://localhost:8901/m3u?username=adocampo&password=adocampo.secret

I will get exactly what i filtered, all Moviestar+ channels and 2 Channels replaced with their HD Quality

In the log you need to look at the processed part -> "processed": {"groups": 1, "channels": 245}

{"stats": {"name": iptv_example, "type": m3u, "errors": 0, "raw": {"groups": 105, "channels": 27779}, "processed": {"groups": 1, "channels": 245}}}

[adocampo]

In the provided file is no group named ES: MOVIESTAR+

Sorry, but as I saw this

and everywhere where I saw, the group name shown was "ES MOVISTAR+", that's why I said you that...

I don't really understand the changes, I see now the filter is 'Group ~ "^ES\s+MOVISTAR\+$"' instead of 'Group ~ "^ES\s+MOVISTAR\+"' and in sources, filter is "!moviestar!" instead of "!moviestar_group!".
But now IT WORKS! I need to play with cherry-picking all the channels but having a couple of examples, I will do it! :D

Thank you for your awesome support and time @euzu!! You're the best! ;)

PS: my two cents: If UI would let the user like me to choose the groups and channels by using checkboxes, or using a dialog like xteve does

I think it would make m3u-filter way more user-friendly.

Again, thank you A LOT for your help!

[euzu]

@adocampo you are welcome,

The difference is: In the source m3u playlist there are 2 spaces ES<space><space>MOVIESTAR+
And \s matches only one space \s+ matches one or more.

The !moviestar_group! filter only filters the Moviestar group. !moviestar! does cherry pick.

m3u-filter has a powerfull filter mechanism once you understand regular expressions.
Yes it is complicated for users who are not familiar with it.
When you only check checkboxes inside the UI this filter mechanism is not used.

The m3u-filter are regexp comparisons which you can combine with AND and OR logical operators,
you can group them with paranthesis ( ) and negate them with Not.

In your example your filter is:

Group ~ "^ES\s+MOVISTAR\+$" 
AND 
    ( 
      (  Name ~ ".*FHD$" 
         AND 
         NOT ( Name ~ "ES\:\sANTENA\s3\sFHD$" 
                      OR 
                     Name ~ "ES\:\sTELECINCO\sFHD$"
                  )
       ) 
       OR (Name ~ "ES\:\sANTENA\s3\sHD$" OR Name ~ "ES\:\sTELECINCO\sHD$")
     )