Doesn't work for big channels

[smaugfm]

Describe the bug
When I setup simple connection for just newly created test channel to forward to another channel - it works fine.
When I try to set it up for some real world channels I use every day - it doen't see (and forward) new messages or skips them (forwards like 1 out of 5).

To Reproduce
Setup connection for the Telegram channel@novinach, for example.

Expected behavior
Must forward all the messages from the channel.

System information:
Ubuntu 20, Docker, x64

There is nothing in the logs as well:

tgcf
Running latest tgcf version 1.1.8


[03/13/24 08:21:40] INFO     Verbosity turned on! This is suitable for cli.py:57
                             debugging                                          
                    INFO     cryptg detected, it will be used for      aes.py:19
                             encryption                                         
                    INFO     tgcf.config.json detected!            config.py:110
                    INFO     config.py got executed                config.py:247
                    INFO     using session string                  config.py:252
                    INFO     Connecting to                  mtprotosender.py:226
                             149.154.167.51:443/TcpFull...                      
                    INFO     Connection to                  mtprotosender.py:277
                             149.154.167.51:443/TcpFull                         
                             complete!                                          
[03/13/24 08:21:41] INFO     config.is_bot=False                     live.py:140
                    INFO     Command prefix is . for userbot and live_bot.py:136
                             / for bot                                          
                    INFO     Loaded admins are [42489179,          config.py:207
                             6503451555]                                        
                    INFO     Added event handler for new             live.py:151
                    INFO     Added event handler for edited          live.py:151
                    INFO     Added event handler for start           live.py:151
                    INFO     Added event handler for forward         live.py:151
                    INFO     Added event handler for remove          live.py:151
                    INFO     Added event handler for style           live.py:151
                    INFO     Added event handler for help            live.py:151
                    INFO     From to dict is {-1002136041380:      config.py:200
                             [-1002043201462], -1001220606936:                  
                             [-1002043201462], -1001134948258:                  
                             [-1002043201462]}                                  
[03/13/24 08:34:32] INFO     Got difference for account updates   updates.py:302
[03/13/24 08:37:59] INFO     Got difference for channel           updates.py:371
                             1876571966 updates                                 
[03/13/24 08:49:32] INFO     Got difference for account updates   updates.py:302
                    INFO     Got difference for channel           updates.py:371
                             1220606936 updates                                 
                    INFO     Message edited in -1001220606936         live.py:67
[03/13/24 08:52:59] INFO     Got difference for channel           updates.py:371
                             1876571966 updates                                 
[03/13/24 09:04:32] INFO     Got difference for account updates   updates.py:302
                    INFO     Got difference for channel           updates.py:371
                             1220606936 updates                                 
                    INFO     New message received in -1001220606936   live.py:26
[03/13/24 09:08:15] INFO     Got difference for channel           updates.py:371
                             1876571966 updates

[manojsitapara]

I am also facing the same issue, have you found any workaround?

[poliandro]

I heard that this is a Telegram issue, apparently they sent updates from big channels with some delay, while for smaller channels they send the updates more quickly. Some guys were changing something in telethon, but I forgot what exactly. If anybody knows, please share.

[manojsitapara]

I have observed this issue since last 4-5 days, earlier it was working fine

[smaugfm]

@manojsitapara I don't think it's a problem with telegram. Here is why:

• tgcf is using an official tdlib library (python bindings of it) so it should work the same as any other desktop/mobile telegram client. My iOS tg app receives updates from any channel just fine, as they happen.
• I ended up coding a barebones solution myself using tglib-go golang bindings - and it works without issues.

[manojsitapara]

Thank you, I will take a look

[ShamanValius]

Unfortunately, I encountered this issue (( The problem is most likely in this particular service (there's also suspicion of insufficient RAM on the server). The logs are clean.
Here's how it happens:

• Initial settings: no plugins, user mode, live, Show 'Forwarded from', forwarding from multiple channels to one (1-3 messages per hour in each channel), launched using a Docker image from the instruction (Docker Hub).
• From some channels, there's initially silence (no forwarded messages since start), but once a day, 1 forwarded message may appear (so far, only seen one) and then silence again.
• From some channels, messages are forwarded initially, but at some point either complete silence or some messages are lost (example: 3 messages arrived, 1 forwarded, 2 not forwarded, 3 forwarded).
• And the strangest moment, forwarding messages with absolutely random time. For example, a message arrives at one time, forwarding can be instantaneous, or after a minute, two, three, five, ten, twenty, or even several hours. At first, I tried to understand the pattern, but when the time passed 20 minutes, I got lost)))

Now I'll try it on my own computer, where the hardware isn't as restricted as on the server, I'll monitor and report back.

[jimmy0435]

I met the same situation. However, I noticed that when I opened the web browser to that channel. The messages can be forwarded properly. Any ideas?

[Navneet-Choudhary]

@manojsitapara I don't think it's a problem with telegram. Here is why:

• tgcf is using an official tdlib library (python bindings of it) so it should work the same as any other desktop/mobile telegram client. My iOS tg app receives updates from any channel just fine, as they happen.
• I ended up coding a barebones solution myself using tglib-go golang bindings - and it works without issues.

Can you help how to run your script

[ShamanValius]

I'm back, unfortunately so late due to work. Here are my logs for approximately a day's worth of work on my device. Unfortunately, the situation hasn't improved, but critical errors have emerged. I've also noticed a large number of 'Sleeping for 2s' instances, despite very few messages in the target channels (around 5 per hour). Attached is the log file.
tgcf_logs-orig.txt

[ShamanValius]

@jimmy0435, I still couldn't get @smaugfm's solution to run, constantly getting a 503 error.
But I came across a project in Golang, haven't tested it thoroughly yet, but at first glance, it works quite well. Unfortunately, there's no documentation available. This can be resolved with a small script that scans the entire project and creates a single mega txt file with all the project contents. Then you can upload it to any AI for context and ask questions to figure out what you want to do.

P.S. I've found that now this project, along with its creator, is no longer accessible (404 error). I don't understand what's causing this, but GitHub remembers everything))) and you can find traces/forks if someone really needs it. It's a pity it happened this way, the project was really well-written.

P.P.S. It's still available at pkg.go.dev
P.P.P.S. repo gogram

[SaiMun92]

Any solutions?

[mureev]

I only can add that if I switch from "live" to "past" mode forwarding works.

This could cover my needs. I'll continue to look for options if I can implement my own scheduling and run bot in past mode every 30 mins or so.