Bump

[weakit]

I can (actually would want to cause I'm bored) work on this if it's still of any use.

I tried to reach the IFF a few months ago about this and then forgot I even did that since I didn't get a response. Maybe an issue here will get me one?

I actually have an idea on how to get things to work. Will need a bit of help.

[jace]

This repo isn't actively maintained. Try the IFF forum for a more active community. https://forum.internetfreedom.in/

[anoop-b]

Hey @weakit , this project is still doable. You can work on your own fork, happy to help/collaborate.

[PickleRickkkkk]

Hi @weakit are you or anyone else working on this? I can also contribute on this or its fork. Otherwise we can discuss the tech stack that we can use to build this and I can start working on this accordingly. I am a backend developer mainly expertising in Python(and it's web frameworks like django, flask and fastAPI)

[weakit]

Hello @PickleRickkkkk👋

I had come up with a rough idea on how I'd go about doing this, and then saved it for another day since I found something else to do and then later got a bit busy.

I'd love to discuss and help out!

What do you have in mind for the project?

[PickleRickkkkk]

So I was looking up any projects with similar implementation so that we have a little clearer picture of what we are trying to build. I came across this project- https://www.blocked.org.uk/

This is an opensource project. Here are the repo links-

• Frontend Repo
• Backend Repo
• Cli Tool

I have been going through the backend repo and I'll also mention my findings here. The code is majorly PHP and some command line tools written in python. I have only gone through python code(Don't really know PHP).

I still couldn't figure how they are verifying what ISPs are blocking certain sites. How I think this will be implemented is having multiple proxies using different ISPs and sending requests through them to know the status of the site. This doesn't seems to be viable as it will be very difficult finding proxies from every ISP. Do you have a better approach from this?

Here's some findings from the project-

• blocked API documentation
• blocked DB schema
• API which categorises the website(Being used in this project internally)

I haven't yet thought of the specific features that we'll implement initially. I am just kind of overwhelmed with the info I gathered from this project. Let me know your thoughts on this and how you are planning to implement this.

[weakit]

I had something similar in mind.

A backend server that delegates websites to test and probes running on devices preferably spread out pretty evenly.

This paper talks about a bunch of common techniques used by ISPs here, and more importantly how to test for those. I think the probes should implement that along with anything else that's required to accurately report blocked websites.

The backend needs to be able to assign and maintain a database of blocked sites along with additional data like region and ISP. Maybe a nice website where people can report blocked websites too.

[anoop-b]

@weakit @PickleRickkkkk, perhaps this might help.

[weakit]

OONI only tests websites on the list from citizen labs, which is not a lot IIRC

Good testing methodology tho will come in handy.

I think we should aim on trying to compile a big exhaustive list, no?

[PickleRickkkkk]

I have gone through the paper. I also found this repo(Based on python) for SNI detection while I was checking out Kushagra Singh's github(one of the author of this paper). Also ooni's backend is also in python, we can refer that as well.
How do you say we start working on this project? We can figure out the stack we are going to use and I can atleast being with the boilerplate code and then we can start working on various implementations

[PickleRickkkkk]

Hi @weakit @anoop-b, any update?

[weakit]

Sorry I wasn't able to get back to you until now. Kinda swamped with other stuff at the moment.

How do you say we start working on this project?

That's great. I think working on the probe first would be a better idea (or one implementation). I feel like golang would be a good choice since it's probably easier to port to mobile OSes.

I intended to have this project track exactly what content is blocked, with data on when and where, instead of a general censorship tracker. I don't think we need to be as comprehensive as OONI for now; IMO we should focus on just webpage blocks, however they're carried out.

For the backend, we'll have to figure out how to maintain a database of websites along with time and region-specific accessibility reports. and decide on how to delegate webpage checks to probes once one is available to use.

I'll help as much as I can, but I don't think I'll have enough time to focus on this for a little while.

Maybe we should start implementing the tests in go and see if they still work properly today, to begin with.

[PickleRickkkkk]

I feel like golang would be a good choice since it's probably easier to port to mobile OSes.

Can we keep the development in python, since that's the only language I've worked on. Also why do we need it to run on mobile? Wouldn't we be doing all the testing on a server?
Also to test on different networks, we would require proxies for all the networks. How are we planning to go about that?

[weakit]

Also why do we need it to run on mobile?

Having probes to run on mobile phones would be better for data collection. It would help with covering different networks in different regions, we wouldn't really have a need for proxies for mobile networks at least.

And to test broadband connections, a simple RPi probe at home should suffice. They're not really expensive—a Pi Zero W is ~900 INR, just kinda hard to find at the moment. Having them set up all over the place would be a hassle, but definitely cheaper than renting out a bunch of proxy servers or probes everywhere.

Python would be fine for the RPi but will be a problem if you want it to run on a mobile phone. Go is pretty snappy and not that hard to pick up. We can have a python probe implementation for now, but I feel go would be a better choice for cross-platform compatibility. And as far as the backend is concerned, it can be anything as long as it gets the job done efficiently.

Again sorry for the late reply, I should be a bit freer from now so we can get started on the project.

[PickleRickkkkk]

Let's just start with python for now. We can also use kivy for android and iso development as long as we are targeting a bare minimum(from UI perspective) application.

[PickleRickkkkk]

Fun fact- ooni had their codebase in python and then they moved to golang, don't know if we should be repeating the same. It would actually be easier to just write it in golong since we can also refer ooni's repo. What are you guys @anoop-b @weakit comfortable writing in?

[weakit]

I'm fine with either. Go isn't that hard to pick up really, you should give it a shot maybe?

[weakit]

I think we should figure out what we want the probe to check and report about a webpage first. Have anything in mind @PickleRickkkkk?