[Up for Adoption] Allow to scan barcodes offline

[teolemon]

What

• Ensure server support is still working
  • Set up CRON for offline export openfoodfacts-server#1698
  • Offline scan dump is partly broken (Nutri-Score field rename) openfoodfacts-server#4648
  • Make exports more flexible, fixes #4932 openfoodfacts-server#4933
  • Add support for the Eco-Score for Offline-Scan openfoodfacts-server#4484
• Consider adding per-country dumps
• Add offline scanning with a reduced local copy of the database so that users can scan offline.
• This feature has been documented
  • Design document
• Behaviour
  • we have toggle button in left menu "Enable offline mode" which tuns on/off search from local DB or from server
  • We have a screen with available regions to download
  • Download the Offline scan dump
    • the app stores somewhere info about region and DB version
  • Load it in the local database
  • Parse and store the downloaded file in background and show user a progress bar.
  • Implement a refresh mechanism
    • once per N day the app check on server is there a new version of DB for user
  • Implement a double-response mechanism in the scan screen (first offline, then online refresh if network is available)
    • When the user scans a barcode in offline mode the data will be shown in bottom sheet but the user will unable to swipe it up (unless the network response arrived)
  • For this we will need to store the nutriscore, nova group (or CO2) , eco score and ingredient analysis.

Why

• No network or slow in supermarkets

Datasource

I've created a special excerpt of the database, which has the required data to scan products for France. This is 11 Mb zipped, and 37Mb unzipped.

http://fr.openfoodfacts.org/data/offline/fr.openfoodfacts.org.products.small.csv.tgz
https://world.openfoodfacts.org/data/offline/en.openfoodfacts.org.products.small.csv.zip

Can we download this data and store it in the DB ?
I can create it for other countries on demand, but right now it's a test.
And I believe the current export is broken, here's the Python script that converts the daily CSV export: https://github.com/openfoodfacts/openfoodfacts-server/blob/main/scripts/generate_dump_for_offline_apps_off.py

I have fixed the CSV export for NOVA which wasn't exported correctly:
https://world.openfoodfacts.org/data/offline/en.openfoodfacts.org.products.small.csv.zip

Equivalent code on iOS

• Swift code for the iOS implem if you'd like to have a look: openfoodfacts/openfoodfacts-ios@20723da

Data structure

It's basically a very minified version of the database.
code product_name quantity brands nutriscore_grade nova_group ecoscore_grade
8938506580127 Dien Bao sweet potato vermiceilli
8938506854082 Virgin Coconut Oil 1 l Vietcoco
8938506854471 Lait De Coco Sterilisé 4.0
8938506854518 Thanh vinh Organic
8938506854563 coconut cream 13.5 fl.oz vietcoco 4.0

Proposed visual assets

https://storage.googleapis.com/icons.google.com.a.appspot.com/google-assets/system_icons/quantum/file/res-export/ic_cloud_off_24px.svg

Part of

• What can I work on ? #4169

Screenshots from the existing PR and the iOS app and other apps based on OFF

Random

https://world.openfoodfacts.org/data/output-small.csv
Database at 1GO, new export: 87MB
https://github.com/wmentzel/dump-creator

[prashantkh19]

We can download the file on the first run and then store it in the DB. I will try that.

[prashantkh19]

@teolemon please review... as of now, since the data is too big (~700000 products), so it's taking too much time to store in the database.

[teolemon]

@jinqian @huzaifaiftikhar @PrajwalM2212 any ideas to massively increase it client side ?
Can we perhaps generate and store a separate db server side ?
@prashantkh19 Did you manage to load it completely ? if so, how long did it take ?

[prashantkh19]

No, I didn't. I did for a subset of 10,000 products and it took around 5-6 minutes.

[teolemon]

Do you have an idea of the size of the DB for 10k products ?

[PrajwalM2212]

Your code is trying to download it during the first run. That is great but I am more worried about the time that will take.
Even at present without the database the splash screen takes a lot of time to load. In fact we must try to reduce the splash screen time as it is.

1. I think we should allow the user to schedule a time for download if he wants offline capability just like how we can schedule software updates on the phone. This way we don't end up consuming more time or memory.
2. Allow we should allow them to download it in packets if they should feel lot is being downloaded

[prashantkh19]

@teolemon No, I didn't check that.
@PrajwalM2212 Yeah, I was thinking the same, maybe we can divide it into two processes, downloading and saving and make an Intent service for both. I tried this already by the way, but it was taking more time through an intent service. For the second point, the size for approx. 700000 products is only 11 mb (zipped). I think it's fine.

[PrajwalM2212]

What if we don't save the data in a database. When a user scans a product, we will
Scan the csv file if the product exists and then fetch that record and display it to them.
We can improve the organization of the csv file so that we take less time to fetch the product from csv
For example we can try to create some sort of group indexing so that when a barcode starts with abc... we move that group and find the product there. This way we may not have to loop through the entire file.
@teolemon @prashantkh19

[PrajwalM2212]

@prashantkh19 That is why I was saying that we download like 10000 products at every attempt rather than as whole

[prashantkh19]

@PrajwalM2212 yeah, it can be good, we can divide the csv file based on sorted barcodes, each file having let say, 10000 products and a separate file which contains the metadata of all the csv files. That way, we will need to scan just 2 files for each product scan, and it will take hardly 5 seconds to get the results. And maybe we can try hashing each csv also.

[teolemon]

weird: http://greenrobot.org/wordpress/wp-content/uploads/2016-07-08-android-orm-performance-benchmark.png
15k entities per second.

[prashantkh19]

Optimized Version: took around 4 minutes to read csv(~ 2 minutes) and saving to the database(~ remaining 2 minutes) apart from downloading time for ~700000 products (complete csv file data).

[prashantkh19]

It can still be optimized more, as I am saving a list of max 15000 entities each time as of now. If I increase the list size, then time can be reduced but increasing the size of the list may result in crashes due to memory. A better Data Structure (space optimized) than ArrayList may help or if we somehow can store List<String[]> records returned by reader.readAll() directly to greenDao database.

[prashantkh19]

@teolemon @PrajwalM2212 can you review the latest commit?

[serpico]

I've created a special excerpt of the database, which has the required data to scan products for France. This is 11 Mb zipped, and 37Mb unzipped.
http://fr.openfoodfacts.org/data/offline/fr.openfoodfacts.org.products.small.csv.tgz
Can we download this data and store it in the DB ?

@teolemon what about optionally limiting the loading/excerpt to a specific store? Through the preferences maybe. I know I do most of my grocery shopping in one store so...
Also, is there a way to test your file directly from the phone with the current app ?

[teolemon]

@serpico That will be V2 or 3. For now, we can afford to load everything. In the future, we'll try to let people control how much detail they want (like full products) and what countries, what stores they want. Besides, right now, we don't know if a product is sold everywhere.
@prashantkh19 hasn't implemented scanning yet. All it does is load the DB right now :-)

[kartikaysharma01]

@teolemon Can you break this down in small tasks ?

[naivekook]

Hi!
I'm a new here, excuse me if I ask fool questions and tell obvious things and of course for hoping into the train in last moment :)

@teolemon Do you consider an approach with downloading the whole sqlite DB file?
I had a project where we chose this approach and quick googling shows that converting csv to sqlite on server side is quite easy task. The main idea is to download .sqlite database from serer as is, put it on sdcard and just work with it, without parsing csv file and copying all data to local DB.

From top of my head I see offline flow V1 next:

• we have a screen with available regions to download
• the app downloads the database and store somewhere info about region and DB version
• once per N day the app check on server is there a new version of DB for user
• we have toogle button in left menu "Enable offline mode" which tuns on/off search from local DB or from server

[kartikaysharma01]

I'd suggest we keep a Go Offline tab in the sidebar. On the landing page, we can keep a link to the tab , for better user awareness of the feature.
In the tab, we can give user a choice to which country's database he wants to download. And once the download button is clicked, we can run the process in the background with a notification.
I like the idea given by @naivekook , but I don't think parsing and copying data to local DB is a blocker issue as it is a one time process, and we can use server side sqlite database as and when it is available.

[VaiTon]

Also, we keep the size problem. I think the best approach would just to store a country into the local db with just a few attributes: name, ecoscore, nutriscore, brand and quantity maybe

[kartikaysharma01]

@VaiTon @teolemon I would like to implement this.

These are the subtasks :

• The user will have a option in preference to download the data for their country.
• Parse and store the downloaded file in background and show user a progress bar.
• When the user scans a barcode in offline mode the data will be shown in bottom sheet but the user will unable to swipe it up.

For this we will need to store the nutriscore, nova group (or CO2) , eco score and ingredient analysis.
Have I got this right?

[kartikaysharma01]

Also , can you please explain the CSV design. @teolemon
https://world.openfoodfacts.org/data/offline/en.openfoodfacts.org.products.small.csv.zip

[teolemon]

@kartikaysharma01 It's basically a very minified version of the database.
code product_name quantity brands nutriscore_grade nova_group ecoscore_grade
8938506580127 Dien Bao sweet potato vermiceilli
8938506854082 Virgin Coconut Oil 1 l Vietcoco
8938506854471 Lait De Coco Sterilisé 4.0
8938506854518 Thanh vinh Organic
8938506854563 coconut cream 13.5 fl.oz vietcoco 4.0

[teolemon]

And I believe the current export is broken, here's the Python script that converts the daily CSV export: https://github.com/openfoodfacts/openfoodfacts-server/blob/main/scripts/generate_dump_for_offline_apps_off.py