export fetched data into a csv file #777

Vebryn · 2017-08-23T21:43:21Z

Hi,

Here is a clean version of csv export implementation. Old conversation can be found into #692.

Best regards

cpfeiffer

Thanks for the PR, nice!

Can you explain why added two CSV export implementations? One in the SampleProvider and the other in the CSVExport class? I think one should be sufficient :-)

cpfeiffer · 2017-09-03T21:15:48Z

app/src/main/java/nodomain/freeyourgadget/gadgetbridge/devices/AbstractSampleProvider.java

+
+        LOG.info("Exporting samples into csv file: " + outFile.getName());
+        try {
+            BufferedWriter bw = new BufferedWriter(new FileWriter(outFile));


Please use
try (BufferedWriter bw = ....) {
and remove the bw.close()
That way the writer will always be closed (also when an exception occurs).

I didn't know this try block syntax. So, it's a try-with-resources statement. It closes bw at end of execution (cf. https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html). I understand so why you remove bw.close().

cpfeiffer · 2017-09-03T21:17:53Z

app/src/main/java/nodomain/freeyourgadget/gadgetbridge/devices/AbstractSampleProvider.java

+            bw.flush();
+            bw.close();
+            } catch (IOException e) {
+            LOG.error(e.getMessage());


Please use
LOG.error("Explain what failed", e);
otherwise, the stacktrace of the exception will not be logged, making debugging hard.

cpfeiffer · 2017-09-03T21:22:41Z

...n/freeyourgadget/gadgetbridge/service/devices/miband2/operations/FetchActivityOperation.java

+                // export csv on fetching
+                File myPath = null;
+                myPath = FileUtils.getExternalFilesDir();
+                File myFile = new File(myPath, startTimestamp.getTimeInMillis() / 1000 + ".csv");


Maybe add a gadgetbridgedb_ prefix (or something like that, maybe shorter)?

But really, this does not belong here. It should be implemented in one place for all devices, not in the Mi Band 2 FetchActivityOperation. And I guess we should also have an option to turn this on and off.

Right, does data from other devices differs ? Where would you place on/off feature ? I would probably add into Mi Band menu since it's the only device that's supporting it.

Data from other devices may differ, yes. So either we use normalized values (= the same data ranges for all devices) or device specific raw values, or even both.

But it would be good to have this for all devices, not just one.

Since some devices provide the data in a different way, where we have no access to all samples at once, we might call a method like samplesAdded(device, start, end) which would then check if CSV export is enabled, then fetch the samples from the given date range from the db (probably cached in the session already) and perform the export.

Alternatively we could integrate with the new DB export functionality and perform the CSV export repeatedly. We would just need to remember the start and end dates so that we only export the incremental changes.

cpfeiffer · 2017-09-03T21:23:38Z

app/src/main/java/nodomain/freeyourgadget/gadgetbridge/util/CSVExport.java

+
+        LOG.info("Exporting samples into csv file: " + outFile.getName());
+        try {
+            bw = new BufferedWriter(new FileWriter(outFile));


See above with try (BufferedWriter ...) and the bw.close()

cpfeiffer · 2017-09-03T21:26:51Z

app/src/main/java/nodomain/freeyourgadget/gadgetbridge/util/CSVExport.java

+        LOG.info("Exporting samples into csv file: " + outFile.getName());
+        try {
+            bw = new BufferedWriter(new FileWriter(outFile));
+            bw.write("TIMESTAMP" + separator + "DEVICE_ID" + separator + "USER_ID" + separator + "RAW_INTENSITY" + separator + "STEPS" + separator + "RAW_KIND" + separator + "HEART_RATE");


Shouldn't we also export the normalized values?

Normalized values are deduced from a simple matching rule. I think correlation have to be done after exportation.

cpfeiffer · 2017-09-03T21:27:32Z

app/src/main/java/nodomain/freeyourgadget/gadgetbridge/util/CSVExport.java

+public class CSVExport {
+    private static final Logger LOG = LoggerFactory.getLogger(AbstractSampleProvider.class);
+
+    public static void exportToCSV(AbstractActivitySample[] activitySamples, File outFile) {


I think it's sufficient to use ActivitySample here instead of AbstractActivitySample.

use try-with-resources statement

cpfeiffer

Sorry for being so late, I totally missed your updated PR. I've added two more comments.

cpfeiffer · 2018-01-12T20:32:47Z

...n/freeyourgadget/gadgetbridge/service/devices/miband2/operations/FetchActivityOperation.java

+                // export csv on fetching
+                File myPath = null;
+                myPath = FileUtils.getExternalFilesDir();
+                File myFile = new File(myPath, startTimestamp.getTimeInMillis() / 1000 + ".csv");


Data from other devices may differ, yes. So either we use normalized values (= the same data ranges for all devices) or device specific raw values, or even both.

But it would be good to have this for all devices, not just one.

Since some devices provide the data in a different way, where we have no access to all samples at once, we might call a method like samplesAdded(device, start, end) which would then check if CSV export is enabled, then fetch the samples from the given date range from the db (probably cached in the session already) and perform the export.

Alternatively we could integrate with the new DB export functionality and perform the CSV export repeatedly. We would just need to remember the start and end dates so that we only export the incremental changes.

cpfeiffer · 2018-01-12T20:49:30Z

app/src/main/java/nodomain/freeyourgadget/gadgetbridge/util/CSVExport.java

+        String separator = ",";
+
+        LOG.info("Exporting samples into csv file: " + outFile.getName());
+        try (BufferedWriter bw = new BufferedWriter(new FileWriter(outFile))) {


FileWriter always uses the local platform encoding, which, atm, is UTF-8 on Android. Relying on the local platform encoding will in many cases lead to a bug at some point. One has to use the extra verbose
new OutputStreamWriter(new FileOutputStream(outFile), StandardCharsets.UTF_8))

I've spent too much of my lifetime on encoding bugs already (I actually have a T-Shirt with the text "Schei? Encoding".

For the record, Excel' CSV import does CP-1252 (similar to ISO-8859-1), LibreOffice can handle all kinds of encodings including UTF-8.

Rahulaswani · 2018-01-12T22:40:33Z

hI @cpfeiffer @Vebryn was this solution to easily export the activity/sleep data into csv finally implimented, if yes where can i find it ?

I went through the wikis and tried to find a way to export data from the mobile app and found the 'Database Management' option in the side menu which lets you export data but i am not sure of the format, thought it might be sql db but it isn't as DB browser for SQLite cannot recognise and open the file.

Can you help me here or point me in the right direction to be able to easily export my data in a csv file

Thanks

Vebryn · 2018-01-15T19:27:11Z

Hi @Rahulaswani,

Export option is available on my branch csv-export. Feel free to contribute.

Rahulaswani · 2018-01-23T10:50:39Z

hi @Vebryn i and @izBasit couldn't find the CSV export option in the build from you code. The export options given are exactly the same as the original project. Check screenshot from your repo below

Can you please help us understand from where can we trigger a csv export or if there is something else that needs to be done ?

@cpfeiffer can you help here ?

elagin · 2018-08-27T15:07:27Z

I want to try to bring the matter to an end, considering all of the above.

export fetched data into a csv file

5525938

cpfeiffer suggested changes Sep 3, 2017

View reviewed changes

clean old exportToCSV method

a867c06

use try-with-resources statement

cpfeiffer reviewed Jan 12, 2018

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

export fetched data into a csv file #777

export fetched data into a csv file #777

Vebryn commented Aug 23, 2017

cpfeiffer left a comment

cpfeiffer Sep 3, 2017

Vebryn Sep 17, 2017 •

edited

cpfeiffer Sep 3, 2017

cpfeiffer Sep 3, 2017

Vebryn Sep 17, 2017 •

edited

cpfeiffer Jan 12, 2018

cpfeiffer Sep 3, 2017

cpfeiffer Sep 3, 2017

Vebryn Sep 17, 2017

cpfeiffer Sep 3, 2017

cpfeiffer left a comment

cpfeiffer Jan 12, 2018

cpfeiffer Jan 12, 2018

Rahulaswani commented Jan 12, 2018

Vebryn commented Jan 15, 2018

Rahulaswani commented Jan 23, 2018

elagin commented Aug 27, 2018

export fetched data into a csv file #777

Are you sure you want to change the base?

export fetched data into a csv file #777

Conversation

Vebryn commented Aug 23, 2017

cpfeiffer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vebryn Sep 17, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vebryn Sep 17, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpfeiffer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rahulaswani commented Jan 12, 2018

Vebryn commented Jan 15, 2018

Rahulaswani commented Jan 23, 2018

elagin commented Aug 27, 2018

Vebryn Sep 17, 2017 •

edited

Vebryn Sep 17, 2017 •

edited