Extensions of dump_clean.sql for selective and daily dumps #699

sasharevzin · 2020-06-15T13:59:31Z

@matschaffer two new crons to setup on a server:

# https://api.safecast.org/system/dump_clean_daily.tar.gz
cron/dump_clean_daily

matschaffer

Per #427 (comment) we should probably tweak a few things:

Remove the blacklisting (this is for historical stuff and shouldn't be required for a new daily export)
rename it from "mclean" to avoid confusion

@Frangible should be able to advise here if there are any other questions beyond what was on the original PR

sasharevzin · 2020-06-16T13:26:18Z

Remove the blacklisting (this is for historical stuff and shouldn't be required for a new daily export)

@matschaffer did you mean removing these filters: https://github.com/Safecast/safecastapi/pull/699/files#diff-2a92cbfc2514ad867e2c7fa579a3d0c9R21-R38

matschaffer · 2020-06-18T01:16:21Z

I'm pretty sure that's what @Frangible meant. Would be nice if he could confirm.

matschaffer · 2020-06-18T01:19:51Z

Also wondering if we should leave cron/dump_clean_selective.sql out for now. The original ask was for the daily exports (#419)

Not sure how a second bulk export got into the request. @absalomshu can you provide more info on why you need another selective bulk export (instead of just the daily?)

Those full-data-set exports are really expensive in terms of DB resources, so I'd rather not have two.

I'm okay with the one we have today plus a daily incremental export.

Frangible · 2020-06-18T01:21:21Z

Yes, if you’re not querying historical data you shouldn’t need any of the exclusions/blacklisting.

…

On Jun 17, 2020, at 7:16 PM, Mat Schaffer ***@***.***> wrote: I'm pretty sure that's what @Frangible meant. Would be nice if he could confirm. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

sasharevzin · 2020-06-18T15:14:31Z

cron/dump_clean_daily.sql

+     AND captured_at IS NOT NULL
+     AND value IS NOT NULL
+     AND LOCATION IS NOT NULL
+     AND ((unit = 'cpm'
+           AND ((device_id IS NULL
+                 AND value BETWEEN 10.00 AND 350000.0)
+                OR ((device_id <= 24
+                     OR device_id IN (69,
+                                      89))
+                    AND value BETWEEN 10.00 AND 30000.0)))
+          OR (unit = 'celsius'
+              AND value BETWEEN -80 AND 80))
+     AND (ST_X(LOCATION::geometry) != 0.0
+          OR ST_Y(LOCATION::geometry) != 0.0)
+     AND ST_Y(LOCATION::geometry) BETWEEN -85.05 AND 85.05
+     AND ST_X(LOCATION::geometry) BETWEEN -180.00 AND 180.00
+     AND (user_id NOT IN (9,
+                          442)
+          OR value < 35.0
+          OR ST_Y(LOCATION::geometry) NOT BETWEEN 35.4489 AND 35.7278
+          OR ST_X(LOCATION::geometry) NOT BETWEEN 139.5706 AND 139.9186)
+     AND (user_id != 366
+          OR value < 35.0
+          OR ST_Y(LOCATION::geometry) NOT BETWEEN -45.5201 AND -7.6228
+          OR ST_X(LOCATION::geometry) NOT BETWEEN 111.3241 AND 153.8620
+          OR (value < 105.0
+              AND ST_X(LOCATION::geometry) < 111.3241)) ) ;


@Frangible @matschaffer Let me know if we need these filters

julovi · 2020-06-19T00:51:55Z

Hi, I don’t think they’re needed, other filtering can be done locally. By the way, where the daily exports would be located and how many will be stored? Cheers, Julian Villegas, Ph.D.

…

On Jun 19, 2020, at 0:14 AM, Sasha ***@***.***> wrote: @sasharevzin commented on this pull request. In cron/dump_clean_daily.sql: > + AND captured_at IS NOT NULL + AND value IS NOT NULL + AND LOCATION IS NOT NULL + AND ((unit = 'cpm' + AND ((device_id IS NULL + AND value BETWEEN 10.00 AND 350000.0) + OR ((device_id <= 24 + OR device_id IN (69, + 89)) + AND value BETWEEN 10.00 AND 30000.0))) + OR (unit = 'celsius' + AND value BETWEEN -80 AND 80)) + AND (ST_X(LOCATION::geometry) != 0.0 + OR ST_Y(LOCATION::geometry) != 0.0) + AND ST_Y(LOCATION::geometry) BETWEEN -85.05 AND 85.05 + AND ST_X(LOCATION::geometry) BETWEEN -180.00 AND 180.00 + AND (user_id NOT IN (9, + 442) + OR value < 35.0 + OR ST_Y(LOCATION::geometry) NOT BETWEEN 35.4489 AND 35.7278 + OR ST_X(LOCATION::geometry) NOT BETWEEN 139.5706 AND 139.9186) + AND (user_id != 366 + OR value < 35.0 + OR ST_Y(LOCATION::geometry) NOT BETWEEN -45.5201 AND -7.6228 + OR ST_X(LOCATION::geometry) NOT BETWEEN 111.3241 AND 153.8620 + OR (value < 105.0 + AND ST_X(LOCATION::geometry) < 111.3241)) ) ; @Frangible @matschaffer Let me know if we need these filters — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

cron/dump_clean_daily

julovi · 2020-06-19T06:18:07Z

Hi, I think keeping, say, the last week of daily backups could be useful. I imagine someone feeding a third-party analysis missing a couple of days after a system migration or failure, so having those extra days could make the recovery faster… Anyway, I’m really grateful that you guys are doing this. Have a nice weekend, Julian Villegas, Ph.D.

…

On Jun 19, 2020, at 14:52 PM, Andrew Todd ***@***.***> wrote: @auspicacious commented on this pull request. In cron/dump_clean_daily: > +# This differs from dump_measurements, which does not filter data at all. +# The schema returned is also different; redundant/unused columns have been eliminated, and id/user_id finally included. +# The ORDER BY clause was removed for performance. + +set -euo pipefail + +source cron_env.sh + +cd ../public/system + +psql -f "${CRON_DIR}/dump_clean_daily.sql" + +tar -czf dump_clean_daily.tar.gz dump_clean_daily-out.csv +mv dump_clean_daily-out.csv dump_clean_daily.csv + +# Now this is available as https://api.safecast.org/system/dump_clean_daily.tar.gz Is this sufficient for our users' needs? Or should we name these files differently for each day, and retain a few days of dumps? For instance dump_clean_daily_2020-07-01.tar.gz — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

sasharevzin · 2020-06-19T13:34:05Z

Hi, I don’t think they’re needed, other filtering can be done locally. By the way, where the daily exports would be located and how many will be stored? Cheers, Julian Villegas, Ph.D.

@julovi On a server itself and can be found here: https://api.safecast.org/system/dump_clean_daily.tar.gz

matschaffer · 2020-06-20T01:29:39Z

I’d go with dated exports to s3, and just keep them indefinitely. I suspect even a few years of daily data will be well within budget.

On Sat, Jun 20, 2020 at 10:18 Sasha ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In cron/dump_clean_daily <#699 (comment)>: > +# This differs from dump_measurements, which does not filter data at all. +# The schema returned is also different; redundant/unused columns have been eliminated, and id/user_id finally included. +# The ORDER BY clause was removed for performance. + +set -euo pipefail + +source cron_env.sh + +cd ../public/system + +psql -f "${CRON_DIR}/dump_clean_daily.sql" + +tar -czf dump_clean_daily.tar.gz dump_clean_daily-out.csv +mv dump_clean_daily-out.csv dump_clean_daily.csv + +# Now this is available as https://api.safecast.org/system/dump_clean_daily.tar.gz Thank you very much but it might take time to me to understand what I’m looking at, haha. I’m trying to understand what is the task. Do we want to have a list of dumps back to three months? What is our goal? :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#699 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAFMQC3T2PARXW2UAR6ILRXQE7FANCNFSM4N6GSJCQ> .

--

…

-Mat matschaffer.com

matschaffer · 2020-06-20T04:01:31Z

Certainly less than the $75/mo we’d have to pay to run kubernetes ;)

matschaffer · 2020-06-20T04:02:22Z

Btw, we might want to get these moved to s3 first. Then we can implement retention via s3 lifetime rules

auspicacious · 2020-06-20T05:15:13Z

@sasharevzin and @matschaffer I'm thinking that we:

Fix the shell script in this PR for the simplest possible case (every day, we create https://api.safecast.org/system/dump_clean_daily.tar.gz) and accept this PR, so we can see it working
Create a separate task for moving all scheduled dumps to S3 with retention rules

What do you think?

matschaffer · 2020-06-20T06:14:42Z

Yep. I'd say #571 is basically that "move it to s3 issue", just need to make a note about the retention rules.

sasharevzin · 2020-07-08T13:33:08Z

@sasharevzin and @matschaffer I'm thinking that we:

Fix the shell script in this PR for the simplest possible case (every day, we create https://api.safecast.org/system/dump_clean_daily.tar.gz) and accept this PR, so we can see it working

Create a separate task for moving all scheduled dumps to S3 with retention rules

What do you think?

@matschaffer @auspicacious I think we are good to go with the first item, for a second I created a ticket: #710
Let me know if something is still missing in this PR.

Thanks

matschaffer · 2020-07-10T05:18:26Z

Sorry for the back and forth here, but two things:

I think we should probably remove this bit

(measurement_import_id NOT IN
            (SELECT id
             FROM measurement_imports
             WHERE subtype NOT IN ('None',
                                   'Drive'))
          OR measurement_import_id IS NULL)

Just make it a straight measurements export.

Similarly let's take clean out of the name and just call it dump_measurements_daily since we're really not doing any cleaning just dropping whatever came into the table to disk.

matschaffer · 2020-07-10T05:19:59Z

Then yeah, let's do #710 after we set up the s3 storage.

Then we don't have to worry about local clean up we can just s3 cp right after we export, then rm the file.

Extensions of dump_clean.sql for selective and daily dumps

e0f1e94

sasharevzin linked an issue Jun 15, 2020 that may be closed by this pull request

Extensions of dump_clean.sql for selective and daily dumps #698

Open

Update crontab

b3fc523

sasharevzin added this to In progress in API Team board via automation Jun 15, 2020

sasharevzin moved this from In progress to Needs Review in API Team board Jun 15, 2020

matschaffer mentioned this pull request Jun 16, 2020

Extensions of dump_clean.sql for selective and daily dumps. #427

Closed

1 task

matschaffer requested changes Jun 16, 2020

View reviewed changes

Code review

dd0262b

sasharevzin commented Jun 18, 2020

View reviewed changes

auspicacious reviewed Jun 19, 2020

View reviewed changes

cron/dump_clean_daily Show resolved Hide resolved

auspicacious reviewed Jun 19, 2020

View reviewed changes

cron/dump_clean_daily Show resolved Hide resolved

auspicacious reviewed Jun 19, 2020

View reviewed changes

cron/dump_clean_daily Show resolved Hide resolved

Update sql

f2fb652

sasharevzin requested review from matschaffer and auspicacious June 19, 2020 13:39

matschaffer moved this from Needs Review to Soon in API Team board Aug 28, 2020

matschaffer moved this from Soon to Backburner in API Team board Dec 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extensions of dump_clean.sql for selective and daily dumps #699

Extensions of dump_clean.sql for selective and daily dumps #699

sasharevzin commented Jun 15, 2020 •

edited

matschaffer left a comment

sasharevzin commented Jun 16, 2020

matschaffer commented Jun 18, 2020

matschaffer commented Jun 18, 2020

Frangible commented Jun 18, 2020 via email

sasharevzin Jun 18, 2020

julovi commented Jun 19, 2020 via email

julovi commented Jun 19, 2020 via email

sasharevzin commented Jun 19, 2020

matschaffer commented Jun 20, 2020 via email

matschaffer commented Jun 20, 2020

matschaffer commented Jun 20, 2020

auspicacious commented Jun 20, 2020

matschaffer commented Jun 20, 2020

sasharevzin commented Jul 8, 2020

matschaffer commented Jul 10, 2020 •

edited

matschaffer commented Jul 10, 2020

Extensions of dump_clean.sql for selective and daily dumps #699

Are you sure you want to change the base?

Extensions of dump_clean.sql for selective and daily dumps #699

Conversation

sasharevzin commented Jun 15, 2020 • edited

matschaffer left a comment

Choose a reason for hiding this comment

sasharevzin commented Jun 16, 2020

matschaffer commented Jun 18, 2020

matschaffer commented Jun 18, 2020

Frangible commented Jun 18, 2020 via email

sasharevzin Jun 18, 2020

Choose a reason for hiding this comment

julovi commented Jun 19, 2020 via email

julovi commented Jun 19, 2020 via email

sasharevzin commented Jun 19, 2020

matschaffer commented Jun 20, 2020 via email

matschaffer commented Jun 20, 2020

matschaffer commented Jun 20, 2020

auspicacious commented Jun 20, 2020

matschaffer commented Jun 20, 2020

sasharevzin commented Jul 8, 2020

matschaffer commented Jul 10, 2020 • edited

matschaffer commented Jul 10, 2020

sasharevzin commented Jun 15, 2020 •

edited

matschaffer commented Jul 10, 2020 •

edited