draft nuv dark monitor. help. #156

cmagness · 2020-02-06T21:29:40Z

it doesn't break!!! but i think it's clunky and not sure what is happening at the plotting stage. everything seems to be fine up to there.

eventually will close #79
Resolves #157

…anything else useful

codecov · 2020-02-06T21:35:01Z

Codecov Report

Merging #156 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #156   +/-   ##
=======================================
  Coverage   92.68%   92.68%           
=======================================
  Files          18       18           
  Lines        1476     1476           
=======================================
  Hits         1368     1368           
  Misses        108      108

Impacted Files	Coverage Δ
cosmo/filesystem.py	`88.38% <ø> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4bf5484...57ef9d2. Read the comment docs.

jwhite3 · 2020-02-11T19:53:52Z

cosmo/monitors/nuv_dark_monitor.py

+    run = 'monthly'
+
+    def get_data(self):
+        # access data, perform any filtering required for analysis


Suggested change

# access data, perform any filtering required for analysis

"""access data, perform any filtering required for analysis"""

jwhite3 · 2020-02-11T19:54:16Z

cosmo/monitors/nuv_dark_monitor.py

+        return data
+
+    def calculate_dark_rate(self, dataframe, xlim, ylim):
+        # calculate dark rate for one exposure, with a dataframe with TIME,


Suggested change

# calculate dark rate for one exposure, with a dataframe with TIME,

"""calculate dark rate for one exposure, with a dataframe with TIME, XCORR, and YCORR values"""

jwhite3 · 2020-02-11T20:08:14Z

cosmo/monitors/nuv_dark_monitor.py

+    # TODO: update docs
+    # docs = "https://spacetelescope.github.io/cosmo/monitors.html#<darks>"
+    output = COS_MONITORING
+


Suggested change

# Plot stuff

x = 'TIME'

y= 'DARK_RATE'

plot_type = 'scatter'

^^ If you want to use the built-in basic plotting

https://spacetelescope.github.io/monitor-framework/creating_monitors.html#defining-a-new-monitor

cosmo/monitors/data_models.py

cosmo/monitors/nuv_dark_monitor.py

…ing calculated wrong

jwhite3 · 2020-03-16T18:39:10Z

.gitignore

@@ -71,3 +71,7 @@ target/

 # Finder
 .DS_Store
+
+# Misc
+testoutputs/


Was this something for you in particular? If so, I'd be careful about adding it here since .gitignore could grow quite a bit depending on what folks are doing.

Instead, try using the the unit tests! You could even update some of the "cleanup" fixtures to remove "output" on the start of tests rather than the end so that you can inspect the test artifacts. If you'd like some suggestions or help with this, let me know!

yes, actually that would be helpful! but i was planning to remove that stuff before merging it in, it's just for ease of directory access right now

jwhite3 · 2020-03-16T18:45:04Z

cosmo/monitors/data_models.py

+class NUVDarkDataModel(BaseDataModel):
+    """Datamodel for NUV Dark files."""
+    files_source = FILES_SOURCE
+    subdir_pattern = '?????'


Doesn't look like you're using subdir_pattern elsewhere, so I think there are a couple of options:

Get rid of this attribute

Use find_files (which uses subdir_pattern)

I'd recommend the 2nd option just because it might make "forward compatibility" easier (i.e. if dark datasets aren't manually identified in this way in the future). I think @dzhuliya has an example of using find_files on the fuv dark branch

yes i want to use find_files but for some reason it wasn't working for me so i was planning to update that. i'll look at dzhuliya's branch

jwhite3 · 2020-03-16T18:53:30Z

cosmo/monitors/nuv_dark_monitor.py

+
+        return dark_rate_array, dec_year_array
+
+    def track(self):


The track method should return some sort of quantity (scalar, dataframe, etc), not execute plotting

yep that is the end goal, don't worry. once the histogram is working correctly it will get the rates from the histogram and return those

cosmo/monitors/nuv_dark_monitor.py

jwhite3 · 2020-03-16T18:59:02Z

cosmo/monitors/nuv_dark_monitor.py

+        fig = plt.figure(figsize=(12, 9))
+        bin_size = 1e-8
+        n_bins = int((dark_counts.max() - dark_counts.min()) / bin_size)
+        ax = fig.add_subplot(2, 1, 1)
+        ax.hist(dark_counts, bins=n_bins, align='mid', histtype='stepfilled')
+        counts, bins = np.histogram(dark_counts, bins=100)
+        cuml_dist = np.cumsum(counts)
+        count_99 = abs(cuml_dist / float(cuml_dist.max()) - .99).argmin()
+        count_95 = abs(cuml_dist / float(cuml_dist.max()) - .95).argmin()
+
+        mean = dark_counts.mean()
+        med = np.median(dark_counts)
+        std = dark_counts.std()
+        mean_obj = ax.axvline(x=mean, lw=2, ls='--', color='r', label='Mean ')
+        med_obj = ax.axvline(x=med, lw=2, ls='-', color='r', label='Median')
+        two_sig = ax.axvline(x=med + (2 * std), lw=2, ls='-', color='gold')
+        three_sig = ax.axvline(x=med + (3 * std), lw=2, ls='-',
+                               color='DarkOrange')
+        dist_95 = ax.axvline(x=bins[count_95], lw=2, ls='-',
+                             color='LightGreen')
+        dist_99 = ax.axvline(x=bins[count_99], lw=2, ls='-', color='DarkGreen')
+
+        ax.grid(True, which='both')
+        ax.set_title('Histogram of Dark Rates', fontsize=15, fontweight='bold')
+        ax.set_ylabel('Frequency', fontsize=15, fontweight='bold')
+        ax.set_xlabel('Counts/pix/sec', fontsize=15, fontweight='bold')
+        ax.set_xlim(dark_counts.min(), dark_counts.max())
+        ax.xaxis.set_major_formatter(FormatStrFormatter('%3.2e'))
+
+        ax = fig.add_subplot(2, 1, 2)
+        # log_bins = np.logspace(np.log10(dark.min()), np.log10(dark.max()),
+        # 100)
+        ax.hist(dark_counts, bins=n_bins, align='mid', log=True,
+                histtype='stepfilled')
+
+        ax.axvline(x=mean, lw=2, ls='--', color='r', label='Mean')
+        ax.axvline(x=med, lw=2, ls='-', color='r', label='Median')
+        ax.axvline(x=med + (2 * std), lw=2, ls='-', color='gold')
+        ax.axvline(x=med + (3 * std), lw=2, ls='-', color='DarkOrange')
+        ax.axvline(x=bins[count_95], lw=2, ls='-', color='LightGreen')
+        ax.axvline(x=bins[count_99], lw=2, ls='-', color='DarkGreen')
+
+        # ax.set_xscale('log')
+        ax.grid(True, which='both')
+        ax.set_ylabel('Log Frequency', fontsize=15, fontweight='bold')
+        ax.set_xlabel('Counts/pix/sec', fontsize=15, fontweight='bold')
+        ax.set_xlim(dark_counts.min(), dark_counts.max())
+        ax.xaxis.set_major_formatter(FormatStrFormatter('%3.2e'))
+
+        fig.legend([med_obj, mean_obj, two_sig, three_sig, dist_95, dist_99],
+                   ['Median: {0:.2e}'.format(med),
+                    'Mean: {0:.2e}'.format(mean),
+                    r'2$\sigma$: {0:.2e}'.format(med + (2 * std)),
+                    r'3$\sigma$: {0:.2e}'.format(med + (3 * std)),
+                    r'95$\%$: {0:.2e}'.format(bins[count_95]),
+                    r'99$\%$: {0:.2e}'.format(bins[count_99])], shadow=True,
+                   numpoints=1, bbox_to_anchor=[0.8, 0.8])


Just to reiterate, plotting should not be in track, but also, plotly should be used for plotting if possible by creating traces and whatnot and then adding them to the figure attribute:

def plot(self): ... # Make plotly traces here; let's say you have a list called "traces" self.figure.add_traces(traces)

yeah that is the end goal as well, but marc wanted the plots replicated for now

cosmo/monitors/nuv_dark_monitor.py

jwhite3 · 2020-03-16T19:01:53Z

cosmo/monitors/nuv_dark_monitor.py

+        for index, row in plotdf.iterrows():
+            all_dec_year = list(
+                itertools.chain(all_dec_year, row["DECIMAL_YEAR"]))
+            all_dark_rates = list(
+                itertools.chain(all_dark_rates, row["DARK_RATE"]))


This bit might be what's causing the funkiness in your plots perhaps?

cosmo/monitors/nuv_dark_monitor.py

jwhite3 · 2020-03-16T19:42:19Z

cosmo/monitors/nuv_dark_monitor.py

+        if self.data is None:
+            self.data = self.get_data()


I see where you're going with this, and it's not a bad idea, but the precedent with the other monitors has been to not enable "asynchronous" steps in the sense that if you want to call individual steps, you still need to call the other steps that it depends on.

If you enabled this here, it should be enabled for all of the other monitors as well so there's no confusion on expected behavior between monitors.

draft of the nuv dark monitor. it doesn't break. not sure if it does …

67f5cc8

…anything else useful

cmagness added the New Feature label Feb 6, 2020

cmagness assigned jwhite3 Feb 6, 2020

cmagness added this to In progress in COSMO Overhaul via automation Feb 6, 2020

jwhite3 reviewed Feb 11, 2020

View reviewed changes

jwhite3 added this to the Add Dark Monitors milestone Feb 14, 2020

this is NOT FUNCTIONAL but i am pushing it so @jwhite can review. thx.

8b215b0

jwhite3 reviewed Mar 10, 2020

View reviewed changes

cosmo/monitors/data_models.py Show resolved Hide resolved

jwhite3 reviewed Mar 10, 2020

View reviewed changes

cosmo/monitors/nuv_dark_monitor.py Show resolved Hide resolved

cosmo/monitors/nuv_dark_monitor.py Outdated Show resolved Hide resolved

cosmo/monitors/nuv_dark_monitor.py Show resolved Hide resolved

cosmo/monitors/nuv_dark_monitor.py Outdated Show resolved Hide resolved

this is a disaster and i need help to figure out why the rates are be…

57ef9d2

…ing calculated wrong

jwhite3 reviewed Mar 16, 2020

View reviewed changes

cosmo/monitors/nuv_dark_monitor.py Show resolved Hide resolved

jwhite3 reviewed Mar 16, 2020

View reviewed changes

cosmo/monitors/nuv_dark_monitor.py Show resolved Hide resolved

jwhite3 reviewed Mar 16, 2020

View reviewed changes

cosmo/monitors/nuv_dark_monitor.py Show resolved Hide resolved

jwhite3 reviewed Mar 16, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

draft nuv dark monitor. help. #156

draft nuv dark monitor. help. #156

cmagness commented Feb 6, 2020 •

edited by jwhite3

codecov bot commented Feb 6, 2020 •

edited

jwhite3 Feb 11, 2020

jwhite3 Feb 11, 2020 •

edited

jwhite3 Feb 11, 2020 •

edited

jwhite3 Feb 11, 2020

jwhite3 Feb 18, 2020

jwhite3 Mar 16, 2020

cmagness Mar 20, 2020

jwhite3 Mar 16, 2020

cmagness Mar 20, 2020

jwhite3 Mar 16, 2020

cmagness Mar 20, 2020

jwhite3 Mar 16, 2020

cmagness Mar 20, 2020

jwhite3 Mar 16, 2020

jwhite3 Mar 16, 2020

	# access data, perform any filtering required for analysis
	"""access data, perform any filtering required for analysis"""

	# calculate dark rate for one exposure, with a dataframe with TIME,
	"""calculate dark rate for one exposure, with a dataframe with TIME, XCORR, and YCORR values"""

+    # Plot stuff
+    x = 'TIME'
+    y= 'DARK_RATE'
+    plot_type = 'scatter'

draft nuv dark monitor. help. #156

Are you sure you want to change the base?

draft nuv dark monitor. help. #156

Conversation

cmagness commented Feb 6, 2020 • edited by jwhite3

codecov bot commented Feb 6, 2020 • edited

Codecov Report

Choose a reason for hiding this comment

jwhite3 Feb 11, 2020 • edited

Choose a reason for hiding this comment

jwhite3 Feb 11, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmagness commented Feb 6, 2020 •

edited by jwhite3

codecov bot commented Feb 6, 2020 •

edited

jwhite3 Feb 11, 2020 •

edited

jwhite3 Feb 11, 2020 •

edited