Merge pull request #367 from mwcraig/finish-detection

Finish off source detection

_toc.yml

@@ -100,8 +100,21 @@ parts:
         - file: notebooks/photometry/01.03-Local-peak-detection.ipynb
           title: Local peak detection
           sections:
+          - file: notebooks/photometry/01.03.01-peak_detection-simulated-image.ipynb
+            title: Overview with simulated image
           - file: notebooks/photometry/01.03.02-peak_detection-XDF.ipynb
             title: Peak detection in the XDF
+          - file: notebooks/photometry/01.03.03-peak_detection-Feder.ipynb
+            title: Peak detection in a ground-based stellar image
+        - file: notebooks/photometry/01.04-image-segmentation.ipynb
+          title: Image segmentation
+          sections:
+          - file: notebooks/photometry/01.04.01-image-segmentation-simulated-image.ipynb
+            title: Overview with simulated image
+          - file: notebooks/photometry/01.04.02-image-segmentation-XDF.ipynb
+            title: Image segmentation in the XDF
+          - file: notebooks/photometry/01.04.03-image-segmentation-Feder.ipynb
+            title: Image segmentation in a ground-based stellar image
 
 
 # - url: https://github.com/mwcraig/ccd-reduction-and-photometry-guide

notebooks/photometry/00.00-Preface.ipynb

@@ -22,19 +22,14 @@
     "```{note}\n",
     "Several notebooks in this part of the guide were origninally developed by Lauren CHambers and Erik Tollerud with support from the Community Software Initiative at the Space Telescope Science Institute, the Astropy Project and ScienceBetter Consulting. Those notebooks are:\n",
     "\n",
-    "+ A notebook\n",
-    "+ Another notebook \n",
+    "+ [Handling masking before background removal](01.01.02-Background-estimation-XDF.ipynb)\n",
+    "+ [Detecting sources in the XDF](01.02.02-IRAF-like-source-detection-XDF.ipynb) \n",
+    "+ [Peak finding in the XDF](01.03.02-peak_detection-XDF.ipynb)\n",
+    "+ [Image segmentation in the XDF](01.04.02-image-segmentation-XDF)\n",
     "\n",
-    "<img style=\"float: right;\" src=\"https://raw.githubusercontent.com/spacetelescope/notebooks/master/assets/stsci_pri_combo_mark_horizonal_white_bkgd.png\" alt=\"STScI logo\" width=\"200px\"/>\n",
+    "<img style=\"float: left;\" src=\"https://raw.githubusercontent.com/spacetelescope/notebooks/master/assets/stsci_pri_combo_mark_horizonal_white_bkgd.png\" alt=\"STScI logo\" width=\"200px\"/>\n",
     "```"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
@@ -53,7 +48,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.3"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

notebooks/photometry/01.00-Source-detection.ipynb

@@ -15,7 +15,7 @@
     "\n",
     "+ Find sources in your image and perform photometry on every source you can detect in the image.\n",
     "+ Perform photometry on objects from a catalog (either one you have created or from some other source); this presumes there is WCS information in the images so that catalog RA/Dec can be translated to pixel position on the image.\n",
-    "+ Stack your images, detect the sources in that much deeper stacked image, treat those sources as your catalog and use that catlog to erform photometry."
+    "+ Stack your images, detect the sources in that much deeper stacked image, treat those sources as your catalog and use that catlog to perform photometry."
    ]
   },
   {
@@ -46,7 +46,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.4"
+   "version": "3.11.8"
   }
  },
  "nbformat": 4,

notebooks/photometry/01.01-Background-removal.ipynb

@@ -11,7 +11,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The first step in detecting sources is to remove any background from the image. There are a number of ways to do this, ranging from subtracting a single value (typically the median or mean) from every pixel to more sophisticated methods either fit a slowly-varying shape to the background or that smooth the input image in some way.\n",
+    "The first step in detecting sources is to remove any background from the image. There are a number of ways to do this, ranging from subtracting a single value (typically the median or mean) from every pixel to more sophisticated methods that either fit a slowly-varying shape to the background or that smooth the input image in some way.\n",
     "\n",
     "There is probably not a single \"best\" way to do this; what matters is whether you are able to detect the sources you need to detect in the images you have.\n",
     "\n",
@@ -35,7 +35,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.3"
+   "version": "3.11.8"
   }
  },
  "nbformat": 4,

notebooks/photometry/01.01.01-Background-removal-simulated-data.ipynb

@@ -11,11 +11,31 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The first step in detecting sources is to remove any background from the image. There are a number of ways to do this, ranging from subtracting a single value (typically the median or mean) from every pixel to more sophisticated methods either fit a slowly-varying shape to the background or that smooth the input image in some way.\n",
+    "The first step in detecting sources is to remove any background from the image. This notebook walks through a couple of ways to do that: using a single value for the background or constructing a two dimensional background image.\n",
     "\n",
-    "There is probably not a single \"best\" way to do this; what matters is whether you are able to detect the sources you need to detect in the images you have.\n",
+    "The imports for this notebook are in the cell below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "\n",
+    "%matplotlib inline\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "from photutils.background import Background2D, MeanBackground, MADStdBackgroundRMS\n",
+    "from photutils.datasets import make_100gaussians_image\n",
+    "from photutils.segmentation import detect_threshold, detect_sources\n",
+    "from photutils.utils import circular_footprint\n",
     "\n",
-    "Note that the way you do background removal for source detection does *not* need to be the same way you do it for photometry. It is perfectly fine to use the same method for both, of course, but it is not required."
+    "from astropy.visualization import SqrtStretch\n",
+    "from astropy.visualization.mpl_normalize import ImageNormalize\n",
+    "from astropy.visualization import hist\n",
+    "from astropy.stats import sigma_clipped_stats, SigmaClip"
    ]
   },
   {
@@ -39,9 +59,11 @@
     "\n",
     "We begin by constructing a histogram of pixel values in a sample image. The code below illustrates a few useful things from astropy and photutils:\n",
     "\n",
-    "+ photutils comes with useful simulated images.\n",
-    "+ astropy comes with a histogram function that will automatically bin the data in a way that brings out features in your data that uniform-width bins might miss.\n",
-    "+ astropy has several stretches and normalizations to bring out features in astronomical images.\n"
+    "+ [`photutils`](https://photutils.readthedocs.io/en/stable/) comes with useful simulated images.\n",
+    "+ [`astropy`](https://docs.astropy.org/en/stable/) comes with a histogram function that will automatically bin the data in a way that brings out features in your data that uniform-width bins might miss.\n",
+    "+ [`astropy`](https://docs.astropy.org/en/stable/) has several stretches and normalizations to bring out features in astronomical images.\n",
+    "\n",
+    "The simulated image we will work with in this section is below.\n"
    ]
   },
   {
@@ -50,34 +72,21 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import numpy as np\n",
-    "\n",
-    "%matplotlib inline\n",
-    "import matplotlib.pyplot as plt\n",
-    "\n",
-    "from photutils.background import Background2D, MeanBackground, MADStdBackgroundRMS\n",
-    "from photutils.datasets import make_100gaussians_image\n",
-    "from photutils.segmentation import detect_threshold, detect_sources\n",
-    "from photutils.utils import circular_footprint\n",
-    "\n",
-    "from astropy.visualization import SqrtStretch\n",
-    "from astropy.visualization.mpl_normalize import ImageNormalize\n",
-    "from astropy.visualization import hist\n",
-    "from astropy.stats import sigma_clipped_stats, SigmaClip\n",
-    "\n",
     "data = make_100gaussians_image()\n",
     "\n",
+    "# Create an image stretch that will be applied when we display the image.\n",
     "norm = ImageNormalize(stretch=SqrtStretch())\n",
-    "plt.imshow(data, norm=norm, origin='lower', cmap='Greys_r')\n"
+    "\n",
+    "plt.figure(figsize=(10, 5))\n",
+    "plt.imshow(data, norm=norm, origin='lower', cmap='Greys_r')\n",
+    "plt.title(\"Simulated image with 100 Gaussian sources\")"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "There are a couple of things to note about this distrubtion. \n",
-    "\n",
-    "The bulk of the pixel values are noise. The function [`make_100gaussians_image`](https://photutils.readthedocs.io/en/stable/api/photutils.datasets.make_100gaussians_image.html#photutils.datasets.make_100gaussians_image) adds a background with a Gaussian distribution centered at 5 with a standard deviation of 2. The long tail at higher pixel values are from the sources in the image."
+    "Next we plot a histogram of the pixel values in the image. As suggested at the beginning of this section, the distribution is not symmetric."
    ]
   },
   {
@@ -95,17 +104,33 @@
     "                        alpha=0.5, label='Pixel values')\n",
     "plt.xlabel('Pixel value')\n",
     "plt.ylabel('Number of pixels')\n",
-    "# The semi-colon supresses a display of the grid object representation\n",
+    "plt.title(\"Histogram of pixel values\")\n",
+    "# The semi-colon suppresses a display of the grid object representation\n",
     "plt.grid();"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "There are a couple of things to note about this distribution. \n",
+    "\n",
+    "The bulk of the pixel values are noise. The function [`make_100gaussians_image`](https://photutils.readthedocs.io/en/stable/api/photutils.datasets.make_100gaussians_image.html#photutils.datasets.make_100gaussians_image) adds a background with a Gaussian distribution centered at 5 with a standard deviation of 2. The long tail at higher pixel values are from the sources in the image."
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "### Estimating the background value\n",
     "\n",
-    "The plot below shows the result of several ways of estimating the background from the data. None of these straightforward estimates give a good measure of the true background level, indicated by the green line."
+    "The plot below shows the result of several ways of estimating the background from the data:\n",
+    "\n",
+    "+ mean of all of the pixels\n",
+    "+ median of all of the values\n",
+    "+ a sigma-clipped median of all of the values.\n",
+    "\n",
+    "None of these straightforward estimates give a good measure of the true background level, which is indicated by the green line."
    ]
   },
   {
@@ -133,6 +158,7 @@
     "plt.axvline(clipped_med, label='Sigma-clipped median', color='yellow')\n",
     "plt.xlabel('Pixel value')\n",
     "plt.ylabel('Number of pixels')\n",
+    "plt.title(\"Methods of estimating background value\")\n",
     "plt.legend();"
    ]
   },
@@ -223,6 +249,13 @@
     "source_mask = segment_img.make_source_mask(footprint=footprint)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We now add the sigma-clipped median of just the pixels without a source in them."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -240,8 +273,9 @@
     "plt.axvline(avg_all_pixels, label='Mean of all values', color='red')\n",
     "plt.axvline(median_all_pixels, label='Median of all values', color='cyan')\n",
     "plt.axvline(clipped_med, label='Sigma-clipped median', color='yellow')\n",
-    "plt.axvline(mask_clip_med, label='Sigma-clipped, sources masked',\n",
+    "plt.axvline(mask_clip_med, label='Sigma-clipped median, sources masked',\n",
     "            color='violet', linestyle='dashed')\n",
+    "plt.title(\"Methods of estimating background value\")\n",
     "plt.legend();"
    ]
   },
@@ -295,7 +329,8 @@
     "data2 = data + gradient\n",
     "\n",
     "plt.figure(figsize=(10, 5))\n",
-    "plt.imshow(data2, norm=norm, origin='lower', cmap='Greys_r')"
+    "plt.imshow(data2, norm=norm, origin='lower', cmap='Greys_r')\n",
+    "plt.title(\"Simulated image with bakground gradient\");"
    ]
   },
   {
@@ -363,7 +398,8 @@
    "outputs": [],
    "source": [
     "plt.figure(figsize=(10, 5))\n",
-    "plt.imshow(bgd.background, norm=norm, origin='lower', cmap='Greys_r')"
+    "plt.imshow(bgd.background, norm=norm, origin='lower', cmap='Greys_r')\n",
+    "plt.title(\"Calculated image background\");"
    ]
   },
   {
@@ -383,10 +419,9 @@
    "source": [
     "back_sub_data = data2 - bgd.background\n",
     "\n",
-    "sub_mean, sub_med, sub_std  = sigma_clipped_stats(back_sub_data,\n",
-    "                                                  sigma=3)\n",
+    "sub_mean, sub_med, sub_std  = sigma_clipped_stats(back_sub_data, sigma=3)\n",
     "\n",
-    "print(sub_mean, sub_med, sub_std)"
+    "print(f\"Mean: {sub_mean}, Median: {sub_med}, Standard deviation: {sub_std}\")"
    ]
   },
   {
@@ -415,7 +450,7 @@
    "source": [
     "## Summary\n",
     "\n",
-    "An image needs to be background-subtracted before you detect the sources in the image. That background can be represented by either a single number (e.g. the mean or median) or by a two-dimensional image. In both cases the best estimate of the background is obtained when sources are masked out before estimating the background. Though this means doing two rounds of source detection, one to estimate the background and one for the actual source detection, the method of finding sources used above is reasonably fast.\n",
+    "An image needs to be background-subtracted before you detect the sources in the image. That background can be represented by either a single number, e.g. the mean or median, or by a two-dimensional image. In both cases the best estimate of the background is obtained when sources are masked out before estimating the background. Though this means doing two rounds of source detection, one to estimate the background and one for the actual source detection, the method of finding sources used above is reasonably fast.\n",
     "\n",
     "Next, we will look at how to do background estimation for a couple of actual science images to get a more nuanced understanding of the choices in background subtraction."
    ]
@@ -437,7 +472,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.3"
+   "version": "3.11.8"
   }
  },
  "nbformat": 4,

notebooks/photometry/01.01.02-Background-estimation-XDF.ipynb

@@ -106,7 +106,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Let's look at the data, including the background we've added. The cell below sets up an image normaliztaion we will use for all of our later views of the image. It also defines a small function, `format_colorbar` that is used to set up the colorbar in the subsequent plots."
+    "Let's look at the data, including the background we've added. The cell below sets up an image normaliztaion we will use for all of our later views of the image. It also defines a small function, `format_colorbar`, that is used to set up the colorbar in the subsequent plots."
    ]
   },
   {
@@ -281,7 +281,7 @@
    "source": [
     "### Mask sources, then calculate scalar background\n",
     "\n",
-    "As discussed in [FILL IN LINK](FILL IN LINK), the most accurate estimate of the scalar background is obtained when the sources are masked first. The cell below does that procedure."
+    "As discussed in [the section on background estimation with a simulated image](01.01.01-Background-removal-simulated-data.ipynb), the most accurate estimate of the scalar background is obtained when the sources are masked first. The cell below does that procedure."
    ]
   },
   {
@@ -441,14 +441,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The `Background2D` class in `photutils` allows users to model 2-dimensional backgrounds, by calculating the mean or median in small boxes, and smoothing these boxes to reconstruct a continuous 2D background. The class includes the following arguments/attributes:\n",
+    "The [`Background2D`](https://photutils.readthedocs.io/en/stable/api/photutils.background.Background2D.html#photutils.background.Background2D) class in [`photutils`](https://photutils.readthedocs.io/en/stable/index.html) allows users to model 2-dimensional backgrounds, by calculating the mean or median in small boxes, and smoothing these boxes to reconstruct a continuous 2D background. The class includes the following arguments/attributes:\n",
     "* **`box_size`** — the size of the boxes used to calculate the background. This should be larger than individual sources, yet still small enough to encompass changes in the background.\n",
     "* **`filter_size`** — the size of the median filter used to smooth the final 2D background. The dimension should be odd along both axes.\n",
     "* **`filter_threshold`** — threshold below which the smoothing median filter will not be applied.\n",
-    "* **`sigma_clip`** — an ` astropy.stats.SigmaClip` object that is used to specify the sigma and number of iterations used to sigma-clip the data before background calculations are performed.\n",
+    "* **`sigma_clip`** — an [`astropy.stats.SigmaClip`](https://docs.astropy.org/en/stable/api/astropy.stats.SigmaClip.html#astropy.stats.SigmaClip) object that is used to specify the sigma and number of iterations used to sigma-clip the data before background calculations are performed.\n",
     "* **`bkg_estimator`** — the method used to perform the background calculation in each box (mean, median, SExtractor algorithm, etc.).\n",
     "\n",
-    "For this example, we will use the `MeanBackground` estimator. Note though, that there is little or no spatial variation apparent in the background of this image."
+    "For this example, we will use the [`MeanBackground`](https://photutils.readthedocs.io/en/stable/api/photutils.background.MeanBackground.html#photutils.background.MeanBackground) estimator. Note though, that there is little or no spatial variation apparent in the background of this image."
    ]
   },
   {
@@ -592,7 +592,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.3"
+   "version": "3.11.8"
   }
  },
  "nbformat": 4,