notebooks/02-dimension-reduction-and-discretization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 02 - Dimension reduction and discretization\n",
    "\n",
    "<a rel=\"license\" href=\"http://creativecommons.org/licenses/by/4.0/\"><img alt=\"Creative Commons Licence\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by/4.0/88x31.png\" title='This work is licensed under a Creative Commons Attribution 4.0 International License.' align=\"right\"/></a>\n",
    "\n",
    "In this notebook, we will cover how to perform dimension reduction and discretization of molecular dynamics data. We also repeat data loading and visualization tasks from [Notebook 01 ➜ 📓](01-data-io-and-featurization.ipynb).\n",
    "\n",
    "Maintainers: [@cwehmeyer](https://github.com/cwehmeyer), [@marscher](https://github.com/marscher), [@thempel](https://github.com/thempel), [@psolsson](https://github.com/psolsson)\n",
    "\n",
    "**Remember**:\n",
    "- to run the currently highlighted cell, hold <kbd>&#x21E7; Shift</kbd> and press <kbd>&#x23ce; Enter</kbd>;\n",
    "- to get help for a specific function, place the cursor within the function's brackets, hold <kbd>&#x21E7; Shift</kbd>, and press <kbd>&#x21E5; Tab</kbd>;\n",
    "- you can find the full documentation at [PyEMMA.org](http://www.pyemma.org)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import mdshare\n",
    "import pyemma"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case 1: preprocessed, two-dimensional data (toy model)\n",
    "\n",
    "We load the two-dimensional trajectory from an archive using numpy\n",
    "(see [Notebook 01 ➜ 📓](01-data-io-and-featurization.ipynb))\n",
    "and visualize the marginal and joint distributions of both components.\n",
    "In order to make the important concept of metastability easier to understand,\n",
    "an excerpt from the original trajectory is added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file = mdshare.fetch('hmm-doublewell-2d-100k.npz', working_directory='data')\n",
    "with np.load(file) as fh:\n",
    "    data = fh['trajectory']\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n",
    "pyemma.plots.plot_feature_histograms(data, feature_labels=['$x$', '$y$'], ax=axes[0])\n",
    "for i, dim in enumerate(['y', 'x']): \n",
    "    axes[0].plot(data[:300, 1 - i], np.linspace(-0.2 + i, 0.8 + i, 300), color='C2', alpha=0.6)\n",
    "    axes[0].annotate(\n",
    "        '${}$(time)'.format(dim),\n",
    "        xy=(3, 0.6 + i),\n",
    "        xytext=(3, i),\n",
    "        arrowprops=dict(fc='C2', ec='None', alpha=0.6, width=2))\n",
    "pyemma.plots.plot_density(*data.T, ax=axes[1])\n",
    "axes[1].set_xlabel('$x$')\n",
    "axes[1].set_ylabel('$y$')\n",
    "axes[1].set_xlim(-4, 4)\n",
    "axes[1].set_ylim(-4, 4)\n",
    "axes[1].set_aspect('equal')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the low dimensionality of this data set,\n",
    "we can directly attempt to discretize,\n",
    "e.g., with $k$-means with $100$ centers and a stride of $5$ to reduce the computational effort.\n",
    "In real world examples we also might encounter low dimensional feature spaces\n",
    "which do not require further dimension reduction techniques to be clustered efficiently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster_kmeans = pyemma.coordinates.cluster_kmeans(data, k=100, stride=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... or with a regspace technique where all centers should have a minimal pairwise distance of $0.5$ units of length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster_regspace = pyemma.coordinates.cluster_regspace(data, dmin=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We visualize both sets of centers on top of the joint distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n",
    "for ax, cls in zip(axes.flat, [cluster_kmeans, cluster_regspace]):\n",
    "    pyemma.plots.plot_density(*data.T, ax=ax, cbar=False, alpha=0.1)\n",
    "    ax.scatter(*cls.clustercenters.T, s=15, c='C1')\n",
    "    ax.set_xlabel('$x$')\n",
    "    ax.set_xlim(-4, 4)\n",
    "    ax.set_ylim(-4, 4)\n",
    "    ax.set_aspect('equal')\n",
    "axes[0].set_ylabel('$y$')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have you noticed how the $k$-means centers follow the density of the data points while the regspace centers are spread uniformly over the whole area? \n",
    "\n",
    "If your are only interested in well sampled states, you should use a density based method to discretize.\n",
    "If exploring new states is one of your objectives,\n",
    "it might be of advantage to place states also in rarely observed regions.\n",
    "The latter is especially useful in adaptive sampling approaches,\n",
    "because in the initial phase you want to explore the phase space as much as possible.\n",
    "The downside of placing states in areas of low density is that we will have poor statistics on these states. \n",
    "\n",
    "Another advantage of regular space clustering is that it is fast in comparison to $k$-means:\n",
    "regspace clustering runs in linear time while $k$-means is superpolynomial in time.\n",
    "\n",
    "⚠️ For large datasets we also offer a mini batch version of $k$-means which has the same semantics as the original method but trains the centers on subsets of your data.\n",
    "This tutorial does not cover this case, but you should keep in mind that $k$-means requires your low dimensional space to fit into your main memory.\n",
    "\n",
    "As the number of cluster centers has to be chosen by the modeler, we will analyze its implications on the MSM analysis in [Notebook 03 ➜ 📓](03-msm-estimation-and-validation.ipynb). A systematic approach to choose this number is proposed in [Notebook 00 ➜ 📓](00-pentapeptide-showcase.ipynb).\n",
    "\n",
    "The main result of a discretization for Markov modeling, however,\n",
    "is not the set of centers but the time series of discrete states.\n",
    "These are accessible via the `dtrajs` attribute of any clustering object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(cluster_kmeans.dtrajs)\n",
    "print(cluster_regspace.dtrajs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each trajectory passed to the clustering object, we get a corresponding discrete trajectory.\n",
    "\n",
    "Please note that as an alternative to clustering algorithms such as $k$-means and regspace,\n",
    "it is possible to manually assign the data to cluster centers using the `pyemma.coordinates.assign_to_centers()` function.\n",
    "\n",
    "Instead of discretizing the full (two-dimensional) space, we can attempt to find a one-dimensional subspace which\n",
    "1. describes the slow dynamics of the data set equally well but\n",
    "2. is easier to discretize.\n",
    "\n",
    "One widespread method for dimension reduction is the principal component analysis (PCA) which finds a subspace with maximized variance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = pyemma.coordinates.pca(data, dim=1)\n",
    "pca_output = pca.get_output()\n",
    "print(pca_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another technique is the time-lagged independent component analysis (TICA) which finds a subspace with maximized autocorrelation <a id=\"ref-1\" href=\"#cite-tica2\">molgedey-94</a>, <a id=\"ref-2\" href=\"#cite-tica\">perez-hernandez-13</a>.\n",
    "To compute the autocorrelation, we need a time shifted version of the data.\n",
    "This time shift is specified by the `lag` argument.\n",
    "For the current example, we choose a lag time of $1$ step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tica = pyemma.coordinates.tica(data, dim=1, lag=1)\n",
    "tica_output = tica.get_output()\n",
    "print(tica_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of TICA, we can also employ the variational approach for Markov processes (VAMP) to obtain a coordinate transform <a id=\"ref-3\" href=\"#cite-vamp-preprint\">wu-17</a>.\n",
    "In contrast to TICA, VAMP can be applied to non-equilibrium / non-reversible data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vamp = pyemma.coordinates.vamp(data, dim=1, lag=1)\n",
    "vamp_output = vamp.get_output()\n",
    "print(vamp_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "⚠️ While there are many cases where PCA can find a suitable subspace,\n",
    "there are also many cases where the PCA-based subspace neglects the slow dynamics.\n",
    "\n",
    "In our example, the slow process is the jump between both wells along the $y$-axis while the $x$-axis contains only random noise. For all three methods, we show the distribution after projecting the full dynamics onto a one-dimensional subspace (left) and the direction of projection (right)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_concatenated = pca_output[0]\n",
    "tica_concatenated = tica_output[0]\n",
    "vamp_concatenated = vamp_output[0]\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    np.concatenate([pca_concatenated, tica_concatenated, vamp_concatenated], axis=1),\n",
    "    feature_labels=['PCA', 'TICA', 'VAMP'],\n",
    "    ax=axes[0])\n",
    "pyemma.plots.plot_density(*data.T, ax=axes[1], cbar=False, alpha=0.1)\n",
    "axes[1].plot(\n",
    "    [0, 3 * pca.eigenvectors[0, 0]],\n",
    "    [0, 3 * pca.eigenvectors[1, 0]],\n",
    "    linewidth=3,\n",
    "    label='PCA')\n",
    "axes[1].plot(\n",
    "    [0, 3 * tica.eigenvectors[0, 0]],\n",
    "    [0, 3 * tica.eigenvectors[1, 0]],\n",
    "    linewidth=3,\n",
    "    label='TICA')\n",
    "axes[1].plot(\n",
    "    [0, 3 * vamp.singular_vectors_right[0, 0]],\n",
    "    [0, 3 * vamp.singular_vectors_right[1, 0]],\n",
    "    linewidth=3,\n",
    "    label='VAMP')\n",
    "axes[1].set_xlabel('$x$')\n",
    "axes[1].set_ylabel('$y$')\n",
    "axes[1].set_xlim(-4, 4)\n",
    "axes[1].set_ylim(-4, 4)\n",
    "axes[1].set_aspect('equal')\n",
    "axes[1].legend()\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that TICA and VAMP project along the $y$-axis and, thus, yield a subspace which clearly resolves both metastable states.\n",
    "PCA on the other hand projects closely along the $x$-axis and does not resolve both metastable states.\n",
    "This is a case in point where variance maximization does not find a subspace which resolves the relevant dynamics of the system.\n",
    "\n",
    "This effect can also be seen when we plot the subspace time series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 3))\n",
    "ax.plot(pca_concatenated[:300], label='PCA')\n",
    "ax.plot(tica_concatenated[:300], label='TICA')\n",
    "# note that for better comparability, we enforce the same direction as TICA\n",
    "ax.plot(vamp_concatenated[:300] * -1, label='VAMP')\n",
    "ax.set_xlabel('time / steps')\n",
    "ax.set_ylabel('feature values')\n",
    "ax.legend()\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case of TICA/VAMP, we observe that the projected coordinate jumps between two clearly separated plateaus.\n",
    "For PCA, we observe only random fluctuations without any hint of metastablility.\n",
    "\n",
    "In many applications, however, we also need to understand what our coordinate transform means in physical terms.\n",
    "This, in general, might be less obvious.\n",
    "Hence, it might be instructive to inspect the correlations of features to the independent components:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "i = ax.imshow(tica.feature_TIC_correlation, cmap='bwr', vmin=-1, vmax=1)\n",
    "\n",
    "ax.set_xticks([0])\n",
    "ax.set_xlabel('IC')\n",
    "\n",
    "ax.set_yticks([0, 1])\n",
    "ax.set_ylabel('input feature')\n",
    "\n",
    "fig.colorbar(i);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this simple example, we clearly see a significant correlation between the $y$ component of the input data and the first independent component."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Case 2: low-dimensional molecular dynamics data (alanine dipeptide)\n",
    "\n",
    "We fetch the alanine dipeptide data set, load the backbone torsions into memory, and visualize the margial and joint distributions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdb = mdshare.fetch('alanine-dipeptide-nowater.pdb', working_directory='data')\n",
    "files = mdshare.fetch('alanine-dipeptide-*-250ns-nowater.xtc', working_directory='data')\n",
    "\n",
    "feat = pyemma.coordinates.featurizer(pdb)\n",
    "feat.add_backbone_torsions(periodic=False)\n",
    "data = pyemma.coordinates.load(files, features=feat)\n",
    "data_concatenated = np.concatenate(data)\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    np.concatenate(data), feature_labels=['$\\Phi$', '$\\Psi$'], ax=axes[0])\n",
    "pyemma.plots.plot_density(*data_concatenated.T, ax=axes[1], logscale=True)\n",
    "axes[1].set_xlabel('$\\Phi$')\n",
    "axes[1].set_ylabel('$\\Psi$')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following the previous example, we perform a $k$-means ($100$ centers, stride of $5$) and a regspace clustering ($0.3$ radians center distance) on the full two-dimensional data set and visualize the obtained centers. In [Notebook 03 ➜ 📓](03-msm-estimation-and-validation.ipynb), we show the effect of different numbers of cluster centers on MSM estimation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster_kmeans = pyemma.coordinates.cluster_kmeans(data, k=100, max_iter=50, stride=5)\n",
    "cluster_regspace = pyemma.coordinates.cluster_regspace(data, dmin=0.3)\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4), sharex=True, sharey=True)\n",
    "for ax, cls in zip(axes.flat, [cluster_kmeans, cluster_regspace]):\n",
    "    pyemma.plots.plot_density(*data_concatenated.T, ax=ax, cbar=False, alpha=0.1, logscale=True)\n",
    "    ax.scatter(*cls.clustercenters.T, s=15, c='C1')\n",
    "    ax.set_xlabel('$\\Phi$')\n",
    "    ax.set_ylabel('$\\Psi$')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, notice the difference between $k$-means and regspace clustering.\n",
    "\n",
    "Now, we use a different featurization for the same data set and revisit how to use PCA, TICA, and VAMP.\n",
    "\n",
    "⚠️ In practice you almost never would like to use PCA as dimension reduction method in MSM building,\n",
    "as it does not preserve kinetic variance. We are showing it here in these exercises to make this point clear."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Streaming memory discretization\n",
    "For real world case examples it is often not possible to load entire datasets into main memory. We can perform the whole discretization step without the need of having the dataset fit into memory. Keep in mind that this is not as efficient as loading into memory, because certain calculations (e.g. featurization), will have to be recomputed during iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reader = pyemma.coordinates.source(files, top=pdb) # create reader\n",
    "reader.featurizer.add_backbone_torsions(periodic=False) # add feature\n",
    "tica = pyemma.coordinates.tica(reader) # perform tica on feature space\n",
    "cluster = pyemma.coordinates.cluster_mini_batch_kmeans(tica, k=10, batch_size=0.1, max_iter=3) # cluster in tica space\n",
    "# get result\n",
    "dtrajs = cluster.dtrajs\n",
    "print('discrete trajectories:', dtrajs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should mention that regular space clustering does not require to load the TICA output into memory, while $k$-means does. Use the minibatch version if your TICA output does not fit memory. Since the minibatch version takes more time to converge, it is therefore desirable to to shrink the TICA output to fit into memory. We split the pipeline for cluster estimation, and re-use the reader to for the assignment of the full dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster = pyemma.coordinates.cluster_kmeans(tica, k=10, stride=3) # use only 1/3 of the input data to find centers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have you noticed how fast this converged compared to the minibatch version?\n",
    "We can now just obtain the discrete trajectories by accessing the property on the cluster instance.\n",
    "This will get all the TICA projected trajectories and assign them to the centers computed on the reduced data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dtrajs = cluster.dtrajs\n",
    "print('Assignment:', dtrajs)\n",
    "dtrajs_len = [len(d) for d in dtrajs]\n",
    "for dtraj_len, input_len in zip(dtrajs_len, reader.trajectory_lengths()):\n",
    "    print('Input length:', input_len, '\\tdtraj length:', dtraj_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 1: data loading \n",
    "\n",
    "Load the heavy atoms' positions into memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "feat = pyemma.coordinates.featurizer(pdb)\n",
    "feat. #FIXME\n",
    "data = pyemma.coordinates.load(files, features=feat)\n",
    "\n",
    "print('We have {} features.'.format(feat.dimension()))\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 7))\n",
    "pyemma.plots.plot_feature_histograms(np.concatenate(data), feature_labels=feat, ax=ax)\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden"
   },
   "source": [
    "###### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "feat = pyemma.coordinates.featurizer(pdb)\n",
    "feat.add_selection(feat.select_Heavy())\n",
    "data = pyemma.coordinates.load(files, features=feat)\n",
    "\n",
    "print('We have {} features.'.format(feat.dimension()))\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 7))\n",
    "pyemma.plots.plot_feature_histograms(np.concatenate(data), feature_labels=feat, ax=ax)\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Discretizing a $30$-dimensional feature space is impractical.\n",
    "Let's use PCA to find a low-dimensional projection and visualize the marginal distributions of all principal components (PCs) as well as the joint distributions for the first two PCs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = pyemma.coordinates.pca(data)\n",
    "pca_concatenated = np.concatenate(pca.get_output())\n",
    "\n",
    "fig, axes = plt.subplots(1, 3, figsize=(12, 3), sharex=True)\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    pca_concatenated, ['PC {}'.format(i + 1) for i in range(pca.dimension())], ax=axes[0])\n",
    "pyemma.plots.plot_density(*pca_concatenated[:, :2].T, ax=axes[1], cbar=False, logscale=True)\n",
    "pyemma.plots.plot_free_energy(*pca_concatenated[:, :2].T, ax=axes[2], legacy=False)\n",
    "for ax in axes.flat[1:]:\n",
    "    ax.set_xlabel('PC 1')\n",
    "    ax.set_ylabel('PC 2')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the default parameters, PCA will return as many dimensions as necessary to explain $95\\%$ of the variance;\n",
    "in this case, we have found a five-dimensional subspace which does seem to resolve some metastability in the first three principal components.\n",
    "\n",
    "#### Exercise 2: TICA visualization\n",
    "\n",
    "Apply TICA and visualize the marginal distributions of all independent components (ICs) as well as the joint distributions of the first two ICs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "tica =  #FIXME\n",
    "tica_concatenated = np.concatenate(tica.get_output())\n",
    "\n",
    "fig, axes = plt.subplots(1, 3, figsize=(12, 3))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    tica_concatenated, ['IC {}'.format(i + 1) for i in range(tica.dimension())], ax=axes[0])\n",
    "pyemma.plots.plot_density(*tica_concatenated[:, :2].T, ax=axes[1], cbar=False, logscale=True)\n",
    "pyemma.plots.plot_free_energy(*tica_concatenated[:, :2].T, ax=axes[2], legacy=False)\n",
    "for ax in axes.flat[1:]:\n",
    "    ax.set_xlabel('IC 1')\n",
    "    ax.set_ylabel('IC 2')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden"
   },
   "source": [
    "###### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "tica = pyemma.coordinates.tica(data)\n",
    "tica_concatenated = np.concatenate(tica.get_output())\n",
    "\n",
    "fig, axes = plt.subplots(1, 3, figsize=(12, 3))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    tica_concatenated, ['IC {}'.format(i + 1) for i in range(tica.dimension())], ax=axes[0])\n",
    "pyemma.plots.plot_density(*tica_concatenated[:, :2].T, ax=axes[1], cbar=False, logscale=True)\n",
    "pyemma.plots.plot_free_energy(*tica_concatenated[:, :2].T, ax=axes[2], legacy=False)\n",
    "for ax in axes.flat[1:]:\n",
    "    ax.set_xlabel('IC 1')\n",
    "    ax.set_ylabel('IC 2')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TICA, by default, uses a lag time of $10$ steps, kinetic mapping and a kinetic variance cutoff of $95\\%$ to determine the number of ICs.\n",
    "We observe that this projection does resolve some metastability in both ICs.\n",
    "Whether these projections are suitable for building Markov state models, though, remains to be seen in later tests\n",
    "([Notebook 03 ➜ 📓](03-msm-estimation-and-validation.ipynb)).\n",
    "\n",
    "As we discussed in the first example, the physical meaning of the TICA projection is not directly clear.\n",
    "We can analyze the feature TIC correlation as we did above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(3, 8))\n",
    "i = ax.imshow(tica.feature_TIC_correlation, cmap='bwr')\n",
    "\n",
    "ax.set_xticks(range(tica.dimension()))\n",
    "ax.set_xlabel('IC')\n",
    "\n",
    "ax.set_yticks(range(feat.dimension()))\n",
    "ax.set_yticklabels(feat.describe())\n",
    "ax.set_ylabel('input feature')\n",
    "\n",
    "fig.colorbar(i);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is not helpful as it only shows that some of our $x, y, z$-coordinates correlate with the TICA components.\n",
    "Since we rather expect the slow processes to happen in backbone torsion space, this comes to no surprise. \n",
    "\n",
    "To understand what the TICs really mean, let us do a more systematic approach and scan through some angular features.\n",
    "We add some randomly chosen angles between heavy atoms and the backbone angles that we already know to be a good feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_test = pyemma.coordinates.featurizer(pdb)\n",
    "feat_test.add_backbone_torsions(periodic=False)\n",
    "feat_test.add_angles(feat_test.select_Heavy()[:-1].reshape(3, 3), periodic=False)\n",
    "data_test = pyemma.coordinates.load(files, features=feat_test)\n",
    "data_test_concatenated = np.concatenate(data_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the sake of simplicity, we use scipy's implementation of Pearson's correlation coefficient which we compute between our test features and TICA projected $x, y, z$-coordinates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import pearsonr\n",
    "test_feature_TIC_correlation = np.zeros((feat_test.dimension(), tica.dimension()))\n",
    "\n",
    "for i in range(feat_test.dimension()):\n",
    "    for j in range(tica.dimension()):\n",
    "        test_feature_TIC_correlation[i, j] = pearsonr(\n",
    "            data_test_concatenated[:, i],\n",
    "            tica_concatenated[:, j])[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm = abs(test_feature_TIC_correlation).max()\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "i = ax.imshow(test_feature_TIC_correlation, vmin=-vm, vmax=vm, cmap='bwr')\n",
    "\n",
    "ax.set_xticks(range(tica.dimension()))\n",
    "ax.set_xlabel('IC')\n",
    "\n",
    "ax.set_yticks(range(feat_test.dimension()))\n",
    "ax.set_yticklabels(feat_test.describe())\n",
    "ax.set_ylabel('input feature')\n",
    "\n",
    "fig.colorbar(i);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this simple analysis, we find that features that correlated most with our TICA projection are indeed the backbone torsion angles used previously.\n",
    "We might thus expect the dynamics in TICA space to be similar to the one in backbone torsion space.\n",
    "\n",
    "⚠️ Please note that in general, we do not know which feature would be a good observable.\n",
    "Thus, a realistic scenario might require a much broader scan of a large set of different features.\n",
    "\n",
    "However, it should be mentioned that TICA projections do not necessarily have a simple physical interpretation.\n",
    "The above analysis might very well end with feature TIC correlations that show no significant contributor and rather hint towards a complicated linear combination of input features.\n",
    "\n",
    "As an alternative to understanding the projection in detail at this stage,\n",
    "one might go one step further and extract representative structures,\n",
    "e.g., from an MSM, as shown in [Notebook 05 📓](05-pcca-tpt.ipynb).\n",
    "\n",
    "#### Exercise 3: PCA parameters\n",
    "\n",
    "Perform PCA on the heavy atoms' positions data set with a target dimension of two;\n",
    "then discretize the two-dimensional subspace using $k$-means with $100$ centers and a stride of $5$ to reduce the computational effort.\n",
    "\n",
    "**Hint:** Look up the parameters of `pyemma.coordinates.pca()`, especially the `dim` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "pca =  # FIXME\n",
    "pca_concatenated =  # FIXME\n",
    "\n",
    "cluster = pyemma.coordinates.cluster_kmeans(pca, k=100, max_iter=50, stride=5)\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    pca_concatenated, ['PC {}'.format(i + 1) for i in range(pca.dimension())], ax=axes[0])\n",
    "pyemma.plots.plot_density(*pca_concatenated.T, ax=axes[1], cbar=False, alpha=0.1, logscale=True)\n",
    "axes[1].scatter(*cluster.clustercenters.T, s=15, c='C1')\n",
    "axes[1].set_xlabel('PC 1')\n",
    "axes[1].set_ylabel('PC 2')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden"
   },
   "source": [
    "###### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "pca = pyemma.coordinates.pca(data, dim=2)\n",
    "pca_concatenated = np.concatenate(pca.get_output())\n",
    "\n",
    "cluster = pyemma.coordinates.cluster_kmeans(pca, k=100, max_iter=50, stride=5)\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    pca_concatenated, ['PC {}'.format(i + 1) for i in range(pca.dimension())], ax=axes[0])\n",
    "pyemma.plots.plot_density(*pca_concatenated.T, ax=axes[1], cbar=False, alpha=0.1, logscale=True)\n",
    "axes[1].scatter(*cluster.clustercenters.T, s=15, c='C1')\n",
    "axes[1].set_xlabel('PC 1')\n",
    "axes[1].set_ylabel('PC 2')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 4: TICA parameters\n",
    "\n",
    "Perform TICA at lag time $1$ step on the heavy atoms' positions data set with a target dimension of two;\n",
    "then discretize the two-dimensional subspace using $k$-means with $100$ centers and a stride of $5$ to reduce the computational effort.\n",
    "\n",
    "**Hint:** Look up the parameters of `pyemma.coordinates.tica()`, especially the `dim` and `lag` parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "tica =  # FIXME\n",
    "tica_concatenated =  # FIXME\n",
    "\n",
    "cluster =  # FIXME\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    tica_concatenated, ['IC {}'.format(i + 1) for i in range(tica.dimension())], ax=axes[0])\n",
    "pyemma.plots.plot_density(*tica_concatenated.T, ax=axes[1], cbar=False, alpha=0.1, logscale=True)\n",
    "axes[1].scatter(*cluster.clustercenters.T, s=15, c='C1')\n",
    "axes[1].set_xlabel('IC 1')\n",
    "axes[1].set_ylabel('IC 2')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden"
   },
   "source": [
    "###### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "tica = pyemma.coordinates.tica(data, lag=1, dim=2)\n",
    "tica_concatenated = np.concatenate(tica.get_output())\n",
    "\n",
    "cluster = pyemma.coordinates.cluster_kmeans(tica, k=100, max_iter=50, stride=5)\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    tica_concatenated, ['IC {}'.format(i + 1) for i in range(tica.dimension())], ax=axes[0])\n",
    "pyemma.plots.plot_density(*tica_concatenated.T, ax=axes[1], cbar=False, alpha=0.1, logscale=True)\n",
    "axes[1].scatter(*cluster.clustercenters.T, s=15, c='C1')\n",
    "axes[1].set_xlabel('IC 1')\n",
    "axes[1].set_ylabel('IC 2')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have you noticed the difference in the first two ICs for lag times $10$ steps vs. $1$ step (e.g., result of exercises $2$ and $3$)?\n",
    "\n",
    "## Case 3: another molecular dynamics data set (pentapeptide)\n",
    "\n",
    "Before we start to load and discretize the pentapeptide data set,\n",
    "let us discuss what the difficulties with larger protein systems are.\n",
    "The goal of this notebook is to find a state space discretization for MSM estimation.\n",
    "This means that an algorithm such as $k$-means has to be able to find a meaningful state space partitioning.\n",
    "In general, this works better in lower dimensional spaces because Euclidean distances become less meaningful with increasing dimensionality <a id=\"ref-4\" href=\"#cite-aggarwal_surprising_2001\">aggarwal-01</a>.\n",
    "The modeler should be aware that a discretization of hundreds of dimensions will be computationally expensive and most likely yield unsatisfactory results. \n",
    "\n",
    "The first goal is thus to map the data to a reasonable number of dimensions, e.g.,\n",
    "with a smart choice of features and/or by using TICA.\n",
    "Large systems often require significant parts of the kinetic variance to be discarded in order to obtain a balance between capturing as much of the kinetic variance as possible and achieving a reasonable discretization.\n",
    "\n",
    "Another point about discretization algorithms is that one should bear in mind the distribution of density.\n",
    "The $k$-means algorithm tends to conserve density, i.e.,\n",
    "data sets that incorporate regions of extremely high density as well as poorly sampled regions might be problematic, especially in high dimensions.\n",
    "For those cases, a regular spatial clustering might be worth a try. \n",
    "\n",
    "More details on problematic data situations and how to cope with them are explained in [Notebook 08 📓](08-common-problems.ipynb).\n",
    "\n",
    "Now, we fetch the pentapeptide data set, load several different input features into memory and perform a VAMP estimation/scoring of each.\n",
    "Since we want to evaluate the VAMP score on a disjoint test set,\n",
    "we split the available files into a train and test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdb = mdshare.fetch('pentapeptide-impl-solv.pdb', working_directory='data')\n",
    "files = mdshare.fetch('pentapeptide-*-500ns-impl-solv.xtc', working_directory='data')\n",
    "\n",
    "feat = pyemma.coordinates.featurizer(pdb)\n",
    "feat.add_backbone_torsions(cossin=True, periodic=False)\n",
    "feat.add_sidechain_torsions(which='all', cossin=True, periodic=False)\n",
    "\n",
    "train_files = files[:-2]\n",
    "test_file = files[-2]  # last trajectory is our test data set\n",
    "assert set(train_files) & set(test_file) == set()  # ensure test and train sets do not overlap\n",
    "\n",
    "data_torsions = pyemma.coordinates.load(train_files, features=feat)\n",
    "data_torsions_test = pyemma.coordinates.load(test_file, features=feat)\n",
    "\n",
    "feat.active_features = []\n",
    "feat.add_distances_ca(periodic=False)\n",
    "\n",
    "data_dists_ca = pyemma.coordinates.load(train_files, features=feat)\n",
    "data_dists_ca_test = pyemma.coordinates.load(test_file, features=feat)\n",
    "\n",
    "feat.active_features = []\n",
    "pairs = feat.pairs(feat.select_Heavy())\n",
    "feat.add_contacts(pairs, periodic=False)\n",
    "\n",
    "data_contacts = pyemma.coordinates.load(train_files, features=feat)\n",
    "data_contacts_test = pyemma.coordinates.load(test_file, features=feat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_for_lag(ax, lag, dim=3):\n",
    "    vamp_torsions = pyemma.coordinates.vamp(data_torsions, lag=lag, dim=dim)\n",
    "    vamp_dist_ca = pyemma.coordinates.vamp(data_dists_ca, lag=lag, dim=dim)\n",
    "    vamp_contacts = pyemma.coordinates.vamp(data_contacts, lag=lag, dim=dim)\n",
    "\n",
    "    vamps = (vamp_torsions, vamp_dist_ca, vamp_contacts)\n",
    "    test_data = (data_torsions_test, data_dists_ca_test, data_contacts_test)\n",
    "    labels = ('torsions', 'CA distances', 'contacts')\n",
    "    for i, (v, test_data) in enumerate(zip(vamps, test_data)):\n",
    "        s = v.score(test_data=test_data)\n",
    "        ax.bar(i, s)\n",
    "    ax.set_title('VAMP2 @ lag = {} ps'.format(lag))\n",
    "    ax.set_xticks(range(len(vamps)))\n",
    "    ax.set_xticklabels(labels)\n",
    "    fig.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 4, figsize=(15, 3), sharey=True)\n",
    "plot_for_lag(axes[0], 5)\n",
    "plot_for_lag(axes[1], 10)\n",
    "plot_for_lag(axes[2], 20)\n",
    "plot_for_lag(axes[3], 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the small lag time we see that there is a large gap between torsion angles on the one hand and CA distances and contacts on the other hand.\n",
    "For increasing lag times this gap vanishes, but also the overall score is decreasing.\n",
    "Because we have capped the maximum dimension or equivalently the score to contain only the three largest components of the underlying dynamical model,\n",
    "we can expect only a maximum score of three.\n",
    "As we increase the lag time, more of the fast kinetic processes have already decayed.\n",
    "So these are not contributing to the score anymore. \n",
    "\n",
    "We have learned that backbone and sidechain torsions are better suited than the other features for modeling the kinetics,\n",
    "so we will continue with this feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_concatenated = data_torsions + [data_torsions_test] # concatenate two lists\n",
    "type(data_concatenated)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now perform a principal component analysis (PCA) with default parameters and visualize the marginal distributions of all PCs and the joint distributions of the first two PCs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = pyemma.coordinates.pca(data_concatenated, dim=2)\n",
    "pca_concatenated = np.concatenate(pca.get_output())\n",
    "\n",
    "fig, axes = plt.subplots(1, 3, figsize=(12, 3))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    pca_concatenated,\n",
    "    ['PC {}'.format(i + 1) for i in range(pca.dimension())],\n",
    "    ax=axes[0])\n",
    "pyemma.plots.plot_density(*pca_concatenated[:, :2].T, ax=axes[1], cbar=False)\n",
    "pyemma.plots.plot_free_energy(*pca_concatenated[:, :2].T, ax=axes[2], legacy=False)\n",
    "for ax in axes.flat[1:]:\n",
    "    ax.set_xlabel('PC 1')\n",
    "    ax.set_ylabel('PC 2')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a look at some TICA projections estimated with the lag times given below and, for each lag time,\n",
    "we visualize the marginal distributions of all ICs and the joint distributions of the first two ICs.\n",
    "The actual dimension is determined by the default variance cutoff of $95\\%$.\n",
    "This could be customized either by adjusting the `var_cutoff` or by requesting a certain number of dimensions with the `dim` keyword argument of `tica()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lags = [1, 2, 5, 10, 20, 50]\n",
    "\n",
    "fig, axes = plt.subplots(6, 3, figsize=(10, 18))\n",
    "for i, lag in enumerate(lags):\n",
    "    tica = pyemma.coordinates.tica(data_concatenated, lag=lag)\n",
    "    tica_concatenated = np.concatenate(tica.get_output())\n",
    "    pyemma.plots.plot_feature_histograms(\n",
    "        tica_concatenated,\n",
    "        ['IC {}'.format(i + 1) for i in range(tica.dimension())],\n",
    "        ax=axes[i, 0])\n",
    "    axes[i, 0].set_title('lag time = {} steps'.format(lag))\n",
    "    axes[i, 1].set_title(\n",
    "        'Density, actual dimension = {}'.format(tica.dimension()))\n",
    "    pyemma.plots.plot_density(\n",
    "        *tica_concatenated[:, :2].T, ax=axes[i, 1], cbar=False)\n",
    "    pyemma.plots.plot_free_energy(\n",
    "        *tica_concatenated[:, :2].T, ax=axes[i, 2], legacy=False)\n",
    "for ax in axes[:, 1:].flat:\n",
    "    ax.set_xlabel('IC 1')\n",
    "    ax.set_ylabel('IC 2')\n",
    "axes[0, 2].set_title('Pseudo free energy')\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have you noticed that increasing the lag time\n",
    "1. leads to a rotation of the projection and\n",
    "2. reduces the number of TICs to explain $95\\%$ (default) of the kinetic variance?\n",
    "\n",
    "Note that, while we can get lower and lower dimensional subspaces with increased lag times, we also loose information from the faster processes.\n",
    "\n",
    "How to choose the optimal lag time for a TICA projection often is a hard problem and there are only heuristic approaches to it.\n",
    "For example, you can search for the amount of dimensions where the variance cutoff does not change anymore.\n",
    "\n",
    "#### Exercise 5: PCA\n",
    "\n",
    "Perform PCA with target dimension $3$ on the current feature set and discretize the projected space using $k$-means with $100$ centers and a stride of $5$ to reduce the computational effort."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "pca =  # FIXME\n",
    "pca_concatenated = np.concatenate(pca.get_output(stride=5))\n",
    "\n",
    "cluster =  # FIXME\n",
    "\n",
    "fig, axes = plt.subplots(2, 2, figsize=(10, 8))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    pca_concatenated, ['PC {}'.format(i + 1) for i in range(pca.dimension())], ax=axes[0, 0])\n",
    "for ax, (i, j) in zip(axes.flat[1:], [[0, 1], [1, 2], [0, 2]]):\n",
    "    pyemma.plots.plot_density(\n",
    "        *pca_concatenated[:, [i, j]].T, ax=ax, cbar=False, alpha=0.1)\n",
    "    ax.scatter(*cluster.clustercenters[:, [i, j]].T, s=15, c='C1')\n",
    "    ax.set_xlabel('PC {}'.format(i + 1))\n",
    "    ax.set_ylabel('PC {}'.format(j + 1))\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden"
   },
   "source": [
    "###### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "pca = pyemma.coordinates.pca(data_concatenated, dim=3)\n",
    "pca_concatenated = np.concatenate(pca.get_output(stride=5))\n",
    "\n",
    "cluster = pyemma.coordinates.cluster_kmeans(pca, k=100, max_iter=50, stride=5)\n",
    "\n",
    "fig, axes = plt.subplots(2, 2, figsize=(10, 8))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    pca_concatenated, ['PC {}'.format(i + 1) for i in range(pca.dimension())], ax=axes[0, 0])\n",
    "for ax, (i, j) in zip(axes.flat[1:], [[0, 1], [1, 2], [0, 2]]):\n",
    "    pyemma.plots.plot_density(*pca_concatenated[:, [i, j]].T, ax=ax, cbar=False, alpha=0.1)\n",
    "    ax.scatter(*cluster.clustercenters[:, [i, j]].T, s=15, c='C1')\n",
    "    ax.set_xlabel('PC {}'.format(i + 1))\n",
    "    ax.set_ylabel('PC {}'.format(j + 1))\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 6: TICA\n",
    "\n",
    "Perform TICA with target dimension $3$ and lag time $20$ steps on the current feature set and discretize the projected space using $k$-means with $100$ centers and a stride of $5$ to reduce the computational effort."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "tica =  # FIXME\n",
    "tica_concatenated =  # FIXME\n",
    "\n",
    "cluster =  # FIXME\n",
    "\n",
    "fig, axes = plt.subplots(2, 2, figsize=(10, 8))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    tica_concatenated, ['IC {}'.format(i + 1) for i in range(tica.dimension())], ax=axes[0, 0])\n",
    "for ax, (i, j) in zip(axes.flat[1:], [[0, 1], [1, 2], [0, 2]]):\n",
    "    pyemma.plots.plot_density(\n",
    "        *tica_concatenated[:, [i, j]].T, ax=ax, cbar=False, alpha=0.1)\n",
    "    ax.scatter(*cluster.clustercenters[:, [i, j]].T, s=15, c='C1')\n",
    "    ax.set_xlabel('IC {}'.format(i + 1))\n",
    "    ax.set_ylabel('IC {}'.format(j + 1))\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden"
   },
   "source": [
    "###### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "tica = pyemma.coordinates.tica(data_concatenated, dim=3)\n",
    "tica_concatenated = np.concatenate(tica.get_output(stride=5))\n",
    "\n",
    "cluster = pyemma.coordinates.cluster_kmeans(tica, k=100, max_iter=50, stride=5)\n",
    "\n",
    "fig, axes = plt.subplots(2, 2, figsize=(10, 8))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    tica_concatenated, ['IC {}'.format(i + 1) for i in range(tica.dimension())], ax=axes[0, 0])\n",
    "for ax, (i, j) in zip(axes.flat[1:], [[0, 1], [1, 2], [0, 2]]):\n",
    "    pyemma.plots.plot_density(\n",
    "        *tica_concatenated[:, [i, j]].T, ax=ax, cbar=False, alpha=0.1)\n",
    "    ax.scatter(*cluster.clustercenters[:, [i, j]].T, s=15, c='C1')\n",
    "    ax.set_xlabel('IC {}'.format(i + 1))\n",
    "    ax.set_ylabel('IC {}'.format(j + 1))\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 7: VAMP\n",
    "\n",
    "Perform VAMP with target dimension $3$ and lag time $20$ steps on the current feature set and discretize the projected space using $k$-means with $100$ centers and a stride of $5$ to reduce the computational effort."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "vamp =  # FIXME\n",
    "vamp_concatenated =  # FIXME\n",
    "\n",
    "cluster =  # FIXME\n",
    "\n",
    "fig, axes = plt.subplots(2, 2, figsize=(10, 8))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    tica_concatenated, ['IC {}'.format(i + 1) for i in range(vamp.dimension())], ax=axes[0, 0])\n",
    "for ax, (i, j) in zip(axes.flat[1:], [[0, 1], [1, 2], [0, 2]]):\n",
    "    pyemma.plots.plot_density(\n",
    "        *vamp_concatenated[:, [i, j]].T, ax=ax, cbar=False, alpha=0.1)\n",
    "    ax.scatter(*cluster.clustercenters[:, [i, j]].T, s=15, c='C1')\n",
    "    ax.set_xlabel('IC {}'.format(i + 1))\n",
    "    ax.set_ylabel('IC {}'.format(j + 1))\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden"
   },
   "source": [
    "###### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "vamp = pyemma.coordinates.vamp(data_concatenated, lag=20, dim=3)\n",
    "vamp_concatenated = np.concatenate(vamp.get_output(stride=5))\n",
    "\n",
    "cluster = pyemma.coordinates.cluster_kmeans(vamp, k=100, max_iter=50, stride=5)\n",
    "\n",
    "fig, axes = plt.subplots(2, 2, figsize=(10, 8))\n",
    "pyemma.plots.plot_feature_histograms(\n",
    "    tica_concatenated, ['IC {}'.format(i + 1) for i in range(tica.dimension())], ax=axes[0, 0])\n",
    "for ax, (i, j) in zip(axes.flat[1:], [[0, 1], [1, 2], [0, 2]]):\n",
    "    pyemma.plots.plot_density(*vamp_concatenated[:, [i, j]].T, ax=ax, cbar=False, alpha=0.1)\n",
    "    ax.scatter(*cluster.clustercenters[:, [i, j]].T, s=15, c='C1')\n",
    "    ax.set_xlabel('IC {}'.format(i + 1))\n",
    "    ax.set_ylabel('IC {}'.format(j + 1))\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping up\n",
    "\n",
    "In this notebook, we have learned how to reduce the dimension of molecular simulation data and discretize the projected dynamics with PyEMMA.\n",
    "In detail, we have used\n",
    "- `pyemma.coordinates.pca()` to perform a principal components analysis,\n",
    "- `pyemma.coordinates.tica()` to perform a time-lagged independent component analysis, and\n",
    "- `pyemma.coordinates.vamp()` to analyze the quality of some feature spaces, perform dimension reduction, and\n",
    "- `pyemma.coordinates.cluster_kmeans()` to perform a $k$-means clustering, and\n",
    "- `pyemma.coordinates.cluster_regspace()` to perform a regspace clustering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "<a id=\"cite-tica2\"/><sup><a href=#ref-1>[^]</a></sup>Molgedey, L. and Schuster, H. G.. 1994. _Separation of a mixture of independent signals using time delayed correlations_. [URL](http://dx.doi.org/10.1103/PhysRevLett.72.3634)\n",
    "\n",
    "<a id=\"cite-tica\"/><sup><a href=#ref-2>[^]</a></sup>Guillermo Pérez-Hernández and Fabian Paul and Toni Giorgino and Gianni De Fabritiis and Frank Noé. 2013. _Identification of slow molecular order parameters for Markov model construction_. [URL](https://doi.org/10.1063/1.4811489)\n",
    "\n",
    "<a id=\"cite-vamp-preprint\"/><sup><a href=#ref-3>[^]</a></sup>Wu, H. and Noé, F.. 2017. _Variational approach for learning Markov processes from time series data_. [URL](https://arxiv.org/pdf/1707.04659.pdf)\n",
    "\n",
    "<a id=\"cite-aggarwal_surprising_2001\"/><sup><a href=#ref-4>[^]</a></sup>Aggarwal, Charu C. and Hinneburg, Alexander and Keim, Daniel A.. 2001. _On the Surprising Behavior of Distance Metrics in High Dimensional Space_.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}