Imaging_generate_metadata.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# iRODS for TU Delft TNW\n",
    "This notebook was used to prepare experimental imaging data, collected by Hozanna Miro in March 2018, for Gary Steele and Aurèle Adam.\n",
    "This notebook was run from a Windows 7 laptop using Anaconda to install Python 3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up\n",
    "The code works through the `source` directory, looking for image files. When it finds one, it looks for a `.txt` file of the same name. The `.txt` file contains the metadata.\n",
    "\n",
    "iRODS metadata is prepared as a text file `metadata` for running as an iCommands script. `dest_iRODS` is the name of the iRODS collection where the files with metadata are stored. It is used solely to create the iCommand. The files can be copied from `source` to `dest_iRODS` using Cyberduck, for example.\n",
    "\n",
    "To save time while debugging, the creation of summary files and metadata can be turned on and off."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "source = \"Y:/Hozanna\" #Source of data files - in this case a USB-drive\n",
    "dest_iRODS = \"/tempZone/home/susanb\" #iRODS destination - only used to generate iCommands\n",
    "metadata = \"D:/metadata.txt\" #text file to write iCommands to\n",
    "\n",
    "list_types_on_disk = True\n",
    "make_metadata = True\n",
    "\n",
    "line_end = \" \\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Metadata\n",
    "This section parses the metadata written to text files automatically by the imaging set-up.\n",
    "\n",
    "The text files look something like this:\n",
    "\n",
    "`[SemImageFile]`\n",
    "\n",
    "`InstructName=S-1234`\n",
    "\n",
    "`DataNumber=y-1x0`\n",
    "\n",
    "`SampleName=`\n",
    "\n",
    "`ImageName=images.jpg`\n",
    "\n",
    "`Directory=D:\\TU Delft\\2016-05-31_wafer3\\`\n",
    "\n",
    "`Date=05/31/2016`\n",
    "\n",
    "`Time=11:11:11`\n",
    "\n",
    "`DataSize=2560x1920`\n",
    "\n",
    "`PixelSize=1.1234567`\n",
    "\n",
    "`AcceleratingVoltage=1234 Volt`\n",
    "\n",
    "`Magnification=12345`\n",
    "\n",
    "`LensMode=High`\n",
    "\n",
    "`ScreenMode=Full Screen`\n",
    "\n",
    "`Condition=Vacc=25kV   Mag=x12.0k   WD=2.1mm`\n",
    "\n",
    "`DataDisplayCombine=1`\n",
    "\n",
    "This is converted to imeta iCommands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def write_metadata(text_file, data_object):\n",
    "\n",
    "    lines = [line.rstrip('\\n') for line in open(text_file, \"r\")]\n",
    "\n",
    "    for line in lines:\n",
    "        if (line.count(\"=\") == 0):\n",
    "            pass\n",
    "        elif (line.count(\"Condition\") == 1):\n",
    "            # Assume line has the form: Condition=Vacc=??kV   Mag=x250   WD=???mm\n",
    "            fields = line.split(\" \")\n",
    "            for field in fields:\n",
    "                if (field.find(\"Vacc\") >= 0):\n",
    "                    attname = \"Vacc\"\n",
    "                    attvalue = field.split(\"Vacc=\")[1][:-2]\n",
    "                    attunit = field[-2:]\n",
    "                    metadata_file.write(\"imeta add -d '\"+data_object+\"' '\"+attname+\"' '\"+attvalue+\"' '\"+attunit+\"'\"+line_end)\n",
    "                elif (field.find(\"Mag\") >= 0):\n",
    "                    attname = \"Mag\"\n",
    "                    attvalue = field.split(\"=\")[1]\n",
    "                    metadata_file.write(\"imeta add -d '\"+data_object+\"' '\"+attname+\"' '\"+attvalue+\"' ''\"+line_end)\n",
    "                elif (field.find(\"WD\") >= 0):\n",
    "                    attname = \"WD\"\n",
    "                    attvalue = field.split(\"WD=\")[1][:-2]\n",
    "                    attunit = field[-2:]\n",
    "                    metadata_file.write(\"imeta add -d '\"+data_object+\"' '\"+attname+\"' '\"+attvalue+\"' '\"+attunit+\"'\"+line_end)\n",
    "                else:\n",
    "                    pass\n",
    "        elif (line.count(\"=\") == 1):\n",
    "            attname = line.split(\"=\")[0]\n",
    "            attvalue = line.split(\"=\")[1]\n",
    "            if (attvalue == \"\"):\n",
    "                pass\n",
    "            elif ((line.count(\" \") == 0) or (attname == \"Directory\") or (attname == \"Media\") or \n",
    "                  (attname == \"ScreenMode\") or (attname == \"ImageName\") or (attname == \"SampleName\")):\n",
    "                # Assume that these are strings that can have spaces, but no units\n",
    "                if (attvalue.find(\"'\") > -1): attvalue = attvalue.replace(\"'\", \"\")\n",
    "#SEB could do better than just removing quotes in attribute value\n",
    "                metadata_file.write(\"imeta add -d '\"+data_object+\"' '\"+attname+\"' '\"+attvalue+\"' ''\"+line_end)\n",
    "            elif (line.count(\" \") == 1):\n",
    "                metadata_file.write(\"imeta add -d '\"+data_object+\"' '\"+attname+\"' '\"+attvalue.split(\" \")[0]+\"' '\"+\\\n",
    "                                    attvalue.split(\" \")[1]+\"'\"+line_end)\n",
    "            else:\n",
    "                problems_file.write(\"Could not interpret metadata line '\"+line+\"' in file \"+text_file+line_end)            \n",
    "        else:\n",
    "            problems_file.write(\"Could not interpret metadata line '\"+line+\"' in file \"+text_file+line_end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search through files looking for metadata\n",
    "This section searches a (sub-)directory recursively, looking for images of type:\n",
    "-  .jpg or .JPG\n",
    "-  .tif or .TIF\n",
    "-  .bmp or .BMP\n",
    "-  .jpeg or .JPEG\n",
    "\n",
    "When it finds one, it calls the code above to create the metadata script, assuming that the `.txt` file containing the metadata is in the same directory as the image file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exploretree(dir, formats):\n",
    "\n",
    "    names = os.listdir(dir)\n",
    "\n",
    "    for name in names:\n",
    "        pathname = dir+\"/\"+name\n",
    "        rodsname = dest_iRODS+pathname[len(source):]\n",
    "# SEB if path name is too long, Windows cannot handle it. Skip this path. Could do better\n",
    "        if (len(pathname) > 156):\n",
    "            problems_file.write(\"path name too long \"+pathname+line_end)\n",
    "            continue \n",
    "\n",
    "        if os.path.isdir(pathname):\n",
    "            exploretree(pathname, formats)\n",
    "        elif os.path.isfile(pathname):\n",
    "            filename, extension = os.path.splitext(name)\n",
    "            formats.append(extension)\n",
    "            if (extension == \".jpg\") or (extension == \".tif\") or (extension == \".bmp\") or (extension == \".jpeg\") or \\\n",
    "               (extension == \".JPG\") or (extension == \".TIF\") or (extension == \".BMP\") or (extension == \".JPEG\"):\n",
    "# SEB search for files with these extensions only. This should be extended\n",
    "                if make_metadata:\n",
    "# SEB only search for metadata in txt files in the same directory. This should be changed\n",
    "                    if os.path.exists(dir+\"/\"+filename+\".txt\"):\n",
    "                        write_metadata(dir+\"/\"+filename+\".txt\",rodsname)\n",
    "                    else:\n",
    "                        problems_file.write(\"no .txt file available to extract metadata for \"+pathname+line_end)\n",
    "        else:\n",
    "            problems_file.write(\"neither collection nor file\"+line_end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Main section\n",
    "This section opens files, calls the code above and closes files again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if(make_metadata): metadata_file = open(metadata, \"w\")\n",
    "if list_types_on_disk: types_on_disk = open(\"types_on_disk.txt\", \"w\")\n",
    "problems_file = open(\"problems_file.txt\", \"w\")\n",
    "\n",
    "formats = []\n",
    "\n",
    "exploretree(source, formats)\n",
    "d = {x:formats.count(x) for x in formats}\n",
    "if list_types_on_disk: types_on_disk.write(str(d))\n",
    "\n",
    "if(make_metadata): metadata_file.close()\n",
    "if list_types_on_disk: types_on_disk.close()\n",
    "problems_file.close()\n",
    "\n",
    "if list_types_on_disk:\n",
    "    types_on_disk = open(\"types_on_disk.txt\", \"w\")\n",
    "    types_on_disk.write(str(d))\n",
    "    types_on_disk.close()\n",
    "\n",
    "with open(\"problems_file.txt\", 'r') as fin:\n",
    "    print(fin.read())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}