-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve oxford_iiit_pet dataset #5349
base: master
Are you sure you want to change the base?
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,7 @@ | ||
The Oxford-IIIT pet dataset is a 37 category pet image dataset with roughly 200 | ||
images for each class. The images have large variations in scale, pose and | ||
lighting. All images have an associated ground truth annotation of breed. | ||
|
||
Please take note of the following: | ||
- Valid bounding boxes of the pets' heads are only provided for the training split. The test split does not contain annotations. | ||
- A few image files in the original dataset are corrupt or in a wrong format. These are being skipped. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
<annotation><folder>OXIIIT</folder><filename>Abyssinian_1.jpg</filename><source><database>OXFORD-IIIT Pet Dataset</database><annotation>OXIIIT</annotation><image>flickr</image></source><size><width>600</width><height>400</height><depth>3</depth></size><segmented>0</segmented><object><name>cat</name><pose>Frontal</pose><truncated>0</truncated><occluded>0</occluded><bndbox><xmin>333</xmin><ymin>72</ymin><xmax>425</xmax><ymax>158</ymax></bndbox><difficult>0</difficult></object></annotation> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
<annotation><folder>OXIIIT</folder><filename>Abyssinian_10.jpg</filename><source><database>OXFORD-IIIT Pet Dataset</database><annotation>OXIIIT</annotation><image>flickr</image></source><size><width>375</width><height>500</height><depth>3</depth></size><segmented>0</segmented><object><name>cat</name><pose>Frontal</pose><truncated>0</truncated><occluded>0</occluded><bndbox><xmin>72</xmin><ymin>105</ymin><xmax>288</xmax><ymax>291</ymax></bndbox><difficult>0</difficult></object></annotation> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
<annotation><folder>OXIIIT</folder><filename>Abyssinian_11.jpg</filename><source><database>OXFORD-IIIT Pet Dataset</database><annotation>OXIIIT</annotation><image>flickr</image></source><size><width>192</width><height>288</height><depth>3</depth></size><segmented>0</segmented><object><name>cat</name><pose>Frontal</pose><truncated>0</truncated><occluded>0</occluded><bndbox><xmin>43</xmin><ymin>57</ymin><xmax>94</xmax><ymax>105</ymax></bndbox><difficult>0</difficult></object></annotation> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
<annotation><folder>OXIIIT</folder><filename>Abyssinian_18.jpg</filename><source><database>OXFORD-IIIT Pet Dataset</database><annotation>OXIIIT</annotation><image>flickr</image></source><size><width>335</width><height>500</height><depth>3</depth></size><segmented>0</segmented><object><name>cat</name><pose>Frontal</pose><truncated>0</truncated><occluded>0</occluded><bndbox><xmin>135</xmin><ymin>87</ymin><xmax>314</xmax><ymax>285</ymax></bndbox><difficult>0</difficult></object></annotation> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
<annotation><folder>OXIIIT</folder><filename>Abyssinian_17.jpg</filename><source><database>OXFORD-IIIT Pet Dataset</database><annotation>OXIIIT</annotation><image>flickr</image></source><size><width>275</width><height>411</height><depth>3</depth></size><segmented>0</segmented><object><name>cat</name><pose>Frontal</pose><truncated>0</truncated><occluded>0</occluded><bndbox><xmin>38</xmin><ymin>92</ymin><xmax>136</xmax><ymax>192</ymax></bndbox><difficult>0</difficult></object></annotation> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
<annotation><folder>OXIIIT</folder><filename>Abyssinian_16.jpg</filename><source><database>OXFORD-IIIT Pet Dataset</database><annotation>OXIIIT</annotation><image>flickr</image></source><size><width>500</width><height>335</height><depth>3</depth></size><segmented>0</segmented><object><name>cat</name><pose>Frontal</pose><truncated>0</truncated><occluded>0</occluded><bndbox><xmin>375</xmin><ymin>53</ymin><xmax>458</xmax><ymax>134</ymax></bndbox><difficult>0</difficult></object></annotation> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
<annotation><folder>OXIIIT</folder><filename>Abyssinian_14.jpg</filename><source><database>OXFORD-IIIT Pet Dataset</database><annotation>OXIIIT</annotation><image>flickr</image></source><size><width>335</width><height>500</height><depth>3</depth></size><segmented>0</segmented><object><name>cat</name><pose>Frontal</pose><truncated>0</truncated><occluded>0</occluded><bndbox><xmin>164</xmin><ymin>71</ymin><xmax>252</xmax><ymax>163</ymax></bndbox><difficult>0</difficult></object></annotation> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
<annotation><folder>OXIIIT</folder><filename>Abyssinian_15.jpg</filename><source><database>OXFORD-IIIT Pet Dataset</database><annotation>OXIIIT</annotation><image>flickr</image></source><size><width>477</width><height>422</height><depth>3</depth></size><segmented>0</segmented><object><name>cat</name><pose>Frontal</pose><truncated>0</truncated><occluded>0</occluded><bndbox><xmin>16</xmin><ymin>16</ymin><xmax>431</xmax><ymax>357</ymax></bndbox><difficult>0</difficult></object></annotation> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
<annotation><folder>OXIIIT</folder><filename>Abyssinian_13.jpg</filename><source><database>OXFORD-IIIT Pet Dataset</database><annotation>OXIIIT</annotation><image>flickr</image></source><size><width>245</width><height>280</height><depth>3</depth></size><segmented>0</segmented><object><name>cat</name><pose>Frontal</pose><truncated>0</truncated><occluded>0</occluded><bndbox><xmin>145</xmin><ymin>25</ymin><xmax>218</xmax><ymax>96</ymax></bndbox><difficult>0</difficult></object></annotation> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
<annotation><folder>OXIIIT</folder><filename>Abyssinian_12.jpg</filename><source><database>OXFORD-IIIT Pet Dataset</database><annotation>OXIIIT</annotation><image>flickr</image></source><size><width>335</width><height>500</height><depth>3</depth></size><segmented>0</segmented><object><name>cat</name><pose>Frontal</pose><truncated>0</truncated><occluded>0</occluded><bndbox><xmin>94</xmin><ymin>83</ymin><xmax>211</xmax><ymax>190</ymax></bndbox><difficult>0</difficult></object></annotation> |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -16,11 +16,12 @@ | |
"""Oxford-IIIT pet dataset.""" | ||
|
||
import os | ||
import xml.etree.ElementTree | ||
|
||
from tensorflow_datasets.core.utils.lazy_imports_utils import tensorflow as tf | ||
import tensorflow_datasets.public_api as tfds | ||
|
||
_BASE_URL = "http://www.robots.ox.ac.uk/~vgg/data/pets/data" | ||
_BASE_URL = "https://thor.robots.ox.ac.uk/~vgg/data/pets" | ||
|
||
_LABEL_CLASSES = [ | ||
"Abyssinian", | ||
|
@@ -63,6 +64,44 @@ | |
] | ||
_SPECIES_CLASSES = ["Cat", "Dog"] | ||
|
||
# List of samples with corrupt image files | ||
_SKIP_SAMPLES = [ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you know what the problem with these images is? Could it be that they're encoded in a different format? I suggest we check them in code and try our best to recover, similar to: |
||
"beagle_116", | ||
"chihuahua_121", | ||
"Abyssinian_5", | ||
"Abyssinian_34", | ||
"Egyptian_Mau_14", | ||
"Egyptian_Mau_139", | ||
"Egyptian_Mau_145", | ||
"Egyptian_Mau_156", | ||
"Egyptian_Mau_167", | ||
"Egyptian_Mau_177", | ||
"Egyptian_Mau_186", | ||
"Egyptian_Mau_191" | ||
] | ||
|
||
def _get_head_bbox(annon_filepath): | ||
"""Read head bbox from annotation XML file.""" | ||
with tf.io.gfile.GFile(annon_filepath, "r") as f: | ||
root = xml.etree.ElementTree.parse(f).getroot() | ||
|
||
# Disable pytype to avoid attribute-error due to find returning | ||
# Optional[Element] | ||
# pytype: disable=attribute-error | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This disables pytype for the rest of the file, which is undesirable. How about something like |
||
size = root.find("size") | ||
width = float(size.find("width").text) | ||
height = float(size.find("height").text) | ||
|
||
obj = root.find("object") | ||
bndbox = obj.find("bndbox") | ||
xmax = float(bndbox.find("xmax").text) | ||
xmin = float(bndbox.find("xmin").text) | ||
ymax = float(bndbox.find("ymax").text) | ||
ymin = float(bndbox.find("ymin").text) | ||
return tfds.features.BBox( | ||
ymin / height, xmin / width, ymax / height, xmax / width | ||
) | ||
|
||
|
||
class Builder(tfds.core.GeneratorBasedBuilder): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please bump VERSION and add RELEASE_NOTES. |
||
"""Oxford-IIIT pet dataset.""" | ||
|
@@ -79,6 +118,7 @@ def _info(self): | |
"segmentation_mask": tfds.features.Image( | ||
shape=(None, None, 1), use_colormap=True | ||
), | ||
"head": tfds.features.BBoxFeature() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How about a more explicit feature name, e.g. "head_bbox"? |
||
}), | ||
supervised_keys=("image", "label"), | ||
homepage="http://www.robots.ox.ac.uk/~vgg/data/pets/", | ||
|
@@ -126,18 +166,31 @@ def _generate_examples( | |
for line in images_list: | ||
image_name, label, species, _ = line.strip().split(" ") | ||
|
||
# skip corrupt samples | ||
if image_name in _SKIP_SAMPLES: | ||
continue | ||
|
||
trimaps_dir_path = os.path.join(annotations_dir_path, "trimaps") | ||
xmls_dir_path = os.path.join(annotations_dir_path, "xmls") | ||
|
||
trimap_name = image_name + ".png" | ||
xml_name = image_name + ".xml" | ||
image_name += ".jpg" | ||
label = int(label) - 1 | ||
species = int(species) - 1 | ||
|
||
try: | ||
head_bbox = _get_head_bbox(os.path.join(xmls_dir_path, xml_name)) | ||
except tf.errors.NotFoundError: | ||
# test samples do not have an annotation file | ||
head_bbox = tfds.features.BBox(0., 0., 0., 0.) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Move this to a constant |
||
|
||
record = { | ||
"image": os.path.join(images_dir_path, image_name), | ||
"label": int(label), | ||
"species": species, | ||
"file_name": image_name, | ||
"segmentation_mask": os.path.join(trimaps_dir_path, trimap_name), | ||
"head": head_bbox | ||
} | ||
yield image_name, record |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think tensorflow is only used to open files, could you please replace this with
from etils import epath
?