Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

provide better defaults for mimetype to extension mapping #81

Open
wants to merge 31 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
86114bd
initial code from pr !15
mx1up Jan 5, 2023
9f1ef55
update copyright year
mx1up Jan 5, 2023
c9e3e15
improve lookup: case-insensitive and use firstWhereOrNull instead of …
mx1up Jan 6, 2023
0f37a4b
tests cleanup
mx1up Jan 6, 2023
f4c4a4f
check baseline
mx1up Jan 6, 2023
a9ec842
test to check whether each mimetype with multiple extensions has a pr…
mx1up Jan 6, 2023
02b9dcb
reformat
mx1up Jan 6, 2023
1ad29e9
apparently, the function already existed :roll-eyes: but the preferre…
mx1up Jan 6, 2023
84a8f43
probably better not to enforce a preferred extension for mime types w…
mx1up Jan 6, 2023
1f2ab54
added some preferred extensions for common file formats
mx1up Jan 6, 2023
fbf8f68
cleanup old tests
mx1up Jan 6, 2023
6600a3f
test orElse param for unknown mime type
mx1up Jan 16, 2023
6fe7ecb
link to file
mx1up Jan 16, 2023
d57b9fc
fix title to be more accurate
mx1up Jan 16, 2023
0d6c519
rename to existing function
mx1up Jan 16, 2023
bb9fc15
add standard behavior to make example more clear
mx1up Jan 16, 2023
7280615
include text/plain mapping again
mx1up Jan 16, 2023
40107ff
remove collection dependency
mx1up Jan 17, 2023
c384ef3
lookupExtension has been renamed to existing extensionFromMime
mx1up Jan 27, 2023
9ca2e5b
Merge branch 'dart-lang:master' into issue13_contenttype_to_ext
mx1up Dec 13, 2023
76f80f6
drop non-nullable extensionFromMime variant per review suggestion
mx1up Dec 13, 2023
7b2804f
doc update: try using noun phrase; don't refer to implementation details
mx1up Dec 13, 2023
65c7016
don't use extension method
mx1up Dec 13, 2023
e8a6ed5
remove addMimeType and leave it for another time..
mx1up Dec 13, 2023
f8642bd
remove addMimeType documentation
mx1up Dec 13, 2023
8acfbe4
be more specific about extension
mx1up Feb 19, 2024
f20c40f
review feedback: avoid abbreviations, just overwrite function param
mx1up Feb 19, 2024
2be331f
since extensions can no longer by added at runtime, we can optimize a…
mx1up Feb 19, 2024
60e9a7b
local var no longer needed
mx1up Feb 19, 2024
54da123
update docs
mx1up Feb 19, 2024
d190914
update docs
mx1up Feb 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions CONTRIBUTING.md
Expand Up @@ -31,3 +31,7 @@ All files in the project must start with the following header.
Contributions made by corporations are covered by a different agreement than the
one above, the
[Software Grant and Corporate Contributor License Agreement](https://developers.google.com/open-source/cla/corporate).

### Adding an extension / MIME type mapping
If a MIME type ends up with multiple extensions, it is recommended to define a
preferred extension in `_preferredExtensionsMap` in [extension.dart](lib/src/extension.dart).
20 changes: 20 additions & 0 deletions README.md
Expand Up @@ -57,3 +57,23 @@ request
.map((part) => part.fold(0, (p, d) => p + d))
.listen((length) => print('Part with length $length'));
```

## Determining file extension by MIME type

The top level function `extensionFromMime` can be used to determine the
file extension of a given MIME type.

```dart
print(extensionFromMime('text/html')); // Will print html
print(extensionFromMime('image/jpeg')); // Will print jpg
print(extensionFromMime('application/pdf')); // Will print pdf
```

You can override the default MIME type-extension mapping using
`addMimeType`:

```dart
print(extensionFromMime('image/jpeg')); // Will print jpg
addMimeType('image/jpeg', 'jpeg');
print(extensionFromMime('image/jpeg')); // Will print jpeg
```
1 change: 1 addition & 0 deletions lib/mime.dart
Expand Up @@ -9,6 +9,7 @@
/// [Internet media type](http://en.wikipedia.org/wiki/Internet_media_type).
library mime;

export 'src/extension.dart';
export 'src/mime_multipart_transformer.dart';
export 'src/mime_shared.dart';
export 'src/mime_type.dart';
58 changes: 58 additions & 0 deletions lib/src/extension.dart
@@ -0,0 +1,58 @@
// Copyright (c) 2023, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

import 'default_extension_map.dart';

/// Add an override for common extensions since different extensions may map
/// to the same MIME type.
final Map<String, String> _preferredExtensionsMap = <String, String>{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not possible for code to add to this map, right?
If so, it proably should be const.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed, not anymore ;)

but applying the optimization made it no longer possible

'application/vnd.ms-excel': 'xls',
'application/vnd.ms-powerpoint': 'ppt',
'image/jpeg': 'jpg',
'image/tiff': 'tif',
'image/svg+xml': 'svg',
'text/calendar': 'ics',
'text/javascript': 'js',
'text/plain': 'txt',
'text/sgml': 'sgml',
'text/x-pascal': 'pas',
'video/mp4': 'mp4',
'video/mpeg': 'mpg',
'video/quicktime': 'mov',
'video/x-matroska': 'mkv',
};

/// Lookup file extension for a given MIME type.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name is a noun phrase, so treat this as a parameterized getter. That includes documenting it using a noun phrase too.
(At least the first sentence, after that it's OK to talk about returning a value.)

/// File extension for a given MIME type, if available.
///
/// Returns `null` if [mimeType] has no associated extension.
/// If [mimeType] has multiple associated extensions,
/// the preferred extension is returned.

Don't mention the map, it's not part of the public API, so a reader won't know what "the map" is.

Copy link
Author

@mx1up mx1up Dec 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done (had to look up what a 'noun phrase' is, so not sure if I did right ;) )

///
/// If there are multiple extensions for [mimeType], return preferred extension
/// if defined, or the first occurrence in the map.
/// If no extension is found, `null` is returned.
String? extensionFromMimeOrNull(String mimeType) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't have the "OrNull" on the name here. That's just repeating the type.
It distinguishes it from the function below, but see comment there.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

final mimeTypeLC = mimeType.toLowerCase();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Avoid abbreviations, so LC could be lowerCase, but I'd probably just do:

 mimeType = mimeType.toLowerCase();

and not rename.)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought it would be bad practice to overwrite incoming params?
anyway, it turned out the function became so trivial I didn't need the var anymore

if (_preferredExtensionsMap.containsKey(mimeTypeLC)) {
return _preferredExtensionsMap[mimeTypeLC]!;
}
return defaultExtensionMap.entries
Copy link
Member

@lrhn lrhn May 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd just inline that extension method here:

  for (var entry in defaultExtensionsMap.entries) {
     if (entry.value == mimeTypeLC) return entry.key;  
  }
  return null;

It's far too much (unnecessary) syntactic overhead to add your own firstWhereOrNull extension (takes up nine lines further down) to save, basically, one line of code here. And I personally don't find it easier to read either.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

.firstWhereOrNull((entry) => entry.value == mimeTypeLC)
?.key;
}

String extensionFromMime(String mimeType, {String? orElse}) =>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs documentation.
But I'd just drop this method. The user can easily do .extensionFromMime(myMime) ?? orElse() themselves.
Or .extensionFromMime(myMime)! if they know it's there.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, I now remember I did this to preserve backward compatibility..

extensionFromMimeOrNull(mimeType) ?? orElse ?? mimeType;

/// Allow for a user-specified MIME type-extension mapping that overrides the
/// default.
void addMimeType(String mimeType, String extension) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be more comfortable with this PR if it didn't include this API.

Why do we need this one?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @natebosch , thanks for having a look

this library does not know all possible file formats and I think it's neither realistic nor a goal of this lib to achieve that. The library also seems to be seeded with quite a random list of formats. So it can be quite a frequent situation in which a user of this library needs a format added. A pull request could be made, but the release cycle is not that fast and it's also a bit of overhead for maintainers.

That's why I think it may be useful to have this method so users don't immediately get stuck.

Similarly, this library already provides addExtension.

The 2nd usecase is one in which the user of the lib has a different preference than the one that is included. For example, if one prefers 'text/plain' to map to 'conf' as it currently does, one can still configure that old behavior.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the slow response.

I'm worried about having an addMimeType that only impacts extensionFromMime and doesn't impact lookupMimeType.

We could either make this signature addMimeType(String mimeType, String extension, {List<int>? headerBytes}) and try to make this work with lookupMimeType, or we could consider an alternative name like `addExtensionPreference.

@devoncarew @lrhn WDYT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally, I worry about global mutable state.
This is a kind of registry, where any code can add its own preferences, but they then also affect everybody else.

I'd prefer a structure where you have a Mime object instead of global state and functions. Then you can get your own object, modify it, and pass it around to your own code, without affecting other people's Mime objects.
That's a larger change, though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we'll be able to do any refactoring of existing usage.

We could add a MimeRegistry class which can be overridden like this. The top-level APIs would continue to use the global map, the MimeRegistry would fall back on the global map, but it could also store local overrides. @lrhn WDYT about something like this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lrhn - Should I work on a PR with an implementation of a MimeRegistry, or do you think we can get by with the top level approach?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can definitely "get by" with the top-level approach, but it's inherently fragile. I prefer to avoid adding fragile features.

What I'd do, if I had all the time in the world, is to have an immutable registry, and a mutator "builder" type.
The top-level functions should then just forward to the corresponding functions on MimeRegistry.defaultRegistry.

abstract interface class MimeRegistry {
  static get MimeRegistry currentRegistry => ... (probably something that can be zone overridden) ...;

  factory MimeRegistry.empty() = ...;

  int get defaultMagicNumbersMaxLength;
  String lookup(String path, {List<int>? headerBytes});
  String extensionFor(String mimeType);

  MimeRegistryBuilder get build;
  MimeRegistry rebuild(void Function(MimeRegistryBuilder) mutator);
}

abstract interface class MimeRegistryBuilder {
  // I don't know what you'd want, but something like:

  /// Adds mime type, if it doesn't already exist. 
  /// Adds [extension] to the extensions for [mimeType].
  /// Set the header-recognizer function to [headerRecognizer].
  void addMimeType(String mimeType, {String? extension, bool Function(List<int>) headerRecognizer});
  void removeMimeType(String mimeType);
  void addExtension(String mimeType, String additionalExtension);
  void removeExtension(String mimeType, String extension);
  // Maybe some inspection functions, but it's a little weird if those are not available on the registry itself.
  Iterable<String> get allMimeTypes;
  Iterable<String> allExtensionsFor(String mimeType);
  bool Function(List<int>)? headerRecognizerFor(String mimeType);
}

The builder is building on top of an existing registry, and it can either defer to it, lazily copy from it, or eagerly clone.
I'd make the actual implementations hidden.

But I might just be over-engineering. Always a risk 😉

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I'd do, if I had all the time in the world

What do you think we should do given that we have less time than that?

Copy link
Member

@lrhn lrhn May 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same thing! The worry about lack of time for something else afterwards 😁

But the concrete problem we're trying to solve here seems to be solved without the addMimeType, so I'd remove that, not add configurability in this PR, and just give access to the default file-type extensions.

Then, if we still need configurability, we can come back and look at it fresh, and not stall this PR.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed addMimeType

_preferredExtensionsMap[mimeType.toLowerCase()] = extension.toLowerCase();
}

extension _IterableExtension<T> on Iterable<T> {
/// The first element satisfying [test], or `null` if there are none.
T? firstWhereOrNull(bool Function(T element) test) {
for (var element in this) {
if (test(element)) return element;
}
return null;
}
}
14 changes: 0 additions & 14 deletions lib/src/mime_type.dart
Expand Up @@ -23,20 +23,6 @@ int get defaultMagicNumbersMaxLength => _globalResolver.magicNumbersMaxLength;
String? lookupMimeType(String path, {List<int>? headerBytes}) =>
_globalResolver.lookup(path, headerBytes: headerBytes);

/// Returns the extension for the given MIME type.
///
/// If there are multiple extensions for [mime], return the first occurrence in
/// the map. If there are no extensions for [mime], return [mime].
String extensionFromMime(String mime) {
mime = mime.toLowerCase();
for (final entry in defaultExtensionMap.entries) {
if (defaultExtensionMap[entry.key] == mime) {
return entry.key;
}
}
return mime;
}

/// MIME-type resolver class, used to customize the lookup of mime-types.
class MimeTypeResolver {
final Map<String, String> _extensionMap = {};
Expand Down
63 changes: 63 additions & 0 deletions test/extension_test.dart
@@ -0,0 +1,63 @@
// Copyright (c) 2023, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

import 'package:mime/mime.dart';
import 'package:test/test.dart';

void main() {
group('global-lookup-mime-type', () {
test('valid-mime-type', () {
expect(extensionFromMime('text/x-dart'), equals('dart'));
expect(extensionFromMime('text/javascript'), equals('js'));
expect(extensionFromMime('application/java-archive'), equals('jar'));
expect(extensionFromMime('application/json'), equals('json'));
expect(extensionFromMime('application/pdf'), equals('pdf'));
expect(extensionFromMime('application/vnd.ms-excel'), equals('xls'));
expect(extensionFromMime('application/xhtml+xml'), equals('xht'));
expect(extensionFromMime('image/jpeg'), equals('jpg'));
expect(extensionFromMime('image/png'), equals('png'));
expect(extensionFromMime('text/css'), equals('css'));
expect(extensionFromMime('text/html'), equals('htm'));
expect(extensionFromMime('text/plain'), equals('txt'));
expect(extensionFromMime('text/x-c'), equals('c'));
});

test('invalid-mime-type', () {
expect(extensionFromMime('invalid-mime-type'), 'invalid-mime-type');
expect(extensionFromMime('invalid/mime/type'), 'invalid/mime/type');

expect(
extensionFromMime('invalid-mime-type', orElse: 'invalid'), 'invalid');
expect(
extensionFromMime('invalid/mime/type', orElse: 'invalid'), 'invalid');

expect(extensionFromMimeOrNull('invalid-mime-type'), isNull);
expect(extensionFromMimeOrNull('invalid/mime/type'), isNull);
});

test('unknown-mime-type', () {
expect(extensionFromMime('application/to-be-invented'),
'application/to-be-invented');

expect(extensionFromMime('application/to-be-invented', orElse: 'unknown'),
'unknown');

expect(extensionFromMimeOrNull('application/to-be-invented'), isNull);
});
});

group('add-mime-type', () {
test('new-mime-type', () {
expect(extensionFromMimeOrNull('custom/type'), isNull);
addMimeType('custom/type', 'ct');
expect(extensionFromMime('custom/type'), equals('ct'));
});

test('overridden-mime-type', () {
expect(extensionFromMime('image/jpeg'), equals('jpg'));
addMimeType('image/jpeg', 'jpeg');
expect(extensionFromMime('image/jpeg'), equals('jpeg'));
});
});
}
17 changes: 0 additions & 17 deletions test/mime_type_test.dart
Expand Up @@ -184,21 +184,4 @@ void main() {
expect(initialMagicNumbersMaxLength, actualMaxBytes);
});

group('extensionFromMime', () {
test('returns match for mime with single extension', () {
expect(extensionFromMime('application/json'), equals('json'));
expect(extensionFromMime('application/java-archive'), equals('jar'));
});

test('returns first match for mime with multiple extensions', () {
expect(extensionFromMime('text/html'), equals('htm'));
expect(extensionFromMime('application/x-cbr'), equals('cb7'));
});

test('returns inputted string for unrecognized mime', () {
expect(
extensionFromMime('unrecognized_mime'), equals('unrecognized_mime'));
expect(extensionFromMime('i/am/not/a/mime'), equals('i/am/not/a/mime'));
});
});
}