Cookie name with brackets

[seahindeniz]

Is your feature request related to a problem? Please describe.
Because of escaping characters on the key, I can't able to add brackets in key without having escaped characters.

js-cookie/src/api.mjs

Lines 21 to 23 in b1d83a0

	key = encodeURIComponent(key)
	.replace(/%(2[346B]|5E|60|7C)/g, decodeURIComponent)
	.replace(/[()]/g, escape)

Describe the solution you'd like
I'm suggesting to have a additional way to override the key replacers same as with the value.

js-cookie/README.md

Lines 308 to 312 in b1d83a0

	Cookies.withConverter({
	write: function (value, name) {
	return value.toUpperCase()
	}
	})

to have something like this definition

Cookies.withConverter({
  write: function (value, name) {
    return String(value).toUpperCase()
  }
})

and in additional, this definition to have an overloaded return type to keep both definition exist.

Cookies.withConverter({
  write: function(value, key) {
    return {
      key: String(key),
      value: String(value).toUpperCase()
    }
  }
})

Describe alternatives you've considered
Passing additional optional parameter to disable escapers.

Cookies.set('name', 'value', {
  unescapedName: true,
  unescapedValue: true,
  //...
});

Additional context
I know this solution can expect this this answer #590 (comment) . I'm only suggesting to have an optional way to avoid complying to standards.

Such as this

js-cookie/src/rfc6265.mjs

Lines 5 to 10 in b1d83a0

	write: function (value) {
	return encodeURIComponent(value).replace(
	/%(2[346BF]|3[AC-F]|40|5[BDE]|60|7[BCD])/g,
	decodeURIComponent
	)
	}

[carhartl]

Could you explain why you need brackets in the cookie‘s key?

[seahindeniz]

I'll give you a quick example of my real problem..

I'm currently developing a browser extension for a web site.

In this screenshot, Zadanepl_cookie[infobar] cookie has settled by web site and as you can assume, extension tries to override the value inside of it by using this way

cookie.set("Zadanepl_cookie[infobar]", infoBarBase64);

First red arrow is the result

[nreynis]

I think this feature request is kinda related to this one: #560

[carhartl]

@seahindeniz Do you have control over the Zadanepl_cookie[infobar] cookie?

[seahindeniz]

@carhartl could you elaborate more? What do you mean by control over?

If you mean can I delete, update or create a new one? - Yes I can do that.
But consider me as a third party developer and making changes on a web site that would only affect client side and since my extension cooperates with the web site, the web site is also needs that cookie to communicate with their own backend API's without communicating/proxying with a third party service like mine. So if I create any new cookie with a new name(without brackets) will not affect anything at all 🙁

[carhartl]

@seahindeniz

Sorry, I wasn't clear. My question was, would you be in control of the backend writing the cookie in question, and thus change its name to something that is complying with the standard? But you seem to have answered that already.

For the time being you will have to resort to document.cookie = ... it seems.

[FagnerMartinsBrack]

I know this solution can expect this this answer #590 (comment) . I'm only suggesting to have an optional way to avoid complying to standards.

It's always possible to have an option to avoid complying with standards. However, that standard was built based on the way browsers interpret cookies so that if you deviate from it your cookie won't be written or read correctly.

When I built the encoding, I tested every single character in the latest Safari, Chrome, Firefox, Opera, and IE. Many of the ones that didn't comply with the standard couldn't be read or written using document.cookie, which would make your application break for some browsers.

In this case, it makes sense to use the library and discover that you can't use a square brackets in the name than deploy the app in production using document.cookie (or using a "standards deviation config"). It's not that it doesn't work. It's worse, it will work when you deploy and you test but it will break later when Chrome decides for whatever reason to not accept the character in the name anymore because that's not spec-compliant or when you start using the same code for production in other browsers.

Again, it's possible to add an option to override it if you still want to have control over the name encoding. We can add a big BOLD warning in the Readme about avoiding it as much as you can for these reasons.

@carhartl I guess we didn't release 3.0 yet so there might still be time for an API change like this?

[seahindeniz]

Thank you @FagnerMartinsBrack
I know you are absolutely right and when Chrome has naming standards on cookie names, I think it will first broke the product that I'm dealing with. If they decides to change the cookie name, means I can also follow and adjust.

I'm actually Ok complying with the standards but sadly I can't do anything about making them to comply and that's the first problem I couldn't solve 🙂

I currently solved the problem on my end by following the document.cookie suggestion. But IMHO would be good to have a new option for to override, till all browsers strictly follows the naming standards.

[carhartl]

I'm still a bit torn here. There's no value in strictly adhering to a standard when the real world looks different. On the other hand we have to apply some standard for the encoding, be it an official standard or a de-facto one. I wouldn't know though what the latter one would be...

I've also been thinking about what a reasonably simple, possibly backwards-compatible api could look like. I will definitely want to refrain from overloading the return type of the write converter - it will mean extra complexity not just for us, but also for the user. We had just simplified the api for this in v3, in that we removed overloading entirely from the converter api.

Here's my idea: writing, reading and the name are three different things, therefore the solution I have in mind - and I believe, given the real-world, we'll have to add supporting bypassing any encoding -, is adding a third option to the converter api:

Cookies.withConverter({
  write: ...,
  read: ...,
  name: ...
  }
})

This provides the simplicity + flexibility I desire (and even compatibility, making it easy for users to migrate from v2 -> v3).

I guess we didn't release 3.0 yet so there might still be time for an API change like this?

Not really, mostly I didn't find the time, but of course now we still have the opportunity do bake a solution into v3.

[carhartl]

I'm going to give it a try later today, and see how this plays out...

[carhartl]

(Also, I believe a naming converter doesn‘t just apply to writing a cookie, it‘s also involved when getting one.)

[FagnerMartinsBrack]

There's no value in strictly adhering to a standard when the real world looks different

Some browsers won't work correctly with cookies if you deviate from the standard. It's not like the real world is different, only Chrome is more lenient than other browsers with the brackets in the name, and Chrome is not the world (yet).

(Also, I believe a naming converter doesn‘t just apply to writing a cookie, it‘s also involved when getting one.)

Yeah you would require read/write converters for the name too, that's why I was thinking on something like this:

const NameConverter = Cookies.NameConverter;
const ValueConverter = Cookies.ValueConverter;

Cookies.withConverter(NameConverter({
  read: () => {},
  write: () => {}
}));

Cookies.withConverter(ValueConverter({
  read: () => {},
  write: () => {}
}));

This one keeps cohesion of the interface over composing the kind of converter the dev wanna use. This design also don't allow having both read and write for name and value.

or

Cookies.withConverter({
  readName: () => {},
  readValue: () => {},
  writeName: () => {},
  writeValue: () => {}, 
});

Same as what you proposed, only that "write" and "read" are not default to "value". That's an assumption that has to be documented. If we change read/write to readValue/writeValue then it's a self-documented API at the cost of an API change for a major version.

[FagnerMartinsBrack]

Cookies.withValueConverter({
  read: () => {},
  write: () => {}
}).withNameConverter({
  read; () => {},
  write: () => {}
});

/// ----

const usingAlgorithmA = () => {};
const usingAlgorithmB = () => {};

Cookies
  .withValueConverter(usingAlgorithmA)
  .withNameConverter(usingAlgorithmB);

Con: API Breaking Change
Pro: Can use currying on the algorithm function like .withNameConverter(usingAlgorithmWithParams(...)).

or

Cookies({
  processName: {
    read: () => {}
    write: () => {}
  },
  processValue: {
    read: () => {}
    write: () => {}
  }
})

Just throwing some random ideas at this point haha

Con: Law of Demeter

[carhartl]

It's not like the real world is different, only Chrome is more lenient than other browsers with the brackets

I checked in Chrome, Firefox, Safari (all latest) and IE 11 and all of them accepted document.cookie = "foo[bar]=baz". Plus the fact that server-side implementations rely on it as well is sufficiently real-world to me.

[carhartl]

Anyway, I want to absolutely avoid adding the complexity that's originating from adding converters for both key and value.

There was once the idea to make use of the given converters for both name and value, should there come the need for it. Maybe that time is now. I'll investigate whether that's a feasible avenue.

Then, if you have to have "non-standards-mode" like requested here, you could basically implement "noop" converters:

Cookies.withConverter({
  read: value => value,
  write: value => value
})

[FagnerMartinsBrack]

I was misunderstood. I didn't mean to state that all the major browsers today do not accept brackets. I didn't even run that test recently.

It's tempting to dump standards as long as all server-side and client-side implementations we test are working fine. We can even argue that Chromium is going to be everywhere in 5 years' time so why bother supporting different platforms?

However, we don't have proof that all server-side and client-side implementations out there and, most importantly, the ones who are created in the future will accept brackets. They will all base themselves on RFC 6265 as we did here too. Maybe they will be more lenient, maybe not, impossible to know without making assumptions.

The standard is not perfect. For example, some PHP standard functions escape and unescape the plus sign by default while the RFC 6265 allows it to be used on document.cookie. That's clearly a mistake from RFC authors given the PHP "set cookie" functions exist well before the RFC came to life.

Although the standard is not perfect, we don't have proof that how the world looks like today will be the same in the future. The reason the web worked so well and a website from 1995 still renders is because it was built on top of standards, even when those standards didn't make any sense.

There was once the idea to make use of the given converters for both name and value, should there come the need for it. Maybe that time is now. I'll investigate whether that's a feasible avenue.

Every time you escape a character it adds a few more bytes to it. It increases the size of the cookie. The cookie header cannot be compressed by GZip.

According to RFC 6265, the cookie value can have dozens of characters that are not allowed and the cookie name only a few. By coupling the converter to both name and value we are forcing the developer to increase the payload size of an individual cookie unnecessarily, which helps it reach the limit.

Although designing an API for both name and value adds some complexity to js-cookie internals, it doesn't add more complexity or unintended side-effects to the developer using the library.

I believe we should add a converter for each name and value with an example to be completely non-standard, and then warn the developer that by using them you are going against the standard RFC6265 at your own risk.

[nreynis]

I'm also in favour of keeping name and value converters distinct. If you merge them you'll lose the ability to make complex value converters like jsonConverter (object <=> jsonParse/serialize<=> rfc6265). A name converter will always be string to string.

[carhartl]

I‘m not arguing for dropping adherence to a standard. But: we might be adhering to the wrong one.

The DOM specification for document.cookie refers to RFC 2965: https://www.w3.org/TR/DOM-Level-2-HTML/html.html#ID-8747038

We have proof for this in at least a couple of mayor browsers. I’m pretty sure this is not going to change in the next 10 years, if ever, simply because it would “break the web” in myriads of ways. Lesson to be learned here from the HTML 5 spec, which in many ways was modeled after the real world, and not vice-versa.

Furthermore, the status of RFC 6265 is still “Proposed Standard” (emphasis on proposed).

https://www.rfc-editor.org/info/rfc6265

Also quoting MDN:

The cookie value string can use encodeURIComponent() to ensure that the string does not contain any commas, semicolons, or whitespace (which are disallowed in cookie values).

https://developer.mozilla.org/en-US/docs/Web/API/Document/cookie

This is the status quo that developers expect to work with. RFC 6265 isn’t.

We’re sitting in an ivory tower by adhering to a theoretical standard.

[carhartl]

And because of that we’re discussing adding more converters, which is to me the wrong solution to the (self-made) problem..

If we choose to stick to the current encoding implementation (I wouldn‘t) we need to enable developers to work with legacy cookies (let‘s call them that maybe), by way of opting out of default conversion for the entire cookie — because real-world. And not force them to set up ever more converters. In many years of maintaining a cookie library I have never seen the need for encoding a cookie name and value in different ways, which is what the proposed converter setup is providing us.

[FagnerMartinsBrack]

I‘m not arguing for dropping adherence to a standard. But: we might be adhering to the wrong one.

That changes everything. Unfortunately, I don't have answers for that.

Based on a quick search, there are around 393k websites using js-cookie - the search might be skewed due to the use of bundlers. From those, 103k are calling withConverter( - the search might be skewed due to minification.

That could mean 30% of folks using js-cookie are also using converters, but that's just a rough unprecise estimation.

[FagnerMartinsBrack]

The only thing I can do at this point is to summon @abarth (author of the spec), @bagder (author of curl), @montulli (who co-created RFC 2965 and coined the term "cookie") with the question:

Based on the content of this thread, is there a practical reason why we would continue using RFC 6265 as a reference implementation to understand which characters are allowed and which ones are not in a cookie name and cookie value? Which one should we be using to understand how cookie name and cookie value should be encoded?

According to RFC 6265, cookie-name is a token, defined in RFC2616, Section 2.2:

cookie-pair       = cookie-name "=" cookie-value
cookie-name       = token
cookie-value      = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
cookie-octet      = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
                       ; US-ASCII characters excluding CTLs,
                       ; whitespace DQUOTE, comma, semicolon,
                       ; and backslash

RFC 2965 specified the syntax for the header, where "attr" is "token" but doesn't link to any reference (what's "token" there then?):

NAME            =  attr
VALUE           =  value

av-pairs    =     av-pair *(";" av-pair)
av-pair     =     attr ["=" value]              ; optional value
attr        =     token
value       =     token | quoted-string

Different systems (server, clients, and userland applications) are accepting or rejecting the name/value of cookies in different ways and developers are asking for ways to encode/decode a cookie name differently to bypass RFC 6265 guidelines.

We can't encode every cookie-octet by default cause that would increase the size of the cookie. It will also cause server/browser interaction problems (example 1, example 2, Example 3). If we leave as is and don't encode by default, we might be opening another can of worms because that will potentially cause future interactions between browser -> server -> browser to break, for that someone in between could encoding or decoding in unexpected ways and the developer will have a hard time to understand who's correct (we are having a hard time right now, now imagine userland website authors).

From what it seems, RFC 2965 seems to be the most "real world" specification out there which doesn't specify anything and leaves to the developer to research in a given point in time which characters are valid and which ones are not. Are we missing anything here?

[carhartl]

I took another look at RFC 6265.. if I understand it correctly the rules we are adhering to are aimed at server implementations. For user-agents the spec suggests demands to use more liberal rules (to ensure interoperability, the problem we‘re facing here):

User agents MUST implement the more liberal processing rules defined
in Section 5, in order to maximize interoperability with existing
servers that do not conform to the well-behaved profile defined in
Section 4.

Section 5.2 describes the algorithm to be used when reading the set-cookie header.

https://tools.ietf.org/html/rfc6265#section-5.2

On the other hand, I‘m imagining that this applies to browser vendors, not necessarily us.

[abarth]

On the other hand, I‘m imagining that this applies to browser vendors, not necessarily us.

If you're parsing the Set-Cookie header, you should use the algorithm in Section 5.2. Otherwise, you'll often encounter Set-Cookie headers that you cannot parse correctly.

[FagnerMartinsBrack]

@abarth We're using document.cookie to read and write cookies through a new JavaScript API. We're also trying to abstract away the complexities of the encoding using a standard encoder/decoder that only encodes (using percent-encoding) the characters RFC 6265 says it's invalid. Therefore, we're using RFC 6265 as a guideline to what we should encode and what we should not.

The problem is there are userland implementations that are accepting plain characters that are NOT allowed in a cookie-name (or token) as specified per RFC 6265, which are creating issues. According to our tests, brackets in a cookie-name are supported in all major browsers, only not supported in the spec.

Should we be using RFC 6265 as a guideline for document.cookie? If not, is there something else?

[abarth]

That's not really how cookies work. There isn't any such thing as an invalid character from the point of view of cookie consumers. The point of Section 5.2 is to explain how to process every octet sequence.

According to our tests, brackets in a cookie-name are supported in all major browsers, only not supported in the spec.

What do you mean by "not supported in the spec?" Section 5.2 doesn't treat brackets any differently than any other characters. The only character that are treated differently from other characters are %x3B, %x3D, and WSP.

Should we be using RFC 6265 as a guideline for document.cookie?

document.cookie is specified by https://html.spec.whatwg.org/multipage/dom.html#dom-document-cookie , which refers to RFC6265. So, yes. If you want to understand how document.cookie works, you'll end up needing to read RFC6265 in addition to the HTML standard.

[FagnerMartinsBrack]

@abarth

We're talking about the cookie-name, not the cookie-value. The ] and [ characters are allowed in the cookie-value indeed.

I'm looking at the cookie-name syntax for Set-Cookie on section 4.1.1 and assuming that if it's not allowed in Set-Cookie it's also not allowed in document.cookie. After all, how are you going to read something that shouldn't be able to be set in the first place?

This is the cookie-name syntax on RFC 6265:

cookie-pair       = cookie-name "=" cookie-value
cookie-name      = token
 ...
token             = <token, defined in [RFC2616], Section 2.2>

Notice the: token as defined in RFC2616, Section 2.2:

token          = 1*<any CHAR except CTLs or separators>

Note also the "except separators". Separators are specified as:

       separators     = "(" | ")" | "<" | ">" | "@"
                      | "," | ";" | ":" | "\" | <">
                      | "/" | "[" | "]" | "?" | "="
                      | "{" | "}" | SP | HT

The brackets are there, so "any CHAR except CTLs or separators" is the same as "any CHAR except CTLs or square brackets"

[FagnerMartinsBrack]

we could probably manually encode the few characters where it's required, in whichever way it's required.

Yup. The only reason I kept using encode/decode URIComponent functions for the existing standard encoding mechanism is that it helped to golf some bytes.

[carhartl]

I just thought that we could still face interoperability issues given that RFC 6265 doesn't require a particular encoding at all... we can't have a ";" in the cookie name, but the spec doesn't say how it should be encoded (percent-encoding seems to be common though?).

Another possibility for how to support distinct converters for name + value: always return two values (which would at least nicely mirror what we're already passing to the converter):

Cookies.withConverter({
  write: function (value, name) {
    return [value.toUpperCase(), name.toUpperCase()]
  }
})

I just hope we don't need it and accept percent-encoding in the cookie name for ";" and "=" as a given.

[FagnerMartinsBrack]

I just thought that we could still face interoperability issues given that RFC 6265 doesn't require a particular encoding at all

So far we haven't had reports with ; or = problems with percent-encoding, only with other characters. I believe that's enough evidence that percent-encoding is the way to go.

That means, of course, we have to test for cookies that have a percent char in the value and in the name (such as 10% or 100%50) so that we don't interpret the right-hand side of that as a valid percent-encoding value.

[carhartl]

That means, of course, we have to test for cookies that have a percent char in the value and in the name (such as 10% or 100%50) so that we don't interpret the right-hand side of that as a valid percent-encoding value.

With the current implementation we don't have to bother, it's using a regex without any danger of misinterpreting "%". Unless someone would insist on using %3B as a sequence of characters not representing a ";", that is.

[carhartl]

Closed by #608

[carhartl]

I've bumped the v3 beta containing the new implementation.

https://github.com/js-cookie/js-cookie/releases/tag/v3.0.0-beta.4

[carhartl]

Introduced: #623

[carhartl]

Back to the drawing board..

[carhartl]

To remind myself here, reading such a cookie already works, it's all about being able to write it.

[carhartl]

I checked in Chrome, Firefox, Safari (all latest) and IE 11 and all of them accepted document.cookie = "foo[bar]=baz". Plus the fact that server-side implementations rely on it as well is sufficiently real-world to me.

Let me double check this! While Safari doesn't support deviating from RFC 6265 for the cookie value (#623), it seems to allow it for the key. Then, if all browsers we support would support this, we may relax the enocding for the name while writing and be done. There'd be a test to know if one of these browsers would start to change the behavior. I would be ok doing this for the time being as a compromise. We wouldn't be adding new code, might even reduce it slightly.

[carhartl]

It's a mess. While Safari seems to adhere to RFC 6265 more strictly for the value (#623 (comment)), for the name it is deviating from the spec: cookie-name is supposed to be a token, which forbids for instance [ and ] (being separators).

[FagnerMartinsBrack]

Yes, it's a mess. That's why I strongly suggest going with the spec. The mess of today may not be the mess of tomorrow, so compromises may break later.

I believe we can create a more stable library if we follow a common/standard guideline. Creating our own compromises IMHO is a recipe for disaster.

[carhartl]

I agree.. had a weak moment.

[FagnerMartinsBrack]

So here's a summary:

1. This issue was opened.
2. The RFC 6265 spec author advised us to follow section 5 of the spec only
3. We tried to deviate from the server-side encoding and use the Section 5.2 as per the spec author advice and relax the encoding
4. Then we realized that being more permissive breaks cookies in different browsers (Safari doesn't accept weird chars, only basic ASCII with exceptions) js-cookie/js-cookie/issues/623
5. @carhartl decided to revert the encoding implementation to v2 as it was working before without any issues (Special characters #623 (comment))
6. We realized that relaxing restrictions on the cookie-name bring hell to earth and this issue stalled

I suggest to either

a) create a "Converter API" that allows devs to overwrite the standard converter of cookie-name
or
b) leave as it is and whoever wants to deviate from the current encoding can fork the project.

Ping @carhartl

[carhartl]

Likely going to settle with:

const Customized = Cookies.withReadConverter({
  key: (key) => { ... },
  value: (value, key) => { ... }
})
.withWriteConverter({
  key: (key) => { ... }, 
  value: (value, key) => { ... }
})

[FagnerMartinsBrack]

That's a good one @carhartl. Here are my assumptions:

1. If you call withReadConverter(params) the instance originated from that call bypasses the standard read algorithm for either key or value, depending on which property you've declared in the Object Literal params.
2. If you call withWriteConverter(params) the instance originated from that call bypasses the standard write algorithm for either key or value, depending on which property you've declared in the Object Literal params.
3. withConverter() remains there, therefore this is a backwards-compatible change you can add in a v3.1 without the need to wait for a v4.

Are those assumptions correct so far?

I can't see any issues with this approach.

[carhartl]

Are those assumptions correct so far?

Yes to all three @FagnerMartinsBrack

[FagnerMartinsBrack]

It's been a while since the last comment on this issue so I'll reiterate the current state since 2021:

This issue is essentially pending the implementation of the code that @carhartl suggested in this comment. There won't be a need for a major version bump (new functions added).

If you're reading this and willing to open a PR with proper tests implemented for that functionality, we'll review and potentially accept. Otherwise, wait for the maintainers here to implement it, but given the current interest in doing so I don't see this landing soon unless there's some community push with PRs.