v2

[4kimov]

tl&dr: My name is Ivan Akimov, I'm the original creator of Hashids. I'd like to suggest an updated algorithm, better support on my part, rebranding, better docs/website and a few other minor things. Reasons & details below. Asking for feedback.

---

Hey all, it's been a decade since the release of Hashids. I'm very proud of the support this project has received over the years. The initial traction was unexpected, and so were the numerous ports that poured in from all of you. What followed were many happy users and millions of combined installs.

You're probably aware that a few years in, I took a backseat role within the project, and transferred my own versions over to @niieani and @vinkla for maintenance, who did an amazing job not only maintaining but moving them forward. And so did all of you who have been running your own ports.

I've always lurked in the background of Hashids, but throughout the years wanted to improve certain aspects and simplify parts of the implementation.

What follows is ideas/suggestions on v2 and the org itself:

1. House only new ports/versions/repos under one roof (a Github org), as opposed to them being scattered throughout Github. A single repo port would have several maintainers for redundancy. If someone is no longer able to maintain the repo, we'd have the ability to add other maintainers. It would also be easier for new users to find all ports when they're under one roof. Current implementations can and should stay where they are, and we would link to them.

2. Remove the salt parameter. It's confusing and implies having something to do with encryption. If users would like custom ids, they can pass a manually shuffled alphabet to the constructor.

3. All versions should produce the same output. Having the same unit tests & all ports being under one Github org would help.

4. Profanity filtering handled through a wordlist. Current version only works against most common English curse words. I'd like it to support any custom words for multiple languages/countries.

5. Simpler interface: only encode(numbers: u64[]) and decode(id: string) functions. Possibly minValue() and maxValue() functions if the programming language is not clear on what the max value might be (min value would still always be 0)...

6. BigInt support / max integer values would depend on the individual implementations. Hence the maxValue() function.

7. A new name. I've been going back and forth on this for a while, and previous discussions were kind of split. The Hashids name still doesn't sit well with me for the following reasons (even tho I'm the one that came up with it):

   1. Tons of questions about why "hash" is in the name since IDs are decodable and hashes are not
   2. One of the parameters being called "salt" didn't help with explaining this is not an encryption library
   3. The very early versions using method names like encrypt/decrypt [also didn't help]

   I think the new name would help avoid some of the existing confusion, and since the core would change so much, this would be a good time to name v2 something else.

8. New website: simpler docs, up-to-date design, good use cases + bad, features list, why use it, a simple playground, dark mode, etc. all of this is long overdue. Also: proper 301 redirects to the new website not to break SEO (with links to the original Hashids Github implementations) + explanation on why Hashids has been upgraded.

9. I paid for hashids.org to be registered through 2029. I'd like one of my companies to sponsor v2. It can pay for Github teams, domain name registration, hosting costs (Vercel teams for backend support), Cloudflare, Plausible, FortAwesome, etc. Hashids is not a product that makes money, it's a project that costs money -- sponsoring it would help ensure this project can be sustained for years.

10. I've registered sqids.org. It doesn't stand for anything significant, other than: it's shorter than Hashids, doesn't have the word "hash" in it, theoretically could stand for "short quick ids" or "short quality ids", and it's arguably fun/goofy sounding (like squids). If you have better suggestions, please do share.

Random notes:

• I pushed the working changes I described above in a temporary repo here if you're curious on the algo: https://github.com/sqids/sqids-spec.
• The updated website/documentation is almost done, not a problem to push it out soon.

---

I would like to hear from current @hashids/owners (and users) & their thoughts on the above.

• If you're cool with everything, please comment with the thumbs up & I'll pull you into the individual repos as I move this forward.
• If you have questions or comments, let's discuss below.
• If you have tech suggestions on the algo, please use https://github.com/orgs/sqids/discussions

[speps]

sqids looks like sql at quick glance, not sure about the name change to be honest or rather I don't have suggestions... everything else is fine by me.

[miquelfire]

Yes, if we were to use a name like that, the q would need to be changed.

As far as getting things moving for a better name, something like snid or scid (Scrambled Number/Character ID). I don't really like these names, but maybe that could help someone else brainstorm a better name. I like hashids, but the hash in the name really confuses people on the purpose in the wrong way, as said in the OP.

[waynebloss]

Hi, you can fork or remove hashids-tsql since it was never fully complete and is no longer maintained. Thanks!

[dswitzer]

It's really an Obfuscated ID, so maybe a play somewhere off that, like OID or OBID. You could call it ObfusicatID. Just a thought....

…

On Mon, Jun 19, 2023 at 12:40 PM Miquel Fire Burns ***@***.***> wrote: Yes, if we were to use a name like that, the q would need to be changed. As far as getting things moving for a better name, something like snid or scid (Scrambled Number/Character ID). I don't really like these names, but maybe that could help someone else brainstorm a better name. I like hashids, but the hash in the name really confuses people on the purpose in the wrong way, as said in the OP. — Reply to this email directly, view it on GitHub <#92 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG33ENPWEXFOZIDWLL4QHDXMB6GZANCNFSM6AAAAAAZMCK4JQ> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.*** .com>

[vinkla]

It's great to have you back, Ivan. We definitely welcome any improvements, and it's probably a wise decision to rebrand the project with simpler defaults, as you suggested.

I took a look at the repository and I've made an initial attempt to create a PHP version. I used AI to generate most of the code, but there are memory leak issues. We'll probably need to rewrite the logic using a math extension.

[niieani]

Hey @4kimov! Great to hear back from you.
I was recently experimenting with a new, simpler algorithm for a vNext too, I haven't gotten too far though. I feel like the current one is too complex. I'm onboard with skipping salt and adding a word list instead of seps in the next version.
The current API also is confusing as it depends on argument order (at least in JS) which isn't too great. There is also no hard error support when input is malformed. I think we should add explicit exceptions thrown in such cases, or at least document how we handle such behavior.
If we write a good and simple spec, we should be able to generate most of the implementation code using an LLM like CharGPT, since writing code based on spec is something it's really good at.
Also onboard with creating a universal test suite, as that way we can ensure cross-platform compatibility.

One thing I'd like to add to the official spec is support for Unicode characters in the alphabet. I've added it to the JavaScript port, but I'm not sure it's compatible with all the other implementations. This allows things like generating a string of emojis by using an emoji alphabet. Treating the alphabet and individual tokens as an array of strings generally does the trick. It could also unlock scenarios like generating an array of tokens, if the "alphabet" allowed arbitrary multichar words, e.g. if using a dictionary of English words, you could get results like: [pepperoni, lemon, wonder, amazing, tree, open, anxious] which could easily be turned into a word token: ’pepperoni-lemon-wonder-amazing-tree-open-anxious’. Might be useful for use cases where a person might dictate the ID, e.g. error codes.

We should also make sure the new version is at least as performant as the original hashids. I have received complaints about regressions as I developed hashids.js, which I mediated after adding a benchmarking test (and actually made it faster than the original v1 version by using arrays internally, rather than string manipulation).

[4kimov]

Thank you all for the on-going feedback.

@speps @miquelfire @dswitzer, interesting observation regarding SQL. This is what I meant initially regarding rebranding: https://sqids.org/ (logo is temporary; plural ending is a nod to Hashids). My hope is different capitalization and pronunciation would distance it further from SQL. Naming things is hard. Most good names (+domains) are already taken. I'll wait for others to weight in, but if there's a name that has .com|.org available + Github username available + shorter than Hashids + something relatively pronounceable & googleable, then let's check it out.

@waynebloss Roger that.

@vinkla Thank you, great to be back. Wow, you're quick! And it's also impressive how much help AI can be. Copy that regarding PHP math extensions. I was hoping to try and finalize sqids-spec this + next week. I'm still testing shuffling, encoding, etc to see what could be optimized or improved. If you have ideas on how to upgrade & stabilize the API / algorithm, please do share.

@niieani Thank you, great to hear from you as well. I'm still messing around with v2 API over at https://github.com/sqids/sqids-spec/blob/main/index.ts - as you can see it's still missing checks & error handling (please feel free to suggest what could be done better/simpler). I saw your awesome test suites, think you could help me move them over to here? We should discuss support for word tokens separately (this is a big topic, I see some pros + cons). I hear you on the performance. The spec repo right now is optimized for readability & clarity (this is by no means meant to be a JavaScript port). I'm hoping to base other ports on it, but the individual implementations should def take freedom to optimize anything and everything possible for performance. I've added some todos based on your suggestions, am I missing anything?

[fanweixiao]

Hi @4kimov, It's exciting to have you back!

House only new ports/versions/repos under one roof (a Github org), as opposed to them being scattered throughout Github.

I think this should be the first step when we start v2 work, and all v1 repo should add same introductions about v2 in their README file.

sqids

I like this squids and the new logo.

One more thing, I suggest we add WebAssembly implementation in "what language are you using?" section.

[4kimov]

@fanweixiao Any particular langs you'd like to be added to? I see you've forked Go+Java previously

[4kimov]

@fanweixiao Thank you, great to be back. Copy that re: one Github repo + READMEs + sqids. Regarding WebAssembly, do you mean the filter should show repos that support compiling to wasm (JavaScript, Rust, etc), or you had a separate implementation in mind?

[fanweixiao]

Hi @4kimov, I don't know what's the best way for developers deal with WebAssembly, may be we can get suggests from developers there.

[laserpants]

Hi @4kimov. This is a great initiative. Please add me as well. 👍 I'll have a look at the spec and start working on the Haskell port ASAP.

[tzvetkoff]

@4kimov, good to see you're back! :-)

My 2 cents here -- it's nice to have an idea for a new simplified library (even though the blacklist will complicate things a bit.)
I'd gladly do an ANSI C port this weekend, so add me as well.
I can also port to a few other languages, like Golang, Rust, TCL (I almost ported Hashids back in the day but I lost interest in TCL scripting) if nobody signs up for it.

[4kimov]

@laserpants Nice, will do, thank you.

@tzvetkoff Thank you, appreciate it. TCL would be great, as well as the other languages you mentioned. And yes, the custom blacklist would be a bit more effective, but also we'd have to get the list just right (words can't be too short to have too many hits, but also not too specific so the size of the library doesn't blow up).

I'm still adjusting the spec and it will take me a bit of time, so atm it's still early to port versions. However, if anybody would like to take part in shaping the spec, you can find it here.

Btw, there's also a separate Unicode discussion going on here if anybody has any feedback on that.

[4kimov]

Thank you all for the feedback.

I'll close the discussion for now, but feel free to comment if you have more -- I'll continue replying.

Quick update:

1. The spec is complete as far my own todos. Unit tests & full coverage have been added. There's a separate discussion that's an RFC for the algorithm specifically. I'd appreciate some eyes on it for a sanity check.
2. The default blacklist is also done, but I'm sure could use some curating. If anyone speaks multiple languages, please check it out.

I'll wait another few days for the comments on the algorithm's RFC, and if it looks good the next steps would be to write individual implementations (I'll invite individual members to different repos).

[niieani]

Any chance we could source the blocklist from somewhere else, instead of having to curate our own?

[4kimov]

I've been trying to find reliable lists, but so far no luck. Some are very large & a bit inaccurate and many have attribution requirements. I've also been trying not to include everything I find so I don't blow up the library size or degrade performance unnecessarily. If you have suggestions, feel free to open a discussion. I'm curious about this problem myself.

[niieani]

We could ask ChatGPT to generate a comprehensive list for all major languages. This is one of those things that LLMs are really good at.

[4kimov]

Good idea, but ChatGPT wouldn't let me 😅 I keep receiving: "As an AI developed by OpenAI, I am programmed to adhere to strict ethical guidelines, and promoting or sharing inappropriate or offensive content goes against those guidelines."

Maybe somebody can prompt it better, or use a different LLM?

[niieani]

Works with open source LLMs. Here's my prompt:

The following is a blocklist of words that should be avoided when generating alphanumeric URLs:

(insert blocklist)

Expand this list to be more comprehensive.

I think the prompt could be adjusted to "include words in other commonly spoken languages", or specifically asked to generate lists in specific languages.

[4kimov]

Thanks for the link. I got mixed results for now. It replaces stars as some letters, and while I might guess what the words are for the English language, it'd be hard to do so for others 😅 Also, for non-English langs I got mixed results: it recommended "kitchen", "leg" and "smoking".

The most ideal query would be something like: "give me a list of top 50 inappropriate words for the English language and spell them in latin characters" (because we'd try to protect the default alphabet), but I haven't found a model that would give decent results yet.

Edit: And of course we'd replace "English" with other most-common languages too.

[dswitzer]

I haven’t read the spec yet, but I wonder if instead of managing blocklists we could just guarantee no more than like 2 alphabet characters appear before a non-letter.

…

On Sat, Jun 24, 2023 at 8:45 PM Ivan Akimov ***@***.***> wrote: I've been trying to find reliable lists, but so far no luck. Some are very large & a bit inaccurate and many have attribution requirements. I've also been trying not to include everything I find so I don't blow up the library size or degrade performance unnecessarily. If you have suggestions, feel free to open a discussion <https://github.com/orgs/sqids/discussions>. I'm curious about this problem myself. — Reply to this email directly, view it on GitHub <#92 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG33EL7RL3ZW4CMSKOZSHDXM6CY5ANCNFSM6AAAAAAZMCK4JQ> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[4kimov]

That'd be a bit hard to implement since we don't control neighboring chars. It would also not block leetspeak (eg: "he11o").

[dswitzer]

The trick would be to insert a character that should help to avoid ***@***.*** So no zero, one, three, five.

…

On Sat, Jun 24, 2023 at 9:26 PM Ivan Akimov ***@***.***> wrote: That'd be a bit hard to implement since we don't control neighboring chars. It would also not block leetspeak (eg: "he11o"). — Reply to this email directly, view it on GitHub <#92 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG33EM2DX3KGTZTDADLH3TXM6HVTANCNFSM6AAAAAAZMCK4JQ> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[laserpants]

Hey @4kimov,

I noticed that the website says "no dependencies". Is this a strict requirement for all language implementations? Although this is technically possible, it does introduce some limitations. Could you clarify, please?

[4kimov]

Hey @laserpants, good question.

Initially, I didn't notice dependencies for the few languages that I was messing around with, that's why it's there. But I do recognize that some implementations might want/need to use the standard library or common user-defined packages.

The library is small enough that ideally we'd not rely on too many external packages, but I don't think it's a problem if there's a need for it.

I will update the website now to remove that phrase. Thank you for letting me know.

[miquelfire]

Technically, PHP is one of them. It required one of two extensions to be installed, and out of the box, both are not enabled by default and require the user to enable them.

When this was done, it seemed people were mostly using 32-bit compiled versions of PHP to the point 64-bit support wasn't known, and they used numbers that would often cause the algorithm to need bigger than 32-bit integer support them, so they just dropped support of not using one of the two. IIRC, before they dropped support of not needing one of the two extensions, the effective limit was basically around 24-bit as the way the code was written, it needed a bit of a buffer. Didn't help that PHP used signed numbers.

[4kimov]

I'm seeing some more on-going comments, so I'll re-open for more visibility.

[4kimov]

Another quick update:

• Individual language repos are created. I've given access to everyone I can think of -- if I'm missing someone, please let me know.

• At this point I don't think I'm blocking anyone on porting over the spec. I'll slowly start on JavaScript, Rust & Go if nobody else jumps in over the next few days/weeks. I see @laserpants has already started with Haskell 💪

• Website is up & running. I'll redirect hashids.org to it later, once we get some momentum on implementing some codebases.

If questions or feedback, I'm here 🙏

[4kimov]

I'll close this discussion - thank you all for the feedback.

To sum it up:

• Spec has been finalized, if you'd like to discuss something there's a separate discussion here
• Individual repos have been created & everyone that I can think of has been invited. I see some invitations have expired (they automatically expire after 7 days of inactivity), so please let me know if you need another one
• Hashids README has been updated
• The following are already done: Julia, Haskell & Go 💪
• The following are being worked on: JavaScript & Rust
• The rest haven't been started yet -- I could use your help on getting them to the finish line 🏁
• I'll aim to redirect hashids.org to sqids.org next week. It's ok that some repos haven't been implemented yet, since the new website links to existing Hashids implementations anyway, but if you were planning on implementing a repo, now is the best time 💪

Thank you all.

[dswitzer]

Can you send me another invite? I forgot to accept it originally. Thanks!

…

On Thu, Jul 20, 2023 at 9:36 AM Ivan Akimov ***@***.***> wrote: I'll close this discussion - thank you all for the feedback. To sum it up: - Spec has been finalized <https://github.com/sqids/sqids-spec>, if you'd like to discuss something there's a separate discussion here <sqids/sqids-spec#2> - Individual repos have been created & everyone that I can think of has been invited. I see some invitations have expired (they automatically expire after 7 days of inactivity), so please let me know if you need another one - Hashids README <https://github.com/hashids> has been updated - The following are already done: Julia <https://sqids.org/julia?hashids>, Haskell <https://sqids.org/haskell?hashids> & Go <https://sqids.org/go?hashids> 💪 - The following are being worked on: JavaScript <https://github.com/sqids/sqids-javascript> & Rust <https://github.com/sqids/sqids-rust> - The rest haven't been started yet -- I could use your help on getting them to the finish line 🏁 - I'll aim to redirect hashids.org to sqids.org next week. It's ok that some repos haven't been implemented yet, since the new website links to existing Hashids implementations anyway, but if you were planning on implementing a repo, now is the best time 💪 Thank you all. — Reply to this email directly, view it on GitHub <#92 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG33EPHZHZO2N7V2MEQWRDXREX5HANCNFSM6AAAAAAZMCK4JQ> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[4kimov]

Done 👍

[leihog]

Hey folksI also missed my initial invite, please resend. 🙂20 juli 2023 kl. 19:33 skrev Ivan Akimov ***@***.***>: Done 👍 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

[4kimov]

Done 👍