Exploration: Create bookmarklet for pinning content #1267
Comments
Status updateI have being prototyping Shift+hover in as bookmarklet in this gist: You can install bookmarklet from https://runkit.io/gozala/5887e949adda540013624150/branches/master/ it just serves code from that gist https://runkit.com/gozala/5887e949adda540013624150 Once bookmarklet is activated you can start using I’ll likely face other complication when I’ll get to storing assets like css and images while saving selected fragment, as some of those could be from different origins and there for access will be restricted. |
|
@edsilv dropped by a channel today and provided some interesting pointers. He has being drafting proposal for saving / sharing trails (https://github.com/memex/trails-proposal). Turns out there is web annotation proposal https://www.w3.org/TR/annotation-model/ that we need to investigate as it seems really relevant. The selectors are immediately relevant bit as we're attempting to store & present selections from the page. Another interesting bit is https://hypothes.is/ as they have bookmarklet & while I have not tried it, from description it seems to do very much the same thing as us here. Worth looking into it & talking to folks behind it, thy might be able to offer some feedback on limitations they faced and ways to work around them. What is also interesting is that they seem to try and keep annotations even when pages have changed but relevant fragments are present (See https://twitter.com/edsilv/status/773413548756242432), which is also what we aim for. |
|
here's some more prior art we might be able to piggyback on / learn from: |
|
I have tried things ppl have pointed out. Here are some of my thoughts. bullseye
hypothes.is
|
|
I think that most important right now is simply being able to annotate / highlight content, so it's saved and associate it with particular history entry. This way we could do actually both: connect content structurally (trails, see ancestors of the page), but also give user possibility to combine different interesting pieces and connect them by their meaning. Once we have annotations correlated to the particular page in the history, it would be really powerful feature once utilized correctly. Also, if we can draw this connection (saved piece of the content with entry in the history), we don't really need styling, as you can always see the full version. RE: my mockups, I think we should for now just use text selection as you'd suggested and go with the 2nd one (bottom row). |
|
Hello @Gozala I am a developer at Hypothesis. We caught wind that you were interested in potentially using our bookmarklet to tackle some of your annotation and highlighting needs. Let us know if you have any questions - we are spread pretty thin but we can try to answer any questions you bring up. A bit about us, we are a non-profit and, barring confidential bits, all of our work is open source. Part of the team even played a significant role in getting the W3C spec where it is today. With that, we are trying to build a platform so that products like yours can take it and run with it to get your annotation needs knocked out and enable users to annotate the collective knowledge of the web. Regarding some of the 👎 mentioned above:
Our backend is open source and docker based. So if you wanted to, you could host the annotations yourself. 👍
Yes, if you are referring to being aware of particular sites like twitter or other post based sites. We have not quite taken the plunge on app context aware annotating but is something we'd explore further. Let us know how we could help 👍 |
|
@Gozala, FYI: You may enjoy talking with @BigBlueHat when doing anything with the Web Annotation spec and PouchDB. Relevant might be his page-notes, annotator-pouchdb (older), and probably another dozen of his repos. Also @tilgovi's work may be useful, this small demo of his may give a feeling for the structure of an annotation target selector. There are quite a few people happy to collaborate on making reusable components for annotation/memexy tools, including myself. (all three of us have worked at hypothes.is by the way) |
|
Hi @sean-roberts and @Treora thanks for jumping on this thread and providing useful pointers & prior art in form of Hypothesis & W3C spec work, it is amazing to have this foundation to build upon. I could use some input based on your experience to better understand why specific decisions were made.
|
@sean-roberts I did not meant it as a criticism, it's just I think we would rather store them on users machine & not in the cloud.
@sean-roberts What I meant is that it's not a best fit for what we're trying to do, which is - Fix the No 1 problem as per Mozilla user research concerning the Saving / Sharing, according to which most users just resort to screenshots / mail as everything else seems unreliable for different reasons:
With that said we want to build something like pinterest.com where arbitrary page content can be:
Each as a separate milestone. We'd be happy to get any help we can to do this. In longer term we would like to use this foundation to allow annotating navigation trails |
|
Greetings @Gozala! First, thank you for what you're building! I've been a distant fan for awhile. 😄 Pretty sure @Treora aims to fix that. 😉 So. Selectors and anchoring. Essentially, the Web Annotation Data Model allows for storing a set of selectors (or lists or choices or composites) for use in (re)anchoring the annotation. Those selectors can be used in a combination of ways but Hypothes.is and others store several and work through each of them until something hopefully anchors to the resource in its current state. If it doesn't, then the annotation is "orphaned" but still associated to the resource itself. Sometimes, as in editorial reviews, you may actually want an annotation to not re-anchor (i.e. "please remove this sentence."). If the text is gone in the next revision (which may have the original URL and/or a version URL), then the content owner made the change--if it does re-anchor, then someone missed something. 😃 All of what you describe is spot on and matches efforts by loads of people I'd love to connect you with (currently resisting the urge to "mention dump" them all into this thread 😉). Relatedly, @tilgovi and I are working on getting the Apache Annotator project on its feet--which aims to implement the Web Annotation Data Model and Protocol as well as give a migration path for former Annotator.js developers. It's early stages yet, but that demo mentioned earlier is likely to lay the foundation for some of the DOM-connected bits. I'll stop here, but I'd be more than happy to chat about this ad infinitum. Cheers! |
|
Oh! Also, since you're connected to Mozilla, you might enjoy these past projects that helped inspire me along the way (and consequently I was disappointed got abandoned...):
Just some highlights. 😉 |
|
@BigBlueHat thanks for more pointers! I was just actually skimming through the annotator-pouchdb code to get a sense how all pieces do fit together. Also unless I'm mistaken all of these projects seem to leverage http://annotatorjs.org aren't they ? I'll be looking into that next. Would it make sense for folks interested in this subject to join our slack channel ? I also don't mind joining whatever the other good place to discuss this would be. |
|
@Gozala sure thing.
So the pintrest.com approach where it's largely screenshot part of the page? If that's what you're thinking about doing, the annotation stuff is probably a different direction. It's mostly focused on the references to the raw values (such as the actual text or the video content) and not the stylings/visuals that make it a part of that original page. But if that is the type of approach you are wanting, perhaps it makes sense to allow user to select an area of the page for pinning. Then you screenshot that piece and grab any text selection inside of that area as well. Allowing you to capture styling/exact versions of the page as the user sees it and have content to make it searchable? |
|
@BigBlueHat funny that you mention Pancake as @gordonbrander amazing UX engineer who kick started this project used work on it, sadly he's no longer with Mozilla as he's off saving the world from hunger. |
Yeah I have being considering solutions for some, which is why I'm also curios if you have more comprehensive list of issues.
We chatted among the team yesterday and decided that for the first cut we might just capturing selections as markdown, it would simplify styling but possibly provide a slightly better connection. We will aim to capture styling in a followup step, mainly so it's easier for user to identify relevant pins from own library.
That's part of the why we don't store anything other than users machine, at which point it seems no different than a cache. But honestly I'm no expert here and we'd need to discuss these with legal before we can make it available. Also can't text be copyrighted as well ?
Cool, thanks for the pointers!
No we want to allow saving arbitrary fragments of page that user selected. Kind of like bookmarking that actually works and can target arbitrary content on sites not just URL.
That is exactly an interaction we're aiming for.
That's is one possibility we're considering, but saving as pictures isn't really ideal. We want to try & capture actual data & maybe bend rendering engine to manage that so that we can have different presentation depending on context. But we'll have to see I guess. |
|
@Gozala I'd be happy to join a slack channel. We don't yet have a chat thing setup for Apache Annotator, but when we do, you're welcome to join! Also, I'm always find-able on IRC (moz, w3c, or freednode). Many of the projects you'll find do (or did in some fashion) use Annotator.js. However, they've nearly all forked it in some way, and the 2.x pre-release didn't solicit enough interest to "stick"--though it's quite usable. Mostly, it's been a problem of ocean boiling--which is a key thing we hope not to do with Apache Annotator. @tilgovi's We're still working out the details of how much of that Apache Annotator will provide and how much we'll leave the wonderfully crowded world of JS & CSS frameworks. 😉 |
So, I am curious how you plan to convert selections to markdown and have that make styling easier? Because if you're going to try to match styling of the original site, you're better off to use the outerHTML of your selection instead of trying to map html to markdown and then map the stylings of every element. Which points back to your first question about issues. The main issue is that it's easier said than done to grab all of the styling for every element and then have it be presented in a view for the user. On top of that, as users get more and more annotations, your UI becomes a rather random collection of text stylings which includes more complexity to show, load, and maintain. So that's the big reason why we didn't go down that rabbit hole. We opted to keep it simple and target getting conversations going (for our users the need to match styling is almost nil).
So the suggestion I made, helped you do two things, keep style matching easy (it's in the image exactly) and store actual text fragments. Selections are largely immutable once they are made. So having the data backed behind the image, you can present the view you want (styled) and have it be searchable based on the selection. So you would store image and selection contents. The image can be in a dataURI format which is still likely to be much easier than storing html, styling rules, etc. - But again, only necessary if you think the end user wants their selection to match the visuals as well |
You have misunderstood my comment, I said:
Meaning that we'd drop page styles & instead use general styles for used elements. Kind of capture selections in reader view style.
Strategy I was aiming to go for is to walk DOM tree up from the selected nodes dropping all the siblings and inlining all relevant styles while also rewriting positioning to take into account dropped nodes. Then run test out algorithm on http://www.alexa.com/topsites in attempt to get a result we'd be happy with.
Agreed it's not the easy task, question for me if it is doable reliably at all.
There is definitely a challenge in presenting it in a nice & usable way, but I suspect that's to be lesser issue than capturing everything as appropriate.
That makes sense, especially given that Hypothesis is solving somewhat different problem.
You are right it definitely makes capturing a lot simpler, but there is drawbacks, even though library still is searchable actually saved content isn't really interactive (can't select, copy & paste, edit, further annotate, etc...). Which is why I still would like to see if actual capturing is gonig to work. Here is how I envision going about it:
Along the way we can talk to Servo team to find out if we could expose API to decide what styles contribute to the presentation and in which ways. It could be that rendering engine could provide use with all we need as it already has information. |
If this works it would be nice to build a reusable module for this. Would perhaps fiddling a bit with Likewise a module could be made for the backup plan, of creating a screenshot of the area containing a selection. Perhaps using ( I'd gladly consume (and help with) both such modules. |
|
It seems like there's two, somewhat independent, issues here:
|
|
@Treora @Gozala I'd also be more than happy to help extract the more broadly useful bits from the https://github.com/mozilla-services/pageshot/ code (or similar). It's broadly useful in and out of Web Extension code, and it's something I'd very much like Apache Annotator to provide eventually. |
That is an interesting spin on this, I'll have to think more about it if it can be viable solution to problem we're trying to solve. Let me try to elaborate - This work is driven by the user study that Mozilla did and I would really recommend reading it through: Idea of capturing specific fragments, fits really well (at least in my mind) with Organize and even more so for a Synthesize use case. OrganizeI imagine organizing "pinned content" in a galleries or a magazines styled user library that can be organized by subject & tags & captured metadata & well it's searchable. SynthesizeOne important aspect for me in regards to organizing magazines / galleries / catalogs (whatever we end up naming that) is that it should happen without users direct input. Not everyone is organized & only tool that will work for those who are not is the tool that just does it for them. In other words "pinned content" should automatically get synthesized into a magazines that are attractive way to revisit discoveries / results of research. Sure we need to allow users to manually organize & synthesize if they wish to do so, but that's not as important in this context. Now if we were to attempt to build such a library into browser I can't really imagine an attractive and useful version of it which has catalogs presented as series of cards of plain text or a screenshots or full page views. To be clear, I'm not saying it's not possible, it's just I have hard time imagining it, but maybe @patrykadas can ? |
|
@Gozala Sorry for my confusion here but "matching styling" might need to have an exact definition set to it.
or
|
|
There is certainly some overlap here with Page Shot, especially earlier directions Page Shot took. Now as we're getting close to shipping Page Shot the experience has become much more conservative, capturing just images and basic page metadata. Page Shot included DOM freezing as part of the experience. I'm still very interested in this capability, but shipping this under the Mozilla name is difficult (security and privacy concerns). Because of this I've recently forked this to pagearchive and the code will disappear from Page Shot soon. I've only started this split so the code is still messy. I'm hoping to provide the functionality on my own server, as I feel comfortable as an individual providing a use-at-your-own-risk service. There's a few layers to what Page Shot does. The DOM freezing happens primarily in make-static-html.js, with a little of it in extractor-worker.js. The split was between a framescript for make-static-html, which runs at high permissions, and a content worker for extractor-worker that runs as lesser permissions. (Web Extensions offer no high-permission worker that can penetrate cross-domain barriers, which is disappointing.) Before freezing we would try to add ids to every element, to make addressability easier. Though technically creating a CSS selector to address any element (given a static DOM) is not particularly hard either. The DOM freezing tries to eliminate any element that isn't visible, as we are preparing to share the content, and there may be hidden information that the user doesn't realize would be shared. That's probably incidental to these use cases. Of course we also remove scripts, which shouldn't (and generally can't) run. Another topic is all the embedded resources in the page – images and CSS primarily. Page Shot identifies these and rewrites the page replacing each resource URL with a UUID. This makes it a simple string substitution to put those resources elsewhere (we thought about storing them, but out of laziness have only proxied them). CSS can be recursive, so proxying is an incomplete solution. I had pretty good CSS inlining happening (translating all the CSS to a single The frozen DOM is a nice context for extracting or adding other information to a page. For instance, when Page Shot had the ability to save a clipped image and save the DOM, we'd annotate the clip with its position in the DOM (finding nearby anchor nodes, and giving pixel offsets). This made it reasonable to locate the clip in the DOM in a resolution-independent manner. We also captured the readable view of a page, as a way of trying to distill the page in some sense. But Readability.js is a really slow library and that caused problems. An area I had hoped to explore further was extracting other semantic information from pages at that moment. I haven't kept track of Fathom development, but maybe they've continued on that path. There was some mobile work around cards that also went in that direction, but I believe that was put aside. There was also some experimental support for extracting text clips. If you selected text it would capture the selection and some surrounding context (when it worked well, the paragraph the selection was contained in). It would try to maintain limited styling, with just a whitelist of tags. I'm not sure where that code went. The goal with saving the various layers of information was to keep many options open in terms of display. You could display an image, a text clip, you could use the Open Graph information, you can zoom into the frozen page, you have a URL to return to the real page, etc. As a counterexample, if you use the Evernote Web Clipper you have to make lots of choices about what you want to save – I wanted to create an experience in Page Shot where it would save everything it could, and treat other things as annotations (e.g., treating screenshotting as a way of highlighting a certain part of the page). A long time ago we also tried to capture the navigational context as another annotation on the page. So if there was a search page in the history for that tab/navigation then the query would be another kind of annotation on the page, the why for the page. Two shots with serendipitous overlap might be related, and so on. Then as all that gets collected we make a big ol' JSON file that includes everything. In some ways that object is the most interesting part, the other things are all there in the service of producing that object. That itself also opens up a question: at what moment do you create that object? The page can change, but the object can't along with it. We treat the user's intent to save something as the moment in time to save something, and if the user invokes that multiple times on the same page then we create multiple objects. Anyway, there's a big dump of information about Page Shot/Page Archive. This is still something that interests me a great deal, even if Page Shot itself is moving in a more mechanical and less navigational direction, so it would be nice to find a home for some of its ideas. |
Once we can corelate saved snippet / card to an entry in the history (ie. particlar page within particuar trail) I think I can come up with some ideas. This would be a really nice feature, as we could show content that's connected not only structurally, but also semantically. I remember that some people requested topic-oriented browser sessions. |
|
Whatever gets build out of this (and the related conversations), I'd love to see them be as small and modular and community built as possible. To many amazing bits get lumped into larger projects and ultimately lost in the long run. I'd very much like to not see that happen again. This stuff is too promising. 😃 |
|
Alright I have followed the rabbit-hole of web-annotation spec. There are some useful bits but some just seem over-engineered or under-engineered. As I was trying to build up understanding of the spec I've wrote down a type signatures (inline below) for Web annotations Selectors as my No 1 goal would be to create a pair of function that can translate to / from DOM Range and Web Annotation Selectors /* @flow */
type StringEncodedCSSSelector = string
type CSSSelector = {
type: "CssSelector",
value: StringEncodedCSSSelector,
refinedBy?: Selector
}
type XPath = string
type XPathSelector = {
type: "XPathSelector",
value: XPath,
refinedBy?: Selector
}
type FragmentSpecification =
| 'http://tools.ietf.org/rfc/rfc3236'
| 'http://tools.ietf.org/rfc/rfc3778'
| 'http://tools.ietf.org/rfc/rfc5147'
| 'http://tools.ietf.org/rfc/rfc3023'
| 'http://tools.ietf.org/rfc/rfc3870'
| 'http://tools.ietf.org/rfc/rfc7111'
| 'http://www.w3.org/TR/media-frags/'
| 'http://www.w3.org/TR/SVG/'
| 'http://www.idpf.org/epub/linking/cfi/epub-cfi.html'
type FragmentSelector = {
type: "FragmentSelector",
conformsTo: FragmentSpecification,
value: string
}
type TextQuoteSelector = {
exact: string,
prefix: string,
suffix: string,
refinedBy?: Selector
}
type Integer = number
type TextPositionSelector = {
start: Integer,
end: Integer,
refinedBy?: Selector
}
type DataPositionSelector = {
start: Integer,
end: Integer,
refinedBy?: Selector
}
type SerializedSVG = string
type SVGSelector = {
type: "SvgSelector",
value: SerializedSVG,
refinedBy?: Selector
}
type RangeSelector = {
type: "RangeSelector",
startSelector: Selector,
endSelector: Selector,
refinedBy?: Selector
}
type Selector =
| FragmentSelector
| CSSSelector
| XPathSelector
| TextQuoteSelector
| TextPositionSelector
| DataPositionSelector
| SVGSelector
| RangeSelectorNot everything can be entyped due to the direction chosen by a spec here is the list & it would be nice to provide this feedback to the working group somehow:
With that said my plan is to implement following (assumes types from above): const rangeToSelector = (range:Range):null|Selector => { ... }
const rangeFromSelector = (selector:Selector):Range => { ... }I would like to possibly provide a parameter to configure a strategy used for selector generation, like it maybe more preferable to use |
|
I also still need digg into a part of the spec in regards to States might have something relevant. |
|
You will definitely want to look at my libraries, tilgovi/dom-anchor-text-position and tilgovi/dom-anchor-text-quote as these do exactly that conversion for those two selector types. I would jump in to help with XPath as well, if it's useful to you to have this. The libraries don't actually return JSON-LD because they're meant to be deeply unopinionated. |
|
@tilgovi I was actually looking at those, I'm little confused in regards to the Thanks! |
|
To be more clear what i don't understand is why not just use |
|
Sure. Any preference as to tool for chat?
…On Fri, Feb 3, 2017, 15:08 Irakli Gozalishvili ***@***.***> wrote:
To be more clear what i don't understand is why not just use
range.startContainer it seems like resolving it to the another anchor is
somewhat irrelevant in that context.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1267 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABRUlAn_KMyZbTFvqKb5KNcq26ZOkJgks5rY7NcgaJpZM4LyDD3>
.
|
How about our slack channel ? Anything else works as well. |
|
Happy to, but I don't know what Slack organization or channel. Do I need an invite? |
|
There is the link on the the Readme, coping it here for convenience: |
|
The root argument is there to make things explicit and not rely on the browser range to be aware of the appropriate context for the operation. You are welcome to pass the common ancestor container, or anything else. Often, I've found it helpful to measure from some root that the application finds appropriate, like the body element, or main element, or some other block that's meaningful to the application. Again, my libraries try to be flexible, low level, and explicit wherever possible. |
Status update
|
|
@tilgovi If you want to help out you could maybe work on code that would take a selector as described above and produce a visual highlighting similar to the way hypothesis does it if such fragment exists on the page. Another area we could use help with would be with taking a selector as described above and extracting the content from the document as a markdown. In other words mapping DOM |
|
@Gozala really glad you and @tilgovi connected. If possible, I'd love for you to "say hi" on the Apache Annotator mailing list. @tilgovie, I and others are working to get that community on its feet building and growing a code foundation around the Web Annotation specs and the Annotator.js fork-ers. 😃 Also, depending on how quickly this comes together, it might be possible to get this project listed as an implementer of the W3C Web Annotation Data Model. Here's the current test results (fwiw). The specs should be reaching "Published Recommendation" status Real Soon Now. /cc @azaroth42 @iherman I've signed up for a Slack channel invite also in case you have other questions. Cheers! |
|
I did a poor job at posting updates last couple of days, but I'll try to do better - post daily updates. Here is a summary of where things are now:
Thanks everyone for suggestions here on slack and sharing your code, it really helped! I hope to seed some of these work done back eventually. |
devlog entry
|
|
Possibly relevant: https://ctxt.io/ |
Before we go ahead at persisting visited sites in the navigation trail as they appeared to user it would make more sense to start with a less ambitious task as the case study. Given that broken URLs are no 1 issue why browser bookmarks and history is unreliable and users resort to pictures instead let's allow users to select fragment of a site and save it to the local library - kind of like pintereset.com but not just for images but rather for arbitrary content & in a way that it's still select-able, searchable, sharable.
To do an exploratory work let's start with a most basic implementation:
Bookmarklet
Let's start with browser bookmarklet that,
Here are the mockups:
The text was updated successfully, but these errors were encountered: