Readme

Org CRUD

A tool for reading and writing org content via clojure, as well as converting org to markdown.

> Ninety percent of everything is crud. > – Theodore Sturgeon

Status

Alpha. I’ve recently namespaced the keys and flattened the props a bit. Still waiting for things to settle.

I’ve used it internally for months, but the public api still needs to be proven. In particular, I’d like to iron out some things I’m doing with dynamic vars that no user should need to figure out/configure.

It works for my current use-cases, and there are unit tests! Feel free to take it for a spin.

Motivation

This library was pulled out of another tool, a productivity app built on top of org-mode. That tool needed to be able to treat org files like a database of items.

It was pulled out to allow a few other libraries to use it independently (russmatney/ralphie), and also to add support for converting org files to markdown. See Markdown.

Org-crud aims to provide simple interactions with org files to clojure environments.

Background

There is not much to the parser besides a thin layer on top of organum. Organum does not nest the org items - it returns a flattened list, regardless of the items’ hierarchical relationship. Org-crud provides both a flattened and nested option for parsing org items.

This library is also babashka compatible, so you can drop it into a bb script without issue. This was necessary for tools like ralphie to run the exporter. You can see how ralphie consumes it in the ralphie.notes namespace

Features

• Babashka compatible
• List nested or flattened org items
• Update existing org items
  • Updates by :ID:
  • Add/remove tags, properties
  • Change an item’s name
• Delete org items
• Convert org files to markdown files

Org Item Model

This library parses org “items” from a given `.org` file.

An example item looks something like:

{:org/name "My org headline" ;; the name without the bullets/todo block
 :org/headline "** [ ] My org headline" ;; a raw headline
 :org/source-file "" ;; the file this item was parsed from
 :org/id *uuid "" ;; a unique id for the headline (parsed from the item's property bucket)
 :org/tags *{"some" "tags"}
 :org/level 2 ;; 1-6 or :level/root
 :org/body-string "raw body string\nwith lines\nof content"
 :org/body '() ;; a list of parsed lines, straight from organum TODO document this structure
 :org/status :status/not-started ;; parsed based on the
 ;; also supports :status/in-progress, :status/done, :status/cancelled

 ;; these dates are pulled through as strings
 :org/closed "2022-04-30 Sat 17:42"
 :org/deadline "2022-04-30 Sat"
 :org/scheduled "2022-04-30 Sat"

 ;; supports [#A], [#B], [#C] in headlines
 :org/priority "B"

 :org.prop/some-prop "some prop value" ;; props are lower-and-kebab-cased
 :org.prop/some-other-prop "some other prop value"
 :org.prop/created-at "2020-07-21T09:25:50-04:00[America/New_York]" ;; to be parsed by consumer

 :org/items '() ;; nested org-items (if parsed with the 'nested' helpers)

 ;; misc helper attrs
 :org/word-count 3 ;; a basic count of words in the name and body
 :org/urls '() ;; parsed urls from the body - helpful for some use-cases
}

Items were originally implemented to support individual org headlines, but have been adapted to work with single org files as well (to fit org-roam tooling use-cases).

Install

Currently, install requires referencing the git repo.

;; deps.edn
{:deps
 {russmatney/org-crud {:git/url "https://github.com/russmatney/org-crud.git"
                       :sha     "a4b44022c690e1c8fb34512f1aad85bc49569d19"}}}

TODO add to clojars

Usage

TODO do some work on this section!

You can see the test files for example usage.

I’m attempting to hold a public api at `org-crud.api`, but that is a WIP.

Parsing

(ns your.ns
 (:require [org-crud.api :as org-crud]))

;; a nested item represents an entire file, with items as children
(let [item (org-crud/path->nested-item "/path/to/file.org")]
  (println item))

;; parses every '.org' file in a directory into a list of nested items
(let [items (org-crud/dir->nested-items "/path/to/org/dir")]
  (println (first items)

;; 'flattened' items have no children - just a list of every headline
;; (starting with the root itself)
(let [items (org-crud/path->flattened-items "/path/to/file.org")]
  (println (first items)))

Updating

Updates are performed with a passed item and an update map that resembles the org-item itself. It will use the passed item’s id and source-file to find the item to be updated, merge the updates in memory, then rewrite it.

(ns your.ns
 (:require [org-crud.api :as org-crud]))

(-> (org-crud/path->flattened-items "/path/to/file.org")
    second ;; grabbing some item
    (org-crud/update!
      {:org/name "new item name" ;; changing the item name
       :org/tags "newtag" ;; adding a new tag
       :org.prop/some-prop "some-prop-value"
      }))

TODO document props-as-lists features TODO document refile!, add-item!, delete-item!

Markdown

Org-crud provides a namespace for converting org files to markdown, and a babashka-based cli tool for running this conversion on the command line.

In order for this to work, you’ll need to have Babashka (and clojure installed and available on the command line as `bb` and `clojure`.

bb org-crud.jar org-to-markdown ~/Dropbox/notes tmp-out

Note that this support targets a use-case for publishing an org-roam directory as markdown, but otherwise is probably not a complete org->markdown conversion solution. If you have more use-cases that you’d like to see supported, please open an issue describing the use-case, and I’d be happy to take a shot at it.

Note that Emacs/Org supports export that is fairly similar as well - I enjoyed putting this together and not needing to leave the joy of clojure-land.

An org file like `20200618104339-dated-example.org`:

*+TITLE: Dated Example
*+ROAM_TAGS: dated

Another org file, now with a link!

- [[file:example.org][example link]]

Dated to match the org-roam default style.

Will be converted to:

---
title: "Dated Example"
date: 2020-06-18
tags:
  - dated
  - note
---


Another org file, now with a link!

- [example link](/notes/example)

Dated to match the org-roam default style.

• The frontmatter pulls tags from `*+ROAM_TAGS`.
  • TODO prevent `note` from being added every time.
• The date is parsed from the filename.
  • TODO support alternate sources for the date, if users don’t have timestamps in filenames.
• The links to other notes are prepended with `/notes/<filename>`
  • TODO support custom link handling options, not just this hardcoded notes prefix.

Appended `Backlinks` section

When run over a directory, a `Backlinks` section is built up as a basic markdown list.

<... rest of file>
\* Backlinks

- [Index](/notes/20200704184516-index)

Notes

Item IDs (UUIDs)

Item IDs are more or less required for updating. Things will fallback to matching on name if there are no ids, but this approach has a few issues, because names are not necessarily unique throughout files.

I’ve updated my personal org templates/snippets in places to include IDs when creating new items, and org-mode provides helpers that can be used to add them without too much trouble. (Ex: `org-id-get-create`).

TODO share links to templates/snippets that create uuids

If this is a problem, let me know, there are other workarounds. Using IDs allows for cases with repeated headlines in the same file - otherwise you might get into tracking line numbers or parents, which did not seem worth it, especially as my usage benefitted from the IDs elsewhere.

Relevant/Related tools

• ox-hugo
• organum
• org-roam

Development

Running the cli using the source

Rather than the built uberjar:

# from this repo's root
bb -cp $(clojure -Spath) -m org-crud.cli org-to-markdown ~/Dropbox/notes tmp-out

Rebuild the uberjar

To rebuild the cli-based uberjar via babashka:

bb -cp $(clojure -Spath) -m org-crud.cli --uberjar org-crud.jar

Running tests

./bin/kaocha