TEI translator full rewrite #3245

glorieux-f · 2024-01-30T13:38:26Z

New features

Rich html text in notes exported as TEI
Rich text in titles, more Zotero conformant
Extra field parsed for extra CSL properties
XML/TEI indented
More conformant TEI

Best is to see how is handled a quite complex item.

{
	"version": 3038
	"itemType": "journalArticle"
	"title": "Not closing <b> or<strong>unknown tag</strong>, kept as text nodes; <i>italic</i>, is an element"
	"date": "1929"
	"language": "fr"
	"callNumber": "piaget1929a02"
	"extra": "Start of text note.\nGenre: Review of\nReviewed Title: Rich text >> <i>italic</i>\nMedium: Paris, <Payot>, 1927\nUnknown CSL property: CLA1929_5\nNote continuing.\nEditorial Director: One || John\nEditorial Director: Two || John"
	"volume": "22"
	"pages": "117-118"
	"publicationTitle": "Archives de psychologie"
	"issue": "85"
	"creators": [
		"0": {
			"firstName": "Jean"
			"lastName": "Piaget"
			"creatorType": "author"
		}
		"1": {
			"name": "Eugène Minkowski"
			"creatorType": "reviewedAuthor"
		}
		"2": {
			"firstName": "J."
			"lastName": "Contributor"
			"creatorType": "contributor"
		}
		"3": {
			"firstName": "M."
			"lastName": "Editor"
			"creatorType": "editor"
		}
	]
	"tags": [
		"0": {
			"tag": "AddATag"
		}
	]
	"collections": [
		"0": "U8JSTJZ5"
	]
	"relations": {
		"owl:sameAs": [
			"0": "http://zotero.org/groups/5048422/items/G99GTU5D"
		]
	}
	"dateAdded": "2024-01-29T19:58:13Z"
	"dateModified": "2024-01-30T12:55:46Z"
	"uri": "http://zotero.org/users/8989645/items/RLRXRWYM"
	"attachments": []
	"notes": [
		"0": {
			"key": "6SE87GSB"
			"version": 3020
			"itemType": "note"
			"parentItem": "RLRXRWYM"
			"note": "<div data-schema-version=\"8\"><p>A zotero note with some <em>formatting</em>.</p>\n<ol>\n<li>\nHate Numbered List\n</li>\n<li>\nGo to 1.\n</li>\n</ol>\n</div>"
			"tags": []
			"relations": {}
			"dateAdded": "2024-01-30T12:10:49Z"
			"dateModified": "2024-01-30T12:12:30Z"
			"uri": "http://zotero.org/users/8989645/items/6SE87GSB"
		}
	]
}

What Zotero do with this item in APA7.

Piaget, J. (1929). Not closing orunknown tag, kept as text nodes; italic, is an element [Review of Rich text >> italic, par Eugène Minkowski; Paris, <Payot>, 1927]. Archives de psychologie, 22(85), 117‑118.

This TEI export.

<biblStruct type="journalArticle" corresp="http://zotero.org/users/8989645/items/RLRXRWYM">
	<analytic xml:lang="fr">
		<author>
		<forename>Jean</forename>
		<surname>Piaget</surname>
		</author>
		<respStmt>
		<resp>contributor</resp>
		<persName>
			<forename>J.</forename>
			<surname>Contributor</surname>
		</persName>
		</respStmt>
		<respStmt>
		<resp>editorial-director</resp>
		<persName>
			<forename>John</forename>
			<surname>One</surname>
		</persName>
		</respStmt>
		<respStmt>
		<resp>editorial-director</resp>
		<persName>
			<forename>John</forename>
			<surname>Two</surname>
		</persName>
		</respStmt>
		<title level="a">Not closing &lt;b&gt; or&lt;strong&gt;unknown tag&lt;/strong&gt;,
 kept as text nodes; <hi rend="italic">italic</hi>, is an element</title>
		<idno type="callNumber">piaget1929a02</idno>
	</analytic>
	<monogr>
		<editor>
		<forename>M.</forename>
		<surname>Editor</surname>
		</editor>
		<title level="j">Archives de psychologie</title>
		<imprint>
		<date when="1929">1929</date>
		<biblScope unit="volume">22</biblScope>
		<biblScope unit="issue">85</biblScope>
		<biblScope unit="page">117-118</biblScope>
		</imprint>
	</monogr>
	<relatedItem type="reviewed">
		<bibl>
		<author>
			<name>Eugène Minkowski</name>
		</author>
		<title>Rich text &gt;&gt; <hi rend="italic">italic</hi></title>
		<edition>Paris, &lt;Payot&gt;, 1927</edition>
		</bibl>
	</relatedItem>
	<note type="extra">Start of text note.
	
	Unknown CSL property: CLA1929_5
	Note continuing.</note>
	<note corresp="http://zotero.org/users/8989645/items/6SE87GSB"><p>A zotero note with some <emph>formatting</emph>.</p>
	<list rend="numbered">
	<item>
	Hate Numbered List
	</item>
	<item>
	Go to 1.
	</item>
	</list>
	</note>
	<note type="tags">
		<term type="tag">AddATag</term>
	</note>
</biblStruct>

The previous TEI export (original is not well idented)

<biblStruct type="journalArticle">
	<analytic>
		<title level="a">Not closing <hi rend="bold"> or<strong>unknown tag</strong>,
		 kept as text nodes; <hi rend="italics">italic</hi>, is an element</hi></title>
		<author><forename>Jean</forename><surname>Piaget</surname></author>
		<respStmt>
			<resp>reviewedAuthor</resp>
			<persName><name>Eugène Minkowski</name></persName>
		</respStmt>
		<respStmt>
			<resp>contributor</resp>
			<persName><forename>J.</forename><surname>Contributor</surname></persName>
		</respStmt>
	</analytic>
	<monogr>
		<title level="j">Archives de psychologie</title>
		<idno type="callNumber">piaget1929a02</idno>
		<editor><forename>M.</forename><surname>Editor</surname></editor>
		<imprint>
			<biblScope unit="volume">22</biblScope>
			<biblScope unit="issue">85</biblScope>
			<biblScope unit="page">117-118</biblScope>
			<date>1929</date>
		</imprint>
	</monogr>
	<note>A zotero note with some formatting.
		
		
		
		
		Hate Numbered List
		
		
		Go to 1.
		
		
	</note>
	<note type="tags"><note type="tag">AddATag</note></note>
</biblStruct>

Tested with malformed html in Zotero fields.

…into tei_export

AbeJellinek · 2024-02-08T18:35:44Z

TEI.js

+
+/* 2024, Frédéric Glorieux.
+
+// item produced by Zotero


I wish we had test cases for export translators, but this is a lot to put at the top of the translator. We can keep it in the commit history but should remove from the translator before merging IMO.

What I tried was a kind of documentation for end users. TEI is always an interpretation, there are more than one way to encode an information. To understand how the old translator was working, I had to test various kind of records to trace the fields in TEI output, I thought a good idea to explain the code a bit more. Comment in source code is not the right place, you are right, test case is a good idea, but there is still something to find for a documentation.

AbeJellinek · 2024-02-08T18:39:04Z

TEI.js

+ * Imitated from zotero source code
+ * https://github.com/zotero/zotero/blob/main/chrome/content/zotero/itemTree.jsx#L2472


FWIW, Zotero is imitating the behavior of Citeproc.js here, so let's link straight to the source too.

Link added. Below the spec, @dstillman #3054 (comment) make me rewrite the parser to handle tricky cases like
Not closing orunknown tag, kept as text nodes; italic, is an element
Reading code helps me to make the right choices on each case.

AbeJellinek · 2024-02-08T18:40:28Z

TEI.js

+		let discardedNode = nodeStack.pop();
+		nodeStack[0].append(discardedMarkup.token, ...discardedNode.childNodes);
+	}
+	// return textContent; // lint see it’s not used


Old code used for debug. Deleted.

AbeJellinek · 2024-02-08T18:40:48Z

TEI.js

+	if (!html) return;
+	// import html as dom
+	let dom = xmlParser.parseFromString(html, "text/html");
+	let body = dom.getElementsByTagName("body").item(0);


Suggested change

let body = dom.getElementsByTagName("body").item(0);

let body = dom.body;

AbeJellinek · 2024-02-08T18:42:32Z

TEI.js

+ * @param {*} item
+ * @returns
+ */
+function parseExtraFields(item) {


Why do we need this? Won't it happen upon saving the item anyway?

I’m sorry to not understand your question. In the context of a translator, do you mean I can get a state of an item where extra field is parsed to CSL? I would be glad if extraToCSL() have been available through ZU, but it is not.

AbeJellinek · 2024-02-08T18:43:55Z

TEI.js

+		// unicode classes seems not supported
+		// xmlid = xmlid.normalize("NFD").replace(/\p{M}+/u, '');


Use XRegExp:

translators/eLibrary.ru.js

Line 138 in edf44a1

var nameFormat1RE = new ZU.XRegExp("^\\p{Letter}+\\s\\p{Letter}+\\s\\p{Letter}+$");

This will be supported natively in Z7+.

Thanks for the tip.

AbeJellinek · 2024-02-08T18:44:50Z

TEI.js

+	let iso = null;
+	let year = Number(date.year);
+	if (isNaN(year)) return iso;
+	iso = String(date.year).padStart(4, '0');
+	let month = Number(date.month);
+	if (isNaN(month)) return iso;
+	// january = 0
+	iso += '-' + String(date.month + 1).padStart(2, '0');
+	if (!date.day) return iso;
+	iso += '-' + String(date.day).padStart(2, '0');
+	return iso;


Why not strToISO?

Sorry, I missed ZU.strToISO(item.date). Deleted.

glorieux-f added 24 commits June 15, 2023 12:52

Import html rich text as TEI elements

f62f537

Tested with malformed html in Zotero fields.

Big refacroring

3104d92

The january surprise ! 0 is not null but first month

6d28b0f

Rewrite for indentation and better control on variable scope.

8173431

Export abstract as a note

f313105

Reviewed author and formating

41686fa

Better tagsoup handling

7ecbe91

Element.appendChild() > append()

9de4d67

APA7, reviewed-title

bc362e3

Reviewed-Title and Call-Number, better TEI place ; some refactoring

0bab265

better TEI tag for extra.format

e74bdec

language, some refactor

c49681f

Merge branch 'zotero:master' into master

5cc1434

Merge branch 'zotero:master' into tei_export

f298eec

Review, more semantic ; indent refactor

e04e3b8

Merge branch 'zotero:master' into master

6a70732

Merge branch 'zotero:master' into tei_export

7e25f37

Rewrite of Extra parsing, inspired by Zotero.Utilities.extraToCSL()

2e056aa

Merge branch 'tei_export' of https://github.com/glorieux-f/translators …

923f905

…into tei_export

eslint is happy

4cca6b0

Merge branch 'tei_export'

290b3bc

Documentation and better handling of creators in Extra field

0006d4d

date update, eslint

bfc2486

Merge branch 'zotero:master' into master

382473f

AbeJellinek requested changes Feb 8, 2024

View reviewed changes

glorieux-f added 3 commits February 10, 2024 17:58

suggestions from @AbeJellinek zotero#3245

fe9efaa

Merge branch 'zotero:master' into master

3b5949b

Merge branch 'zotero:master' into master

a6ee87d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TEI translator full rewrite #3245

TEI translator full rewrite #3245

glorieux-f commented Jan 30, 2024 •

edited

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024 •

edited

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024 •

edited

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024 •

edited

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024

		* Imitated from zotero source code
		* https://github.com/zotero/zotero/blob/main/chrome/content/zotero/itemTree.jsx#L2472

	let body = dom.getElementsByTagName("body").item(0);
	let body = dom.body;

		// unicode classes seems not supported
		// xmlid = xmlid.normalize("NFD").replace(/\p{M}+/u, '');

TEI translator full rewrite #3245

Are you sure you want to change the base?

TEI translator full rewrite #3245

Conversation

glorieux-f commented Jan 30, 2024 • edited

Choose a reason for hiding this comment

glorieux-f Feb 10, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glorieux-f Feb 10, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glorieux-f Feb 10, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glorieux-f commented Jan 30, 2024 •

edited

glorieux-f Feb 10, 2024 •

edited

glorieux-f Feb 10, 2024 •

edited

glorieux-f Feb 10, 2024 •

edited