Configuration file question

[airone506]

Good day to you!

I was very excited to find this project. Stork Search looks to be an awesome piece of software and I like how it actually works in the live demo on the project's website. It looks like a tool I was exactly looking for. I also see a big amount of a great work that has already been done on Stork Search. Congratulations & Thank You Mr. James Little!

I already decided to give it a try for a website, that uses a generator for static content. Now I would like to ask a question regarding the .toml configuration file. Here is an example of basic.toml file copied from the Stork Documentation.

[input]
base_directory = "federalist"
url_prefix = "https://www.gutenberg.org/files/1404/1404-h/1404-h.htm#link2H_4_"
files = [
    {path = "federalist-1.txt", url = "0001", title = "General Introduction"},
    {path = "federalist-2.txt", url = "0002", title = "Concerning Dangers from Foreign Force and Influence"},
    {path = "federalist-3.txt", url = "0003", title = "Concerning Dangers from Foreign Force and Influence 2"},
    {path = "federalist-4.txt", url = "0004", title = "Concerning Dangers from Foreign Force and Influence 3"},
]

Now I understand that I need to have the content of each article (content to be included in search) stored on a local computer, I need to know the future url of each article and I need to know the title of each article. And I need to know all three listed parameters in advance. Do I understad it correctly?

[jameslittle230]

@airone506 - Yup, that's an accurate understanding, except for one thing: you don't necessarily need to have the content that you want indexed stored on a local computer; instead, you can index content from published web pages and the Stork indexer will fetch that content from the web.

I have yet to write any really good documentation, but you can play around with the src_url field of the configuration file object. In the example you gave above, you could do something like the following:

files = [
    {src_url = "https://whatever.com/my-page", url = "https://whatever.com/my-page", title = "General Introduction"},
    # ...
]

Happy to help clarify anything further, and good luck with your website!

[airone506]

Thanks for your reply @jameslittle230.

Well, the possibility of using src_url option is a feature I was not aware of, but I was secretly hoping for. Awesome!

Now I would like to politely ask some additional questions.

1. If HTML content is being indexed by stork build, would also the values of the the alt and the title attributes of an img element being included in the final .st file? Or eventually the value of the title attribute of any element having this attribute? It makes perfect sense to me to include these in the .st file.

2. What would be the best practice for the multi-lingual sites? To build (A) one .st file with all the languages together or to build (B) a separate .st for each of the languages?

   The A variant makes sense here, the user would start to type the phrase in their language, not taking care of anything, and only the results containing the phrase would appear, of course matching the language of the searched phrase. The B variant would require to create search form for each of the languages and connect the form it to corresponding .st file. But which variant seems to be the better one?

[airone506]

I just finished writing of a script that creates the configuration .toml file with all the parameters like path, url, title and stemming_override. It also puts the content of the respective page into Markdown file. The source for all the data is the directory tree and all data files originating from the static website generator I use. I can also decide which of the additional metadata should be included in the Markdown file to be indexed. Now it is also possible for the images' alt and title to be included in the Markdown file to be indexed.

For the multi-lingual site I went with the (A) variant and I adjusted the stemming_override parameter for each page/article individually, based on its language of course.

Now the testing search with stork test --input ... works as expected, but I already had high expectations 🙂. Yes, the Stork Search works great. 👍

Now I would expect my approach to build the index (& prepare the data before the building) would differ each time, based on specific scenario.

[jameslittle230]

If HTML content is being indexed by stork build, would also the values of the the alt and the title attributes of an img element being included in the final .st file? Or eventually the value of the title attribute of any element having this attribute? It makes perfect sense to me to include these in the .st file.

Not yet! That's been a feature request I've heard, and I'm tracking it in #183.

What would be the best practice for the multi-lingual sites? To build (A) one .st file with all the languages together or to build (B) a separate .st for each of the languages?

It depends on what you want your search experience to be -- Stork can support either. For a corpus with content in different languages, you can configure each document to use a different stemming algorithm and Stork can handle mixing it all into one index. If you have the same content translated into different languages and the user expects to only see one language of content when they search, then you'd probably want to build one index file per language and use the user's settings to determine which index to load.

Now I would expect my approach to build the index (& prepare the data before the building) would differ each time, based on specific scenario.

Yup, that sounds right to me.

Glad you're enjoying Stork!

James