Examples in the manual as MWE #640
Comments
|
@tallmarmot @Mo-Gul I have now more or less finished the extractor script. You can give it a go using the following bash script #!/bin/bash
mkdir -p mwe
texlua extract.lua text-en/ mwe/
cd mwe
n=1
N=$(ls -1q *.tex | wc -l)
for i in *.tex; do
echo -n "Processing $n/$N $i"
if [ -f "${i%%.tex}.pdf" ]; then
echo -n " Skipped." # skip existing
else
lualatex -interaction=batchmode -halt-on-error "$i" > /dev/null
if [ "x$?" == "x1" ]; then
echo -n " Failed!"
fi
fi
echo ""
n=$((n+1))
doneSave this script as, say As you can see not all of the examples typeset out-of-the-box. For the failed ones, you can inspect the log to find the error. As you can see from the extractor script pgf/doc/generic/pgf/extract.lua Lines 2 to 18 in 710bd7f I currently just include all libraries in the preamble. This of course has to change. We only want to use the libraries which are necessary for each example. |
|
@hmenke, thank you very much for providing the extractor script. I gave it a try and it works perfectly fine. And you are right, it takes a lot of time. Since I don't have clue about Bash scripts: Is there a chance to parallize the compilation? @tallmarmot, what do you think about the extractor script? Do you see a chance to modify it according to "our understanding"? |
|
Here is a script which runs in parallel on all available processors. #!/bin/bash
mkdir -p mwe
texlua extract.lua text-en/ mwe/
cd mwe
ls -1q *.tex | sort | xargs -I @ -P `nproc` bash -c '
i="@"
str="Building $i"
if [ -f "${i%%.tex}.pdf" ]; then
str="$str Skipped."
else
lualatex -interaction=batchmode -halt-on-error "$i" > /dev/null
if [ "x$?" == "x1" ]; then
str="$str Failed!"
fi
fi
echo $str' |
|
Works like a charm. Thank you for that. The only downside is that now one doesn't have a clue about the progress. But I think one has to live with this minor issue. |
|
I could add a counter again but that would require some sort of locking mechanism because |
|
@Mo-Gul I needed this for something else, so I had a look at how to lock file descriptors in Bash. Here is a new script with counter. #!/bin/bash
mkdir -p mwe
texlua extract.lua text-en/ mwe/
cd mwe
touch /tmp/pgfmanuallock
echo 1 > /tmp/pgfmanualcount
ls -1q *.tex | sort | xargs -I @ -P `nproc` bash -c '
i="@"
{ flock -x 200;
n=$(cat /tmp/pgfmanualcount);
echo $((n+1)) > /tmp/pgfmanualcount;
} 200> /tmp/pgfmanuallock
str="Building $n/$(ls -1q *.tex | wc -l) $i"
if [ -f "${i%%.tex}.pdf" ]; then
str="$str Skipped."
else
lualatex -interaction=batchmode -halt-on-error "$i" > /dev/null
if [ "x$?" == "x1" ]; then
str="$str Failed!"
fi
fi
echo $str' |
|
I have also run the script to see how many examples currently fail even in the presence of all libraries and there are luckily only few (181 out of 2128).
|
|
@hmenke, great work (as always). It seems that the both keys pgf/doc/generic/pgf/text-en/pgfmanual-en-tutorial-chains.tex Lines 253 to 266 in 4c3ab05 (which corresponds to the extracted pgfmanual-en-tutorial-chains-4.tex) where the terminal style is no longer defined/repeated ...
The definitions of the styles are given at the beginning of the file pgf/doc/generic/pgf/text-en/pgfmanual-en-tutorial-chains.tex Lines 25 to 50 in 4c3ab05 Maybe you have an idea how to add grab/store them and then either state the needed styles or just all of them. But in the latter case there is needed some kind of a reset thus that not all following examples contain stuff that is not needed any more. To make it not too easy there is a need to grab several pgf/doc/generic/pgf/text-en/pgfmanual-en-tutorial-chains.tex Lines 336 to 338 in 4c3ab05 Also we need to grab definitions pgf/doc/generic/pgf/text-en/pgfmanual-en-tutorial-chains.tex Lines 542 to 573 in 4c3ab05 (which corresponds to the extracted pgfmanual-en-tutorial-chains-15.tex).
If my above suggestion cannot be implemented (easily), maybe there is a possibility to abuse the |
|
We also need something that allows us to load pgfmodules (e.g. for nonlinear transformations) unless we agree on changing the corresponding libraries in such a way that they always load the corresponding modules automatically. Whether or not we need the |
|
@tallmarmot, you are right. Just having a look at the manual you don't need to "catch" the |
|
Ok, for now I use the pgf/doc/generic/pgf/text-en/pgfmanual-en-tutorial-chains.tex Lines 676 to 679 in 5d17930 But of course then also in the extracted files the hashes are doubled, but need to be in single form. In all these cases I added comment lines on top, thus the user has an idea what he has to do to make the example work. Of course this doesn't help for the automatic testing. These lines start with Also I have found out that pgf/doc/generic/pgf/text-en/pgfmanual-en-xxcolor.tex Lines 36 to 50 in 5d17930 isn't extracted. When I remove the optional argument it is. Is this a bug in |
|
|
|
Fixed in 0a6f197 |
|
Following the request by @tallmarmot I have changed the way libraries are listed. I have removed the pgf/doc/generic/pgf/text-en/pgfmanual-en-base-decorations.tex Lines 27 to 29 in 36803f7 Right now the content of preamble is simply piped through \detokenize. I might add more elaborate formatting in the future but I think this approach is flexible enough for now. Remember, the main objective currently is to identify which libraries are required to make each example compile. Pretty printing in the manual is only secondary at this point.
|
|
@hmenke, it would be nice if you could add some more features
To 1: Background is that sometimes styles or definitions are only needed for a section and not for the whole chapter. Simply enclosing this will nonetheless add this collected stuff to a To 2: Sometimes there are simple examples to introduce some stuff before this is applied to the "real" example. Of course for these all the collected stuff is not needed and might confuse users when they see all these definitions. |
|
You have clearly not understood what this task is about. Let me make this clear: |
|
Hi Henri,
Leider bekomme ich
mratz@MacBook-Pro-2:~/Documents/GitHub/pgf/doc/generic/pgf\> ./build.sh
Usage: texlua extract.lua <source-dir> <target-dir>
./build.sh: line 8: nproc: command not found
xargs: max. processes must be >0
ls: */**/*.tex: No such file or directory
Was mache ich falsch?
Ciao,
Michael
…On Thu, Jul 4, 2019 at 12:43 AM Henri Menke ***@***.***> wrote:
I've updated the extractor script to also work with the graphdrawing
examples and to ignore remember picture. Here is an updated build script:
#!/bin/bash
mkdir -p mwe
texlua extract.lua text-en/ ../../../tex/generic/pgf/graphdrawing/lua/pgf/ mwe/cd mwe
touch /tmp/pgfmanuallockecho 1 > /tmp/pgfmanualcount
ls -1q *.tex */**/*.tex | sort | xargs -I @ -P `nproc` bash -c ' d=`dirname "@"`; f=`basename "@"`; { flock -x 200; n=$(cat /tmp/pgfmanualcount); echo $((n+1)) > /tmp/pgfmanualcount; } 200> /tmp/pgfmanuallock str="Building $n/$(ls -1q *.tex */**/*.tex | wc -l) $d/$f" cd $d; if [ -f "${f%%.tex}.pdf" ]; then str="$str Skipped." else lualatex -interaction=batchmode -halt-on-error "$f" > /dev/null if [ "x$?" == "x1" ]; then str="$str Failed!" fi fi echo $str'
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#640?email_source=notifications&email_token=AISROXABUTDMQCGCQTSZNKTP5UTQRA5CNFSM4HD7XP72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZF3PYQ#issuecomment-508278754>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AISROXHUTYNZXC5POIFKDXTP5UTQRANCNFSM4HD7XP7Q>
.
|
|
I see, macOS doesn't seem to have |
|
@hmenke, and for your information. Under Win7 That's why I am currently "just" running the solution with parallelization but without the counter (#640 (comment)). A "dummy" solution could be to clear help files (AUX, LOG), count all TEX files minus the number of PDF files to get the total number of files that need to be processed. Then counting the number of LOG files gives number of processed files ... |
Maybe I should have mentioned that I had modified |
|
For the record: I have written a PowerShell script myself to have parallel processing of the extracted # -----------------------------------------------------------------------------
# adapted from <https://hkeylocalmachine.com/?p=612>
# (2do: have a look at
# <https://devblogs.microsoft.com/scripting/weekend-scripter-a-look-at-the-poshrsjob-module/>
# maybe the module can help to simplify this script)
# -----------------------------------------------------------------------------
$DestFolder = "mwe"
### create `$DestFolder` if it doesn't exist already
If (-Not (Test-Path -Path $DestFolder -PathType Container )) {
New-Item -Path "$PSScriptRoot\" -Name $DestFolder -ItemType Directory
}
### extract `codeexample`s from the manual files
texlua extract.lua text-en/ ../../../tex/generic/pgf/graphdrawing/lua/pgf/ $DestFolder/
### TeX all extracted files in parallel
$Folder = $PSScriptRoot + "\$DestFolder"
$TexFiles = Get-ChildItem $Folder -Recurse -Include *.tex `
| Where {-Not (Test-Path -Path "$($_.Directory)\$($_.BaseName).pdf" -Pathtype Leaf)}
# Create Runspace Pool with `$MaxRunspaces` threads
[int]$MaxRunspaces = [int]$env:NUMBER_OF_PROCESSORS
$Pool = [RunspaceFactory]::CreateRunspacePool(1, $MaxRunspaces)
$Pool.ApartmentState = "MTA"
$Pool.Open()
$Runspaces = @()
# The script you want run against each host
$ScriptBlock = {
# Take a TeX file as a parameter
Param ($File);
$DirectoryName = $File.DirectoryName
$FileName = $File.Name
lualatex -interaction=batchmode -halt-on-error --output-directory="$DirectoryName" "$FileName"
}
# Loop through the files
ForEach ($TexFile In $TexFiles) {
$Runspace = [powershell]::Create()
# Add script block to runspace (use $Null to avoid noise)
$Null = $Runspace.AddScript($ScriptBlock)
# Add IP address as an argument to the scriptblock (use $Null to avoid noise)
$Null = $Runspace.AddArgument($TexFile)
# Add/create new runspace
$Runspace.RunspacePool = $Pool
$Runspaces += [pscustomobject]@{Pipe=$Runspace; Status=$Runspace.BeginInvoke() }
}
# -------------------------------------------------------------------------
# Prepare the progress bar
$CurrentCount = 0;
$TotalCount = ($Runspaces | Measure-Object).Count;
# Pause until all runspaces have completed
While ($Runspaces.Status -ne $Null)
{
$Completed = $Runspaces | Where { $_.Status.IsCompleted -eq $True };
# Update progress bar
$CurrentCount = $CurrentCount + ($Completed | Measure-Object).Count;
$StatusText = "Completed $CurrentCount/$TotalCount"
Write-Progress `
-Activity "TeXing files..." `
-Status $StatusText `
-PercentComplete (([int]$CurrentCount/[int]$TotalCount)*100);
# Clear completed runspaces
ForEach ($Runspace In $Completed)
{
$Runspace.Pipe.EndInvoke($Runspace.Status)
$Runspace.Status = $Null
}
}
# -------------------------------------------------------------------------
# Clean-up Runspace Pool
$Pool.Close();
$Pool.Dispose();
# recursively delete empty folders
# (from <https://www.powershelladmin.com/wiki/PowerShell_script_to_recursively_delete_empty_folders>)
Get-ChildItem -LiteralPath $Folder -Force -Recurse `
| Where-Object {
$_.PSIsContainer -and `
@(Get-ChildItem -LiteralPath $_.Fullname -Force -Recurse `
| Where { -not $_.PSIsContainer }).Count -eq 0 `
} `
| Remove-Item -Recurse |
|
@hmenke, it seems that there are even more (code)examples in the (Lua) documentation. Unfortunately they have another syntax. Here an example. pgf/tex/generic/pgf/graphdrawing/lua/pgf/gd/circular/doc.lua Lines 94 to 101 in a927d72 I guess you will easily be able to adapt |
|
@hmenke, I have worked through the I also searched myself for more |
|
Sorry for stealing your time and disappointing you. After the events at
texstackexchange last month my motivation to help the TeX community just
dropped below zero. As long as Stefan Kottwitz is around there is no way I
will be part of this. In any case, I cannot handle github so I was never of
any use.
Best,
M.
…On Mon, Jul 15, 2019 at 9:25 AM Mo-Gul ***@***.***> wrote:
@hmenke <https://github.com/hmenke>, I have worked through the codeexamples
and example(s) in the Lua part of the documentation as well now.
Regarding the examples which need to have a preamble option I have marked
with % TODOsp: codeexamples: and stated what is needed there.
I also searched myself for more examples and didn't find any more (that
are not commented), so I would say I am ready for a pull request. Shall I
do a pull request for the branch or first merge it to the master and then
do a pull request?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#640?email_source=notifications&email_token=AISROXBSZNFTFAEVEM25RWDP7SQGZA5CNFSM4HD7XP72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ6HCMQ#issuecomment-511471922>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AISROXHX5IFRN6RLTVEOQ2DP7SQGZANCNFSM4HD7XP7Q>
.
|
|
@Mo-Gul I've put some stuff in place so that you can add options to the Lua examples. A table of examples can be converted as such: - examples = {[["
- \tikz \graph [spring electrical layout, horizontal=0 to 1]
- { 0 [electric charge=1] -- subgraph C_n [n=10] };
- "]],[["
- \tikz \graph [spring electrical layout, horizontal=0 to 1]
- { 0 [electric charge=5] -- subgraph C_n [n=10] };
- "]],[["
- \tikz \graph [spring electrical layout, horizontal=0 to 1]
- { [clique] 1 [electric charge=5], 2, 3, 4 };
- "]]
- }
+ examples = {
+ {
+ options = [["preamble={\usetikzlibrary{graphs,graphdrawing} \usegdlibrary{force}}"]],
+ code = [["
+ \tikz \graph [spring electrical layout, horizontal=0 to 1]
+ { 0 [electric charge=1] -- subgraph C_n [n=10] };
+ "]]
+ },{
+ code = [["
+ \tikz \graph [spring electrical layout, horizontal=0 to 1]
+ { 0 [electric charge=5] -- subgraph C_n [n=10] };
+ "]]
+ },{
+ code = [["
+ \tikz \graph [spring electrical layout, horizontal=0 to 1]
+ { [clique] 1 [electric charge=5], 2, 3, 4 };
+ "]]
+ }
+ }Sometimes though there are single examples, which are given as a string. These have to be wrapped in two tables (note the double braces in the example): - examples = [["
- \tikz \graph [spring electrical layout, horizontal=0 to 1]
- { 0 [electric charge=1] -- subgraph C_n [n=10] };
- "]]
+ examples = {{
+ options = [["preamble={\usetikzlibrary{graphs,graphdrawing} \usegdlibrary{force}}"]],
+ code = [["
+ \tikz \graph [spring electrical layout, horizontal=0 to 1]
+ { 0 [electric charge=1] -- subgraph C_n [n=10] };
+ "]]
+ }}Please also disregard the code in |
|
Thank you for the additions. I'll have a look at them. But it seems that I am not able to test them, because
pgf/tex/generic/pgf/graphdrawing/lua/pgf/gd/force/ControlStart.lua Lines 24 to 29 in d96c3f2 is extracted as \documentclass{standalone}
\usepackage{fp,pgf,tikz,xcolor}
\usetikzlibrary{graphs,graphdrawing}
-- \usegdlibrary{force}
\begin{document}
-- \tikz \graph [spring layout, node distance=7mm] { subgraph C_n[n=3] };
-- \tikz \graph [spring layout] { subgraph C_n[n=3] };
-- \tikz \graph [spring layout, node distance=15mm]{ subgraph C_n[n=3] };
--
\end{document}Could you fix this as well, i.e. remove the leading |
|
For the record: From the files in https://github.com/pgf-tikz/pgf/tree/master/doc/generic/pgf/text-en the extracted
don't compile. |
|
@hmenke, applying the changes as you have stated in your comment above for multiple examples in one go show the
So either you see a possibility to state the examples = {
+ options = [["preamble={\usetikzlibrary{graphs,graphdrawing} \usegdlibrary{force}}"]],
{
- options = [["preamble={\usetikzlibrary{graphs,graphdrawing} \usegdlibrary{force}}"]],
code = [["
\tikz \graph [spring electrical layout, horizontal=0 to 1]
{ 0 [electric charge=1] -- subgraph C_n [n=10] };
"]]
},{
code = [["
\tikz \graph [spring electrical layout, horizontal=0 to 1]
{ 0 [electric charge=5] -- subgraph C_n [n=10] };
"]]
},{
code = [["
\tikz \graph [spring electrical layout, horizontal=0 to 1]
{ [clique] 1 [electric charge=5], 2, 3, 4 };
"]]
}
}for your given code above or I add the examples = {
{
options = [["preamble={\usetikzlibrary{graphs,graphdrawing} \usegdlibrary{force}}"]],
code = [["
\tikz \graph [spring electrical layout, horizontal=0 to 1]
{ 0 [electric charge=1] -- subgraph C_n [n=10] };
"]]
},{
+ options = [["preamble={\usetikzlibrary{graphs,graphdrawing} \usegdlibrary{force}}"]],
code = [["
\tikz \graph [spring electrical layout, horizontal=0 to 1]
{ 0 [electric charge=5] -- subgraph C_n [n=10] };
"]]
},{
+ options = [["preamble={\usetikzlibrary{graphs,graphdrawing} \usegdlibrary{force}}"]],
code = [["
\tikz \graph [spring electrical layout, horizontal=0 to 1]
{ [clique] 1 [electric charge=5], 2, 3, 4 };
"]]
}
}How shall I proceed? |
So far this is done as @hmenke suggested, i.e. the libraries are only shown for the first example if multiple examples are given.
|
Hello. The first example of phylogenetics with |
Migrated from SourceForge
Author: krichter722
Timestamp: 2018-07-03 19:40:40.026000
I'm aware that the following is a huge pile of work, however I hope that you devs don't close it immediately because of that. It's intended to make this awesome software easier to use for beginners and trained users which tend to forget or can't keep up with the latest major changes:
It'd be nice to have a link to a Short, Self Contained, Correct (Compilable), Example for every code sample so that if you click on a link to a code sample like
on page xy you're redirected to the plain text file https://sourceforge.net/p/pgf/wiki/samples/001.tex which contains a SSCCE which can be directly passed to the compiler.
The manual describes what the user needs to do in a satisfying way, starting with
However, it's sometimes hard to follow all the packages which have been introduced and settings which have been changed over the course of the chapter where the snippet is shown (which can be dozens of pages long). The plain text links would avoid this and reduce the number of mistakes can make while using the snippet almost to zero.
This would also improve the incapacity of most PDF viewers to copy code accurate, i.e. without introducing unwanted linebreaks and replacing spaces with linebreaks, even though that's not PGF's job.
If you think that's a good idea regardless of the effort, then you might want to keep those SSCCEs coming piece by piece, maybe through merge requests. I can provide some, however, I often find myself trying to get the snippets to compile for hours which is the main reason why I'm suggesting this.
The SSCCEs could be provided for LaTeX, TeX, ConTeX and LuaLaTeX.
The text was updated successfully, but these errors were encountered: