Skip to content

Commit

Permalink
Merge pull request #204 from IBM/plotting_refinements
Browse files Browse the repository at this point in the history 
Plotting refinements
  • Loading branch information
@ppalmes
ppalmes committed Apr 7, 2023
2 parents ca5e949 + b16a87d commit fe0206a
Show file tree
Hide file tree
Showing 11 changed files with 573 additions and 581 deletions.
6 changes: 3 additions & 3 deletions Project.toml
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,21 @@
name = "TSML"
uuid = "198dc43e-9e51-5cd7-9d40-d9794d335912"
authors = ["Paulito Palmes <ppalmes@gmail.com>"]
version = "2.7.4"
version = "2.7.5"

[deps]
AMLPipelineBase = "e3c3008a-8869-4d53-9f34-c96f99c8a2b6"
ArgParse = "c7e460c6-2fb9-53a9-8c5b-16f535851c63"
CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
GR = "28b8d3ca-fb5f-59d9-8090-bfdbd6d07a71"
Impute = "f7bf1975-0170-51b9-8c5f-a992d46b9575"
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
MLDataUtils = "cc2ba9b6-d476-5e6d-8eaf-a92d5412d41d"
MultivariateStats = "6f286f6a-111f-5878-ab1e-185364afe411"
Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
RecipesBase = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
Serialization = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
Expand All @@ -27,7 +28,6 @@ DataFrames = "0.17, 0.18, 0.19, 0.20, 0.21, 0.22, 1.0, 1.1, 1.2"
Impute = "0.5, 0.6"
MLDataUtils = "0.2, 0.3, 0.4, 0.5"
MultivariateStats = "0.5, 0.6, 0.7, 0.8, 0.9, 0.10"
RecipesBase = "1.1"
StatsBase = "0.29, 0.30, 0.31, 0.32, 0.33"
julia = "1"

Expand Down
2 changes: 1 addition & 1 deletion docs/notebooks/JuliaCon2019.jl.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -304,7 +304,7 @@
}
],
"source": [
"pltr=Plotter(Dict(:interactive => true))\n",
"pltr=Plotter(Dict(:pdfoutput => true))\n",
"\n",
"mypipeline = pltr\n",
"\n",
Expand Down
964 changes: 464 additions & 500 deletions docs/notebooks/StaticPlotting.jl.ipynb
View file

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/src/man/dateproc.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ Another way is to use the symbolic pipeline to
describe the transformation and concatenation in
just one line of expression.
```@example dateifier
ppl = @pipeline dtr + mtr
ppl = dtr + mtr
features = fit_transform!(ppl,X)
nothing #hide
```
Expand Down
54 changes: 32 additions & 22 deletions docs/src/tutorial/monotonic_plotting.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,12 @@ We can use the `Plotter` filter to visualize the generated data.
using TSML
Random.seed!(123)
pltr = Plotter(Dict(:interactive => false,:pdfoutput => false))
pltr = Plotter(Dict(:pdfoutput => false))
mdates = DateTime(2017,12,1,1):Dates.Hour(1):DateTime(2017,12,31,10) |> collect
mvals = rand(length(mdates)) |> cumsum
df = DataFrame(Date=mdates ,Value = mvals)
fit_transform!(pltr,df)
fit_transform!(pltr,df);
nothing #hide
```

Now that we have a monotonic data, let's use the `Monotonicer` to normalize and plot the result:
Expand All @@ -31,9 +32,10 @@ using TSML
mono = Monotonicer(Dict())
pipeline = @pipeline mono |> pltr
pipeline = mono |> pltr
res=fit_transform!(pipeline,df)
res=fit_transform!(pipeline,df);
nothing #hide
```

## Real Data Example
Expand Down Expand Up @@ -61,10 +63,10 @@ monofilecsv = CSVDateValReader(Dict(:filename=>monofile,:dateformat=>"dd/mm/yyyy
dailymonofilecsv = CSVDateValReader(Dict(:filename=>dailymonofile,:dateformat=>"dd/mm/yyyy HH:MM"))
valgator = DateValgator(Dict(:dateinterval=>Dates.Hour(1)))
valnner = DateValNNer(Dict(:dateinterval=>Dates.Hour(1)))
valnner = DateValLinearImputer(Dict(:dateinterval=>Dates.Hour(1)))
stfier = Statifier(Dict(:processmissing=>true))
mono = Monotonicer(Dict())
pltr = Plotter(Dict(:interactive => false))
pltr = Plotter(Dict(:pdfoutput => false))
nothing #hide
```

Expand All @@ -75,16 +77,18 @@ Let's test by feeding the regular time series type to the pipeline. We expect th

- Pipeline with `Monotonicer`: regular time series
```@example mono
pipeline = @pipeline regularfilecsv |> valgator |> valnner |> mono |> pltr
pipeline = regularfilecsv |> valgator |> valnner |> mono |> pltr
fit_transform!(pipeline)
fit_transform!(pipeline);
nothing #hide
```

- Pipeline without `Monotonicer`: regular time series
```@example mono
pipeline = @pipeline regularfilecsv |> valgator |> valnner |> pltr
pipeline = regularfilecsv |> valgator |> valnner |> pltr
fit_transform!(pipeline)
fit_transform!(pipeline);
nothing #hide
```

Notice that the plots are the same with or without the `Monotonicer` instance.
Expand All @@ -94,16 +98,18 @@ Let's now feed the same pipeline with a monotonic csv data.

- Pipeline without `Monotonicer`: monotonic time series
```@example mono
pipeline = @pipeline monofilecsv |> valgator |> valnner |> pltr
pipeline = monofilecsv |> valgator |> valnner |> pltr
fit_transform!(pipeline)
fit_transform!(pipeline);
nothing #hide
```

- Pipeline with `Monotonicer`: monotonic time series
```@example mono
pipeline = @pipeline monofilecsv |> valgator |> valnner |> mono |> pltr
pipeline = monofilecsv |> valgator |> valnner |> mono |> pltr
fit_transform!(pipeline)
fit_transform!(pipeline);
nothing #hide
```

Notice that without the `Monotonicer` instance, the data is monotonic. Applying
Expand All @@ -118,17 +124,19 @@ We can use the `Outliernicer` filter to remove outliers. Let's apply this filter
using TSML: Outliernicer
outliernicer = Outliernicer(Dict(:dateinterval=>Dates.Hour(1)));
pipeline = @pipeline monofilecsv |> valgator |> valnner |> mono |> outliernicer |> pltr
fit_transform!(pipeline)
pipeline = monofilecsv |> valgator |> valnner |> mono |> outliernicer |> pltr
fit_transform!(pipeline);
nothing #hide
```

## Daily Monotonic TS Processing
Lastly, let's feed the daily monotonic data using similar pipeline and examine its plot.

- Pipeline without `Monotonicer`: daily monotonic time series
```@example mono
pipeline = @pipeline dailymonofilecsv |> valgator |> valnner |> pltr
fit_transform!(pipeline)
pipeline = dailymonofilecsv |> valgator |> valnner |> pltr
fit_transform!(pipeline);
nothing #hide
```

This plot is characterized by monotonically increasing trend but resets to certain baseline value
Expand All @@ -138,8 +146,9 @@ the correct normalization.

- Pipeline with `Monotonicer`: daily monotonic time series
```@example mono
pipeline = @pipeline dailymonofilecsv |> valgator |> valnner |> mono |> pltr
fit_transform!(pipeline)
pipeline = dailymonofilecsv |> valgator |> valnner |> mono |> pltr
fit_transform!(pipeline);
nothing #hide
```

While the `Monotonicer` filter is able to transform the data into a regular time series,
Expand All @@ -149,8 +158,9 @@ Let's remove the outliers by applying the `Outliernicer` filter and examine the

- Pipeline with `Monotonicer` and `Outliernicer`: daily monotonic time series
```@example mono
pipeline = @pipeline dailymonofilecsv |> valgator |> valnner |> mono |> outliernicer |> pltr
fit_transform!(pipeline)
pipeline = dailymonofilecsv |> valgator |> valnner |> mono |> outliernicer |> pltr
fit_transform!(pipeline);
nothing #hide
```

The `Outliernicer` filter effectively removed the outliers as shown in the plot.
52 changes: 31 additions & 21 deletions docs/src/tutorial/monotonic_plotting_pdf.md → docs/src/tutorial/monotonic_plotting_pdf.md1
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,12 @@ We can use the `Plotter` filter to visualize the generated data.
using TSML

Random.seed!(123)
pltr = Plotter(Dict(:interactive => false,:pdfoutput => true))
pltr = Plotter(Dict(:pdfoutput => true))
mdates = DateTime(2017,12,1,1):Dates.Hour(1):DateTime(2017,12,31,10) |> collect
mvals = rand(length(mdates)) |> cumsum
df = DataFrame(Date=mdates ,Value = mvals)
fit_transform!(pltr,df)
fit_transform!(pltr,df);
nothing #hide
```

Now that we have a monotonic data, let's use the `Monotonicer` to normalize and plot the result:
Expand All @@ -31,9 +32,10 @@ using TSML

mono = Monotonicer(Dict())

pipeline = @pipeline mono |> pltr
pipeline = mono |> pltr

res=fit_transform!(pipeline,df)
nothing #hide
```

## Real Data Example
Expand Down Expand Up @@ -61,10 +63,10 @@ monofilecsv = CSVDateValReader(Dict(:filename=>monofile,:dateformat=>"dd/mm/yyyy
dailymonofilecsv = CSVDateValReader(Dict(:filename=>dailymonofile,:dateformat=>"dd/mm/yyyy HH:MM"))

valgator = DateValgator(Dict(:dateinterval=>Dates.Hour(1)))
valnner = DateValNNer(Dict(:dateinterval=>Dates.Hour(1)))
valnner = DateValLinearImputer(Dict(:dateinterval=>Dates.Hour(1)))
stfier = Statifier(Dict(:processmissing=>true))
mono = Monotonicer(Dict())
pltr = Plotter(Dict(:interactive => false))
pltr = Plotter(Dict(:pdfoutput => true))
nothing #hide
```

Expand All @@ -75,16 +77,18 @@ Let's test by feeding the regular time series type to the pipeline. We expect th

- Pipeline with `Monotonicer`: regular time series
```@example mono
pipeline = @pipeline regularfilecsv |> valgator |> valnner |> mono |> pltr
pipeline = regularfilecsv |> valgator |> valnner |> mono |> pltr

fit_transform!(pipeline)
fit_transform!(pipeline);
nothing #hide
```

- Pipeline without `Monotonicer`: regular time series
```@example mono
pipeline = @pipeline regularfilecsv |> valgator |> valnner |> pltr
pipeline = regularfilecsv |> valgator |> valnner |> pltr

fit_transform!(pipeline)
fit_transform!(pipeline);
nothing #hide
```

Notice that the plots are the same with or without the `Monotonicer` instance.
Expand All @@ -94,16 +98,18 @@ Let's now feed the same pipeline with a monotonic csv data.

- Pipeline without `Monotonicer`: monotonic time series
```@example mono
pipeline = @pipeline monofilecsv |> valgator |> valnner |> pltr
pipeline = monofilecsv |> valgator |> valnner |> pltr

fit_transform!(pipeline)
fit_transform!(pipeline);
nothing #hide
```

- Pipeline with `Monotonicer`: monotonic time series
```@example mono
pipeline = @pipeline monofilecsv |> valgator |> valnner |> mono |> pltr
pipeline = monofilecsv |> valgator |> valnner |> mono |> pltr

fit_transform!(pipeline)
fit_transform!(pipeline);
nothing #hide
```

Notice that without the `Monotonicer` instance, the data is monotonic. Applying
Expand All @@ -118,17 +124,19 @@ We can use the `Outliernicer` filter to remove outliers. Let's apply this filter
using TSML: Outliernicer
outliernicer = Outliernicer(Dict(:dateinterval=>Dates.Hour(1)));

pipeline = @pipeline monofilecsv |> valgator |> valnner |> mono |> outliernicer |> pltr
fit_transform!(pipeline)
pipeline = monofilecsv |> valgator |> valnner |> mono |> outliernicer |> pltr
fit_transform!(pipeline);
nothing #hide
```

## Daily Monotonic TS Processing
Lastly, let's feed the daily monotonic data using similar pipeline and examine its plot.

- Pipeline without `Monotonicer`: daily monotonic time series
```@example mono
pipeline = @pipeline dailymonofilecsv |> valgator |> valnner |> pltr
fit_transform!(pipeline)
pipeline = dailymonofilecsv |> valgator |> valnner |> pltr
fit_transform!(pipeline);
nothing #hide
```

This plot is characterized by monotonically increasing trend but resets to certain baseline value
Expand All @@ -138,8 +146,9 @@ the correct normalization.

- Pipeline with `Monotonicer`: daily monotonic time series
```@example mono
pipeline = @pipeline dailymonofilecsv |> valgator |> valnner |> mono |> pltr
fit_transform!(pipeline)
pipeline = dailymonofilecsv |> valgator |> valnner |> mono |> pltr
fit_transform!(pipeline);
nothing #hide
```

While the `Monotonicer` filter is able to transform the data into a regular time series,
Expand All @@ -149,8 +158,9 @@ Let's remove the outliers by applying the `Outliernicer` filter and examine the

- Pipeline with `Monotonicer` and `Outliernicer`: daily monotonic time series
```@example mono
pipeline = @pipeline dailymonofilecsv |> valgator |> valnner |> mono |> outliernicer |> pltr
fit_transform!(pipeline)
pipeline = dailymonofilecsv |> valgator |> valnner |> mono |> outliernicer |> pltr
fit_transform!(pipeline);
nothing #hide
```

The `Outliernicer` filter effectively removed the outliers as shown in the plot.
6 changes: 3 additions & 3 deletions docs/src/tutorial/pipeline.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ Let's use the pipeline transformer to aggregate and impute:
using TSML
dtvalgator = DateValgator(Dict(:dateinterval => Dates.Hour(1)))
dtvalnner = DateValNNer(Dict(:dateinterval => Dates.Hour(1)))
dtvalnner = DateValLinearImputer(Dict(:dateinterval => Dates.Hour(1)))
mypipeline = @pipeline dtvalgator |> dtvalnner
mypipeline = dtvalgator |> dtvalnner
results = fit_transform!(mypipeline,X)
nothing #hide
Expand Down Expand Up @@ -120,7 +120,7 @@ and process it by aggregation and imputation.


```@example pipeline
mypipeline = @pipeline csvreader |> dtvalgator |> dtvalnner
mypipeline = csvreader |> dtvalgator |> dtvalnner
results = fit_transform!(mypipeline)
nothing #hide
Expand Down
8 changes: 4 additions & 4 deletions docs/src/tutorial/statistics.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ dtvalnner = DateValNNer(Dict(:dateinterval => Dates.Hour(1)))
dtvalizer = DateValizer(Dict(:dateinterval => Dates.Hour(1)))
stfier = Statifier(Dict(:processmissing => true))
mypipeline = @pipeline dtvalgator |> stfier
mypipeline = dtvalgator |> stfier
results = fit_transform!(mypipeline,X)
nothing #hide
Expand All @@ -70,7 +70,7 @@ blocks stat summary by indicating `:processmissing => false` in the instance arg
```@example stat
stfier = Statifier(Dict(:processmissing=>false))
mypipeline = @pipeline dtvalgator |> stfier
mypipeline = dtvalgator |> stfier
results = fit_transform!(mypipeline,X)
nothing #hide
Expand All @@ -89,7 +89,7 @@ the stats for missing blocks will all be NaN because stats of empty set is an Na
```@example stat
stfier = Statifier(Dict(:processmissing=>true))
mypipeline = @pipeline dtvalgator |> dtvalnner |> stfier
mypipeline = dtvalgator |> dtvalnner |> stfier
results = fit_transform!(mypipeline,X)
nothing #hide
Expand All @@ -108,7 +108,7 @@ missing values based on the stats.
```@example stat
stfier = Statifier(Dict(:processmissing=>true))
mypipeline = @pipeline dtvalgator |> dtvalizer |> stfier
mypipeline = dtvalgator |> dtvalizer |> stfier
results = fit_transform!(mypipeline,X)
nothing #hide
Expand Down
2 changes: 1 addition & 1 deletion docs/src/tutorial/tsclassifier.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ mdirname = tscl.model[:modeldirectory]
modelfname=tscl.model[:juliarfmodelname]
trstatfname = joinpath(mdirname,modelfname*".csv")
res = CSV.read(trstatfname) |> DataFrame
res = CSV.read(trstatfname,DataFrame)
nothing #hide
```

Expand Down

2 comments on commit fe0206a

@ppalmes
Copy link
Collaborator Author

@ppalmes ppalmes commented on fe0206a Apr 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JuliaRegistrator register

@JuliaRegistrator
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Registration pull request created: JuliaRegistries/General/81209

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a v2.7.5 -m "<description of version>" fe0206ae2db3fccd57fd74f22ae3ae79917cfb31
git push origin v2.7.5

Please sign in to comment.