Introduce a TestSummary #53

hazard999 · 2020-07-30T09:03:43Z

Fixes #51

…9/Tasty into topic/report-helpers-51

…hazard999/Tasty into topic/report-helpers-51" This reverts commit 9c60366, reversing changes made to 8629308.

biohazard999 · 2020-07-30T09:14:16Z

src/Xenial.Tasty/Reporters/SummaryReporter.cs

+                result[outcome.Key] = new TestResult(outcome.Value.Count, outcome.Value.Sum(t => t.Duration));
+            }
+
+            OutCome = tests.Where(t => t.TestOutcome > TestOutcome.Ignored).MinOrDefault(t => t.TestOutcome); ;


Casing:
OutCome -> Outcome
testsPerOutCome -> testsPerOutcome

biohazard999 · 2020-07-30T09:15:20Z

src/Xenial.Tasty/Execution/TestRuntime/ReportSummaryMiddleware.cs

@@ -17,11 +16,25 @@ public static TestExecutor UseSummaryReporters(this TestExecutor executor)
                }
                finally
                {
-                    var cases = context.Scope.Descendants().OfType<TestCase>().ToList();
+                    async Task<IEnumerable<TestSummary>> Summaries()


Should we use an IAsyncEnumerable here?

Possible, but I couldn't make it work properly

public static class ReportSummaryMiddleware { public static TestExecutor UseSummaryReporters(this TestExecutor executor) => executor.UseRuntime(async (context, next) => { try { await next(); } finally { async IAsyncEnumerable<TestSummary> Summaries() { foreach (var provider in context.Scope.SummaryProviders) { yield return await provider.Invoke(); } } await Task.WhenAll(context.Scope.SummaryReporters .Select(async r => { await foreach (var summary in Summaries()) { await r.Invoke(summary); } }).ToArray()); } }); }

biohazard999 · 2020-07-30T09:16:39Z

src/Xenial.Tasty/Remote/NamedPipeRemoteHook.cs

-            Task SummaryReporter(IEnumerable<Metadata.TestCase> @cases)
-                => remote.Report(@cases.Select(test => MapToSerializableTestCase(test)));
+            Task SummaryReporter(TestSummary summary)
+                => remote.Report(summary.Select(test => MapToSerializableTestCase(test)));


I guess we change the signature of remote summary reporters in the future?

In this form remote reporters still do their stuff on their own and won't get any profit for now, cause it's IEnumerable<SerializableTestCase> over the wire. So we can change that to an SerializableTestSummary in the future.

biohazard999 · 2020-07-30T09:18:37Z

src/Xenial.Tasty/Reporters/SummaryReporter.cs

+
+namespace Xenial.Delicious.Reporters
+{
+    public static class SummaryReporter


I'm not sure about this type. The pipeline only pumps trough collected tests to the original summary reporters anyway.

So it would be something like -> AsyncTestReporters -> All Tests done in the pipeline -> AsyncSummaryCollectors? -> AsyncSummaryReporters? in the pipeline? Sounds like something we should pack into the context stages in middleware

biohazard999 · 2020-07-30T09:21:09Z

src/Xenial.Tasty/Reporters/SummaryReporter.cs

+
+            testsPerOutCome = tests.GroupBy(t => t.TestOutcome).ToDictionary(t => t.Key, t => t.ToList());
+
+            foreach (var outcome in Enum.GetValues(typeof(TestOutcome)).OfType<TestOutcome>())


We can simplify this by doing an earlier init. Move it up and do a select to dictionary on all enum values. Then we can gurantee that the dictionary is initialized correctly.

biohazard999 · 2020-07-30T09:22:15Z

src/Xenial.Tasty/Reporters/SummaryReporter.cs

+                }
+            }
+
+            result = new Dictionary<TestOutcome, TestResult>();


That also makes this into a simple linq to dictionary

biohazard999 · 2020-07-30T09:23:50Z

src/Xenial.Tasty/Reporters/SummaryReporter.cs

+        private readonly Dictionary<TestOutcome, List<TestCase>> testsPerOutCome;
+        private readonly Dictionary<TestOutcome, TestResult> result;
+
+        public TestSummary(IEnumerable<TestCase> tests)


I am not 100% happy with putting so much logic into the ctor, we should move that to a static factory method and do a normal value assign public ctor

src/Xenial.Tasty/Tasty.Scope.cs

ghost · 2020-08-18T21:25:44Z

src/Xenial.Tasty/Reporters/SummaryReporter.cs

+
+        public int Count => tests.Count;
+
+        public TestOutcome OutCome { get; }


OutCome->Outcome

ghost · 2020-08-18T21:31:55Z

src/Xenial.Tasty/Reporters/Reporters.NyanCat.cs

            if (failedCount > 0) { Console.WriteLine($"\t{ColorScheme.Default.ErrorIcon} {failedCount} failed"); }

            Console.ForegroundColor = ColorScheme.Default.SuccessColor;
-            if (tests.Count() == successCount) { Console.WriteLine($"\t{ColorScheme.Default.SuccessIcon} All tests passed"); }
+            if (summary.Count() == successCount) { Console.WriteLine($"\t{ColorScheme.Default.SuccessIcon} All tests passed"); }



Total count was counted at line 80. Maybe save and reuse?

ghost · 2020-08-18T22:16:26Z

src/Xenial.Tasty/Reporters/Reporters.NyanCat.cs

        {
            Console.WriteLine();
            Console.ForegroundColor = ColorScheme.Default.DefaultColor;
-            Console.WriteLine($"\t{tests.Count()} total ({(int)Math.Round(TimeSpan.FromTicks(tests.Sum(t => t.Duration.Ticks)).TotalSeconds, 0, MidpointRounding.AwayFromZero)} s)");
+            Console.WriteLine($"\t{summary.Count()} total ({(int)Math.Round(TimeSpan.FromTicks(summary.Sum(t => t.Duration.Ticks)).TotalSeconds, 0, MidpointRounding.AwayFromZero)} s)");


@biohazard999 Thoughts on potentially supporting pretty timespans for summary output? For example here you count seconds, but what if all results are a fraction. It could output XX ms or XX ns depending on the lowest value or some preset smallest unit. Something like this could help: https://humanizr.net/#humanize-timespan

A good example of this is BenchmarkDotNet summaries:

Yeah I know humanize, the problem is, they don't support short notation yet. So we have to write it our selfs.

hazard999 · 2020-08-24T11:20:24Z

Diverged to much from master, needs rework

hazard999 added 4 commits July 30, 2020 10:51

SummaryReporter

9e5691d

SummaryReporter

8629308

Merge branch 'topic/report-helpers-51' of https://github.com/hazard99…

9c60366

…9/Tasty into topic/report-helpers-51

Revert "Merge branch 'topic/report-helpers-51' of https://github.com/…

3349a9c

…hazard999/Tasty into topic/report-helpers-51" This reverts commit 9c60366, reversing changes made to 8629308.

biohazard999 reviewed Jul 30, 2020

View reviewed changes

src/Xenial.Tasty/Tasty.Scope.cs Show resolved Hide resolved

style: whitespace

5bb812a

ghost reviewed Aug 18, 2020

View reviewed changes

src/Xenial.Tasty/Reporters/SummaryReporter.cs

public int Count => tests.Count;

public TestOutcome OutCome { get; }

Copy link

ghost Aug 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OutCome->Outcome

ghost reviewed Aug 18, 2020

View reviewed changes

hazard999 closed this Aug 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce a TestSummary #53

Introduce a TestSummary #53

hazard999 commented Jul 30, 2020 •

edited

biohazard999 Jul 30, 2020

ghost Jul 30, 2020

biohazard999 Jul 30, 2020

hazard999 Jul 30, 2020

biohazard999 Jul 30, 2020

biohazard999 Jul 30, 2020

hazard999 Jul 30, 2020

biohazard999 Jul 30, 2020

biohazard999 Jul 30, 2020

biohazard999 Jul 30, 2020 •

edited

biohazard999 Jul 30, 2020

biohazard999 Jul 30, 2020

biohazard999 Jul 30, 2020

ghost Aug 18, 2020

ghost Aug 18, 2020

ghost Aug 18, 2020 •

edited by ghost

ghost Aug 18, 2020

biohazard999 Aug 19, 2020

hazard999 commented Aug 24, 2020


		testsPerOutCome = tests.GroupBy(t => t.TestOutcome).ToDictionary(t => t.Key, t => t.ToList());

		foreach (var outcome in Enum.GetValues(typeof(TestOutcome)).OfType<TestOutcome>())


		public int Count => tests.Count;

		public TestOutcome OutCome { get; }

Introduce a TestSummary #53

Introduce a TestSummary #53

Conversation

hazard999 commented Jul 30, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

biohazard999 Jul 30, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost Aug 18, 2020 • edited by ghost

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hazard999 commented Aug 24, 2020

hazard999 commented Jul 30, 2020 •

edited

biohazard999 Jul 30, 2020 •

edited

ghost Aug 18, 2020 •

edited by ghost