EnumerableSelectIterator invokes selector in MoveNext()

[ActivistInvestor]

re: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Linq/src/System/Linq/Select.cs

I understand why invoking the selector function is necessary in iterators that use predicates. However, I must be missing the reason why this is necessary in the case of Select(). If Count() or Skip() is used on such an iterator, the result of the selector function is repeatedly assigned to a member field, and tossed on each subsequent call to MoveNext().

Wouldn't a more-lazy pattern that only sets a flag in MoveNext(), telling the getter for Current that it must invoke the selector be more-efficient?

A conceptual distillation (sans error checks):

public class LazySelectEnumerableIterator<TInput, TOutput> : IEnumerator<TOutput>
{
   bool _isvalid = false;
   Func<TInput, TOutput> _selector;
   IEnumerable<TInput> _source;
   IEnumerator<TInput> _enumerator;
   TOutput _current;

   public LazySelectEnumerableIterator(IEnumerable<TInput> source, Func<TInput, TOutput> selector)
   {
      this._source = source;
      this._selector = selector;
   }

   public bool MoveNext()
   {
      if(_enumerator == null)
         _enumerator = _source.GetEnumerator();
      _isvalid = false;
      return _enumerator.MoveNext();
   }

   public TOutput Current
   {
      get
      {
         if(!_isvalid)
         {
            _current = _selector(_enumerator.Current);
            _isvalid = true;
         }
         return _current;
      }
   }

   /// balance omitted for brevity
}

[stephentoub]

Can you point to the existing line(s) of code you're asking about?

[ActivistInvestor]

Sorry, the title I gave this topic is a bit misleading (the correct class name is EnumerableSelectIterator) and the method MoveNext(), is from Select.cs, starting at line 121.

I think @bartonjs offered a reasonable explanation as to why things are the way they are, even if the examples are edge-case scenarios.

[bartonjs]

If the selector function throws, or has other side effects, those should be observed by Count(); etc.

e.g.

private static int s_count = 0;

...

var enumerable = data.Select(e => { s_count++; return e; });
enumerable.Count();

foreach (var item in enumerable)
{
    ...
}

Changing the implementation to only invoke the selector when the item is being extracted would change the value of s_count produced by the above code. Furthermore, it would be quite surprising if

A)

private static int Count<T>(IEnumerable<T> data)
{
    IEnumerable<T> enumerable = data.Select(selector);
    int count = 0;
    IEnumerator<T> iterator = enumerable.GetEnumerator();

    while (iterator.MoveNext()) { count++; }

    return count;
}

and B)

private static int Count<T>(IEnumerable<T> data)
{
    IEnumerable<T> enumerable = data.Select(selector);
    int count = 0;

    foreach (T item in enumerable) { count++; }

    return count;
}

invoked the selector a different number of times.

So, while it could improve performance; changing the number of times that a callback is called is a complicated breaking change (in the event of side effects) that we try really hard to avoid.

[ActivistInvestor]

Ah, side-effects. Well, that's certainly makes a case for it being the way it is.

While on the topic of side-effects of delegates:

/// SelectSpeedOpt.cs[495]

/// IListSkipTakeIterator.GetCount()


public override int GetCount(bool onlyIfCheap)
{
    // In case someone uses Count() to force evaluation of
    // the selector, run it provided `onlyIfCheap` is false.

    int count = _source.Count;

    if (!onlyIfCheap)
    {
        for (int i = 0; i < count; i++)
        {
            _selector(_source[i]);
        }
    }

    return count;
}

Is it not true that the selector function used in the above code could have side-effects that could alter the number of elements in the source?

[stephentoub]

It could, but that would be considered undefined behavior, eg if the source were a List it might throw.

[ActivistInvestor]

Typically, I would write a for() loop so that it reads the List's Count property on each iteration, but I notice in the above code the count is read into a variable and that's the number of iterations.

So, what I'm now trying to understand is the difference between the above case, and the case of the original topic (a selector called by MoveNext() that has material side-effects).

Thanks.

[stephentoub]

Mutating the source of a LINQ query while executing that query is undefined behavior and will likely blow up, behave unexpectedly, miss data, re-evaluate data, etc. You can mutate other things (even though doing so is considered an anti-pattern), but not the very thing you're in the process of iterating. That's the difference.