Skip to content

make Iterators.take and Iterators.drop return views for vector input#61234

Closed
nsajko wants to merge 7 commits intoJuliaLang:masterfrom
nsajko:arrays-Iterators-take-drop-view
Closed

make Iterators.take and Iterators.drop return views for vector input#61234
nsajko wants to merge 7 commits intoJuliaLang:masterfrom
nsajko:arrays-Iterators-take-drop-view

Conversation

@nsajko
Copy link
Copy Markdown
Member

@nsajko nsajko commented Mar 4, 2026

Improvements:

  • Make Iterators.take and Iterators.drop preserve the Base.Fix2(isa, AbstractVector) property of iterators.

  • For Iterators.drop: improved iteration performance.

@nsajko nsajko added performance Must go faster arrays [a, r, r, a, y, s] iteration Involves iteration or the iteration protocol labels Mar 4, 2026
@jakobnissen

This comment was marked as resolved.

@nsajko nsajko force-pushed the arrays-Iterators-take-drop-view branch from 06a2c4c to 58c3d8c Compare March 5, 2026 07:49
@nsajko

This comment was marked as resolved.

@nsajko nsajko force-pushed the arrays-Iterators-take-drop-view branch from 58c3d8c to 8e410f5 Compare March 5, 2026 08:15
Improvements:

* Make `Iterators.take` and `Iterators.drop` preserve the
  `Base.Fix2(isa, AbstractVector)` property of iterators.

* For `Iterators.drop`: improved iteration performance.
@nsajko nsajko force-pushed the arrays-Iterators-take-drop-view branch from 8e410f5 to 4a7bdec Compare March 5, 2026 09:25
@adienes adienes added the triage This should be discussed on a triage call label Mar 5, 2026
@nsajko
Copy link
Copy Markdown
Member Author

nsajko commented Mar 5, 2026

Test failures:

  • Throws, but is not supposed to throw:

    @test (@inferred length(drop(StepRangeLen(1, 1, UInt(2)), 3))) == 0

    • The round here throws:

      offset = L(max(min(1 + round(L, (r.offset - first(s))/sstep), last(ind)), first(ind)))

    • Seems like this may just be a pre-existing bug in view. Have yet to investigate.

  • Three Base.show_delim_array tests fail here:

    julia/test/show.jl

    Lines 2380 to 2388 in 3ff8e7b

    @testset "show_delim_array" begin
    sdastr(f, n) = # sda: Show Delim Array
    sprint((io, x) -> Base.show_delim_array(io, x, "[", ",", "]", false, f, n), Iterators.take(1:f+n, f+n))
    @test sdastr(1, 0) == "[1]"
    @test sdastr(1, 1) == "[1]"
    @test sdastr(1, 2) == "[1, 2]"
    @test sdastr(2, 2) == "[2, 3]"
    @test sdastr(3, 3) == "[3, 4, 5]"
    end

    • Base.show_delim_array has different semantics for AbstractArray than for other iterators, regarding the f argument and the n argument. That feels like a (pre-existing) bug. Have to investigate more. Source:

      julia/base/show.jl

      Lines 1433 to 1497 in 3ff8e7b

      function show_delim_array(io::IO, itr::Union{AbstractArray,SimpleVector}, op, delim, cl,
      delim_one, i1=first(LinearIndices(itr)), l=last(LinearIndices(itr)))
      print(io, op)
      if !show_circular(io, itr)
      recur_io = IOContext(io, :SHOWN_SET => itr)
      first = true
      i = i1
      if l >= i1
      while true
      if !isassigned(itr, i)
      print(io, undef_ref_str)
      else
      x = itr[i]
      show(recur_io, x)
      end
      if i == l
      delim_one && first && print(io, delim)
      break
      end
      i += 1
      first = false
      print(io, delim)
      print(io, ' ')
      end
      end
      end
      print(io, cl)
      end
      function show_delim_array(io::IO, itr, op, delim, cl, delim_one, i1=1, n=typemax(Int))
      print(io, op)
      if !show_circular(io, itr)
      recur_io = IOContext(io, :SHOWN_SET => itr)
      y = iterate(itr)
      first = true
      i0 = i1-1
      while i1 > 1 && y !== nothing
      y = iterate(itr, y[2])
      i1 -= 1
      end
      if y !== nothing
      typeinfo = get(io, :typeinfo, Any)
      while true
      x = y[1]
      y = iterate(itr, y[2])
      show(IOContext(recur_io, :typeinfo => itr isa typeinfo <: Tuple ?
      fieldtype(typeinfo, i1+i0) :
      typeinfo),
      x)
      i1 += 1
      if y === nothing || i1 > n
      delim_one && first && print(io, delim)
      break
      end
      first = false
      print(io, delim)
      print(io, ' ')
      end
      end
      end
      print(io, cl)
      end
      show(io::IO, t::Tuple) = show_delim_array(io, t, '(', ',', ')', true)
      show(io::IO, v::SimpleVector) = show_delim_array(io, v, "svec(", ',', ')', false)
  • A test fails here:

    julia/test/iterators.jl

    Lines 209 to 221 in a25da50

    # double take
    # and take/drop canonicalization
    # -----------
    for xs in Any["abc", [1, 2, 3]]
    @test take(take(xs, 2), 3) === take(xs, 2)
    @test take(take(xs, 4), 2) === take(xs, 2)
    @test drop(drop(xs, 1), 1) === drop(xs, 2)
    @test take(drop(xs, 1), 1) === drop(take(xs, 2), 1)
    @test take(drop(xs, 3), 0) === drop(take(xs, 2), 3)
    @test @inferred isempty(drop(drop(xs, 2), 2))
    @test drop(take(drop(xs, 1), 2), 1) === take(drop(xs, 2), 1)
    @test take(drop(take(xs, 3), 1), 1) === take(drop(xs, 1), 1)
    end

    • Two iterators (ranges) compare nonidentical, even though they are both empty. I suppose the test could just be relaxed. Have to investigate more.

@adienes
Copy link
Copy Markdown
Member

adienes commented Mar 5, 2026

Make Iterators.take and Iterators.drop preserve the Base.Fix2(isa, AbstractVector) property of iterators.

I don't understand this statements as this is not any kind of consistent property AFAICT

performance wins could be a decent argument

@nsajko
Copy link
Copy Markdown
Member Author

nsajko commented Mar 5, 2026

I don't understand this statements as this is not any kind of consistent property AFAICT

What I wanted to say is that keeping AbstractVector as AbstractVector (SubArray) is useful because AbstractVector has nice properties taken advantage of in the ecosystem and not shared by other iterators. Indexing, for example.

@nsajko
Copy link
Copy Markdown
Member Author

nsajko commented Mar 5, 2026

performance wins could be a decent argument

Benchmark script:

using BenchmarkTools
function last_alt(i)
    len = length(i)
    j = Iterators.drop(i, len - 1)
    only(j)
end
@btime last_alt(v) evals=200 samples=10000 seconds=Inf setup=(v = rand(Int8, 5000);)

Results:

[nsajko@aceramd jl]$ ./julia-3ff8e7baa5/bin/julia /tmp/b.jl 
  1.295 μs (0 allocations: 0 bytes)
[nsajko@aceramd jl]$ ./julia-3ff8e7baa5/bin/julia /tmp/b.jl 
  1.295 μs (0 allocations: 0 bytes)
[nsajko@aceramd jl]$ ./julia-3ff8e7baa5/bin/julia /tmp/b.jl 
  1.295 μs (0 allocations: 0 bytes)
[nsajko@aceramd jl]$ ./julia-3ff8e7baa5/bin/julia /tmp/b.jl 
  1.295 μs (0 allocations: 0 bytes)
[nsajko@aceramd jl]$ ./julia-c1fb08a2c8/bin/julia /tmp/b.jl 
  3.705 ns (0 allocations: 0 bytes)
[nsajko@aceramd jl]$ ./julia-c1fb08a2c8/bin/julia /tmp/b.jl 
  3.705 ns (0 allocations: 0 bytes)
[nsajko@aceramd jl]$ ./julia-c1fb08a2c8/bin/julia /tmp/b.jl 
  3.755 ns (0 allocations: 0 bytes)
[nsajko@aceramd jl]$ ./julia-c1fb08a2c8/bin/julia /tmp/b.jl 
  3.705 ns (0 allocations: 0 bytes)
[nsajko@aceramd jl]$ 
[nsajko@aceramd jl]$ ./julia-3ff8e7baa5/bin/julia -e 'using InteractiveUtils; versioninfo()'
Julia Version 1.14.0-DEV.1852
Commit 3ff8e7baa50 (2026-03-05 11:43 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 8 × AMD Ryzen 3 5300U with Radeon Graphics
  WORD_SIZE: 64
  LLVM: libLLVM-20.1.8 (ORCJIT, znver2)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 8 virtual cores)
Environment:
  JULIA_NUM_PRECOMPILE_TASKS = 2
  JULIA_PKG_PRECOMPILE_AUTO = 0
[nsajko@aceramd jl]$ ./julia-c1fb08a2c8/bin/julia -e 'using InteractiveUtils; versioninfo()'
Julia Version 1.14.0-DEV.1859
Commit c1fb08a2c8d (2026-03-05 12:41 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 8 × AMD Ryzen 3 5300U with Radeon Graphics
  WORD_SIZE: 64
  LLVM: libLLVM-20.1.8 (ORCJIT, znver2)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 8 virtual cores)
Environment:
  JULIA_NUM_PRECOMPILE_TASKS = 2
  JULIA_PKG_PRECOMPILE_AUTO = 0

Interpretation: huge speedup. Presumably because this PR turns a $O(n)$ operation into a $O(1)$ operation.

@JeffBezanson
Copy link
Copy Markdown
Member

From triage: in a context generic enough to use drop you would not expect to be able to index the output, so we are not really convinced. Personally I think this is too "fussy"; drop should just return a drop iterator, period. But we could be convinced if there is a compelling real-world example where this really improves some code.

@LilithHafner LilithHafner removed the triage This should be discussed on a triage call label Mar 26, 2026
@nsajko nsajko deleted the arrays-Iterators-take-drop-view branch March 28, 2026 10:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrays [a, r, r, a, y, s] iteration Involves iteration or the iteration protocol performance Must go faster

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants