Ability to return an iterator on a "many" query

Hi, when a query return a myriad of objects, the slice based solution is not very efficient, it would be great if it was possible to have a way to return some kind of iterator on the rows, something like sql.Rows but type safe.

The first solution that comes to mind is to add a new type of function in the sql queries builder.
Instead of writing -- name: Name :many we would write -- name: Name :rows or another relevant keyword

and for the generated code, it would look something like this

CREATE TABLE pilots (
          id integer NOT NULL,
          name text NOT NULL
);

ALTER TABLE pilots ADD CONSTRAINT pilot_pkey PRIMARY KEY (id);

-- name: ListPilots :rows
SELECT * FROM pilots;

type Pilot struct {
	ID   int32
	Name string
}

type PilotRows struct {
	rows *sql.Rows
}

func (p *PilotRows) Close() error {
	return p.rows.Close()
}
func (p *PilotRows) Next() bool {
	return p.rows.Next()
}
func (p *PilotRows) Scan(dest *Pilot) error {
	return p.rows.Scan(dest.ID, dest.Name)
}
func (p *PilotRows) Err() error {
	return p.rows.Err()
}

func (q *Queries) ListPilots(ctx context.Context) (*PilotRows, error) {
	rows, err := q.query(ctx, q.listPilotsStmtn, listPilots)
	if err != nil {
		return nil, err
	}
	return &PilotRows{rows: rows}, nil
}

@encendre

@encendre thanks for the feature request. You're suggestion fixes a few of the issues we discussed when this first came up in #155. I'd ask you the same question I asked in that thread: can you use some type of paging to work through this query? It's not usually a good idea to request thousands of rows from a single query.

@encendre

@encendre @kyleconroy In #155 I linked to a poc for streaming via an iterator func. It's used in a project that needs to stream out alot of entries. Paging should oc be preferred, but sometimes its not an option.

My team and I are considering using sqlc for our services and having this feature would be a huge pro for going for sqlc. Anyway, how could we help with this?

Edit: We also need this for streaming records

@eivindr

@eivindr Did you find a nice way to extend sqlc to support the iterator?

@pcriv

@pcriv I changed the generator to produce code like #155 (comment) - so you pass in a iterator func that handles the streaming. If the func returns error the stream is aborted. You tell me if its a nice way or not :P

That one looks good! but I assume you are using your fork for now. I was hoping there was a nice way of extending sqlc without having to rely on a fork 😅

@kyleconroy

Yes, I'm just using the fork. I could create a PR if @kyleconroy pre-approves the fundamentals in the approach. The PR would also need to have some documentation, and maybe also generate similar for Kotlin..? 😅

Having this in sqlc would be a huge win for us :) If I can help somehow please let me know

@eivindr

@eivindr maybe you can just open the PR and we can continue the discussion there?

We're planning on adding support for this if the "range over func" proposal is accepted: golang/go#61405

@kyleconroy

Hi @kyleconroy , the mentioned proposal has been accepted already. Would it be possible to go ahead with the requested feature? We also really need it for the case of exporting data in the streaming mode and where it is not convenient to chunk the data by running multiple queries.

Thanks a lot in advance!

Range over func will be enabled by default on Go 1.23, so we're now unblocked from the Go side of things.

@kyleconroy

Hi @kyleconroy 👋

We're also quite interested in start using the range over func for these queries - do you know if they'll be available any time soon? Or if there's a workaround to configure the generator to use these?

Thanks!

@kyleconroy

Hi @kyleconroy
Can I help you with implementation of this feature?

For me, the following implementation of iteration is preferable:

type IterCitiesRows struct{
	rows *sql.Rows
	err  error
}

func (r *IterCitiesRows) Iterate() iter.Seq[City] {
	if r.rows == nil {
		return func(yield func(City) bool) {}
	}

	return func(yield func(City) bool) {
		for r.rows.Next() {
			var i City
			err := r.rows.Scan(&i.Slug, &i.Name)
			if err != nil {
				r.err = err
				return
			}

			if !yield(i) {
				break
			}
		}

		if err = r.rows.Close(); err != nil {
			r.err = err
		}
	}
}

func (r *IterCitiesRows) Err() error {
	if r.err != nil {
		return r.err
	}
	return r.rows.Err()
}

func (q *Queries) IterCities(ctx context.Context) IterCitiesRows {
	rows, err := q.query(ctx, q.listCitiesStmt, listCities)
	if err != nil {
		return IterCitiesRows{err: err}
	}
	return IterCitiesRows{rows: rows}
}

func example(ctx context.Context, q *Queries) {
	rows := q.IterCities(ctx)
	for city := range rows.Iterate() {
		// do something with city
	}
	if err := rows.Err(); err != nil {
		// handle error
	}
}

I found this article really helpful in understanding how to range over functions in Go 1.23 https://eli.thegreenplace.net/2024/ranging-over-functions-in-go-123/

I don't think I've seen this pattern anywhere. However, passing a predicate into the function creating the iter.Seq seems like it aligns with the spirit of iterators.

https://go.dev/play/p/7es2uIvtUqI

package main

import (
	"fmt"

	"play/iterate"
)

func main() {
	for n := range iterate.Numbers(5, func(err error) bool {
		fmt.Println("Error:", err)
		return false
	}) {
		fmt.Println(n)
	}
}

-- iterate/numbers.go --
package iterate

import (
	"errors"
	"iter"
)

var Err3 = errors.New("3 is a crowd")

func Numbers(n int, errPredicate func(error) bool) iter.Seq[int] {
	return func(yield func(int) bool) {
		for i := range n {
			if i == 3 {
				if !handle(Err3) {
					return
				}
				continue
			}
			if !yield(i) {
				return
			}
		}
	}
}

Here is a variation with the error handling outside the loop: https://go.dev/play/p/GN0bjrx4_Pl

I am using the following in production at the moment:

How to use

results, err := db.Get(ctx)
if err != nil {
	return err
}

for result, rangeErr := range results {
	if rangeErr != nil {
		return rangeErr
	}
	
	fmt.Println("%v\n", result)
}

return nil

Generated code

type Result struct {
	Id     uuid.UUID
	Domain string
}

func (q *Queries) Get(ctx context.Context) (iter.Seq2[*Result error], error) {
	rows, err := q.db.Query(ctx, query)
	if err != nil {
		return nil, err
	}

	return func(yield func(*Result, error) bool) {
		defer rows.Close()

		for {
			// @todo .Next and .Scan do not stop on context cancellation, is this expected of pgx?
			select {
			case <-ctx.Done():
				yield(nil, ctx.Err())
				return
			default:
			}

			if !rows.Next() {
				if returnErr := rows.Err(); returnErr != nil {
					yield(nil, returnErr)
				}
				return
			}

			var id pgtype.UUID
			var domain pgtype.Text
			if scanErr := rows.Scan(&id, &domain); scanErr != nil {
				yield(nil, scanErr)
				return
			}

			if !yield(&Result{
				Id:     id.Bytes,
				Domain: domain.String,
			}, nil) {
				return
			}
		}
	}, nil
}

I still have to look into whether context cancellation not being handled by pgx is expected.

Range over func will be enabled by default on Go 1.23, so we're now unblocked from the Go side of things.

Any updates regarding this feature, since there's no limitations from Go now?

@kyleconroy

Proposal: opt-in iterator generation for `:many` queries (reviving #720)

Following up on the Feb 2026 ask for updates, and cross-linking #4464 — there is no open PR for this today (#3631 was closed without merge). Go 1.23+ iterators are stable; the remaining gap is an agreed API and config surface.

Below is a consolidated proposal that tries to resolve the open design threads from #3631 while keeping zero breaking changes.

Problem

sqlc's :many queries always materialize into []T. That is great DX for small result sets, but for exports, sync jobs, ETL, and backfills it forces O(n) heap allocation even when callers only need row-by-row processing. Workarounds (manual paging, forks, skipping sqlc on hot paths) all sacrifice type safety or maintenance.

Go 1.23 shipped iter.Seq / iter.Seq2. Kyle noted that this unblocks native iterator generation. The community need is still very much alive (#720, #4464, production patterns shared by @MatthiasKunnen).

Design principles

Do not change default :many behavior — ListAuthors(ctx) ([]Author, error) stays as-is.
Opt-in via config (and optionally per-query annotation).
Go 1.23 native idioms — primary output is iter.Seq2[T, error].
Lazy by default — the DB query runs on the first iteration step, not at method call (avoids dangling queries if the iterator is never consumed).
Correct lifecycle — defer rows.Close() inside the iterator closure (addresses gbarr's concern in #3631).

Proposed `sqlc.yaml` options

gen:
  go:
    emit_iterators: false          # opt-in; default false

    iterator_scope: global         # global | explicit_only
    iterator_method_prefix: "Iter" # ListAuthors → IterAuthors (or "Stream")
    iterator_style: seq2             # seq2 | callback | rows
    iterator_start: lazy             # lazy | eager

Scope modes:

Mode	Behavior
`emit_iterators: true` + `iterator_scope: global`	Every `:many` query gets a streaming companion
`iterator_scope: explicit_only`	Only annotated queries, e.g. `-- name: ListAuthors :many:stream` or `-- name: StreamAuthors :stream` (#4464)

Teams that want simple DX flip one flag; teams that want fine-grained control use annotations.

Generated API (recommended default: `seq2` + `lazy`)

SQL — unchanged:

-- name: ListAuthors :many
SELECT id, name, bio FROM authors ORDER BY name;

Go — existing + new companion:

// unchanged
func (q *Queries) ListAuthors(ctx context.Context) ([]Author, error) { ... }

// new (when emit_iterators: true)
func (q *Queries) IterAuthors(ctx context.Context) iter.Seq2[Author, error] {
    return func(yield func(Author, error) bool) {
        rows, err := q.db.QueryContext(ctx, listAuthors)
        if err != nil {
            yield(Author{}, err)
            return
        }
        defer rows.Close()

        for rows.Next() {
            var i Author
            if err := rows.Scan(&i.ID, &i.Name, &i.Bio); err != nil {
                yield(Author{}, err)
                return
            }
            if !yield(i, nil) {
                return
            }
        }
        if err := rows.Err(); err != nil {
            yield(Author{}, err)
        }
    }
}

Usage:

for author, err := range q.IterAuthors(ctx) {
    if err != nil {
        return fmt.Errorf("list authors: %w", err)
    }
    if err := process(author); err != nil {
        return err
    }
}

This aligns with Kyle's later preference in #3631 for direct for x, err := range q.Method(ctx) — no wrapper type in the common case, hard to misuse, early break still closes rows via defer.

Alternate styles (configurable, not all generated by default)

`iterator_style`	Generated method	Notes
`seq2` (default)	`IterAuthors(ctx) iter.Seq2[Author, error]`	Recommended
`callback`	`EachAuthor(ctx, func(Author) error) error`	Optional sugar; can also be a one-liner at call sites
`rows`	`*IterAuthorsRows` with `All()` / `Err()` / `Close()`	Compatibility with #720 original request and #3631

Not proposing three methods per query by default — List* + Iter* covers slice vs stream; Each* is optional config.

iterator_start: eager (alternative): return (iter.Seq2[T, error], error) so connection errors surface before iteration — useful for some pgx setups, but lazy should be the default.

Parameterized queries

Same pattern — iterators mirror :many signatures:

-- name: ListAuthorsByIDs :many
SELECT id, name, bio FROM authors WHERE id = ANY($1::int[]);

func (q *Queries) ListAuthorsByIDs(ctx context.Context, ids []int32) ([]Author, error)
func (q *Queries) IterAuthorsByIDs(ctx context.Context, ids []int32) iter.Seq2[Author, error]

Suggested v1 scope

Go codegen only
database/sql driver path first (pgx follow-up)
PostgreSQL example + e2e test (pattern from Ability to return an iterator on rows #3631's examples)
Min Go 1.23+ when emit_iterators: true (document in release notes)
Python/Kotlin: coordinate separately (#4464, @borissmidt's offer)

Open questions for @kyleconroy / maintainers

Method prefix: Iter vs Stream?
Lazy default: agree lazy is correct for v1?
Config vs annotation: ship both emit_iterators: global and :stream explicit mode?
Eager mode: v1 or defer?
pgx: same PR or immediate follow-up?

Offer to implement

Happy to put together a minimal PoC PR once there is sign-off on the config surface and default API (seq2 + lazy + Iter prefix). Can start with one :many query, one engine, and an end-to-end test.

One-liner: sqlc already generates excellent type-safe Go; an opt-in emit_iterators flag would add lazy iter.Seq2[T, error] companions for :many — same type safety, O(1) memory per row, no breaking changes.

Related: #4464, #4108, closed reference #3631.

Opened a proof-of-concept PR (not merge-ready) implementing the proposal above for early feedback:

#4488

PoC branch / reference: https://github.com/Tilzen/sqlc/tree/feat/emit-iterators-poc

Includes emit_iterators config, lazy iter.Seq2[T, error] companions for :many, :stream / :many:stream annotations, stdlib + pgx/v5 templates, PRD doc, and runtime + codegen tests. Looking for maintainer sign-off on API/config before polishing toward a production PR.

For what it's worth, I've been using a iter.Seq heavy API library which uses this format:

objects, errf := c.GetObjects(ctx)
for object := range objects {
  // ....
}
if err := errf(); err != nil {
  // ...
}

In my experience working with this, this has been quite natural, any final GetObjects error is not inside the loop, and the Go linter gives an error when errf is not referenced (just like err is when not explicitly checked).

@sgielen

Agreed with @sgielen - the smart pointer approach also mimics what sql.Rows does (where .Next() is replaced by the iter.Seq pattern).

Navigation Menu

Repository navigation

Ability to return an iterator on a "many" query #720

How to use

Generated code

Proposal: opt-in iterator generation for `:many` queries (reviving #720)

Problem

Design principles

Proposed `sqlc.yaml` options

Generated API (recommended default: `seq2` + `lazy`)

Alternate styles (configurable, not all generated by default)

Parameterized queries

Suggested v1 scope

Open questions for @kyleconroy / maintainers

Offer to implement

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Ability to return an iterator on a "many" query #720

Description

Activity

kyleconroy commented on Oct 3, 2020

eivindr commented on Nov 6, 2020

pcriv commented on Feb 12, 2021

pcriv commented on Mar 17, 2021

eivindr commented on Mar 17, 2021

pcriv commented on Mar 17, 2021

eivindr commented on Mar 17, 2021

pcriv commented on Mar 17, 2021

pcriv commented on May 4, 2021

kyleconroy commented on Sep 22, 2023

dskarataev commented on May 17, 2024

kyleconroy commented on May 17, 2024

tgmendes commented on Sep 9, 2024

viewsharp commented on Sep 26, 2024

kyleconroy commented on Nov 4, 2024

crhntr commented on Dec 21, 2024

MatthiasKunnen commented on Nov 18, 2025

How to use

Generated code

e-gloo commented on Feb 25, 2026

Tilzen commented on Jun 14, 2026

Proposal: opt-in iterator generation for :many queries (reviving #720)

Problem

Design principles

Proposed sqlc.yaml options

Generated API (recommended default: seq2 + lazy)

Alternate styles (configurable, not all generated by default)

Parameterized queries

Suggested v1 scope

Open questions for @kyleconroy / maintainers

Offer to implement

Tilzen commented on Jun 14, 2026

sgielen commented on Jul 7, 2026

iitsdani commented on Jul 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Proposal: opt-in iterator generation for `:many` queries (reviving #720)

Proposed `sqlc.yaml` options

Generated API (recommended default: `seq2` + `lazy`)