Inspired by a recent tweet from @freekmurze, his upcoming Mailcoach PHP SDK contains a nice API for paginating through all records:

1// listing all subscribers of a list
2$subscribers = $mailcoach->emailList('use-a-real-email-list-uuid-here')->subscribers();
3 
4do {
5    foreach($subscribers as $subscriber) {
6        echo $subscriber->email;
7    }
8} while($subscribers = $subscribers->next())

Whilst I definitely like the idea of the ->next() method, I think this could be simplified/improved such that "consumers" can just loop over all the records without having to worry about the pagination.

In a previous gig, we worked extensively with APIs and so were often fetching large amounts of paginated data. We've used similar approaches where we would expose service classes that returned the results and expose methods for checking/getting the next page. However, we found that this didn't always create the best developer experience.

Rather than wrapping the foreach in a do/while loop, what if we could just loop the all results?

1$subscribers = $mailcoach->emailList('use-a-real-email-list-uuid-here')->subscribers();
2 
3foreach($subscribers as $subscriber) {
4    echo $subscriber->email;
5}

Whilst a small change in this case, we've completely removed the need for any logic to fetch the next "page" of results.

Let's look at how we could use generators to achieve this.

Using generators for paginating through all records

To bring this to life, I like to use a service class to encapsulate the API and return the records.

As a simple example, let's create a service class to fetch Pokémon from the PokeAPI:

 1<?php
 2 
 3namespace App\Services;
 4 
 5use Generator;
 6use Illuminate\Support\Facades\Http;
 7 
 8class Pokemon
 9{
10    public function all($page = 1): Generator
11    {
12        do {
13            $response = Http::get('https://pokeapi.co/api/v2/pokemon', [
14                'offset' => ($page - 1) * 20,
15                'limit' => 20,
16            ]);
17 
18            foreach ($response['results'] as $result) {
19                yield $result;
20            }
21 
22            $page++;
23        } while ($response['next'] !== null);
24    }
25}

By using the Generator returned from the Pokemon class, we can loop over all the Pokémon without needing to manually fetch further pages. To me, this creates a much nicer experience for the consumer who no longer has to be concerned with fetching the next page of results; they simply ask for all Pokémon and get all the Pokémon.

To see how we'd use this service class in action, let's write all the results to our terminal when we run a command:

 1<?php
 2 
 3namespace App\Console\Commands;
 4 
 5use App\Services\Pokemon;
 6use Illuminate\Console\Command;
 7 
 8class ListPokemon extends Command
 9{
10    protected $signature = 'pokemon:list';
11 
12    protected $description = 'Lists all the pokemon names, with support for providing name or type filters';
13 
14    public function handle(Pokemon $pokemon)
15    {
16        foreach ($pokemon->all() as $pokemon) {
17            $this->info($pokemon['name']);
18        }
19 
20        return Command::SUCCESS;
21    }
22}

I work with Laravel a lot and as a personal preference I like to work with collections over arrays. Laravel has the concept of a LazyCollection that can consume our generator and provide some nice collection affordances. If you're unfamiliar with the LazyCollection, they're very similar to a regular collection, with almost all the same methods.

Let's update our service class to use a LazyCollection:

 1<?php
 2 
 3namespace App\Services;
 4 
 5use Illuminate\Support\Facades\Http;
 6use Illuminate\Support\LazyCollection;
 7 
 8class Pokemon
 9{
10    public function all($page = 1): LazyCollection
11    {
12        return LazyCollection::make(
13            function () use ($page) {
14                do {
15                    $response = Http::get('https://pokeapi.co/api/v2/pokemon', [
16                        'offset' => ($page - 1) * 20,
17                        'limit' => 20,
18                    ]);
19 
20                    foreach ($response['results'] as $result) {
21                        yield $result;
22                    }
23 
24                    $page++;
25                } while ($response['next'] !== null);
26            }
27        );
28    }
29}

Now that we're returning a LazyCollection, let's leverage some of the nice collection affordances I was talking about to make the implementation a little more concise:

 1<?php
 2 
 3namespace App\Console\Commands;
 4 
 5use App\Services\Pokemon;
 6use Illuminate\Console\Command;
 7 
 8class ListPokemon extends Command
 9{
10    protected $signature = 'pokemon:list';
11 
12    protected $description = 'Lists all the pokemon names';
13 
14    public function handle(Pokemon $pokemon)
15    {
16        $pokemon->all()
17            ->each(fn (array $pokemon) => $this->info($pokemon['name']));
18 
19        return Command::SUCCESS;
20    }
21}

Whilst the PokeAPI doesn't support query params for filtering, if you wanted to expose methods for filtering, then I find something like this quite nice:

 1<?php
 2 
 3namespace App\Services;
 4 
 5use Illuminate\Support\Facades\Http;
 6use Illuminate\Support\LazyCollection;
 7 
 8class Pokemon
 9{
10    private ?string $name = null;
11 
12    private ?string $type = null;
13 
14    public function name(?string $name): self
15    {
16        $this->name = $name;
17 
18        return $this;
19    }
20 
21    public function type(?string $type): self
22    {
23        $this->type = $type;
24 
25        return $this;
26    }
27 
28    public function all($page = 1): LazyCollection
29    {
30        return LazyCollection::make(
31            function () use ($page) {
32                do {
33                    $response = Http::get('https://pokeapi.co/api/v2/pokemon', [
34                        'offset' => ($page - 1) * 20,
35                        'limit' => 20,
36                        'name' => $this->name,
37                        'type' => $this->type,
38                    ]);
39 
40                    foreach ($response['results'] as $result) {
41                        yield $result;
42                    }
43 
44                    $page++;
45                } while ($response['next'] !== null);
46            }
47        );
48    }
49}

We've created some fluent methods for setting the name and/or type of Pokémon that we want to query for and thanks to the Laravel HTTP client, these parameters will be ignored if they're not set.

If we update our previous implementation, we could utilise these new filters like so:

 1<?php
 2 
 3namespace App\Console\Commands;
 4 
 5use App\Services\Pokemon;
 6use Illuminate\Console\Command;
 7 
 8class ListPokemon extends Command
 9{
10    protected $signature = 'pokemon:list {--name=} {--type=}';
11 
12    protected $description = 'Lists all the pokemon names, with support for providing name or type filters';
13 
14    public function handle(Pokemon $pokemon)
15    {
16        $pokemon->name($this->option('name'))
17            ->type($this->option('type'))
18            ->all()
19            ->each(fn (array $pokemon) => $this->info($pokemon['name']));
20 
21        return Command::SUCCESS;
22    }
23}

Hopefully these examples have made sense, and you can see the benefits of using this approach to help clean up your interfaces for working with service classes that interact with APIs.

---

I hope you found this useful. If you have any feedback, thoughts, or different approaches to this then I'd love to hear from you; I'm @jryd_13 on Twitter.

Thanks!