The Producer-Consumer Problem

Part of Golang Mastery course

~15 min read
Interactive
Hands-on
Beginner-friendly

The Producer-Consumer Problem

The Producer-Consumer pattern is a classic concurrency problem that involves two types of processes: producers that generate data and consumers that process that data. This pattern is widely used in software systems that need to decouple data production from consumption.

Understanding the Producer-Consumer Problem#

In the Producer-Consumer problem:

  1. Producers generate data items and put them into a shared buffer
  2. Consumers take data items from the buffer and process them
  3. The buffer has a limited capacity
  4. Both producers and consumers can work concurrently

The challenge is to coordinate the interaction between producers and consumers efficiently and safely.

Why Use the Producer-Consumer Pattern?#

The Producer-Consumer pattern is useful for:

  1. Decoupling: Separating data production from consumption
  2. Throttling: Managing different rates of production and consumption
  3. Parallelism: Allowing multiple producers and consumers to work in parallel
  4. Resource management: Controlling access to limited resources

Implementation in Go#

Go's channels are perfectly suited for implementing the Producer-Consumer pattern. The channel acts as the buffer between producers and consumers.

A Simple Producer-Consumer Example#

Let's start with a simple example:

example.go
package main
 
import (
    string">"fmt"
    string">"math/rand"
    string">"time"
)
 
func producer(id int, jobs chan<- int) {
    for i := 0; i < 5; i++ {
        job := rand.Intn(100) string">"comment">// Generate a random job
        fmt.Printf(string">"Producer %d: Created job %d\n", id, job)
        jobs <- job string">"comment">// Send job to the jobs channel
        time.Sleep(time.Millisecond * time.Duration(rand.Intn(1000)))
    }
}
 
func consumer(id int, jobs <-chan int) {
    for job := range jobs {
        fmt.Printf(string">"Consumer %d: Processing job %d\n", id, job)
        time.Sleep(time.Millisecond * time.Duration(rand.Intn(1000)))
    }
}
 
func main() {
    rand.Seed(time.Now().UnixNano())
    
    jobs := make(chan int, 10) string">"comment">// Buffered channel with capacity 10
    
    string">"comment">// Start 3 producers
    for i := 1; i <= 3; i++ {
        go producer(i, jobs)
    }
    
    string">"comment">// Start 2 consumers
    for i := 1; i <= 2; i++ {
        go consumer(i, jobs)
    }
    
    string">"comment">// Let the producers and consumers run for a while
    time.Sleep(5 * time.Second)
    
    string">"comment">// Close the jobs channel to signal consumers to stop
    close(jobs)
    
    string">"comment">// Give consumers time to finish processing remaining jobs
    time.Sleep(2 * time.Second)
    fmt.Println(string">"All done!")
}
 

In this example:

  1. We create a buffered channel jobs with a capacity of 10
  2. We start 3 producer goroutines that generate random jobs
  3. We start 2 consumer goroutines that process jobs from the channel
  4. Producers send jobs to the channel, and consumers receive jobs from the channel

A More Complex Example: Pizza Delivery Service#

Let's implement a more complex Producer-Consumer pattern that simulates a pizza delivery service:

example.go
package main
 
import (
    string">"fmt"
    string">"math/rand"
    string">"time"
    
    string">"github.com/fatih/color"
)
 
const NumberOfPizzas = 10
 
var pizzasMade, pizzasFailed, total int
 
type PizzaOrder struct {
    pizzaNumber int
    message     string
    success     bool
}
 
type Producer struct {
    data chan PizzaOrder
    quit chan chan error
}
 
func (p *Producer) Close() error {
    ch := make(chan error)
    p.quit <- ch
    return <-ch
}
 
func makePizza(pizzaNumber int) *PizzaOrder {
    pizzaNumber++
    if pizzaNumber <= NumberOfPizzas {
        delay := rand.Intn(5) + 1
        fmt.Printf(string">"Received order #%d!\n", pizzaNumber)
        
        rnd := rand.Intn(12) + 1
        msg := string">""
        success := false
        
        if rnd < 5 {
            pizzasFailed++
        } else {
            pizzasMade++
        }
        total++
        
        fmt.Printf(string">"Making pizza #%d. It will take %d seconds....\n", pizzaNumber, delay)
        string">"comment">// Simulate making pizza
        time.Sleep(time.Duration(delay) * time.Second)
        
        if rnd <= 2 {
            msg = fmt.Sprintf(string">"*** We ran out of ingredients for pizza #%d!", pizzaNumber)
        } else if rnd <= 4 {
            msg = fmt.Sprintf(string">"*** The cook quit while making pizza #%d!", pizzaNumber)
        } else {
            success = true
            msg = fmt.Sprintf(string">"Pizza order #%d is ready!", pizzaNumber)
        }
        
        p := PizzaOrder{
            pizzaNumber: pizzaNumber,
            message:     msg,
            success:     success,
        }
        
        return &p
    }
    
    return &PizzaOrder{
        pizzaNumber: pizzaNumber,
    }
}
 
func pizzeria(pizzaMaker *Producer) {
    string">"comment">// Keep track of which pizza we're making
    var i = 0
    
    string">"comment">// Run until we receive a quit notification
    string">"comment">// Try to make pizzas
    for {
        currentPizza := makePizza(i)
        if currentPizza != nil {
            i = currentPizza.pizzaNumber
            select {
            case pizzaMaker.data <- *currentPizza:
                string">"comment">// Send the pizza to the channel
            case quitChan := <-pizzaMaker.quit:
                string">"comment">// We received a quit signal
                close(pizzaMaker.data)
                close(quitChan)
                return
            }
        }
    }
}
 
func main() {
    string">"comment">// Seed the random number generator
    rand.Seed(time.Now().UnixNano())
    
    string">"comment">// Print out a welcome message
    color.Cyan(string">"Pizza delivery service!")
    color.Cyan(string">"----------------------------------")
    
    string">"comment">// Create a producer
    pizzaJob := &Producer{
        data: make(chan PizzaOrder),
        quit: make(chan chan error),
    }
    
    string">"comment">// Run the producer in the background
    go pizzeria(pizzaJob)
    
    string">"comment">// Process the pizzas as they're made
    for i := range pizzaJob.data {
        if i.pizzaNumber <= NumberOfPizzas {
            if i.success {
                color.Green(string">"%s", i.message)
                color.Green(string">"Order #%d is out for delivery!", i.pizzaNumber)
            } else {
                color.Red(string">"%s", i.message)
                color.Red(string">"Customer is not happy!")
            }
        } else {
            color.Cyan(string">"Done making pizzas!")
            err := pizzaJob.Close()
            if err != nil {
                color.Red(string">"Error closing channel: %v", err)
            }
        }
    }
    
    string">"comment">// Print out the ending message
    color.Cyan(string">"----------------------------------")
    color.Cyan(string">"Done for the day!")
    color.Cyan(string">"We made %d pizzas, but failed to make %d, with %d attempts in total", pizzasMade, pizzasFailed, total)
    
    switch {
    case pizzasFailed > 9:
        color.Red(string">"It was a terrible day...")
    case pizzasFailed >= 6:
        color.Red(string">"It was not a very good day...")
    case pizzasFailed >= 4:
        color.Yellow(string">"It was an okay day.")
    case pizzasFailed >= 2:
        color.Yellow(string">"It was a pretty good day!")
    default:
        color.Green(string">"It was a great day!")
    }
}
 

In this more complex example:

  1. We define a PizzaOrder struct to represent a pizza order
  2. We define a Producer struct that has a data channel for pizza orders and a quit channel for shutdown
  3. The pizzeria function runs in a goroutine and produces pizza orders
  4. The main function acts as the consumer, processing pizza orders as they come in
  5. We use select statements to handle both normal operation and shutdown gracefully
  6. We use colored output to make the example more engaging

Advantages of Using Channels for Producer-Consumer#

Using channels for the Producer-Consumer pattern in Go has several advantages:

  1. Built-in synchronization: Channels handle the synchronization between producers and consumers
  2. Buffering: Buffered channels can handle temporary rate differences
  3. Clear ownership: Producers own the sending end, consumers own the receiving end
  4. Signaling completion: Closing a channel can signal when production is complete

Common Patterns and Best Practices#

1. Use Buffered Channels for Temporary Rate Differences#

Buffered channels can help when producers and consumers work at different rates:

example.go
string">"comment">// Buffer up to 100 items
jobs := make(chan Job, 100)
 

2. Multiple Producers and Consumers#

You can have multiple producers and consumers working with the same channel:

example.go
string">"comment">// Start multiple producers
for i := 1; i <= 3; i++ {
    go producer(i, jobs)
}
 
string">"comment">// Start multiple consumers
for i := 1; i <= 5; i++ {
    go consumer(i, jobs)
}
 

3. Graceful Shutdown#

For graceful shutdown, have a way to signal producers to stop and allow consumers to finish processing:

example.go
string">"comment">// Signal producers to stop
close(quit)
 
string">"comment">// Wait for all consumers to finish
wg.Wait()
 

4. Error Handling#

Include error handling in your Producer-Consumer implementations:

example.go
type Result struct {
    value interface{}
    err   error
}
 
string">"comment">// Producer
go func() {
    defer close(results)
    for _, job := range jobs {
        value, err := process(job)
        results <- Result{value, err}
    }
}()
 
string">"comment">// Consumer
for result := range results {
    if result.err != nil {
        log.Printf(string">"Error: %v", result.err)
        continue
    }
    string">"comment">// Use result.value
}
 

Summary#

  • The Producer-Consumer pattern helps decouple data production from consumption
  • Go's channels are perfect for implementing this pattern
  • Use buffered channels to handle rate differences
  • Multiple producers and consumers can work with the same channel
  • Consider graceful shutdown and error handling in your implementation

In the next lab, we'll learn about using the range keyword to iterate over values received from a channel, which is a common pattern in consumer code.

Your Progress

11 of 19 modules
58%
Started58% Complete
←Previous
SpaceComplete
→Next