Showing posts with label golang. Show all posts
Showing posts with label golang. Show all posts

Thursday, May 20, 2021

Never Start a Goroutine You Can't Finish

The Go programming language has a pair of features that work well together for assembling a sequence of steps into a pipeline: goroutines and channels. In order to use these successfully, the goroutine needs to listen for communication it expects and watch for communication that might happen. We expect a channel to feed the goroutine items and be closed when no more items are forthcoming. Meanwhile, we need to watch a Context in case the goroutine should exit before the channel is closed. This article will focus on handling some of the edge cases that will keep your goroutines from finishing.
There are some fundamental things you should understand before using channels and goroutines. Start by completing the section of the tour of concurrency. With an understanding of the fundamentals, we can explore the details of goroutines communicating over a channel. The goroutine functions will each be responsible for detecting when it's time to finish: It is important to check in with your context regularly to see if it is "Done." A closed channel will give a waiting receiver the zero value. Ranging on channel loops until that channel is closed. Let's take a close look at two of the more common approaches I've seen. There's a lot to learn by looking at the trade-offs between these two approaches.

Infinite for loop

func ForInfinity(ctx context.Context, inputChan chan string) func() error {
	return func() error {
		for {
			select {
			case input := <-inputChan:
				if len(input) == 0 {
					return ctx.Err()
				}
				fmt.Println("logic to handle input ", input)
			case <-ctx.Done():
				return ctx.Err()
			}
		}
	}
}
  • When the inputChan channel is closed, you have to look for the zero value on the channel. The logic of that select case will need to detect the zero value to finish the goroutine – perhaps with a return nil or return ctx.Err().
  • When the context done case is selected, it feels natural to return ctx.Err(). By doing so, the goroutine is reporting the underlying condition that caused it to finish. Depending on the type of context and its state, the ctx.Err() may be nil.
  • If more than one select case is ready then one will be chosen at random. Given the undefined nature of having both of these case statements ready, you might consider having the zero-value-detecting logic return ctx.Err(). This will ensure your goroutine returns as accurately as possible, even if the channel case was selected.


Range on a channel

func ForRangeChannel(ctx context.Context, inputChan chan string) func() error {
  return func() error {
    for input := range inputChan { 
      select {
      case <-ctx.Done():
        return ctx.Err()
      default:
        fmt.Println("logic to handle input ", input)
      }
    }
    return nil
  }
}
  • While the goroutine is waiting to receive on inputChan, it will not exit unless the channel is closed. Now our pipeline func is dependent on the channel close. If the Context is “Done,” we won't know it until an item is received from the range inputChan. Upstream pipeline functions should close their stream when finishing.
  • Range won't give us the zero-value-infinite-loop, as in the earlier example. The Range will drop out to our final return nil when the channel is closed.
  • The context Done case has the same impact here as it did in the earlier example. The difference here is that the Done context will not be discovered until the channel receive occurs — making it even more important that the channels are closed.

Be mindful of the flow inside your goroutine to ensure it finishes appropriately. That and lots of tests will ensure your goroutines under normal and exceptional scenarios. Here are a couple of tests to get you started. These are written to exercise the same scenarios for each of the above goroutines.

func TestForInfinity(t *testing.T) {
	t.Run("context is canceled", func(t *testing.T) {
		inputChan := make(chan string)

		ctx, cancel := context.WithCancel(context.Background())
		cancel()
		f := ForInfinity(ctx, inputChan)
		err := f()

		assert.EqualError(t, err, "context canceled")
	})
	t.Run("closed channel returns without processing", func(t *testing.T) {
		inputChan := make(chan string)
		close(inputChan)

		ctx := context.Background()
		f := ForInfinity(ctx, inputChan)
		err := f()

		assert.NoError(t, err, "closed chanel return nil from ctx.Err()")
	})
}
func TestForRangeChannel(t *testing.T) {
	t.Run("context is canceled", func(t *testing.T) {
		inputChan := make(chan string)

		ctx, cancel := context.WithCancel(context.Background())
		cancel()
		f := ForRangeChannel(ctx, inputChan)
		go func() {
			//this test will hang without this goroutine
			<-time.After(time.Second)
			inputChan <- "some value"
		}()
		err := f()

		assert.EqualError(t, err, "context canceled")
	})
	t.Run("closed channel returns without processing", func(t *testing.T) {
		inputChan := make(chan string)
		close(inputChan)

		f := ForRangeChannel(context.Background(), inputChan)
		err := f()

		assert.NoError(t, err, "note there is no need to cancel the context, 'range' ends for us")
	})
}


Summary

By understanding how channels interact with range and select you can ensure your goroutine exits when it should. We have used both examples above successfully. Each different design has trade-offs. No matter the logic flow in your goroutines, always ask yourself: “how will this exit?” Then test it.

I think it's time for another Go Proverb: Never start a goroutine you can't finish!


Tuesday, January 19, 2021

Golang Method Receivers

There are different ways for a method to be assigned to a struct. One of the first thing Golang developers learn is that your method can have a value receiver or a pointer receiver.

Notice below that foo's bar method has a value receiver and accepts zero arguments. Standard stuff. Where it gets interesting is how I call foo.bar in the example main method.

package main

import (
 "fmt"
)

type foo struct {
 msg string
}

func (f foo) bar() string {
 return f.msg
}

func main() {
 receiver := foo{msg: "the value"}
 result := foo.bar(receiver)

 fmt.Println(result)
}

Friday, June 5, 2020

Golang Shutdown Flow



While doing some enhancements to the Golang microservices at work I came across quite a few calls to logrus.Fatal late in the execution of the service. Some of these particular services are long running processes that consume from Kafka and write to GCP Spanner. The problem with logrus.Fatal when called late in these services lifecycle is that Fatal internally calls os.Exit(1). So let’s examine why this is bad for the system in which the service is running.

On the surface I wonder if a non-recoverable error encountered late in the process is “Fatal”. The interpretation of what “Fatal” means is subjective. At the beginning of the process when reading the configuration file, making external system connections -- this is a Fatal problem because the service can even get started. Some precondition failure -- yeah, that’s Fatal. But if a service has been happily consuming messages and writing transformed data to a database only to encounter a non-recoverable error -- is that “Fatal”?

But that’s not what I wanted to show you. What I wanted to show you is why logrus.Fatal(...) is the wrong way to shutdown a service that has encountered a fatal error.

First some basics: https://play.golang.org/p/b8CAlmiVZPH

package main

import (
"errors"
"fmt"
)

func main() {
defer func() {
if err := recover(); err != nil {
fmt.Println("chance to recover:", err)
panic(err)
} else {
fmt.Println("nothing to recover")
}
}()

fmt.Println("Hello, playground")

panic(errors.New("It's a perfect time to panic! -- Woody"))
}


This is basic Golang: the app panics, the defer will execute, recover() will consume/catch the error, and then we re-panic (just as a naive solution). The panic is reported, and the application returns a non-zero status code.

But what about os.Exit(1)? We know that logrus calls os.Exit(int). We need to be aware of the impact upon our defer statements when we use os.Exit(int). As noted in the godoc, os.Exit(int) does not take time to run defer functions. It's just going to shutdown: https://play.golang.org/p/fB8GnFxEATs
package main
import (
"fmt"
"os"
) 
func main() {
defer func() {
if err := recover(); err != nil {
fmt.Println("chance to recover:", err)
} else {
fmt.Println("nothing to recover")
}
}()
fmt.Println("Hello, playground")
os.Exit(0)
}

Here the defer does not run. If you were going to gracefully release the connections to Kafka, Spanner, or any other external resource -- that did not happen. For illustration sake I also have this example returning the success zero status code.

There is another way to exit a Golang application. Well, sort of: runtime.Goexit(). As noted in the godoc all registered defer will be executed.  However, it only exits from one goroutine.  So if this is called in your last goroutine, your service will crash -- in the same fashion as when all goroutines are blocked causing deadlock: https://play.golang.org/p/z0k56ZMoF8N
package main

import (
"log"
"runtime"
)

func main() {
defer func() {
if err := recover(); err != nil {
log.Println("chance to recover:", err)
} else {
log.Println("nothing to recover")
}
}()

log.Println("Hello, playground")

runtime.Goexit()
}

Where does that leave us? Well, it's important to understand the impact of your libraries upon the flow of execution. The logrus.Fatal(...) method is definitely useful. Use it with the full understanding of what it is doing. Use it during service initialization before any defer functions have been registered. Use it when you know you want defer statements to be skipped.

Bonus:


It is important when fixing these sorts of problems with service shutdown to recognize the significance of your services exit code. Your services exit code is its last communication with the software architecture -- it's dying breath used to wheeze out one little death rattle.  Are you running your services in Docker? Kubernetes? 

The service exit code is going to communicate to the container if it exited successfully or crashed with an error. In those situations where you were calling logrus.Fatal, you likely do not want to log the error and simply return. That would have your service return zero as exit code communicating success to the software architecture system. Make sure you take into account the pod and the configured restartPolicy. If your service is shutting down because of a non-recoverable error you likely want to wheeze out a death rattle of non-zero.