SANDEEP DINESH: Distributed
systems can be hard to manage.
A big reason is that there
are many moving parts
that all need to work in order
for the system to function.
If a small part
breaks, the system
has to detect it, route
around it, and fix it.
And this all needs to
be done automatically.
In this episode of
"Kubernetes Best Practices,"
let's learn how you can set up
readiness and liveness probes
in your Kubernetes Cluster.
[MUSIC PLAYING]
Health checks are
a simple way to let
the system know if an instance
of your app is working or not.
If an instance of your
app is not working,
then other services should
not access it or send requests
to it.
Instead, the request
should be sent
to another instance of
the app that is ready
or retried at a later time.
The system should also
bring your app back
to a healthy state.
By default, Kubernetes will
start to send traffic to a pod
when all the containers
inside the pods
start, and will restart
containers when they crash.
While this default
behavior can be good enough
when you're starting
out, you can
make your deployments
more robust
by creating custom
health checks.
Fortunately,
Kubernetes makes this
relatively straightforward,
so there's no excuse not to.
Kubernetes gives you two
types of health checks,
and it's important to
understand the differences
and uses of each.
Readiness probes
are designed to let
Kubernetes know when your app
is ready to serve traffic.
Kubernetes will make sure
the readiness probe passes
before allowing the service
to send traffic to the pod.
If a readiness probe
starts to fail,
Kubernetes will stop
sending traffic to the pod
until it passes again.
Liveness probes
lets Kubernetes know
if your app is alive or dead.
If your app is alive, then
Kubernetes will leave it alone.
If your app is dead, then
Kubernetes will remove the pod
and start a new
one to replace it.
Let's imagine a scenario
where your app takes a minute
to warm up and start.
Your service won't
work until it's
up and running, even though the
process has already started.
You will also have
issues if you want
to scale up this deployment
to have multiple copies.
New copies shouldn't
receive traffic
until they're fully ready.
But by default,
Kubernetes will start
sending traffic as soon as the
processes inside the container
start.
By using a readiness
probe, Kubernetes
will wait until the
app is fully started
before it allows the service to
send traffic to the new copy.
Let's imagine another
scenario where
you have an app with a
nasty case of deadlock
that happens in an
edge case, causing
it to hang indefinitely
and stop serving requests.
Because the process
continues to run, by default,
Kubernetes will think
that everything is fine
and continue to send
requests to the broken pod.
By using a liveness
probe, Kubernetes
will detect the app is no
longer serving requests,
and will restart the
offending pod by default.
The next step is
actually defining
the probes that will test
readiness and liveness.
There are three
types of probes--
HTTP, command, and TCP.
You can use any of them for
liveness and readiness checks.
HTTP probes are probably
the most common type
of custom probe.
Even if your app
isn't an HTTP server,
you can usually create
a lightweight HTTP
server inside your app to
respond to the liveness probe.
Now, Kubernetes
will ping a path.
And if it gets an HTTP response
in the 200 or 300 range,
it will mark it as healthy.
Otherwise, it'll be
marked as unhealthy.
For command probes,
Kubernetes will run a command
inside your container.
If the command returns
with an exit code zero,
then the container
will be marked healthy.
Otherwise, it will
be marked unhealthy.
This type of probe
is useful when
you can't or don't want
to run an HTTP server,
but you can run a
command that can check
if your app is healthy or not.
The last type of probe
is the TCP probe.
Kubernetes will try to
establish a TCP connection
on the specified port.
If it can establish
a connection,
the container is
considered healthy.
If it can't, it's
considered a failure.
Now, these can come
in handy if you
have a scenario where the HTTP
probe or the command probe
don't work well.
For example, a
gRPC or FTP service
will be a prime candidate
for this type of probe.
Probes can be
configured in many ways.
You can specify how
often they should run,
what the success and
failure thresholds are,
and how long to
wait for responses.
See the documentation
for more details.
However, there is
one very important
setting that you
need to configure
when using liveness probes.
This is the initial
delay second setting.
As I mentioned before,
a liveness probe failure
will cause a pod to restart.
You need to make sure
the probe doesn't
start until the app is ready.
Otherwise, the app will just
constantly restart in a loop,
and never be ready.
I recommend using
the P99 startup
time as the initial
delay seconds,
or just take the average
startup time and add a buffer.
As your app startup time
gets faster or slower,
make sure you
update this number.
Most people will tell you
that health checks are
a requirement for any
distributed system,
and Kubernetes is no exception.
Using health checks gives
your Kubernetes services
a solid foundation, better
reliability, and higher uptime.
Thankfully, Kubernetes
makes this easy to do.
I'll see you on the next episode
of "Kubernetes Best Practices".
[MUSIC PLAYING]
