Docker swarm – Load balancing asp.net core Ubuntu – Scaling and Upgrading – Part 2

Introduction

Here we’ll build up on the previous article (Part 1). It is mandatory that you finish reading or completing the steps in the previous article. We’ll add some finishing touches to docker swarm based load balancing of asp.net core on Ubuntu. We’ll also scaling and upgrading your web application with newer versions. Both scaling and upgrading will replicate your site on multiple containers on the Leader and Worker nodes.

Fun note: At the end of the article in Part 1, you might have guessed that your web application is already load balanced but it was not very apparent. I’ll make it easy to verify here. In the hind sight you had already reached what you had set out for, when you completed Part 1.

Pre-requisites

  1. All steps in Part 1 are completed. At least thoroughly read it.
  2. All environment and node preparations steps in the previous part need to be completed.
  3. Assuming you will prefix all commands with sudo OR run sudo -s in the beginning to skip typing sudo everywhere.
  4. By web application, here, I mean you asp.net core hello world web application.
  5. If you have recently, rebooted/resumed the Leader or Worker nodes, please run the following on all Leader and Worker nodes.
    1. systemctl restart docker. This is required since docker sometimes does not bring up the containers in a running and reachable state after rebooting.
    2. Another way to test is that when you type curl http://localhost:8000 (On all or either of nodes), nothing happens (error – connection refused). That’s another clue to restart the docker service.

Focus

  1. Hosting and testing the load balancing of your web application
  2. Scaling your web application to multiple containers on Leader and Worker nodes. (Doing this is a piece of cake!)
  3. Upgrading your web application with a newer version and getting it replicated on all Leader and Worker nodes.
  4. To upgrade, we’ll create a second web application and then upgrade the previous one with this one.

Create a new version of your web application

  1. Switch to /apps/aspnet-hw folder
  2. Run: dotnet new webapp -o aspnetcoreapp-v2 --no-https --force
  3. Switch to aspnetcoreapp-v2 folder
  4. Type nano Program.cs. The contents should look like this. (There is only one line change).
    Important: Do not change the port number since your site needs to be running on the same port as the previous version.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

namespace aspnetcoreapp_v2
{
    public class Program
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<startup>().UseKestrel().UseUrls("http://0.0.0.0:5000");
                });
    }
}

5. Switch to ./Pages folder. Type nano Index.cshtml. Replace the content below in this file:

@page
@model IndexModel
@{
    ViewData["Title"] = "Home page";
}
<div class="text-center">
    <h1 class="display-4">Welcome - from machine/node: @Environment.MachineName</h1>
    <p>Learn about <a href="https://docs.microsoft.com/aspnet/core">building Web apps with ASP.NET Core</a>.</p>
</div>

6. Notice that H1 tag will show the Machine name @Environment.MachineName under which the site is running. Ctrl-X, Yes, Enter. Save the file.

Important: Adding the machine name here is very important. This will allow you to verify if you site is running from different nodes or containers while testing load balancing.

7. Switch to you application folder. Run, cd ..

8. Run, dotnet build. There should be no compilation errors.

8. nano Dockerfile and paste the following contents. Notice the last line ENTRYPOINT where the new DLL name is entered.

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build-env
WORKDIR /app

# Copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore

# Copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out

# Build runtime image
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "aspnetcoreapp-v2.dll"]

9. Run, sudo docker build -t counter-image-v2 -f Dockerfile . This will build the new docker image with the suffix of “-v2”.

10. Run, docker images to check your newly created image. It should look like this:

REPOSITORY                             TAG                 IMAGE ID            CREATED             SIZE
<none>                                 <none>              629d8a395d25        6 hours ago         702MB
counter-image-v2                       latest              f0a855bcb06f        6 hours ago         212MB
<none>                                 <none>              b63d497aac59        4 days ago          713MB
127.0.0.1:5000/counter-image           latest              01c13ac310db        6 days ago          212MB
<none>                                 <none>              7fbb03aefc0a        6 days ago          713MB
mcr.microsoft.com/dotnet/core/sdk      3.1                 4aa6a74611ff        3 weeks ago         691MB
mcr.microsoft.com/dotnet/core/aspnet   3.1                 79e79777c3bf        3 weeks ago         207MB
registry                               <none>              708bc6af7e5e        3 months ago        25.8MB

Scale your web application

Run, docker service ls. Note down the service name: counter-image_web. We’ll first scale our existing application to 5 instances

ID                  NAME                MODE                REPLICAS            IMAGE                     PORTS
kngu8zmvnxbt        counter-image_web   replicated          1/1                 counter-image-v2:latest   *:8000->5000/tcp
kug3cnpctqad        registry            replicated          1/1                 registry:2                *:5000->5000/tcp

2. Let us scale our web application to 5 instances. This can be done by just one single command, boom! Run,
docker service scale counter-image_web=5

counter-image_web scaled to 5
overall progress: 5 out of 5 tasks
1/5: running   [==================================================>]
2/5: running   [==================================================>]
3/5: running   [==================================================>]
4/5: running   [==================================================>]
5/5: running   [==================================================>]
verify: Service converged

You can run this command multiple times by increasing the number 5, in this case to any number of instances. If you re-run say with 15, docker will create additional 10 instances.

3. Run, docker service ps counter-image_web. It will show you the replication status across Leader and Worker nodes

ID                  NAME                      IMAGE                                 NODE                DESIRED STATE       CURRENT STATE                 ERROR                              PORTS
0wzvbhzx4ko8        counter-image_web.1       127.0.0.1:5000/counter-image:latest   saturn              Running             Running about a minute ago
iu3qs03tn7dp         \_ counter-image_web.1   127.0.0.1:5000/counter-image:latest   neptune             Shutdown            Rejected about a minute ago   "No such image: 127.0.0.1:5000…"
bqnjn6rk7a5b         \_ counter-image_web.1   127.0.0.1:5000/counter-image:latest   neptune             Shutdown            Rejected about a minute ago   "No such image: 127.0.0.1:5000…"
psw4kpjse7zp         \_ counter-image_web.1   127.0.0.1:5000/counter-image:latest   neptune             Shutdown            Rejected 2 minutes ago        "No such image: 127.0.0.1:5000…"
m6cqhr8im1u0         \_ counter-image_web.1   127.0.0.1:5000/counter-image:latest   neptune             Shutdown            Rejected 2 minutes ago        "No such image: 127.0.0.1:5000…"
hxukcvm4zat5        counter-image_web.2       127.0.0.1:5000/counter-image:latest   jupiter             Running             Running 29 seconds ago
vfl05i56xw5t        counter-image_web.3       127.0.0.1:5000/counter-image:latest   saturn              Running             Running 30 seconds ago
dzazigr46wgm        counter-image_web.4       127.0.0.1:5000/counter-image:latest   saturn              Running             Running 14 seconds ago
jz8tlkvl8x57         \_ counter-image_web.4   127.0.0.1:5000/counter-image:latest   neptune             Shutdown            Rejected 25 seconds ago       "No such image: 127.0.0.1:5000…"
ch16658of0t3         \_ counter-image_web.4   127.0.0.1:5000/counter-image:latest   neptune             Shutdown            Rejected 30 seconds ago       "No such image: 127.0.0.1:5000…"
pja0fz1eom2a         \_ counter-image_web.4   127.0.0.1:5000/counter-image:latest   neptune             Shutdown            Rejected 30 seconds ago       "No such image: 127.0.0.1:5000…"
pn1mpr072jz3        counter-image_web.5       127.0.0.1:5000/counter-image:latest   jupiter             Running             Running 14 seconds ago
kdiww4aoqgcs         \_ counter-image_web.5   127.0.0.1:5000/counter-image:latest   neptune             Shutdown            Rejected 25 seconds ago       "No such image: 127.0.0.1:5000…"
yvplbal3wpon         \_ counter-image_web.5   127.0.0.1:5000/counter-image:latest   neptune             Shutdown            Rejected 30 seconds ago       "No such image: 127.0.0.1:5000…"
pkhj6r847b6m         \_ counter-image_web.5   127.0.0.1:5000/counter-image:latest   neptune             Shutdown            Rejected 30 seconds ago       "No such image: 127.0.0.1:5000…"

4. For some reason docker is not able to start a replicated container of Worker node 3. But you can see that the container is now scaled to 5 instances. 2 instances on jupiter node and 3 instances on saturn node.

5. You can check the logs of replication/scaling by running: docker service logs counter-image_web

6. Inspect service. Run docker service inspect --pretty counter-image_web. On a side note, if you ignore --pretty it will give you an exhaustive output. Observe that the service is running with older version of the image (counter-image), Check the ContainerSpec section.

ID:             kngu8zmvnxbtaftslirgk74is
Name:           counter-image_web
Labels:
 com.docker.stack.image=127.0.0.1:5000/counter-image
 com.docker.stack.namespace=counter-image
Service Mode:   Replicated
 Replicas:      5
Placement:
UpdateConfig:
 Parallelism:   1
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:   1
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first
ContainerSpec:
 Image:         127.0.0.1:5000/counter-image:latest@sha256:cdb85f4a62aaaddbb6f32ce900fd03e8c5d10997715f0c70fcc162c3ae5b8051
Resources:
Networks: counter-image_default
Endpoint Mode:  vip
Ports:
 PublishedPort = 8000
  Protocol = tcp
  TargetPort = 5000
  PublishMode = ingress

Upgrade your application with a newer version

Now we will upgrade the web application with the newer image counter-image-v2. Run,

docker service update --with-registry-auth --image counter-image-v2 counter-image_web

Note: --with-registry-auth parameter sends registry authentication details to Swarm agents. Check how we are using the new image of our web application in the --image parameter. The last parameter is our current service name counter-image_web.

The output will be:

image counter-image-v2:latest could not be accessed on a registry to record
its digest. Each node will access counter-image-v2:latest independently,
possibly leading to different nodes running different
versions of the image.

counter-image_web
overall progress: 0 out of 5 tasks
1/5: preparing [=================================>                 ]
2/5:
3/5:
4/5:
5/5:
service update paused: update paused due to failure or early termination of task r7gnu8jy738fcdzhnodor7off

Notice how it updates only partially on the first run. It actually pauses the update since it got an error updating a container. To resume, run:

docker service update counter-image_web

I had to run this command literally 4 times. You might have to run more if you have more instances. There could be a parameter in the docker service update which automatically retries until success but I haven’t tried it.

docker service update counter-image_web
counter-image_web
overall progress: 1 out of 5 tasks
1/5: running   [==================================================>]
2/5: preparing [=================================>                 ]
3/5:
4/5:
5/5:
service update paused: update paused due to failure or early termination of task ty7lyntxd3fbtgq94p6t3rqb0

docker service update counter-image_web
counter-image_web
overall progress: 1 out of 5 tasks
1/5: preparing [=================================>                 ]
2/5: running   [==================================================>]
3/5: preparing [=================================>                 ]
4/5:
5/5:
service update paused: update paused due to failure or early termination of task ul28q8h3fv7ziu5f7vmz9qdeh

docker service update counter-image_web
counter-image_web
overall progress: 1 out of 5 tasks
1/5: running   [==================================================>]
2/5: assigned  [======================>                            ]
3/5: assigned  [======================>                            ]
4/5: preparing [=================================>                 ]
5/5: preparing [=================================>                 ]
service update paused: update paused due to failure or early termination of task qli9y4xbporbqmr3lz2n7opci

docker service update counter-image_web
counter-image_web
overall progress: 5 out of 5 tasks
1/5: running   [==================================================>]
2/5: running   [==================================================>]
3/5: running   [==================================================>]
4/5: running   [==================================================>]
5/5: running   [==================================================>]
verify: Service converged

Finally you will see “Service converged” with 5 or 5 tasks completed.

Congratulations, your new version of web application is now replicated on all Leader and Worker nodes. To check this, run:

docker service ps counter-image_web

ID                  NAME                      IMAGE                     NODE                DESIRED STATE       CURRENT STATE             ERROR                              PORTS
rztwyx8oq2mv        counter-image_web.1       counter-image-v2:latest   jupiter             Running             Running 13 minutes ago
4x3h714sm32w         \_ counter-image_web.1   counter-image-v2:latest   saturn              Shutdown            Rejected 13 minutes ago   "No such image: counter-image-…"
6e1ej3z5g3o9         \_ counter-image_web.1   counter-image-v2:latest   saturn              Shutdown            Rejected 13 minutes ago   "No such image: counter-image-…"
wqj4qtgob2ak         \_ counter-image_web.1   counter-image-v2:latest   saturn              Shutdown            Rejected 13 minutes ago   "No such image: counter-image-…"
wdo71a60n2s2         \_ counter-image_web.1   counter-image-v2:latest   saturn              Shutdown            Rejected 13 minutes ago   "No such image: counter-image-…"
9vcduh1x1n2h        counter-image_web.2       counter-image-v2:latest   jupiter             Running             Running 10 minutes ago
d8f0luk167ej         \_ counter-image_web.2   counter-image-v2:latest   neptune             Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
p4ws3j0quf6r         \_ counter-image_web.2   counter-image-v2:latest   neptune             Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
rjrp1y7eaiah         \_ counter-image_web.2   counter-image-v2:latest   neptune             Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
p2tx356u6rnh         \_ counter-image_web.2   counter-image-v2:latest   neptune             Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
ggvq9kcg72x9        counter-image_web.3       counter-image-v2:latest   jupiter             Running             Running 10 minutes ago
b8eyx4tray5w         \_ counter-image_web.3   counter-image-v2:latest   saturn              Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
dnj5bwk7p6lc         \_ counter-image_web.3   counter-image-v2:latest   saturn              Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
sqvicbd6dfa1         \_ counter-image_web.3   counter-image-v2:latest   saturn              Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
kw4gaftzfhfm         \_ counter-image_web.3   counter-image-v2:latest   saturn              Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
ksmi3v455q3e        counter-image_web.4       counter-image-v2:latest   jupiter             Running             Running 10 minutes ago
vw8rheectrgr         \_ counter-image_web.4   counter-image-v2:latest   neptune             Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
a950vc04oc64         \_ counter-image_web.4   counter-image-v2:latest   neptune             Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
prc7h7qxbxcm         \_ counter-image_web.4   counter-image-v2:latest   neptune             Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
anfj9bsnsgli         \_ counter-image_web.4   counter-image-v2:latest   neptune             Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
6y98fr2fmv96        counter-image_web.5       counter-image-v2:latest   jupiter             Running             Running 10 minutes ago
s6urvk2tvai0         \_ counter-image_web.5   counter-image-v2:latest   saturn              Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
irb1oigxm03i         \_ counter-image_web.5   counter-image-v2:latest   saturn              Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
ye01zdajzs5b         \_ counter-image_web.5   counter-image-v2:latest   saturn              Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"
ibog290bux02         \_ counter-image_web.5   counter-image-v2:latest   saturn              Shutdown            Rejected 11 minutes ago   "No such image: counter-image-…"

Check the IMAGE column and it will show you that all your containers are running the new web application version. Hurray!

Test your load balancing

On your leader machine, type:

curl http://localhost:8000

In an excerpt from the HTML output, you will see our latest addition of displaying the machine name.

Welcome - from machine/node: 3effceef408d. In this case it will show the container Unique Id under which the web application is running.

<div class="text-center">
    <h1 class="display-4">Welcome - from machine/node: 3effceef408d</h1>
    <p>Learn about <a href="https://docs.microsoft.com/aspnet/core">building Web apps with ASP.NET Core</a>.</p>
</div>

Fun part: If you keep on typing the same curl http://localhost:8000 command, the container Id will keep changing, indicating that for each new request, docker is load balancing the request. This is taken care by docker internally.

Even if you type curl http://192.168.26.129:8000 or curl http://192.168.26.130:8000 repeatedly, it will still keep on showing you different container Ids.

Conclusion

This completes our two part series of hosting your web application on multiple containers. We also scaled and upgraded the site to multiple instances. We also tested the load balancing whether individual requests are being redirected to different containers. Hope you enjoyed it! Let me know if you have any questions at code@onezeroeight.co

Dance as much as you can 🙂

You may also like...

2 Responses

  1. 77up says:

    Hi, just wanted to mention, I enjoyed this blog post. It was funny.
    Keep on posting!

  2. sbobet says:

    It’s the best time to make some plans for the longer term
    and it’s time to be happy. I have learn this put up and if
    I could I desire to counsel you some interesting issues or advice.
    Perhaps you could write next articles relating to this article.
    I wish to learn even more issues approximately it!

Leave a Reply

Your email address will not be published. Required fields are marked *