Monitor APIs in realtime using Go

REST is a popular architectural style for providing standards between computer systems on the web, making it easier for systems to communicate with each other. It is mostly used by APIs to provide data to other systems requiring them.

Sometimes, the providers of APIs would like to monitor its use. Monitoring APIs helps provide useful information, such as which endpoints are called most frequently, or what regions are the largest audience using request IP Addresses. This information can then be used to optimize the API.

In this article, we will implement realtime monitoring of a small API built with GoLang, using Pusher. Here’s a preview of what it should look like at the end:

Requirements

To follow along with this article, you will need the following:

• An IDE of your choice e.g. Visual Studio Code.
• Go installed on your computer.
• Basic knowledge of GoLang.
• Basic knowledge of JavaScript (ES6 syntax) and jQuery.
• Basic knowledge of using a CLI tool or terminal.
• Pusher Channels. Create a free sandbox Pusher account or sign in.

Once you have all the above requirements, let’s proceed.

Setting up our codebase

To keep things simple, we’ll be using an already written GoLang CRUD API, which is available on GitHub. We will fork the repository and set it up following the README.md guidelines on installation.

Next, we will set up Pusher in the API project. Pusher is a service that provides a simple implementation of realtime functionality for our web and mobile applications. We'll be using the Channels API to provide realtime updates to our API monitor dashboard.

Create a free sandbox Pusher account or sign in. On the dashboard, create a new Pusher Channels app and copy out the app credentials (App ID, Key, Secret, and Cluster). We will use these credentials in our API.

Now that we have our Channels app, we will install the Pusher Go library by running:

    $ go get github.com/pusher/pusher-http-go

Monitoring our API

We have so far set up a functional CRUD API, and we will now implement monitoring calls to it. In this article, we will monitor:

• The endpoints called with details like name, request type (GET, POST, etc) and URL.
• For each call to an endpoint, we will also take note of:
  • The requesting IP address remove,
  • The response status code for the particular call.

Now that we have defined what to monitor, we will begin by creating models to keep track of the data we acquire.

Creating models for monitoring

Based on our specifications above, we will create two new model files EndPoints.go and EndPointCalls.go. As was used in the base API, we will use the GORM (the GoLang ORM) for managing data storage.

PRO TIP: Our new model files will exist in the models directory and belong to the models package.

In EndPoints.go, we will define the EndPoints object and a method to save endpoints:

bash
1package models
2
3    import (
4        "github.com/jinzhu/gorm"
5    )
6
7    // EndPoints - endpoint model
8    type EndPoints struct {
9        gorm.Model
10        Name, URL string
11        Type      string          `gorm:"DEFAULT:'GET'"`
12        Calls     []EndPointCalls `gorm:"ForeignKey:EndPointID"`
13    }
14
15    // SaveOrCreate - save endpoint called
16    func (ep EndPoints) SaveOrCreate() EndPoints {
17        db.FirstOrCreate(&ep, ep)
18        return ep
19    }
Copy

In the code block above, our model did not re-initialize the GORM instance db, yet it was used. This is because the instance defined in the Movies.go file was global to all members of the package, and so it can be referenced and used by all members of package models.

PRO TIP: Our EndPoints model has an attribute Calls which is an array of EndPointCalls objects. This attribute signifies the one to many relationship between EndPoints and EndPointCalls. For more information on model associations and relationships see the GORM documentation.

Next, we’ll fill in the model definitions and methods for our EndPointCalls model in the EndPointCalls.go file:

bash
1package models
2
3    import (
4        "github.com/jinzhu/gorm"
5        "github.com/kataras/iris"
6    )
7
8    // EndPointCalls - Object for storing endpoints call details
9    type EndPointCalls struct {
10        gorm.Model
11        EndPointID   uint `gorm:"index;not null"`
12        RequestIP    string
13        ResponseCode int
14    }
15
16    // SaveCall - Save the call details of an endpoint
17    func (ep EndPoints) SaveCall(context iris.Context) EndPointCalls {
18        epCall := EndPointCalls{
19            EndPointID:   ep.ID,
20            RequestIP:    context.RemoteAddr(),
21            ResponseCode: context.GetStatusCode(),
22        }
23
24        db.Create(&epCall)
25        return epCall
26    }
Copy

As shown above, our EndPointCalls model defines a SaveCall method, which stores the requesting IP address and the response code of an existing EndPoint object.

Finally, we will update the model migration in the index.go file to include our new models:

bash
1// index.go
2    // ...
3
4    func main() {
5        // ...
6
7        // Initialize ORM and auto migrate models
8        db, _ := gorm.Open("sqlite3", "./db/gorm.db")
9        db.AutoMigrate(&models.Movies{}, &models.EndPoints{}, &models.EndPointCalls{})
10
11        // ...
12    }
Copy

Saving endpoint data for monitoring

Using our newly created models, we will edit the MoviesController.go file to save relevant data when an endpoint is called.

To do this, we will add a private helper method to MoviesController.go, which will save endpoint data with our models. See how below:

bash
1// MoviesController.go
2    // ...
3
4    func (m MoviesController) saveEndpointCall(name string) {
5        endpoint := models.EndPoints{
6            Name: name,
7            URL:  m.Cntx.Path(),
8            Type: m.Cntx.Request().Method,
9        }
10
11        endpoint = endpoint.SaveOrCreate()
12        endpointCall := endpoint.SaveCall(m.Cntx)
13    }
Copy

The saveEndpointCall method takes the name of the endpoint as a parameter. Using the controller’s iris.Context instance, it reads and saves the endpoint path and request method.

Now that this helper method is available, we will call it in each of the endpoint methods in the MoviesController.go file:

bash
1// MoviesController.go
2    // ...
3
4    // Get - get a list of all available movies
5    func (m MoviesController) Get() {
6        movie := models.Movies{}
7        movies := movie.Get()
8
9        go m.saveEndpointCall("Movies List")
10        m.Cntx.JSON(iris.Map{"status": "success", "data": movies})
11    }
12
13    // GetByID - Get movie by ID
14    func (m MoviesController) GetByID(ID int64) {
15        movie := models.Movies{}
16        movie = movie.GetByID(ID)
17        if !movie.Validate() {
18            msg := fmt.Sprintf("Movie with ID: %v not found", ID)
19            m.Cntx.StatusCode(iris.StatusNotFound)
20            m.Cntx.JSON(iris.Map{"status": "error", "message": msg})
21        } else {
22            m.Cntx.JSON(iris.Map{"status": "success", "data": movie})
23        }
24
25        name := fmt.Sprintf("Single Movie with ID: %v Retrieval", ID)
26        go m.saveEndpointCall(name)
27    }
28
29    // ...
Copy

As shown in the snippet above, the saveEndpointCall helper method will be called in each CRUD method.

PRO TIP: The saveEndpointCall method is called as a Goroutine. Calling it this way calls it concurrently with the execution of the endpoint’s method, and allows our monitoring code to not delay or inhibit the response of the API.

Creating the endpoint monitor dashboard

Now that we have implemented monitoring our API’s calls, we will display the data we have accrued on a dashboard.

Registering our template engine

The GoLang framework, Iris, has the ability to implement a range of template engines, which we will take advantage of.

In this section, we will implement the Handlebars template engine, and in our index.go file, we will register it to the app instance:

bash
1// index.go
2    package main
3
4    import (
5        "goggles/controllers"
6        "goggles/models"
7        "github.com/jinzhu/gorm"
8        "github.com/kataras/iris"
9    )
10
11    func main() {
12        app := iris.New()
13
14        tmpl := iris.Handlebars("./templates", ".html")  
15        app.RegisterView(tmpl)
16
17        // ...
18
19        app.Run(iris.Addr("127.0.0.1:1234"))
20    }
Copy

NOTE: We have defined our template engine (Handlebars), to render .html files contained in the templates directory.

Creating the dashboard's route and controller

Now that we have registered our template engine to the application, we will add a route in index.go to render our API monitor dashboard:

bash
1// index.go
2    // ...
3
4    func main() {
5        app := iris.New()
6
7        // ...
8
9        app.Get("/admin/endpoints", func(ctx iris.Context) {
10            dashBoard := controllers.DashBoardController{Cntx: ctx}
11            dashBoard.ShowEndpoints()
12        })
13
14        app.Run(iris.Addr("127.0.0.1:1234"))
15    }
Copy

Above, we have added definitions for the path /admin/endpoints, where we intend to render details of our API endpoints and its calls. We have also specified that the route should be handled by the ShowEndpoints method of DashBoardController.

To create DashBoardController, we will create a DashBoardController.go file in the controllers directory. And in our DashBoardController.go file, we will define the DashBoardController object and its ShowEndpoints method:

bash
1// DashBoardController.go
2    package controllers
3
4    import (
5        "goggles/models"
6        "github.com/kataras/iris"
7        "github.com/kataras/iris/mvc"
8    )
9
10    // DashBoardController - Controller object for Endpoints dashboard
11    type DashBoardController struct {
12        mvc.BaseController
13        Cntx iris.Context
14    }
15
16    // ShowEndpoints - show list of endpoints
17    func (d DashBoardController) ShowEndpoints() {
18        endpoints := (models.EndPoints{}).GetWithCallSummary()
19        d.Cntx.ViewData("endpoints", endpoints)
20        d.Cntx.View("endpoints.html")
21    }
Copy

In ShowEndpoints(), we retrieve our endpoints and a summary of their calls for display. Then we pass this data to our view using d.Cntx.ViewData("endpoints", endpoints), and finally we render our view file templates/endpoints.html using d.Cntx.View("endpoints.html").

Retrieving endpoints and call summaries

To retrieve our list of endpoints and a summary of their calls, we will create a method in the EndPoints.go file called GetWithCallSummary.

Our GetWithCallSummary method should return the endpoints and their call summaries ready for display. For this, we will define a collection object EndPointWithCallSummary with the attributes we need for our display in the EndPoints.go file:

bash
1// EndPoints.go
2    package models
3
4    import (
5        "github.com/jinzhu/gorm"
6    )
7
8    // EndPoints - endpoint model
9    type EndPoints struct {
10        gorm.Model
11        Name, URL string
12        Type      string          `gorm:"DEFAULT:'GET'"`
13        Calls     []EndPointCalls `gorm:"ForeignKey:EndPointID"`
14    }
15
16    // EndPointWithCallSummary - Endpoint with last call summary
17    type EndPointWithCallSummary struct {
18        ID            uint
19        Name, URL     string
20        Type          string
21        LastStatus    int
22        NumRequests   int
23        LastRequester string
24    }
Copy

And then define GetWithCallSummary method to use it as follows:

bash
1// EndPoints.go
2
3    // ...
4
5    // GetWithCallSummary - get all endpoints with call summary details
6    func (ep EndPoints) GetWithCallSummary() []EndPointWithCallSummary {
7        var eps []EndPoints
8        var epsWithDets []EndPointWithCallSummary
9
10        db.Preload("Calls").Find(&eps)
11
12        for _, elem := range eps {
13            calls := elem.Calls
14            lastCall := calls[len(calls)-1:][0]
15            newElem := EndPointWithCallSummary{
16                elem.ID,
17                elem.Name,
18                elem.URL,
19                elem.Type,
20                lastCall.ResponseCode,
21                len(calls),
22                lastCall.RequestIP,
23            }
24            epsWithDets = append(epsWithDets, newElem)
25        }
26
27        return epsWithDets
28    }
29
30    // ...
Copy

Above, the GetWithCallSummary method leverages the Calls attribute of EndPoints, which defines its relationship with EndPointCalls. When retrieving our list of endpoints from the database, we eager load its EndPointCalls data using db.Preload("Calls").Find(&eps).

For more information on eager loading in GORM, see the documentation.

GetWithCallSummary initializes an array of EndPointWithCallSummary, and loops through the EndPoints objects returned from our database to create EndPointWithCallSummary objects.

These EndPointWithCallSummary objects are appended to the initialized array and returned.

💡 The EndPointWithCallSummary is not a model. It is a collection object and does not need to have a table in our database. This is why it does not have its own file and is not passed to index.go for migration.

Implementing the dashboard and displaying data

Now that we have the dashboard’s route, controller and data for display, we will implement the dashboard view to achieve a simple list display of endpoints and their summary data.

Let’s update templates/endpoints.html to have the following code:

bash
1<!-- templates/endpoints.html -->
2    <!DOCTYPE html>
3    <html>
4    <head>
5        <title>Endpoints Monitor Dashboard</title>
6        <link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-beta.3/css/bootstrap.min.css" />
7    </head>
8    <body>
9        <div>
10            <nav class="navbar navbar-default navbar-static-top">
11                <div class="container">
12                    <div class="navbar-header">
13                        <a class="navbar-brand" href="http://127.0.0.1:1234/">
14                            Goggles - A Real-Time API Monitor
15                        </a>
16                    </div>
17                </div>
18            </nav>
19            <div class="container">
20                <div class="row">
21                    <div class="col-xs-12 col-lg-12">
22                        <div class="endpoints list-group">
23                            {{#each endpoints}}
24                                <a id="endpoint-{{ID}}" href="#" class="list-group-item 
25                                list-group-item-{{status_class LastStatus}}">
26                                    <strong>{{name}}</strong>
27                                    <span class="stats">
28                                        {{type}}: <strong>{{url}}</strong> |
29                                        Last Status: <span class="last_status">
30                                        {{LastStatus}}</span> |
31                                        Times Called: <span class="times_called">
32                                        {{NumRequests}}</span> |
33                                        Last Request IP: <span class="request_ip">
34                                        {{LastRequester}}</span>
35                                    </span>
36                                </a>
37                            {{/each}}
38                        </div>
39                    </div>
40                </div>
41            </div>
42        </div>
43        <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
44        <script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-beta.3/js/bootstrap.min.js"></script>
45    </body>
46    </html>
Copy

Above, we render our endpoints list using Bootstrap and our Handlebars template engine. We have also created and used a template function status_class, to colour code our list based on their last call status LastStatus.

We define the status_class template function in index.go after initialising our template engine:

bash
1// index.go
2
3    // ...
4
5    func main() {
6        app := iris.New()
7
8        tmpl := iris.Handlebars("./templates", ".html")
9
10        tmpl.AddFunc("status_class", func(status int) string {
11            if status >= 200 && status < 300 {
12                return "success"
13            } else if status >= 300 && status < 400 {
14                return "warning"
15            } else if status >= 400 {
16                return "danger"
17            }
18            return "success"
19        })
20
21        app.RegisterView(tmpl)
22    }
Copy

Then in our view file we call the function as:

    class="list-group-item list-group-item-{{status_class LastStatus}}"

💡 In the above LastStatus is the function’s parameter.

Adding realtime updates to our dashboard

So far in this article, we have monitored the calls to an API and displayed the data via a dashboard. We will now use Pusher Channels to provide realtime data updates to our dashboard.

Sending realtime data from the backend

Earlier, we installed the Pusher Go library, which we will use to trigger an event when an endpoint is called. In the MoviesController.go file, where the API requests are handled, we will initialize the Pusher client:

bash
1// MoviesController.go
2
3    package controllers
4
5    import (
6        // ...
7        "github.com/pusher/pusher-http-go"
8    )
9
10    // MoviesController - controller object to serve movie data
11    type MoviesController struct {
12        mvc.BaseController
13        Cntx iris.Context
14    }
15
16    var client = pusher.Client{
17        AppId:   "app_id",
18        Key:     "app_key",
19        Secret:  "app_secret",
20        Cluster: "app_cluster",
21    }
22
23    // ...
Copy

Here, we have initialized the Pusher client using the credentials from our earlier created app.

IMORTANT: Replace app_id, app_key, app_secret and app_cluster with your app credentials.

Next, we will use our Pusher client to trigger an event, which would include the endpoint’s data to be displayed in our view. We will do this in the saveEndpointCall method, which logs an endpoint and its call:

bash
1// MoviesController.go
2
3    // ...
4
5    func (m MoviesController) saveEndpointCall(name string) {
6        endpoint := models.EndPoints{
7            Name: name,
8            URL:  m.Cntx.Path(),
9            Type: m.Cntx.Request().Method,
10        }
11        endpoint = endpoint.SaveOrCreate()
12        endpointCall := endpoint.SaveCall(m.Cntx)
13        endpointWithCallSummary := models.EndPointWithCallSummary{
14            ID:            endpoint.ID,
15            Name:          endpoint.Name,
16            URL:           endpoint.URL,
17            Type:          endpoint.Type,
18            LastStatus:    endpointCall.ResponseCode,
19            NumRequests:   1,
20            LastRequester: endpointCall.RequestIP,
21        }
22
23        client.Trigger("goggles_channel", "new_endpoint_request", endpointWithCallSummary)
24    }
Copy

Above, we create an EndPointWithCallSummary object from EndPoints (the endpoint) and EndPointCalls. This EndPointWithCallSummary object has all the data required for display on the dashboard, so will be passed to Pusher for transmission.

Displaying data in realtime on the dashboard

To display the realtime updates of our endpoints, we will use the Pusher JavaScript client and jQuery libraries.

In our view file, templates/endpoints.html, we will first import and initialize a Pusher instance using our app’s credentials:

bash
1<!-- endpoints.html -->
2    <script src="//cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
3    <script src="//cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-beta.3/js/bootstrap.min.js"></script>
4    <script src="//js.pusher.com/4.1/pusher.min.js"></script>
5    <script>
6      const pusher = new Pusher('app_id', {cluster: "app_cluster"});
7    </script>
Copy

IMPORTANT: Replace app_id and app_cluster with values from your app’s credentials.

Next, we will define the following:

• The template for adding new endpoints to our view.
• The functions to append a new endpoint and get the status class of the endpoint.

Finally, we will subscribe to the goggles_channel and listen to the new_endpoint_request event, where our endpoint updates will be transmitted:

bash
1<!-- endpoints.html -->
2    <script>
3    // ...
4
5    const channel = pusher.subscribe("goggles_channel");
6
7    channel.bind('new_endpoint_request', function(data) {
8        let end_point_id = data.ID;
9        if ( $('#endpoint-' + end_point_id).length > 0 ) {
10            let status_class = getItemStatusClass(data['LastStatus']),
11                endpoint     = $('#endpoint-' + end_point_id);
12            let calls = 1 * endpoint.find('span.times_called').text()
13            endpoint.find('span.last_status').text(data['LastStatus']);
14            endpoint.find('span.times_called').text( (calls + 1) )
15            endpoint.removeClass('list-group-item-success');
16            endpoint.removeClass('list-group-item-danger');
17            endpoint.removeClass('list-group-item-warning');
18            endpoint.addClass('list-group-item-' + status_class);
19        } else {
20            addNewEndPoint(data);
21        }
22    });
23
24    // ...
Copy

In the new_endpoint_request event handler, the endpoint data is categorized into either an update scenario (where the endpoint already exists on the dashboard) or a create scenario (where a new list item is created and appended).

Finally, you can build your application and when you run it you should see something similar to what we have in the preview:

Conclusion

In this article, we were able to monitor the realtime requests to a REST API and demonstrate how Pusher works with GoLang applications.