Build a live analytics dashboard using Go and MongoDB

One of the most important step to take while taking a website or app into production is analytics and usage statistics. This is important as it allows you to see how users are actually using your app, improve usability and inform future development decisions.

In this tutorial, I will describe how to monitor all requests an application is going to receive, we will use the data gotten from monitoring to track a few metrics such as:

• Most visited links
• Response time for each link
• Total number of requests
• Average response time

Prerequisites

• Golang >=1.7. You can install the Golang toolchain by following this guide.
• MongoDB >=3. You can install this by following this guide.
• A Pusher account.

Starting out

We will start out by setting up our project directory. You will need to create a directory called analytics-dashboard. The location of this directory will depend on the version of the Go toolchain you have:

• If you are running <=1.11, you should create the directory in $GOPATH/src/github.com/pusher-tutorials/analytics-dashboard
• If you are running 1.12 or greater, you can create the directory anywhere.

In the newly created directory, create a .env in the root directory with the following command:

    $ touch .env

In the .env file, you will need to add your credentials. Copy and paste the following contents into the file:

bash
1// analytics-dashboard/.env
2    PUSHER_APP_ID=PUSHER_APP_ID
3    PUSHER_APP_KEY=PUSHER_APP_KEY
4    PUSHER_APP_SECRET=PUSHER_APP_SECRET
5    PUSHER_APP_CLUSTER=PUSHER_APP_CLUSTER
6    PUSHER_APP_SECURE="1"
Copy

Please make sure to replace the placeholders with your own credentials.

MongoDB

MongoDB is going to be used as a persistent datastore and we are going to make use of it’s calculation abilities to build out the functionality I described above.

Since we are building the application in Golang, we will need to fetch a client library that will assist us in connecting and querying the MongoDB database. To that, you should run the following command:

    $ go get -u -v gopkg.in/mgo.v2/...

Once the above command succeeds, you will need to create a new file called analytics.go. In this file, paste the following code:

go
1// analytics-dashboard/analytics.go
2    
3    package main
4    
5    import (
6            "gopkg.in/mgo.v2"
7            "gopkg.in/mgo.v2/bson"
8    )
9    
10    const (
11            collectionName = "request_analytics"
12    )
13    
14    type requestAnalytics struct {
15            URL         string `json:"url"`
16            Method      string `json:"method"`
17            RequestTime int64  `json:"request_time"`
18            Day         string `json:"day"`
19            Hour        int    `json:"hour"`
20    }
21    
22    type mongo struct {
23            sess *mgo.Session
24    }
25    
26    func (m mongo) Close() error {
27            m.sess.Close()
28            return nil
29    }
30    
31    func (m mongo) Write(r requestAnalytics) error {
32            return m.sess.DB("pusher_tutorial").C(collectionName).Insert(r)
33    }
34    
35    func (m mongo) Count() (int, error) {
36            return m.sess.DB("pusher_tutorial").C(collectionName).Count()
37    }
38    
39    type statsPerRoute struct {
40            ID struct {
41                    Method string `bson:"method" json:"method"`
42                    URL    string `bson:"url" json:"url"`
43            } `bson:"_id" json:"id"`
44            NumberOfRequests int `bson:"numberOfRequests" json:"number_of_requests"`
45    }
46    
47    func (m mongo) AverageResponseTime() (float64, error) {
48    
49            type res struct {
50                    AverageResponseTime float64 `bson:"averageResponseTime" json:"average_response_time"`
51            }
52    
53            var ret = []res{}
54    
55            var baseMatch = bson.M{
56                    "$group": bson.M{
57                            "_id":                 nil,
58                            "averageResponseTime": bson.M{"$avg": "$requesttime"},
59                    },
60            }
61    
62            err := m.sess.DB("pusher_tutorial").C(collectionName).
63                    Pipe([]bson.M{baseMatch}).All(&ret)
64    
65            if len(ret) > 0 {
66                    return ret[0].AverageResponseTime, err
67            }
68    
69            return 0, nil
70    }
71    
72    func (m mongo) StatsPerRoute() ([]statsPerRoute, error) {
73    
74            var ret []statsPerRoute
75    
76            var baseMatch = bson.M{
77                    "$group": bson.M{
78                            "_id":              bson.M{"url": "$url", "method": "$method"},
79                            "responseTime":     bson.M{"$avg": "$requesttime"},
80                            "numberOfRequests": bson.M{"$sum": 1},
81                    },
82            }
83    
84            err := m.sess.DB("pusher_tutorial").C(collectionName).
85                    Pipe([]bson.M{baseMatch}).All(&ret)
86            return ret, err
87    }
88    
89    type requestsPerDay struct {
90            ID               string `bson:"_id" json:"id"`
91            NumberOfRequests int    `bson:"numberOfRequests" json:"number_of_requests"`
92    }
93    
94    func (m mongo) RequestsPerHour() ([]requestsPerDay, error) {
95    
96            var ret []requestsPerDay
97    
98            var baseMatch = bson.M{
99                    "$group": bson.M{
100                            "_id":              "$hour",
101                            "numberOfRequests": bson.M{"$sum": 1},
102                    },
103            }
104    
105            var sort = bson.M{
106                    "$sort": bson.M{
107                            "numberOfRequests": 1,
108                    },
109            }
110    
111            err := m.sess.DB("pusher_tutorial").C(collectionName).
112                    Pipe([]bson.M{baseMatch, sort}).All(&ret)
113            return ret, err
114    }
115    
116    func (m mongo) RequestsPerDay() ([]requestsPerDay, error) {
117    
118            var ret []requestsPerDay
119    
120            var baseMatch = bson.M{
121                    "$group": bson.M{
122                            "_id":              "$day",
123                            "numberOfRequests": bson.M{"$sum": 1},
124                    },
125            }
126    
127            var sort = bson.M{
128                    "$sort": bson.M{
129                            "numberOfRequests": 1,
130                    },
131            }
132    
133            err := m.sess.DB("pusher_tutorial").C(collectionName).
134                    Pipe([]bson.M{baseMatch, sort}).All(&ret)
135            return ret, err
136    }
137    
138    func newMongo(addr string) (mongo, error) {
139            sess, err := mgo.Dial(addr)
140            if err != nil {
141                    return mongo{}, err
142            }
143    
144            return mongo{
145                    sess: sess,
146            }, nil
147    }
148    
149    type Data struct {
150            AverageResponseTime float64          `json:"average_response_time"`
151            StatsPerRoute       []statsPerRoute  `json:"stats_per_route"`
152            RequestsPerDay      []requestsPerDay `json:"requests_per_day"`
153            RequestsPerHour     []requestsPerDay `json:"requests_per_hour"`
154            TotalRequests       int              `json:"total_requests"`
155    }
156    
157    func (m mongo) getAggregatedAnalytics() (Data, error) {
158    
159            var data Data
160    
161            totalRequests, err := m.Count()
162            if err != nil {
163                    return data, err
164            }
165    
166            stats, err := m.StatsPerRoute()
167            if err != nil {
168                    return data, err
169            }
170    
171            reqsPerDay, err := m.RequestsPerDay()
172            if err != nil {
173                    return data, err
174            }
175    
176            reqsPerHour, err := m.RequestsPerHour()
177            if err != nil {
178                    return data, err
179            }
180    
181            avgResponseTime, err := m.AverageResponseTime()
182            if err != nil {
183                    return data, err
184            }
185    
186            return Data{
187                    AverageResponseTime: avgResponseTime,
188                    StatsPerRoute:       stats,
189                    RequestsPerDay:      reqsPerDay,
190                    RequestsPerHour:     reqsPerHour,
191                    TotalRequests:       totalRequests,
192            }, nil
193    }
Copy

In the above, we have implemented a few queries on the MongoDB database:

• StatsPerRoute: Analytics for each route visited
• RequestsPerDay: Analytics per day
• RequestsPerHour: Analytics per hour

The next step is to add some HTTP endpoints a user can visit. Without those, the code above for querying MongoDB for analytics is redundant. You will also need to create a logging middleware that writes analytics to MongoDB. And to make it realtime, Pusher Channels will also be used.

To get started with that, you will need to create a file named main.go. You can do that via the command below:

    $ touch main.go

You will also need to fetch some libraries that will be used while building. You will need to run the command below to fetch them:

bash
1$ go get github.com/go-chi/chi
2    $ go get github.com/joho/godotenv
3    $ go get github.com/pusher/pusher-http-go
Copy

In the newly created main.go file, paste the following code:

go
1// analytics-dashboard/main.go
2    
3    package main
4    
5    import (
6            "encoding/json"
7            "flag"
8            "fmt"
9            "html/template"
10            "log"
11            "net/http"
12            "os"
13            "path/filepath"
14            "strconv"
15            "strings"
16            "sync"
17            "time"
18    
19            "github.com/go-chi/chi"
20            "github.com/joho/godotenv"
21            "github.com/pusher/pusher-http-go"
22    )
23    
24    const defaultSleepTime = time.Second * 2
25    
26    func main() {
27            httpPort := flag.Int("http.port", 4000, "HTTP Port to run server on")
28            mongoDSN := flag.String("mongo.dsn", "localhost:27017", "DSN for mongoDB server")
29    
30            flag.Parse()
31    
32            if err := godotenv.Load(); err != nil {
33                    log.Fatal("Error loading .env file")
34            }
35    
36            appID := os.Getenv("PUSHER_APP_ID")
37            appKey := os.Getenv("PUSHER_APP_KEY")
38            appSecret := os.Getenv("PUSHER_APP_SECRET")
39            appCluster := os.Getenv("PUSHER_APP_CLUSTER")
40            appIsSecure := os.Getenv("PUSHER_APP_SECURE")
41    
42            var isSecure bool
43            if appIsSecure == "1" {
44                    isSecure = true
45            }
46    
47            client := &pusher.Client{
48                    AppId:   appID,
49                    Key:     appKey,
50                    Secret:  appSecret,
51                    Cluster: appCluster,
52                    Secure:  isSecure,
53                    HttpClient: &http.Client{
54                            Timeout: time.Second * 10,
55                    },
56            }
57    
58            mux := chi.NewRouter()
59    
60            log.Println("Connecting to MongoDB")
61            m, err := newMongo(*mongoDSN)
62            if err != nil {
63                    log.Fatal(err)
64            }
65    
66            log.Println("Successfully connected to MongoDB")
67    
68            mux.Use(analyticsMiddleware(m, client))
69    
70            var once sync.Once
71            var t *template.Template
72    
73            workDir, _ := os.Getwd()
74            filesDir := filepath.Join(workDir, "static")
75            fileServer(mux, "/static", http.Dir(filesDir))
76    
77            mux.Get("/", func(w http.ResponseWriter, r *http.Request) {
78    
79                    once.Do(func() {
80                            tem, err := template.ParseFiles("static/index.html")
81                            if err != nil {
82                                    log.Fatal(err)
83                            }
84    
85                            t = tem.Lookup("index.html")
86                    })
87    
88                    t.Execute(w, nil)
89            })
90    
91            mux.Get("/api/analytics", analyticsAPI(m))
92            mux.Get("/wait/{seconds}", waitHandler)
93    
94            log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *httpPort), mux))
95    }
96    
97    func fileServer(r chi.Router, path string, root http.FileSystem) {
98            if strings.ContainsAny(path, "{}*") {
99                    panic("FileServer does not permit URL parameters.")
100            }
101    
102            fs := http.StripPrefix(path, http.FileServer(root))
103    
104            if path != "/" && path[len(path)-1] != '/' {
105                    r.Get(path, http.RedirectHandler(path+"/", 301).ServeHTTP)
106                    path += "/"
107            }
108    
109            path += "*"
110    
111            r.Get(path, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
112                    fs.ServeHTTP(w, r)
113            }))
114    }
115    
116    func analyticsAPI(m mongo) http.HandlerFunc {
117            return func(w http.ResponseWriter, r *http.Request) {
118    
119                    data, err := m.getAggregatedAnalytics()
120                    if err != nil {
121                            log.Println(err)
122    
123                            json.NewEncoder(w).Encode(&struct {
124                                    Message   string `json:"message"`
125                                    TimeStamp int64  `json:"timestamp"`
126                            }{
127                                    Message:   "An error occurred while fetching analytics data",
128                                    TimeStamp: time.Now().Unix(),
129                            })
130    
131                            return
132                    }
133    
134                    w.Header().Set("Content-Type", "application/json")
135                    json.NewEncoder(w).Encode(data)
136            }
137    }
138    
139    func analyticsMiddleware(m mongo, client *pusher.Client) func(next http.Handler) http.Handler {
140            return func(next http.Handler) http.Handler {
141                    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
142    
143                            startTime := time.Now()
144    
145                            defer func() {
146    
147                                    if strings.HasPrefix(r.URL.String(), "/wait") {
148    
149                                            data := requestAnalytics{
150                                                    URL:         r.URL.String(),
151                                                    Method:      r.Method,
152                                                    RequestTime: time.Now().Unix() - startTime.Unix(),
153                                                    Day:         startTime.Weekday().String(),
154                                                    Hour:        startTime.Hour(),
155                                            }
156    
157                                            if err := m.Write(data); err != nil {
158                                                    log.Println(err)
159                                            }
160    
161                                            aggregatedData, err := m.getAggregatedAnalytics()
162                                            if err == nil {
163                                                    client.Trigger("analytics-dashboard", "data", aggregatedData)
164                                            }
165                                    }
166                            }()
167    
168                            next.ServeHTTP(w, r)
169                    })
170            }
171    }
172    
173    func waitHandler(w http.ResponseWriter, r *http.Request) {
174            var sleepTime = defaultSleepTime
175    
176            secondsToSleep := chi.URLParam(r, "seconds")
177            n, err := strconv.Atoi(secondsToSleep)
178            if err == nil && n >= 2 {
179                    sleepTime = time.Duration(n) * time.Second
180            } else {
181                    n = 2
182            }
183    
184            log.Printf("Sleeping for %d seconds", n)
185            time.Sleep(sleepTime)
186            w.Write([]byte(`Done`))
187    }
Copy

While the above might seem like a lot, basically what has been done is:

• Line 31 - 33: Parse environment variables from the .env created earlier.

Another reminder to update the .env file to contain your actual credentials

• Line 36 - 56: A server side connection to Pusher Channels is established
• Line 68 - 95: Build an HTTP server.
• Line 139 - 171: A lot is happening here. analyticsMiddleware is used to capture all requests, and for requests that have the path wait/{seconds} , a log is written to MongoDB. It is also sent to Pusher Channels.

Before running the server, you need a frontend to visualize the analytics. The frontend is going to be as simple and usable as can be. You will need to create a new directory called static in your root directory - analytics-dashboard . That can be done with the following command:

    $ mkdir analytics-dashboard/static

In the static directory, create two files - index.html and app.js. You can run the command below to do just that:

    $ touch static/{index.html,app.js}

Open the index.html file and paste the following code:

html
1// analytics-dashboard/static/index.html
2    
3    <!DOCTYPE html>
4    <html lang="en">
5    
6    <head>
7        <meta charset="UTF-8">
8        <meta name="viewport" content="width=device-width, initial-scale=1.0">
9        <meta http-equiv="X-UA-Compatible" content="ie=edge">
10        <title>Realtime analytics dashboard</title>
11    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"
12              integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
13    </head>
14    <body>
15    <div class="container" id="app"></div>
16    <script src="https://cdnjs.cloudflare.com/ajax/libs/handlebars.js/4.1.2/handlebars.js"></script>
17    <script src="https://cdnjs.cloudflare.com/ajax/libs/axios/0.18.0/axios.min.js"></script>
18    <script src="https://js.pusher.com/4.3/pusher.min.js"></script>
19    <script src="/static/app.js"></script>
20    </body>
21    </html>
Copy

While that is an empty page, you will make use of JavaScript to fill it up with useful data. So you will also need to open up the app.js file. In the app.js file, paste the following code:

js
1// analytics-dashboard/static/app.js
2    
3    const appDiv = document.getElementById('app');
4    
5    const tmpl = `
6    <div class="row">
7        <div class="col-md-5">
8            <div class="card">
9                <div class="card-body">
10                    <h5 class="card-title">Total requests</h5>
11                    <div class="card-text">
12                        <h3>\{{total_requests}}</h3>
13                    </div>
14                </div>
15            </div>
16        </div>
17        <div class="col-md-5">
18            <div class="card">
19                <div class="card-body">
20                    <h5 class="card-title">Average response time</h5>
21                    <div class="card-text">
22                        <h3>\{{ average_response_time }} seconds</h3>
23                    </div>
24                </div>
25            </div>
26        </div>
27    </div>
28    
29    <div class="row">
30        <div class="col-md-5">
31            <div class="card">
32                <div class="card-body">
33                    <h5 class="card-title">Busiest days of the week</h5>
34                    <div class="card-text" style="width: 18rem">
35                        <ul class="list-group list-group-flush">
36                            {{#each requests_per_day}}
37                            <li class="list-group-item">
38                                \{{ this.id }} (\{{ this.number_of_requests }} requests)
39                            </li>
40                            {{/each }}
41                        </ul>
42                    </div>
43                </div>
44            </div>
45        </div>
46        <div class="col-md-5">
47            <div class="card">
48                <div class="card-body">
49                    <h5 class="card-title">Busiest hours of day</h5>
50                    <div class="card-text" style="width: 18rem;">
51                        <ul class="list-group list-group-flush">
52                            {{#each requests_per_hour}}
53                            <li class="list-group-item">
54                                \{{ this.id }} (\{{ this.number_of_requests }} requests)
55                            </li>
56                            {{/each}}
57                        </ul>
58                    </div>
59                </div>
60            </div>
61        </div>
62    </div>
63    
64    <div class="row">
65        <div class="col-md-5">
66            <div class="card">
67                <div class="card-body">
68                    <h5 class="card-title">Most visited routes</h5>
69                    <div class="card-text" style="width: 18rem;">
70                        <ul class="list-group list-group-flush">
71                            {{#each stats_per_route}}
72                            <li class="list-group-item">
73                                \{{ this.id.method }} \{{ this.id.url }} (\{{ this.number_of_requests }} requests)
74                            </li>
75                            {{/each}}
76                        </ul>
77                    </div>
78                </div>
79            </div>
80        </div>
81    </div>
82    `;
83    
84    const template = Handlebars.compile(tmpl);
85    
86    writeData = data => {
87      appDiv.innerHTML = template(data);
88    };
89    
90    axios
91      .get('http://localhost:4000/api/analytics', {})
92      .then(res => {
93        console.log(res.data);
94        writeData(res.data);
95      })
96      .catch(err => {
97        console.error(err);
98      });
99    
100    const APP_KEY = 'PUSHER_APP_KEY';
101    const APP_CLUSTER = 'PUSHER_CLUSTER';
102    
103    const pusher = new Pusher(APP_KEY, {
104      cluster: APP_CLUSTER,
105    });
106    
107    const channel = pusher.subscribe('analytics-dashboard');
108    
109    channel.bind('data', data => {
110      writeData(data);
111    });
Copy

Please replace PUSHER_APP_KEY and PUSHER_CLUSTER with your own credentials.

In the above code, we defined a constant called tmpl, it holds an HTML template which we will run through the Handlebars template engine to fill it up with actual data.

With this done, you can go ahead to run the Golang server one. You will need to go to the root directory - analytics-dashboard and run the following command:

bash
1$ go build
2    $ ./analytics-dashboard
Copy

Make sure you have a MongoDB instance running. If your MongoDB is running on a port other than the default 27017, make sure to add -mongo.dsn "YOUR_DSN" to the above command

Also make sure your credentials are in .env

At this stage, you will need to open two browser tabs. Visit http://localhost:4000 in one and http://localhost:4000/wait/2 in the other. Refresh the tab where you have http://localhost:4000/wait/2 and go back to the other tab to see a breakdown of usage activity.

Note you can change the value of 2 in the url to any other digit.

Conclusion

In this tutorial, we’ve built a middleware that tracks every request, and a Golang application that calculates analytics of the tracked requests. We also built a dashboard that displays the relevant data. With Pusher Channels, we’ve been able to update the dashboard in realtime. The full source code can be found on GitHub.