Analyzing Logs with the pagely-bucket Tool

Here at Pagely, we want to make sure that your site is running as healthy as possible. In addition to providing the best possible level of service to identify and resolve issues for you, we also make several of our tools available for you to use.

One of those tools is the pagely-bucket tool. It allows you to review your site's logs much quicker by giving you the information in an easier-to-read format.

Accessing the pagely-bucket Tool

To review your logs with the pagely-bucket tool, all you need to do is log into your site over SSH and run the pagely-bucket command. By default without defining a specific log file, the command will automatically read from your site's access logs.

Of course, you can further customize the behavior of the pagely-bucket tool to best suit the information that you're gathering. We'll go over those scenarios in the next sections.

Understanding the pagely-bucket Tool's Output

When running the pagely-bucket tool, your output will look something like this:

time by minute  !HIT total 200 301 302 304 404 403 429 499 500 503 504 other Avg Avg>50 Max
2020-03-20 20:57 12   12    11  0   0   0   0   0   0   0   0   0   0   1     13  131    131
2020-03-20 20:58 8    8     8   0   0   0   0   0   0   0   0   0   0   0     1   NAN    3
2020-03-20 20:59 9    9     9   0   0   0   0   0   0   0   0   0   0   0     1   NAN    2
2020-03-20 21:00 8    8     8   0   0   0   0   0   0   0   0   0   0   0     2   NAN    5
2020-03-20 21:01 8    8     8   0   0   0   0   0   0   0   0   0   0   0     1   NAN    2
2020-03-20 21:02 11   11    11  0   0   0   0   0   0   0   0   0   0   0     21  210    210
2020-03-20 21:03 11   11    8   0   0   0   0   0   0   0   0   0   0   3     1   NAN    3
2020-03-20 21:04 8    8     8   0   0   0   0   0   0   0   0   0   0   0     1   NAN    6
2020-03-20 21:05 8    8     8   0   0   0   0   0   0   0   0   0   0   0     1   NAN    1

Response Codes
Code Count Percentage
200 9,901 99%
444 42    0%
400 25    0%
500 19    0%
Total 9987

Response Times
MS # = 5k
0:   9,721 ##
100: 203   #
200: 63    #

Inside the primary table of the command's output, you'll see a per-minute analysis of the requests that are coming into your sites. Within each column, you'll see details on what occurred during that particular minute.

The first set of columns are the !HIT and total columns. The !HIT column contains the total number of non-cached requests that were processed that minute, while the total column contains the total number of cached and uncached requests.

Next, you'll see columns for various different response codes, such as 200 (OK), 404 (Not Found), and 500 (Internal Server Error).

The final set of columns are the Avg, Avg>50, and Max columns. These columns show the time that it took to process the request, in milliseconds.

Below the table, there are the Response Codes and Response Times sections. These sections provide you with a total count of response codes and how long requests take to process. Using these sections can give you a pretty good look at how your site is performing as a whole.

Specifying a Log File

If the data you're gathering is in a different log file or an older log that has since been rotated, you can specify the log file by appending the command with the file path like this:

pagely-bucket /mnt/log/gateway2/access.log

Filtering By Status Codes

When gathering applicable information, you might need to only output information for particular status codes. To do so, all you need to do is append the pagely-bucket command with the first digit of the status code.

For example, if you only want to limit the results to 5xx status codes, you would use the following command:

pagely-bucket 5

Similarly, if you wanted to include both 3xx and 5xx status codes, you'd do so like this:

pagely-bucket 3 5

Displaying Request Method Data

By default, pagely-bucket doesn't display information for the request methods; only status codes. To add request methods to the command output, you can use the m option like this:

pagely-bucket m

Processing From Standard Input

The pagely-bucket tool can also process logs from the standard input. This allows for much more robust filtering of the logs before they're processed. To tell pagely-bucket that you're piping standard input into it, just add the - option.

For example, if we wanted to only process the last 10,000 lines of a log file, we can use the tail command to read the contents of the file, then pass the output on to pagely-bucket by using the "-" option like this:

tail -n 10000 /mnt/log/gateway2/access.log | pagely-bucket -