REDtalks.live #38 – Grafana Dashboard for BigStats

/ Nathan Pearce / 1 Comment

Having good data is only part of a solution. There’s loads of data on my laptop, for example, that goes unused almost everyday. However, the practice of putting data to work via meaningful representation is how we plan effectively and make informed decisions!

For those who haven’t been following the development of BigStats, its an open source project for ‘exporting’ telemetry off of F5 BIG-IP devices to various destinations and databases. Now, with the release of the ‘Grafana Dashboard for BigStats’, you can start realizing the value of that exported data.

dashboard_overview1

Requires BigStats v0.4 or newer (get that here: https://github.com/npearce/BigStats) installed on your BIG-IP.

Take it for a spin: https://grafana.com/dashboards/7958

Would be great to hear your thoughts!

Twitter: @PearceNathan
https://redtalks.live/BigStats/

 

REDtalks.live #37 – BigStats v0.4 released

/ Nathan Pearce / 1 Comment

BigStats v0.4 has been released! Yay!!! BigStats-300dpi

Way back in REDtalks.live #32 – Stats & Dashboards (May 25th) I demonstrated a prototype solution for getting useful data out of a BIG-IP and into various telemetry pipelines… Well, that was version 0.1, and now we’re up to version 0.4! So, what’s happened since?

Highlights from the RELEASE_NOTES.md file:

  1. We now have a RELEASE_NOTES.md file to track all the awesome!
  2. Support for Apache Kafka message brokers
  3. Support for both AS3 declarations AND traditional configurations (it originally only supported the AS3 declarative interface).
  4. Support for output sizing: Small (VIP Stats only), Medium (VIP + Pool stats), Large (working on this….)
  5. Support for Device Stats (RAM and CPU)
  6. Apache Kafka ‘topic’ config:
    1. All data in one topic, or;
    2. Separate topics per BIG-IP Tenant
  7. Enforced minimum polling interval of 10 seconds (play safe, kids)
  8. Added “config.enabled: true|false” to the config because, sometimes you want some peace and quiet.
  9. Provided a BigStatsSettings object schema (schema validator might be coming soon..)

Here’s an sample stats object built from crawling the running config of a BIG-IP every ‘config.interval: n‘ seconds:

{
        "ip-172-31-1-20-us-west-1-compute-internal": {
                "services": {
                        "Tenant_01/App1": {
                                "/Tenant_01/App3/172.31.4.11:80": {
                                        "clientside_curConns": 0,
                                        "clientside_maxConns": 0,
                                        "clientside_bitsIn": 0,
                                        "clientside_bitsOut": 0,
                                        "clientside_pktsIn": 0,
                                        "clientside_pktsOut": 0,
                                        "/Tenant_01/App1/web_pool1": [
                                                {
                                                        "172.31.10.112:80": {
                                                                "serverside_curConns": 0,
                                                                "serverside_maxConns": 0,
                                                                "serverside_bitsIn": 0,
                                                                "serverside_bitsOut": 0,
                                                                "serverside_pktsIn": 0,
                                                                "serverside_pktsOut": 0,
                                                                "monitorStatus": "down"
                                                        }
                                                },
                                                {
                                                        "172.31.10.111:80": {
                                                                "serverside_curConns": 0,
                                                                "serverside_maxConns": 0,
                                                                "serverside_bitsIn": 0,
                                                                "serverside_bitsOut": 0,
                                                                "serverside_pktsIn": 0,
                                                                "serverside_pktsOut": 0,
                                                                "monitorStatus": "down"
                                                        }
                                                },
                                                {
                                                        "172.31.10.113:80": {
                                                                "serverside_curConns": 0,
                                                                "serverside_maxConns": 0,
                                                                "serverside_bitsIn": 0,
                                                                "serverside_bitsOut": 0,
                                                                "serverside_pktsIn": 0,
                                                                "serverside_pktsOut": 0,
                                                                "monitorStatus": "down"
                                                        }
                                                },
                                                {
                                                        "172.31.10.114:80": {
                                                                "serverside_curConns": 0,
                                                                "serverside_maxConns": 0,
                                                                "serverside_bitsIn": 0,
                                                                "serverside_bitsOut": 0,
                                                                "serverside_pktsIn": 0,
                                                                "serverside_pktsOut": 0,
                                                                "monitorStatus": "down"
                                                        }
                                                }
                                        ]
                                }
                        },
                        "Common": {
                                "/Common/172.31.4.200:80": {
                                        "clientside_curConns": 0,
                                        "clientside_maxConns": 0,
                                        "clientside_bitsIn": 0,
                                        "clientside_bitsOut": 0,
                                        "clientside_pktsIn": 0,
                                        "clientside_pktsOut": 0,
                                        "/Common/noAS3_POOL": [
                                                {
                                                        "172.31.10.200:8080": {
                                                                "serverside_curConns": 0,
                                                                "serverside_maxConns": 0,
                                                                "serverside_bitsIn": 0,
                                                                "serverside_bitsOut": 0,
                                                                "serverside_pktsIn": 0,
                                                                "serverside_pktsOut": 0,
                                                                "monitorStatus": "down"
                                                        }
                                                },
                                                {
                                                        "172.31.10.201:8080": {
                                                                "serverside_curConns": 0,
                                                                "serverside_maxConns": 0,
                                                                "serverside_bitsIn": 0,
                                                                "serverside_bitsOut": 0,
                                                                "serverside_pktsIn": 0,
                                                                "serverside_pktsOut": 0,
                                                                "monitorStatus": "down"
                                                        }
                                                },
                                                {
                                                        "172.31.10.202:8080": {
                                                                "serverside_curConns": 0,
                                                                "serverside_maxConns": 0,
                                                                "serverside_bitsIn": 0,
                                                                "serverside_bitsOut": 0,
                                                                "serverside_pktsIn": 0,
                                                                "serverside_pktsOut": 0,
                                                                "monitorStatus": "down"
                                                        }
                                                }
                                        ]
                                }
                        }
                },
                "device": {
                        "memory": {
                                "memoryTotal": 7574732800,
                                "memoryUsed": 1525312880
                        },
                        "cpu0": {
                                "cpuIdle": 161495459,
                                "cpuIowait": 169763,
                                "cpuSystem": 292088,
                                "cpuUser": 973939
                        },
                        "cpu1": {
                                "cpuIdle": 160343033,
                                "cpuIowait": 68690,
                                "cpuSystem": 426881,
                                "cpuUser": 992052
                        }
                }
        }
}

Here’s the code/docs: https://github.com/npearce/BigStats

In addition to all this awesome, I’ve also shared my lab environment setup details! So, if you want to build a Graphite/Grafana demo like in episode #32, or maybe you want to test against a single node Kafka Broker but don’t know how to do this, well, look no further. Here are all my lab setup instructions: https://gist.github.com/npearce/

Thanks for listening!

REDtalks.live #36 – Christian Weber from GitHub on Infrastructure as Code

/ Nathan Pearce / 1 Comment

Delighted to have GitHub’s Christian Weber on the show this week to share his experiences with Infrastructure as Code.

Watch this episode to hear Christian explain some personal experiences, from his days before he worked at GitHub, where Infrastructure/Configuration as Code would have made a big difference. Learn from first-hand experience how certain practices can bring order back to a dire situations.  But if you’re too busy right now, you could just come and hear our talk at the F5 Agility conference in Boston (August 13th – 16th) titled “Super-NetOps: Configuration as Code”

“…by treating everything, like a configuration file, as a template we go back into our previous state without relying on a failure state as a means of recovery…”

Christian Weber, GitHub

 

Enough words from me, Christian’s wisdom awaits you in the video below:

 

Thanks for listening!

REDtalks.live #35 – How to get started

/ Nathan Pearce / 2 Comments

In this episode I’m joined by F5 Consultant, Vinnie Mazza, to hear his experiences with getting started on his programmability and automation journey. Vinnie shares with us how a project outside of work  inspired him to learn new skills that, coincidentally, are directly transferable to the workplace.

In this video you will see how, armed with no more than curiosity and a couple of Raspberry Pie’s, Vinnie revolutionized a local police training facility that was in dire need of improvement. The skills he learned from this project not only helped others but also took him on a journey through scripting languages and frameworks that turned him into the automation-ninja that many customers get to appreciate today.

Remember, listeners! Everyone starts from the beginning. Find a project, doesn’t have to be at work, and see what you can come up with.

Linked referenced in this episode:

Thanks for listening (and sorry for having some mic problems this week… New microphone ordered!)

REDtalks.live #34 – GitHub Integration v0.2 released

/ Nathan Pearce / 1 Comment

Greetings automators!!! I am delighted to be sharing with you the release of the GitHub Webhook Server for BIG-IP v0.2. w00t!

This release was a major re-write that touched almost every line of code. Why? Well, as with many early prototypes, in the first release I just wanted to see if it could be done. And from the experiment I learned a lot about integrating with GitHub, took a lot of notes, and, well, v0.2 is the result of all those findings: a more robust integration that you should totally take a look at yourself!

The two key adds to version 0.2 are:

  • deployment queuing – so we don’t DDoS the Control Plane with many concurrent declaration commits at once
  • octokit –  in v0.1 I used native HTTP calls to the GitHub API. In v0.2 I switched to the GitHub octokit/rest.js node module, which simplified a lot of code. And simple = robust.

There’s a couple other features I snuck in there, which you can see in this demonstration video:

 

If there are any features you’d like to see, please create a GitHub Issue against the repository, here: https://github.com/npearce/BigHook/issues

For more episode on Network Infrastructure as Code, go here: Network Configuration as Code

Thanks for listening!

REDtalks.live #33 – Joel King on What is Code?

/ Nathan Pearce / 1 Comment

Great to have Joel King join me again on REDtalks.live to answer the question, “What is code?”

The answer isn’t as straightforward as you may think, not with practices like Infrastructure as Code at play. In this episode, Joel does a great job at clarifying some misconceptions and clearing up some concepts that should not have been assumed one way or the other.

Also, a significant bonus, I learned more about the Apollo Space Missions in this episode than I thought I’d ever know. Thanks for the wealth of space-launch knowledge, Joe.

Thanks for listening!

REDtalks.live #32 – Stats & Dashboards

/ Nathan Pearce / 2 Comments

We appear to be leaving the era of polling devices for stats and then processing them into something meaningful. This is what happy customers are telling me! So, how should we be operating in the future?

Organizations are standardizing on app-centric performance monitoring and dash-boarding technologies. They’re using open source technologies like ‘StatsD’, ‘GraphiteDB’, ‘Grafana’, and ‘Prometheus’. And its up to the infrastructure vendors to present useful data to these systems, or get out of the way!

To address such asks I set out on a new project called ‘BigStats’. BigStats is an small piece of code installed onto an F5 BIG-IP App Services device that enables it to push application statistics into popular databases and dashboards.

Watch this episode to see how simple this solution really is:

Here’s the code: GitHub

Thanks for listening!

REDtalks.live #31 – Network Configuration as Code – Part 3

/ Nathan Pearce / 3 Comments

Install, Setup and Troubleshooting

Other episodes in this series:

Welcome to part 3 in this Network Infrastructure as Code series. In the previous episodes I provided some demonstrations and also best practices with Role-based Access Control and Change Rollbacks. In this episode we’ll cover how to setup your own Network Infrastructure as Code environments, in addition to some troubleshooting tips should you run into any problems.

Thanks for listening!

REDtalks.live #30 – Network Configuration as Code – Part 2

/ Nathan Pearce / 3 Comments

Role-based Access Control and Change Rollback

Other episodes in this series:
Welcome to part 2 in this Network Infrastructure as Code series. In the previous episode we introduced the concept of Network Infrastructure as Code and performed a quick demonstration. In this episode we’re going to cover best practices around Access Control and how to roll back changes.
Infrastructure as Code is a great stepping stone towards integrated and automated ‘continuous deployment’. However, for those not quite ready to hand over control to an automated system, applying traditional approval processes to network infrastructure as code is probably a good idea. You’ll also notice that Network Infrastructure as Code has great change back-out capabilities, much better than traditional management models.
Enough of the words, take a look at the video below and let me know your thoughts!
Thanks for listening!

REDtalks.live #29 – Network Configuration as Code – Part 1

/ Nathan Pearce / 3 Comments

Introduction to Network Infrastructure as Code

Other episodes in this series:

Greetings viewers and welcome to a short series of technical demonstrations on Infrastructure as Code.

In my field experience I noticed that most Infrastructure as Code implementations were oriented around ‘server’ infrastructure. As a huge fan of developer tooling being used to improve traditional operations, I thought I’d take a look at bringing Infrastructure as Code practices to managing F5’s BIG-IP App Services appliances.

In this video you’ll see the management of L4 – L7 App Services entirely from within GitHub Enterprise. Totes cool, right! In this first episode I’l just run through a quick demo, followed by a more detailed review in Part 2 (coming soon).

 

Thanks for listening!