200.2. Predict Future Resource Needs

Weight: 2
Description: Candidates should be able to monitor resource usage to predict future resource needs.
Key Knowledge Areas:
Use monitoring and measurement tools to monitor IT infrastructure usage.
Predict capacity break point of a configuration
Observe growth rate of capacity usage
Graph the trend of capacity usage
Awareness of monitoring solutions such as Icinga2, Nagios, collectd, MRTG and Cacti
The following is a partial list of the used files, terms and utilities:
diagnose
predict growth
resource exhaustion
Predicting The future is one of the hardest works ever, can you remember "Dorithy and the witches of the OZ"? :) Many System Administrators wish if they had some thing like Glass Ball. They wish they could realize what kind of problems might happen next day, in order to avoid it today. But unfortunately that is impossible. So forget about that and lets talk about real environment. In computer science predicting future is possible just by archiving and processing what has happened in past.Mathematical science and statistics can analyze old data and graph trends to predict future.
there are some tools to help us:
Icinga2
Nagios
collectd
MRTG
Cacti
We can spend a whole course for each of these tools, but for LPI exam we are wanted to know about their existence. At beginning we spend some time on collectd to show atlead one solution and see how does it look like. And then we just Introduce other solutions.
collectd
Collectd, a Unix daemon that collects statistics related to system performance and also provides means for storing the values in different formats like RRD (Round Robin Database) files. The statistics gathered by Collectd help to detect the current performance blocks and predict system load in future.
collectd is installed by default in many distributions how ever it can be easily installed from repositories in redhat or debian systems, here we use CentOS7 system:
1
[[email protected] ~]# yum install collectd
Copied!
To configure what to monitor, modify collect.conf file:
1
[[email protected] ~]# rpm -ql collectd | grep etc
2
/etc/collectd.conf
3
/etc/collectd.d
4
​
5
[[email protected] etc]# ls -l  | grep collectd
6
-rw-r--r--.  1 root root     42077 May 17 04:46 collectd.conf
7
drwxr-xr-x.  2 root root         6 May 17 04:46 collectd.d
8
​
9
[[email protected] etc]# wc -l collectd.conf 
10
1820 collectd.conf
Copied!
and that is a huge configuration file consist of different sections . Which are not part of our exam!(File has been abstracted) :
1
##############################################################################
2
# Global                                                                     #
3
#----------------------------------------------------------------------------#
4
# Global settings for the daemon.                                            #
5
##############################################################################
6
​
7
#Hostname    "localhost"
8
#FQDNLookup   true
9
#BaseDir     "/var/lib/collectd"
10
#PIDFile     "/var/run/collectd.pid"
11
#PluginDir   "/usr/lib64/collectd"
12
#TypesDB     "/usr/share/collectd/types.db"
13
​
14
#----------------------------------------------------------------------------#
15
# When enabled, plugins are loaded automatically with the default options    #
16
# when an appropriate <Plugin ...> block is encountered.                     #
17
# Disabled by default.                                                       #
18
#----------------------------------------------------------------------------#
19
#AutoLoadPlugin false
20
​
21
#----------------------------------------------------------------------------#
22
# When enabled, internal statistics are collected, using "collectd" as the   #
23
# plugin name.                                                               #
24
# Disabled by default.                                                       #
25
#----------------------------------------------------------------------------#
26
#CollectInternalStats false
27
​
28
#----------------------------------------------------------------------------#
29
# Interval at which to query values. This may be overwritten on a per-plugin #
30
# base by using the 'Interval' option of the LoadPlugin block:               #
31
#   <LoadPlugin foo>                                                         #
32
#       Interval 60                                                          #
33
#   </LoadPlugin>                                                            #
34
#----------------------------------------------------------------------------#
35
#Interval     10
36
​
37
#MaxReadInterval 86400
38
#Timeout         2
39
#ReadThreads     5
40
#WriteThreads    5
41
​
42
# Limit the size of the write queue. Default is no limit. Setting up a limit is
43
# recommended for servers handling a high volume of traffic.
44
#WriteQueueLimitHigh 1000000
45
#WriteQueueLimitLow   800000
46
​
47
##############################################################################
48
# Logging                                                                    #
49
#----------------------------------------------------------------------------#
50
# Plugins which provide logging functions should be loaded first, so log     #
51
# messages generated when loading or configuring other plugins can be        #
52
# accessed.                                                                  #
53
##############################################################################
54
​
55
LoadPlugin syslog
56
#LoadPlugin logfile
57
#LoadPlugin log_logstash
58
​
59
#<Plugin logfile>
60
#    LogLevel info
61
#    File STDOUT
62
#    Timestamp true
63
#    PrintSeverity false
64
#</Plugin>
65
​
66
#<Plugin log_logstash>
67
#    LogLevel info
68
#    File "/var/log/collectd.json.log"
69
#</Plugin>
70
​
71
#<Plugin syslog>
72
#    LogLevel info
73
#</Plugin>
74
​
75
##############################################################################
76
# LoadPlugin section                                                         #
77
#----------------------------------------------------------------------------#
78
# Lines beginning with a single `#' belong to plugins which have been built  #
79
# but are disabled by default.                                               #
80
#                                                                            #
81
# Lines beginning with `##' belong to plugins which have not been built due  #
82
# to missing dependencies or because they have been deactivated explicitly.  #
83
##############################################################################
84
.
85
..
86
....
87
​
88
LoadPlugin cpu
89
#LoadPlugin cpufreq
90
#LoadPlugin cpusleep
91
​
92
#LoadPlugin df
93
#LoadPlugin disk
94
#LoadPlugin dns
95
​
96
LoadPlugin interface
97
#LoadPlugin ipc
98
#LoadPlugin ipmi
99
#LoadPlugin iptables
100
#LoadPlugin ipvs
101
#LoadPlugin irq
102
#LoadPlugin java
103
LoadPlugin load
104
##LoadPlugin lpar
105
#LoadPlugin lua
106
#LoadPlugin lvm
107
#LoadPlugin madwifi
108
#LoadPlugin mbmon
109
#LoadPlugin mcelog
110
#LoadPlugin md
111
#LoadPlugin memcachec
112
#LoadPlugin memcached
113
LoadPlugin memory
114
##LoadPlugin mic
115
#LoadPlugin modbus
116
#LoadPlugin mqtt
117
#LoadPlugin multimeter
118
#LoadPlugin mysql
119
##LoadPlugin netapp
120
#LoadPlugin netlink
121
LoadPlugin network
122
#LoadPlugin nfs
123
​
124
#LoadPlugin rrdtool
125
......
126
....
127
..
128
.
129
##############################################################################
130
# Plugin configuration                                                       #
131
#----------------------------------------------------------------------------#
132
# In this section configuration stubs for each plugin are provided. A desc-  #
133
# ription of those options is available in the collectd.conf(5) manual page. #
134
##############################################################################
135
.
136
..
137
...
138
#<Plugin apache>
139
#  <Instance "local">
140
#    URL "http://localhost/status?auto"
141
#    User "www-user"
142
#    Password "secret"
143
#    CACert "/etc/ssl/ca.crt"
144
#  </Instance>
145
#</Plugin>
146
​
147
#<Plugin rrdtool>
148
#       DataDir "/var/lib/collectd/rrd"
149
#       CreateFilesAsync false
150
#       CacheTimeout 120
151
#       CacheFlush   900
152
#       WritesPerSecond 50
153
#</Plugin>
154
​
155
...
156
..
157
##############################################################################
158
# Filter configuration                                                       #
159
#----------------------------------------------------------------------------#
160
# The following configures collectd's filtering mechanism. Before changing   #
161
# anything in this section, please read the `FILTER CONFIGURATION' section   #
162
# in the collectd.conf(5) manual page.                                       #
163
##############################################################################
164
​
165
# Load required matches:
166
#LoadPlugin match_empty_counter
167
#LoadPlugin match_hashed
168
#LoadPlugin match_regex
169
#LoadPlugin match_value
170
#LoadPlugin match_timediff
171
..
172
...
173
..
174
#############################################################################
175
# Threshold configuration                                                    #
176
#----------------------------------------------------------------------------#
177
# The following outlines how to configure collectd's threshold checking      #
178
# plugin. The plugin and possible configuration options are documented in    #
179
# the collectd-threshold(5) manual page.                                     #
180
##############################################################################
181
​
182
#LoadPlugin "threshold"
183
#<Plugin threshold>
184
#  <Type "foo">
185
#    WarningMin    0.00
186
#    WarningMax 1000.00
187
#    FailureMin    0.00
188
#    FailureMax 1200.00
189
#    Invert false
190
#    Instance "bar"
191
#  </Type>
192
#
193
#  <Plugin "interface">
194
#    Instance "eth0"
195
#    <Type "if_octets">
196
#      FailureMax 10000000
197
#      DataSource "rx"
198
#    </Type>
199
#  </Plugin>
200
#
201
#  <Host "hostname">
202
#    <Type "cpu">
203
#      Instance "idle"
204
#      FailureMin 10
205
#    </Type>
206
...
207
..
Copied!
In order to monitor specific item we can enable it by removing comment sign in Load Plugin and plugin configuration sections. What we have done here is enabling network module .
Lets restart the service :
1
[[email protected] etc]# systemctl restart collectd.service
Copied!
Now we have to give collectd enough time to gather information about network and other modules which have been enabled.Unfortunately collectd doesn't come with an interface for viewing collected data by default, so we need some extra works to install and a web interface for that.
Configure Collectd-web to monitor a server
Collectd-web is a web front-end monitoring tool based on RRDtool (Round-Robin Database Tool), which interprets and graphical outputs the data collected by the Collectd service on Linux systems.
So first lets install and enable rrdtool module:
1
[[email protected] ~]# yum install  collectd-rrdtool.x86_64 rrdtool rrdtool-devel rrdtool-perl perl-HTML-Parser perl-JSON perl-CGI
Copied!
and modify this parts of collectd.conf file:
1
.
2
..
3
...
4
LoadPlugin rrdtool
5
...
6
..
7
​
8
<Plugin rrdtool>
9
        DataDir "/var/lib/collectd/rrd"
10
#       CreateFilesAsync false
11
#       CacheTimeout 120
12
#       CacheFlush   900
13
#       WritesPerSecond 50
14
</Plugin>
15
...
16
..
17
.
Copied!
installing git "
1
[[email protected] ~]# yum install git
Copied!
and Cloning collectd-web from the its repository:
1
[[email protected] opt]# ls -l | grep collectd
2
drwxr-xr-x. 7 root root 251 Oct 15 19:29 collectd-web
3
[[email protected] opt]# cd collectd-web/
4
[[email protected] collectd-web]# ls
5
AUTHORS  CHANGELOG      COPYING  index.html  media       runserver.py
6
cgi-bin  check_deps.sh  docs     iphone      README.rst
Copied!
We use Collectd-web CGI scripts which interprets and generates the graphical html page statistics. First we need to make it executable for further use :
1
[[email protected] collectd-web]# chmod +x cgi-bin/graphdefs.cgi
Copied!
By default, collectd-web is configured to run on the loopback address 127.0.0.1. You will need to edit the runserver.py script and change 127.0.0.1 to 0.0.0.0, in order to bind on all network interface IP Adresses and access the collectd-web interface from a remote machine.
1
[[email protected] collectd-web]# vim runserver.py 
2
[[email protected] collectd-web]# cat runserver.py 
3
#!/usr/bin/env python
4
​
5
import CGIHTTPServer
6
import BaseHTTPServer
7
from optparse import OptionParser
8
​
9
class Handler(CGIHTTPServer.CGIHTTPRequestHandler):
10
    cgi_directories = ["/cgi-bin"]
11
​
12
PORT = 8888
13
​
14
def main():
15
    parser = OptionParser()
16
    opts, args = parser.parse_args()
17
    if args:
18
        httpd = BaseHTTPServer.HTTPServer((args[0], int(args[1])), Handler)
19
        print "Collectd-web server running at http://%s:%s/" % (args[0], args[1])
20
    else:
21
        httpd = BaseHTTPServer.HTTPServer(("0.0.0.0", PORT), Handler)
22
        print "Collectd-web server running at http://%s:%s/" % ("127.0.0.1", PORT)
23
    httpd.serve_forever()
24
​
25
if __name__ == "__main__":
26
    main()
Copied!
Next we need to create /etc/collectd directory and collection.conf file inside that, this specifies where collectd-web can read data from:
1
[[email protected] collectd-web]# cd /etc/
2
[[email protected] etc]# mkdir collectd
3
[[email protected] etc]# cd collectd
4
[[email protected] collectd]# touch collection.conf 
5
[[email protected] collectd]# vim collection.conf 
6
[[email protected] collectd]# cat collection.conf 
7
DataDir: "/var/lib/collectd/rrd/"
Copied!
After modifying the Python server script, we can start the server by running runserver.py :
1
[[email protected] collectd-web]# ./runserver.py &
Copied!
If we have firewall up and running , add required port for collect-web insterface.Here we are using CentOS so we should deal with firewalld:
1
[[email protected] ~]# firewall-cmd --zone=public --permanent --add-port=8888/tcp
2
success
3
[[email protected] ~]# firewall-cmd --reload
4
success
Copied!
Finally, we can access the collectd-web interface at the URL http://our-server-ip:8888 in our web browser:
How does it work? Collectd uses data sets which are defines in types.db:
1
[[email protected] ~]# ls /usr/share/collectd/
2
types.db
3
[[email protected] ~]# cat /usr/share/collectd/types.db | grep cpu
4
cpu                     value:DERIVE:0:U
5
cpu_affinity            value:GAUGE:0:1
6
cpufreq                 value:GAUGE:0:U
7
ps_cputime              user:DERIVE:0:U, syst:DERIVE:0:U
8
vcpu                    value:GAUGE:0:U
9
virt_cpu_total          value:DERIVE:0:U
10
virt_vcpu               value:DERIVE:0:U
Copied!
to gather data in a special format(Round Robin DataBase ) in /var/lib/directory/rrd directory which we previously enabled in collectd.conf:
1
[[email protected] ~]# cat /etc/collectd.conf | grep rrd
2
#LoadPlugin rrdcached
3
LoadPlugin rrdtool
4
#<Plugin rrdcached>
5
#    DaemonAddress "unix:/tmp/rrdcached.sock"
6
#    DataDir "/var/lib/collectd/rrd"
7
<Plugin rrdtool>
8
#    DataDir "/var/lib/collectd/rrd"
9
​
10
​
11
  [[email protected] ~]# tree /var/lib/collectd/rrd
12
/var/lib/collectd/rrd
13
└── centos7-2.example.com
14
    ├── cpu-0
15
    │   ├── cpu-idle.rrd
16
    │   ├── cpu-interrupt.rrd
17
    │   ├── cpu-nice.rrd
18
    │   ├── cpu-softirq.rrd
19
    │   ├── cpu-steal.rrd
20
    │   ├── cpu-system.rrd
21
    │   ├── cpu-user.rrd
22
    │   └── cpu-wait.rrd
23
    ├── cpu-1
24
    │   ├── cpu-idle.rrd
25
    │   ├── cpu-interrupt.rrd
26
    │   ├── cpu-nice.rrd
27
    │   ├── cpu-softirq.rrd
28
    │   ├── cpu-steal.rrd
29
    │   ├── cpu-system.rrd
30
    │   ├── cpu-user.rrd
31
    │   └── cpu-wait.rrd
32
    ├── cpu-2
33
    │   ├── cpu-idle.rrd
34
    │   ├── cpu-interrupt.rrd
35
    │   ├── cpu-nice.rrd
36
    │   ├── cpu-softirq.rrd
37
    │   ├── cpu-steal.rrd
38
    │   ├── cpu-system.rrd
39
    │   ├── cpu-user.rrd
40
    │   └── cpu-wait.rrd
41
    ├── cpu-3
42
    │   ├── cpu-idle.rrd
43
    │   ├── cpu-interrupt.rrd
44
    │   ├── cpu-nice.rrd
45
    │   ├── cpu-softirq.rrd
46
    │   ├── cpu-steal.rrd
47
    │   ├── cpu-system.rrd
48
    │   ├── cpu-user.rrd
49
    │   └── cpu-wait.rrd
50
    ├── interface-ens33
51
    │   ├── if_dropped.rrd
52
    │   ├── if_errors.rrd
53
    │   ├── if_octets.rrd
54
    │   └── if_packets.rrd
55
    ├── interface-lo
56
    │   ├── if_dropped.rrd
57
    │   ├── if_errors.rrd
58
    │   ├── if_octets.rrd
59
    │   └── if_packets.rrd
60
    ├── interface-virbr0
61
    │   ├── if_dropped.rrd
62
    │   ├── if_errors.rrd
63
    │   ├── if_octets.rrd
64
    │   └── if_packets.rrd
65
    ├── interface-virbr0-nic
66
    │   ├── if_dropped.rrd
67
    │   ├── if_errors.rrd
68
    │   ├── if_octets.rrd
69
    │   └── if_packets.rrd
70
    ├── load
71
    │   └── load.rrd
72
    └── memory
73
        ├── memory-buffered.rrd
74
        ├── memory-cached.rrd
75
        ├── memory-free.rrd
76
        ├── memory-slab_recl.rrd
77
        ├── memory-slab_unrecl.rrd
78
        └── memory-used.rrd
79
​
80
11 directories, 55 files
Copied!
Okey do not forget that the installation process is not part of LPIC exam and based on your distribution and version the installation steps might be different from what we have done here.
Awareness of other monitoring tools:
There are some monitoring tools which are available both in open source (in core versions ) as well as enterprise versions or are incorporated with other products from enterprise perspective. they offer support and additional functionality.
Nagios
Nagios has been available from late 90s, early 2000. It is really a collection of production of products.
Nagios Core : freely available open source core monitoring system.
Collectd-nagios plugin : Query information form the collectd daemon and then create some beautiful graphs in nagios monitoring dashboard.
Nagios XI : Full enterprise package.
nagios Log Server
Nagios Network analyzer
Nagiod Fusion 
...
web site: https://www.nagios.com​
MRTG
Multi Router Traffic Grapher (MRTG) is a network admins tool that offer enterprise extentions to mrtg including Scrutinizer , FlowPro and replicator.
MRTG monitors network traffic loads, reads that traffic and create the statistics from the informations from routers and switches and creates logs for later use. It is written in perl. It works in Unix, Linux and windows
web site: https://www.mrtg.com/​
Cacti
Cacti is a complete network graphing solution designed to harness the power of RRDTool's data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box. All of this is wrapped in an intuitive, easy to use interface that makes sense for LAN-sized installations up to complex networks with thousands of devices.
Cacti can be used in conjunction with collectd due to the fact both use rrdTools.
web site: https://www.cacti.net/​
and to summarize, take a look at comparison chart below:
​
Cacti
Collectd
Icinga
Nagios
IP SLA Report
Yes
No
Via Plugin
Via Pugin
Logical Grouping
Yes
No
Yes
Yes
Trending
Yes
No
Yes
Yes
Trend Prediction
Yes
No
No
No
Auto Discovery
Yes
Push Model ,Multicast possible
Via Plugin
Via Plugin
Agentless
Yes
Supported
Supported
Supported
SNMP
Yes
Yes
Via Plugin
Via Plugin
Syslog
Yes
Yes
Via Plugin
Via PLugin
Plugins
Yes
Yes
Yes
Yes
Triggers / Alerts
Yes
Yes
Yes
Yes
WebApp
Full Controll
Viewing
Full Control
Yes
Distributed Monitoring
Yes
Yes
Yes
Yes
Inventory
Yes
No
Via Plugin
Via PLugin
Platform
PHP
C
C
C
Data Storage Method
RRD tool,MySQL
RRD tool
MySQL,PostgreSQL,Oracle Database
Flat File, SQL, MySQL
License
Commercial
GPL V2
GPL
GPL
Maps
Yes
No
Via Plugin
Yes
Access Control
Yes
Apache acl
Yes
Yes
Do not forget, Capacity Calculation, Predicting future needs and have a good sense of IT environment need years of experience.
.
.
.
sources:
​https://www.tecmint.com/install-collectd-and-collectd-web-to-monitor-server-resources-in-linux/​
​https://devops.profitbricks.com/tutorials/how-to-install-and-configure-collectd-and-collectd-web-to-monitor-a-centos-7-server/​
200.1. Measure and Troubleshoot Resource Usage
201.1. Kernel Components
Last modified 2yr ago
Copy link
Contents
collectd
Configure Collectd-web to monitor a server
Awareness of other monitoring tools:
Nagios
MRTG
Cacti