Internal Monitoring Application (geonode.monitoring)¶
This application requires MaxMind’s GeoIP database file.
Base concepts and objects¶
GeoNode monitoring is a configurable monitoring application, that allows internal resources and hardware resources monitoring for GeoNode installations, including GeoServer deployments.
Monitoring application is configurable, so different deployment scenarios could be handled - from GeoNode and GeoServer running on single host, through distributed installations, where GeoServer is deployed to several hosts.
Monitoring application uses three base entity classes to describe elements of reality:
Service Type and
Hostis an object describing physical (or virtual) instance of operating system on which GN or GS is running. This object exists only for grouping and is not used directly by monitoring.
Service Typeis a description of kind of Service. Depending on service type, different metrics are stored, and different data collection mechanisms are used. Additionally, for system monitoring, it’s not conducted directly, but with GeoNode or GeoServer as monitoring agent. That means, no additional software installation is needed to monitor system, but also, hosts that don’t have GeoNode or GeoServer installed, won’t be monitored. There are four service types:
hostgeoserver- those types describe system monitoring probes that are running with GeoNode or GeoServer respectively,
geoserver- application-level probes that monitor one specific GeoNode or GeoServer instance.
Servicedescribes one specific instance of probe, either host-level or application-level. Service references Host and Service Type. Each service must be named, and name should be system-wide unique.
As mentioned above, each
Service Type keeps a set of
metrics, specific for that type.
metric is a description of measured value, for example: number of requests, response size or time, cpu usage, free memory etc.
Service Type has it’s own metrics set. Metric value may be either value counter (like country of user),
numeric counter (like number of requests) or rate (like bytes in/out on network interface).
Besides metric data, monitoring will also store exception information for exceptions that were captured during request handling.
Data are collected periodically (at most every 1 minute), aggregated and stored in aggregated form. User can see data from predefined relative periods (last minute, last 10 minutes, last hour, last day, last week).
User can enable and configure automated checks, which will be run after each collection/aggregation cycle, and will emit notifications if metric values in that run exceed configured thresholds.
This plugin requires a Potgresql DB backend enabled
UTCTimezone to your DB
psql -c 'set timezone=UTC;'
enable MONITORING_ENABLED flag and ensure that following code is in your settings:
# Settings for MONITORING plugin CORS_ORIGIN_ALLOW_ALL = ast.literal_eval(os.environ.get('CORS_ORIGIN_ALLOW_ALL', 'False')) GEOIP_PATH = os.getenv('GEOIP_PATH', os.path.join(PROJECT_ROOT, 'GeoIPCities.dat')) MONITORING_ENABLED = ast.literal_eval(os.environ.get('MONITORING_ENABLED', 'True')) MONITORING_CONFIG = os.getenv("MONITORING_CONFIG", None) MONITORING_HOST_NAME = os.getenv("MONITORING_HOST_NAME", HOSTNAME) MONITORING_SERVICE_NAME = os.getenv("MONITORING_SERVICE_NAME", 'local-geonode') # how long monitoring data should be stored MONITORING_DATA_TTL = timedelta(days=int(os.getenv("MONITORING_DATA_TTL", 7))) # this will disable csrf check for notification config views, # use with caution - for dev purpose only MONITORING_DISABLE_CSRF = ast.literal_eval(os.environ.get('MONITORING_DISABLE_CSRF', 'False')) if MONITORING_ENABLED: if 'geonode.monitoring' not in INSTALLED_APPS: INSTALLED_APPS += ('geonode.monitoring',) if 'geonode.monitoring.middleware.MonitoringMiddleware' not in MIDDLEWARE_CLASSES: MIDDLEWARE_CLASSES += \ ('geonode.monitoring.middleware.MonitoringMiddleware',)
DJANGO_SETTINGS_MODULE=<project_name>.settings python manage.py migrate monitoring
to apply db schema changes and insert initial data
DJANGO_SETTINGS_MODULE=<project_name>.settings python manage.py updategeoip
to fetch MaxMind’s GeoIP database file. It will be written to path specified by GEOIP_PATH setting.
DJANGO_SETTINGS_MODULE=<project_name>.settings python manage.py collect_metrics -n -t xml -f --since='<yyyy-mm-dd HH:mm:ss>'
to create first metrics.
<yyyy-mm-dd HH:mm:ss>with a real date time to start with.
Sitesfrom admin; make sure it contains a correct host name
do not forget to enable notifications and configure them from user profile
Enable the collect_metrics
Here below you will find instructions for a Ubuntu 16.04/18.04 based machine, but the procedure is similar for other OSs. The basic concept is that you must allow the system to run the command every minute (without -f and since):
DJANGO_SETTINGS_MODULE=<project_name>.settings python manage.py collect_metrics -n -t xml
sudo crontab -e
# Add the following line at the bottom; this will run the supervisor command every minute * * * * * supervisorctl start geonode-monitoring
sudo apt install supervisor sudo service supervisor restart sudo update-rc.d supervisor enable
sudo vim /etc/supervisor/conf.d/geonode-monitoring.conf
[program:geonode-monitoring] command=<path_to_virtualenv>/geonode/bin/python -W ignore <path_to_your_project>/geonode/manage.py collect_metrics -n -t xml directory = <path_to_your_project> environment=DJANGO_SETTINGS_MODULE="<your_project>.settings" user=<your_user> numproc=1 stdout_logfile=/var/log/geonode-celery.log stderr_logfile=/var/log/geonode-celery.log autostart = true autorestart = true startsecs = 10 stopwaitsecs = 600 priority = 998
sudo service supervisor restart sudo supervisorctl start geonode-monitoring sudo supervisorctl status geonode-monitoring
sudo vim /etc/hosts
127.0.0.1 localhost <public_ip> <your_host.your_domain> <your_host> # The following lines are desirable for IPv6 capable hosts
In order to have working monitoring, at least
Service should be configured.
Let’s assume following deployment scenario:
- there’s one machine,
geo01hosts both GeoNode and GeoServer (including PostgreSQL).
- applications are served with nginx+uwsgi, on port 80, but they are reachable on
- GeoServer is served from
- GeoNode is served from
Here’s step-by-step instruction how to create monitoring setup for deployment scenario:
Log in as admin, and go to admin section:
Go to monitoring section (or type
/admin/monitoring/as a path in URL):
Go to Hosts:
Click on Add host +:
Enter following information: * host: localhost * ip: 127.0.0.1 Note, that host value is arbitrary. You can enter other name if you like. Don’t forget to save.
Go to Services:
Click on Add service +:
Enter following information:
- name: local-geonode
- host: localhost
- service type: geonode
Add another Service Enter following information:
- name: local-system-geonode
- host: localhost
- service type: hostgeonode
- url: http://localhost/ (should point to GeoNode home page)
Add another Service and enter following information:
- name: local-geoserver
- host: localhost
- service type: geoserver
- url: http://localhost/geoserver/ (should point to GeoServer home page)
To summarize, following entries should be created in admin/monitoring:
localhost, with ip: 127.0.0.1
Monitoring interface is available for superusers only. It’s available in profile menu:
Main view offers overview of recent situation in GeoNode deployment.
Top bar and indicators¶
With top bar buttons User can:
- go back from nested interface elements (charts, alerts, errors)
- select time window from which data will be aggregated and shown (last 10 minutes, last 1 hour, last day or last week from now)
- see what’s currently used time window
- enable/disable autorefresh
Below there are four main health indicators:
- aggregated Health Check information.
- This element will be:
- green if there is no alerts nor errors
- yellow if there are alerts
- red if there are errors
- Uptime that shows GeoNode’s system uptime.
- Alerts shows number of notifications from defined checks. When clicked, Alerts box will show detailed information . See Notifications description for details.
- Errors - shows how many errors were captured during request processing. When clicked, Errors box will show detailed list of captured errors. See Errros description for details.
Indicators in error state
Software Performance view shows GeoServer web service statistics, for all requests monitored and detailed, OWS-specific, per service type (WMS, WFS, OCS etc).
will show charts with data history for overall performance and per-OWS performance:
Hardware performance box shows hardware usage statistics for selected host (monitored with any of hostgeonode or hostgeoserver type Service): % of CPU usage and average memory consumption. User can select from which host data will be presented.
will show charts with data history for selected host and time period
Errors view will show list of captured errors in GeoNode and GeoServer. List contents is displayed for selected time window.
- For each error, details are available:
- error class, message and stack trace
- basic request context (IP, path, user agent)
An alert is a descriptive information on situation when observed metric contains values outside allowed range (for example, response time is above 30 seconds, or no requests were served within last 30 minutes). Alerts are generated by notifications mechanism described below.
Alerts view will show list of alerts for current moment (alerts that were generated in past are not displayed here):
Each alert contains more descriptive information what is wrong:
Notification mechanism (not to be confused with notifications application in GeoNode) is a way to inform selected users about situations, where collected metric data would indicate a problem with deployment. Notifications are accessible from Alerts view:
There can be several notification configurations available.
Each notification configuration contains two main elements:
- list of email addressess which should be notified when alert is generated
- list of checks (at least one check must be in invalid state to generate alert)
User can add arbitrary number of emails. Email address doesn’t need to point to user registered in GeoNode instance. If email provided doesn’t belong to any of users, alert will be send as a regular email. If email provided can be associated with specific user, notifications application (and thus, notification settings for that user) will be used to send alert.