- GeoNode Monitoring
This application requires MaxMind’s GeoIP database file.
GeoNode monitoring is a configurable monitoring application, that allows internal resources and hardware resources monitoring for GeoNode installations, including GeoServer deployments.
Monitoring application is configurable, so different deployment scenarios could be handled - from GeoNode and GeoServer running on single host, through distributed installations, where GeoServer is deployed to several hosts.
Monitoring application uses three base entity classes to describe elements of reality: Host, Service Type and Service.
- Host is an object describing physical (or virtual) instance of operating system on which GN or GS is running. This object exists only for grouping and is not used directly by monitoring.
- Service Type is a description of kind of Service. Depending on service type, different metrics are stored, and different data collection mechanisms are used. Additionally, for system monitoring, it’s not conducted directly, but with GeoNode or GeoServer as monitoring agent. That means, no additional software installation is needed to monitor system, but also, hosts that don’t have GeoNode or GeoServer installed, won’t be monitored. There are four service types:
- hostgeonode, hostgeoserver - those types describe system monitoring probes that are running with GeoNode or GeoServer respectively,
- geonode, geoserver - application-level probes that monitor one specific GeoNode or GeoServer instance.
- Service describes one specific instance of probe, either host-level or application-level. Service references Host and Service Type. Each service must be named, and name should be system-wide unique.
As mentioned above, each Service Type keeps a set of metrics, specific for that type. A metric is a description of measured value, for example: number of requests, response size or time, cpu usage, free memory etc. Each Service Type has it’s own metrics set. Metric value may be either value counter (like country of user), numeric counter (like number of requests) or rate (like bytes in/out on network interface).
Besides metric data, monitoring will also store exception information for exceptions that were captured during request handling.
Data are collected periodically (at most every 1 minute), aggregated and stored in aggregated form. User can see data from predefined relative periods (last minute, last 10 minutes, last hour, last day, last week).
User can enable and configure automated checks, which will be run after each collection/aggregation cycle, and will emit notifications if metric values in that run exceed configured thresholds.
This plugin requires a Potgresql DB backend enabled
UTCTimezone to your DB
psql -c 'set timezone=UTC;'
MONITORING_ENABLEDflag and ensure that following code is in your settings:
CORS_ORIGIN_ALLOW_ALL = True MONITORING_ENABLED = True # add following lines to your local settings to enable monitoring if MONITORING_ENABLED: INSTALLED_APPS += ('geonode.contrib.monitoring',) MIDDLEWARE_CLASSES += ('geonode.contrib.monitoring.middleware.MonitoringMiddleware',) MONITORING_CONFIG = None MONITORING_HOST_NAME = 'localhost' MONITORING_SERVICE_NAME = 'local-geonode' MONITORING_HOST_NAME = SITE_HOST_NAME INSTALLED_APPS += ('geonode.contrib.ows_api',) GEOIP_PATH = os.path.join(os.path.dirname(__file__), '..', 'GeoLiteCity.dat')
DJANGO_SETTINGS_MODULE=<project_name>.settings python manage.py migrate monitoringto apply db schema changes and insert initial data
DJANGO_SETTINGS_MODULE=<project_name>.settings python manage.py updategeoipto fetch MaxMind’s GeoIP database file. It will be written to path specified by GEOIP_PATH setting.
DJANGO_SETTINGS_MODULE=<project_name>.settings python manage.py collect_metrics -n -t xml -f --since='<yyyy-mm-dd HH:mm:ss>'to create first metrics.
<yyyy-mm-dd HH:mm:ss>with a real date time to start with.
Sitesfrom admin; make sure it contains a correct host name
do not forget to enable notifications and configure them from user profile
Here below you will find instructions for a Ubuntu 16.04 based machine, but the procedure is similar for other OSs. The basic concept is that you must allow the system to run the command
DJANGO_SETTINGS_MODULE=<project_name>.settings python manage.py collect_metrics -n -t xml (without -f and since) every minute.
sudo crontab -e
# Add the following line at the bottom; this will run the supervisor command every minute * * * * * supervisorctl start geonode-monitoring
sudo apt install supervisor sudo service supervisor restart sudo update-rc.d supervisor enable
sudo vim /etc/supervisor/conf.d/geonode-monitoring.conf
[program:geonode-monitoring] command=<path_to_virtualenv>/geonode/bin/python -W ignore <path_to_your_project>/geonode/manage.py collect_metrics -n -t xml directory = <path_to_your_project> environment=DJANGO_SETTINGS_MODULE="<your_project>.settings" user=<your_user> numproc=1 stdout_logfile=/var/log/geonode-celery.log stderr_logfile=/var/log/geonode-celery.log autostart = true autorestart = true startsecs = 10 stopwaitsecs = 600 priority = 998
sudo service supervisor restart sudo supervisorctl start geonode-monitoring sudo supervisorctl status geonode-monitoring
sudo vim /etc/hosts
127.0.0.1 localhost <public_ip> <your_host.your_domain> <your_host> # The following lines are desirable for IPv6 capable hosts
- In order to have working monitoring, at least Service should be configured. Let’s assume following deployment scenario:
- there’s one machine,
geo01hosts both GeoNode and GeoServer (including PostgreSQL).
- applications are served with nginx+uwsgi, on port 80, but they are reachable on
- GeoServer is served from
- GeoNode is served from
- there’s one machine,
Here’s step-by-step instruction how to create monitoring setup for deployment scenario:
- Log in as admin, and go to admin section:
- Go to monitoring section (or type
/admin/monitoring/as a path in URL):
- Go to Hosts:
- Click on Add host +:
- Enter following information: * host: localhost * ip: 127.0.0.1 Note, that host value is arbitrary. You can enter other name if you like. Don’t forget to save.
- Go to Services:
- Click on Add service +:
- Enter following information:
- name: local-geonode
- host: localhost
- service type: geonode
- Add another Service Enter following information:
- name: local-system-geonode
- host: localhost
- service type: hostgeonode
- url: http://localhost/ (should point to GeoNode home page)
- Add another Service and enter following information:
- name: local-geoserver
- host: localhost
- service type: geoserver
- url: http://localhost/geoserver/ (should point to GeoServer home page)
To summarize, following entries should be created in admin/monitoring:
localhost, with ip: 127.0.0.1
Monitoring interface is available for superusers only. It’s available in profile menu:
With top bar buttons User can:
- go back from nested interface elements (charts, alerts, errors)
- select time window from which data will be aggregated and shown (last 10 minutes, last 1 hour, last day or last week from now)
- see what’s currently used time window
- enable/disable autorefresh
Below there are four main health indicators:
- aggregated Health Check information.
- This element will be:
- green if there is no alerts nor errors
- yellow if there are alerts
- red if there are errors
- Uptime that shows GeoNode’s system uptime.
- Alerts shows number of notifications from defined checks. When clicked, Alerts box will show detailed information . See Notifications description for details.
- Errors - shows how many errors were captured during request processing. When clicked, Errors box will show detailed list of captured errors. See Errros description for details.
Indicators in error state
Software Performance view shows GeoServer web service statistics, for all requests monitored and detailed, OWS-specific, per service type (WMS, WFS, OCS etc).
will show charts with data history for overall performance and per-OWS performance:
Hardware performance box shows hardware usage statistics for selected host (monitored with any of hostgeonode or hostgeoserver type Service): % of CPU usage and average memory consumption. User can select from which host data will be presented.
will show charts with data history for selected host and time period
Errors view will show list of captured errors in GeoNode and GeoServer. List contents is displayed for selected time window.
- For each error, details are available:
- error class, message and stack trace
- basic request context (IP, path, user agent)
An alert is a descriptive information on situation when observed metric contains values outside allowed range (for example, response time is above 30 seconds, or no requests were served within last 30 minutes). Alerts are generated by notifications mechanism described below.
Alerts view will show list of alerts for current moment (alerts that were generated in past are not displayed here):
Each alert contains more descriptive information what is wrong:
Notification mechanism (not to be confused with notifications application in GeoNode) is a way to inform selected users about situations, where collected metric data would indicate a problem with deployment. Notifications are accessible from Alerts view:
There can be several notification configurations available.
Each notification configuration contains two main elements:
- list of email addressess which should be notified when alert is generated
- list of checks (at least one check must be in invalid state to generate alert)
User can add arbitrary number of emails. Email address doesn’t need to point to user registered in GeoNode instance. If email provided doesn’t belong to any of users, alert will be send as a regular email. If email provided can be associated with specific user, notifications application (and thus, notification settings for that user) will be used to send alert.