Use datastore shards in GeoNode¶
Many organizations have hundreds of layers uploaded to GeoNode. In such a case using the default GeoNode configuration, with just one PostGIS database and one GeoServer PostGIS store for all of the layers has several limitations, such as:
- Layer upload and creation time become very large
- Layer upload and creation time tends to become very large when the PostGIS database starts containing many layers. We have seen cases where even 4/5 minutes were needed for uploading a small sized shapefile). This issue is caused by the actual implementation of the PostgreSQL JDBC driver and has been reported in details in GeoServer bug GEOS-7533: https://osgeo-org.atlassian.net/browse/GEOS-7533
- Large backups
- When data are not edited in GeoNode, it is easier to backup smaller PostgreSQL databases rather than a big one, where only a few tables have been changed since the last backup. When using a single database for all of the uploads this is not possible
These problems can be tackled using the Datastore Shards GeoNode contrib module, which automatically creates new shards when some defined conditions changes.
How to use the datastore_shards module¶
As a first thing, add the module in the INSTALLED_APPS section of the settings file:
INSTALLED_APPS = ( ..., 'geonode.contrib.datastore_shards', ... )
Here is a typical extract of the settings that must be used for GeoNode to use the datatabase shards module:
# SHARD DATABASES SETTINGS # SHARD_STRATEGY may be yearly, monthly, layercount SHARD_STRATEGY = 'layercount' SHARD_LAYER_COUNT = 100 SHARD_PREFIX = 'wm_' SHARD_SUFFIX = '' DATASTORE_URL = 'postgis://user:password@localhost:5432/data'
Now syncronize the module models with the database:
python manage.py migrate datastore_shards
Now you are set and datastore shards will be used as soon as GeoNode is restarted. The datastore_shards application will automatically create a new shard whenever it is needed.
The PostrgreSQL ROLE which is used (user in DATASTORE_URL) must have CREATEDB option assigned in order to be able to create the PostgreSQL shards.