Tricks to Increase The GeoServer Performance

Posted by coffeetechgaff on October 20, 2018


200 users accessing same layer simultaneously without any issues. Each user is sending 39 getMap request with different bounding box. So, 7800 getMap requests to 4 GeoServers without any issues.


GeoServer is an open source software server written in Java that allows users to share and edit geospatial data. Designed for interoperability, it publishes data from any major spatial data source using open standards. GeoServer is the reference implementation of the Open Geospatial Consortium (OGC) Web Feature Service (WFS) and Web Coverage Service (WCS) standards, as well as a high performance certified compliant Web Map Service (WMS). GeoServer forms a core component of the Geospatial Web.

ElasticGeo is plugin for GeoServer that provides a GeoTools data store that allows geospatial features from an Elasticsearch index to be published via OGC services using GeoServer. Both geo_point and geo_shape type mappings are supported. OGC filters are converted to Elasticsearch queries and can be combined with native Elasticsearch queries in WMS and WFS requests.


  1. ElasticSearch 6.2
  2. ElasticGeo 2.12.2
  3. GeoServer 2.12.2
  4. gs-control-flow 2.12.2

GeoServer Configuration

  1. GeoServer is running on Docker container
  2. Each container is running with 2CPU and 6GB of Ram
  3. Running 4 instances of GeoServer
  4. 1000500 number of records on ElasticSearch index (with almost 120 attributes on each record)

Issues We Are Having

We are having java.util.concurrent.TimeoutException issue for most of the requests. Error is below. We were thinking that it is because ElasticSearch couldn't handle the number of requests because each RestClient's timeout is 500ms. We thought 500ms is too little if threads are waiting to execute. We played around with ElasticSearch configuration but did not work. We also find out that the Elasticsearch executes our GeoSpatial queries lightining fast. After banging our head around, we decided to take different route by limiting the requests to the GeoServer since each queries runs really fast in ElasticSearch.

                      29 Jun 16:29:33 ERROR [geoserver.ows] -
                      org.geoserver.platform.ServiceException: Rendering process failed
                      	at org.geoserver.wms.GetMap.executeInternal(
                      	at org.geoserver.wms.DefaultWebMapService.getMap(
                      	at sun.reflect.GeneratedMethodAccessor417.invoke(Unknown Source)
                      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(
                      	at java.lang.reflect.Method.invoke(
                        ... 120 more
                      Caused by: java.lang.RuntimeException: Error executing query search
                      	at org.geoserver.feature.RetypingFeatureCollection.features(
                      	at org.geoserver.feature.RetypingFeatureCollection.features(
                      	at org.geotools.renderer.lite.StreamingRenderer.drawPlain(
                      	at org.geotools.renderer.lite.StreamingRenderer.processStylers(
                      	at org.geotools.renderer.lite.StreamingRenderer.paint(
                      	... 129 more
                      Caused by: Error executing query search
                      	... 135 more
                      Caused by: java.lang.RuntimeException: error while performing request
                      	at org.elasticsearch.client.RestClient$SyncResponseListener.get(
                      	at org.elasticsearch.client.RestClient.performRequest(
                      	at org.elasticsearch.client.RestClient.performRequest(
                      	... 137 more
                      Caused by: java.util.concurrent.TimeoutException
                      	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(
                      	at org.apache.http.nio.pool.AbstractNIOConnPool.processNextPendingRequest(
                      	at org.apache.http.nio.pool.AbstractNIOConnPool.release(
                      	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.releaseConnection(
                      	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.releaseConnection(
                      	at org.apache.http.impl.nio.client.MainClientExec.responseCompleted(
                      	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(
                      	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(
                      	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(
                      	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(
                      	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(
                      	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(
                      	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(
                      	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(
                      	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(
                      	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(
                      	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(
                      	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(
                      	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$
                      	... 1 more

Steps We Took To Resolve The Issue

  1. We updated server.xml of tomcat to allow only 20 threads. We reduce the max threads in tomcat connector to 20.
  2. We added gs-control-flow 2.12.2 jar (extension) to GeoServer to control the number of requests being process on GeoServer.
  3. We added file to /data directory of GeoServer so that control flow can be controlled. We can control bunch of stuffs from property file which as follows.
  4.                       #request time out for a request using control flow
                          # timeout=60
                          # don't allow the execution of more than 18 requests total in parallel
                          # don't allow more than 8 GetMap requests in parallel
                          # don't allow more than 8 WFS GetFeature requests with Excel output format
                          # don't allow the execution of more than 16 tile requests in parallel
                          # (assuming a server with 4 cores, GWC empirical tests show that throughput
                          # peaks up at 4 x number of cores. Adjust as appropriate to your system)
    The control flow module, as its name clearly explains, controls the flow of request to GeoServer. That is, every request coming to GeoServer is filtered by the module and then submitted, queued, or rejected. A common use of this extension is to limit the concurrent request from the same user or a service to avoid an overwhelming amount of load that can block your GeoServer.
  5. We re-ran the test again with these changes, All 7800 requests of 200 users ran successfully in about ~600 seconds. This is our desirable outcome. This runs even faster if we use GeoWebCache

Final Implementation

As we test more, following changes are the final as of right now. Following configuration might change as we test more

  1. The max threads in tomcat connector to 100
  2. 4CPU and 6GB RAM for each instance of GeoServer
  3. file as follows
                            #request time out for a request using control flow
                            # don't allow the execution of more than 18 requests total in parallel
                            # don't allow more than 10 GetMap requests in parallel
                            # don't allow more than 10 WFS GetFeature requests with Excel output format
                            # don't allow the execution of more than 16 tile requests in parallel
                            # (assuming a server with 4 cores, GWC empirical tests show that throughput
                            # peaks up at 4 x number of cores. Adjust as appropriate to your system)

Other Possible Considerations for Performance

  1. Adding JAVAOPTS -Xms rather than -Xmx so that JAVA doesn't have to allow memory under load situations
  2. For Controlling/limiting CPU in java 8, -XX:ParallelGCThreads=N, -XXCICompilerCount=N. N is number of threads we want to run. It depends on the number of cores. N should match the number of cores
  3. Using --XX:+UserParallelOldGC and -XX:+UseParallelGc. Only if we have more than 2 CPUs per GeoServer (recommendation to use 4 CPUs)
  4. Using --XX:NewRatio=2
  5. Tuning Native JAI by using Native PNG Acceleration. We need to configure only to use 2x the number of cores for tile threads.
  6. Using GeoWebCache


Updating server.xml and adding file did the trick to solve the problem of timeout. It wasn't issue with ElasticSearch all along. The main issue was the GeoServer handling the resources. Only 20 threads running at a time prevented throttling GeoServer.