Purpose
200 users accessing same layer simultaneously without any issues. Each user is sending 39 getMap request with different bounding box. So, 7800 getMap requests to 4 GeoServers without any issues.
Background
GeoServer is an open source software server written in Java that allows users to share and edit geospatial data. Designed for interoperability, it publishes data from any major spatial data source using open standards. GeoServer is the reference implementation of the Open Geospatial Consortium (OGC) Web Feature Service (WFS) and Web Coverage Service (WCS) standards, as well as a high performance certified compliant Web Map Service (WMS). GeoServer forms a core component of the Geospatial Web.
ElasticGeo is plugin for GeoServer that provides a GeoTools data store that allows geospatial features from an Elasticsearch index to be published via OGC services using GeoServer. Both geo_point and geo_shape type mappings are supported. OGC filters are converted to Elasticsearch queries and can be combined with native Elasticsearch queries in WMS and WFS requests.
Dependency
- ElasticSearch 6.2
- ElasticGeo 2.12.2
- GeoServer 2.12.2
- gs-control-flow 2.12.2
GeoServer Configuration
- GeoServer is running on Docker container
- Each container is running with 2CPU and 6GB of Ram
- Running 4 instances of GeoServer
- 1000500 number of records on ElasticSearch index (with almost 120 attributes on each record)
Issues We Are Having
We are having java.util.concurrent.TimeoutException issue for most of the requests.
Error is below. We were thinking that it is because ElasticSearch couldn't handle the number of requests because each RestClient's timeout is 500ms.
We thought 500ms is too little if threads are waiting to execute. We played around with ElasticSearch configuration but did not work. We also find out that the Elasticsearch executes our GeoSpatial queries lightining fast.
After banging our head around, we decided to take different route by limiting the requests to the GeoServer since each queries runs really fast in ElasticSearch.
29 Jun 16:29:33 ERROR [geoserver.ows] -
org.geoserver.platform.ServiceException: Rendering process failed
at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:609)
at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:284)
at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:141)
at org.geoserver.wms.GetMap.executeInternal(GetMap.java:653)
at org.geoserver.wms.GetMap.run(GetMap.java:285)
at org.geoserver.wms.GetMap.run(GetMap.java:131)
at org.geoserver.wms.DefaultWebMapService.getMap(DefaultWebMapService.java:320)
at sun.reflect.GeneratedMethodAccessor417.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
... 120 more
Caused by: java.lang.RuntimeException: java.io.IOException: Error executing query search
at org.geotools.data.store.ContentFeatureCollection.features(ContentFeatureCollection.java:176)
at org.geoserver.feature.RetypingFeatureCollection.features(RetypingFeatureCollection.java:54)
at org.geoserver.feature.RetypingFeatureCollection.features(RetypingFeatureCollection.java:38)
at org.geotools.renderer.lite.StreamingRenderer.drawPlain(StreamingRenderer.java:2302)
at org.geotools.renderer.lite.StreamingRenderer.processStylers(StreamingRenderer.java:1937)
at org.geotools.renderer.lite.StreamingRenderer.paint(StreamingRenderer.java:835)
at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:579)
... 129 more
Caused by: java.io.IOException: Error executing query search
at mil.nga.giat.data.elasticsearch.ElasticFeatureSource.getReaderInternal(ElasticFeatureSource.java:132)
at org.geotools.data.store.ContentFeatureSource.getReader(ContentFeatureSource.java:647)
at org.geotools.data.store.ContentFeatureCollection.features(ContentFeatureCollection.java:173)
... 135 more
Caused by: java.lang.RuntimeException: error while performing request
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:684)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:222)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:194)
at mil.nga.giat.data.elasticsearch.RestElasticClient.performRequest(RestElasticClient.java:191)
at mil.nga.giat.data.elasticsearch.RestElasticClient.search(RestElasticClient.java:185)
at mil.nga.giat.data.elasticsearch.ElasticFeatureSource.getReaderInternal(ElasticFeatureSource.java:118)
... 137 more
Caused by: java.util.concurrent.TimeoutException
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:364)
at org.apache.http.nio.pool.AbstractNIOConnPool.processNextPendingRequest(AbstractNIOConnPool.java:344)
at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:318)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.releaseConnection(PoolingNHttpClientConnectionManager.java:303)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.releaseConnection(AbstractClientExchangeHandler.java:239)
at org.apache.http.impl.nio.client.MainClientExec.responseCompleted(MainClientExec.java:387)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:168)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
... 1 more
Steps We Took To Resolve The Issue
- We updated
server.xmlof tomcat to allow only 20 threads. We reduce the max threads in tomcat connector to 20. - We added
gs-control-flow 2.12.2 jar (extension)to GeoServer to control the number of requests being process on GeoServer. - We added
controlflow.propertiesfile to /data directory of GeoServer so that control flow can be controlled. We can control bunch of stuffs from property file which as follows. - We re-ran the test again with these changes, All 7800 requests of 200 users ran successfully in about ~600 seconds. This is our desirable outcome. This runs even faster if we use GeoWebCache
#request time out for a request using control flow
# timeout=60
# don't allow the execution of more than 18 requests total in parallel
ows.global=16
# don't allow more than 8 GetMap requests in parallel
ows.wms.getmap=8
# don't allow more than 8 WFS GetFeature requests with Excel output format
ows.wfs.getfeature=8
# don't allow the execution of more than 16 tile requests in parallel
# (assuming a server with 4 cores, GWC empirical tests show that throughput
# peaks up at 4 x number of cores. Adjust as appropriate to your system)
#ows.gwc=16
The control flow module, as its name clearly explains, controls the flow of request to GeoServer.
That is, every request coming to GeoServer is filtered by the module and then submitted, queued, or rejected.
A common use of this extension is to limit the concurrent request from the same user or a service to avoid an overwhelming amount of load that can block your GeoServer.
Final Implementation
As we test more, following changes are the final as of right now. Following configuration might change as we test more
- The max threads in tomcat connector to 100
- 4CPU and 6GB RAM for each instance of GeoServer
- controlflow.properties file as follows
#request time out for a request using control flow timeout=60 # don't allow the execution of more than 18 requests total in parallel ows.global=20 # don't allow more than 10 GetMap requests in parallel ows.wms.getmap=10 # don't allow more than 10 WFS GetFeature requests with Excel output format ows.wfs.getfeature=10 # don't allow the execution of more than 16 tile requests in parallel # (assuming a server with 4 cores, GWC empirical tests show that throughput # peaks up at 4 x number of cores. Adjust as appropriate to your system) #ows.gwc=16
Other Possible Considerations for Performance
- Adding
JAVAOPTS -Xmsrather than-Xmxso that JAVA doesn't have to allow memory under load situations - For Controlling/limiting CPU in java 8,
-XX:ParallelGCThreads=N, -XXCICompilerCount=N. N is number of threads we want to run. It depends on the number of cores. N should match the number of cores - Using
--XX:+UserParallelOldGCand-XX:+UseParallelGc. Only if we have more than 2 CPUs per GeoServer (recommendation to use 4 CPUs) - Using
--XX:NewRatio=2 - Tuning
Native JAIby using Native PNG Acceleration. We need to configure only to use 2x the number of cores for tile threads. - Using GeoWebCache
Conclusion
Updating server.xml and adding controlflow.properties file did the trick to solve the problem of timeout. It wasn't issue with ElasticSearch all along.
The main issue was the GeoServer handling the resources. Only 20 threads running at a time prevented throttling GeoServer.
References
Copyright of geoserver image is © wikipedia.org
http://geoserver.org
https://github.com/ngageoint/elasticgeo
https://docs.docker.com/engine/reference/commandline/build/