Paging Elasticsearch Data Through GeoServer

GeoServer is an open source software server written in Java that allows users to share and edit geospatial data. Designed for interoperability, it publishes data from any major spatial data source using open standards like Elasticsearch. We can page Elasticsearch data using GeoServer. GeoServer provides out of the box query parameters for all supported data stores.

Purpose

Investigating to fix the issue of paging elasticsearch data through Geoserver

Findings

We can page data through GeoServer on any given data store for WFS. We can get up to 1000 records if user doesn't provide the maxFeatures query parameter on the REST call in Elasticsearch data store. If User provides maxFeatures, GeoServer only returns the maxFeatures number of records. However, there is a limitation on Elasticsearch side that how much records you can retrieve on one call. If user wants to retrieve more than the Elasticsearch threshold, Elasticsearch scroll api should be use. When we create a Elasticsearch datastore, we enable scrolling so that Geoserver can use scroll api to page the data for the user. If user wants to use Elasticsearch scroll api, startIndex query parameter needs to be used on the URL. REST call returns first maxFeatures number records if user don't provide the startIndex query parameter. When the user provides the startIndex, Geoserver returns next maxFeatures from the startIndex. Call should look like this.

First Call

                      https://localhost:8080/geoserver/elasticsearch/ows?service=WFS
                        &version=1.1.0
                        &request=GetFeature
                        &typeName=test:98cd2e44
                        &maxFeatures=1000
                        &outputFormat=application%2Fjson
                    

second call to get next 1000 records after the first call

Second Call With Paging

second call to get next 1000 records after the first call

                      https://localhost:8080/geoserver/elasticsearch/ows?service=WFS
                        &version=1.1.0
                        &request=GetFeature
                        &typeName=test:98cd2e44
                        &maxFeatures=1000
                        &outputFormat=application%2Fjson
                        &startIndex=1001
                    

Conclusion

It is very easy to do paging data using Geoserver from Elasticsearch. We can use native Geoserver query parameters for paging and sorting eventhough we have Elasticsearch data store

References

Geo-Solutions