A visual depiction of our order journey

Blinkit (formerly Grofers) is a data-driven organization, and that means we want to leverage all of our data effectively, including location data! As a platform that works end to end to get groceries delivered to your doorstep, we very well have an abundance of geographic data. We constantly depend on this data for insights, visibility, analytics, and prediction.

This includes the geospatial data captured from the delivery addresses of orders and routes made by delivery executives. We also have established service boundaries which are Polygons strategically created by the operations team on the ground to help divide and conquer the orders we receive across a city. These service boundaries help run the supply chain with a network of stores that power our delivery.

When people noticed how powerful this data could be, use-cases started popping up around creating visualizations of location data using fancy maps and managing our internal service boundaries. We initially used Google MyMaps for creating and sharing user-defined service boundaries. In contrast, for visualizations, we download the data that we query from our databases and upload it over on https://kepler.gl/demo and share the link using dropbox as our backend.

As the requirements grew and more people started looking at this data, we thought it was high time we built our own GIS platform.

Options & an Obvious choice

Given that we had specific requirements in mind, we set out to explore open-source projects in this space and built PoCs around some of the tools while keeping our analytics requirements in mind. Here are our shortlists:

nebula.gl to handle editing of geometries and local deployment of kepler.gl to visualize analytical datasets

• Pros: Both Nebula and Kepler were developed by Uber and go hand-in-hand
• Cons: Users who edit service boundaries need to be assisted with analytical data like order density maps. It seemed like a lot of work to integrate and maintain Nebula and Kepler in the same page/react container. We also felt we have less control over Nebula’s API.

Mapbox-gl-draw to edit geometries and build a toolbox to add/render custom analytical data.

• Pros: Greater control over the map canvas, simpler API.
• Cons: Since Nebula was built on top of mapbox-gl-draw, this option would be a downgrade.

Leaflet.Editable: leaflet can be used to edit geometries with Kepler as a separate app to view analytical data.

• Pros: Simple VanillaJS API gives more control than mapbox-gl-draw or Nebula.
• Cons: Visualization and editing are decoupled now, and hence would be hard to integrate them in the future when needed. This meant moving away from Mapbox base maps that we had grown fond of.

kepler.gl: This PR allowed users to draw polygons on Kepler using Nebula. So the idea was to build on top of this feature to enable editing of service boundaries, and within the same Kepler instance, one can visualize analytical data as well.

• Pros: One application handles both edit and visualization flows. Can cater to cases where one needs to view supporting datasets before editing a geometry. Since Kepler emits Redux events on clicking a point, this can be extended to edit points, paths, lines if needed in the future.
• Cons: Less control over editing flows; one can only edit polygons with the current setup, whereas we also wanted the capability to edit Point and Path data in the near future.

We decided to go with the 4th option and build everything on top of Kepler. Since in the former options, we had to create a toolbox very similar to what Kepler gave us out of the box which does not make sense.

We ❤️ kepler.gl @ letsblinkit!

What did we build to share custom maps?

The first goal was to build a portal for users to create and share maps. This would mean the following things:

• One can upload custom geospatial data in the form of GeoJSON, CSV, shapefiles, and these files would persist in a mounted volume.
• The visualizations and the different map styles used and created by the users would also persist as a configuration.
• A unique URL would identify every map. Users will also be able to ‘fork’/’clone’ maps to collaborate.

How did we do it?

Now that our goals were clear, we turned our eyes to the implementation. There were several existing data tools in the Blinkit data ecosystem that people had gotten accustomed to, so it made a lot of sense to couple our GIS tool with something already existing in the space. That tool ended up being Redash, an Open source software that allows users to explore, query, visualize and share data from any data source.

Whenever a user writes a query on Redash to fetch data, it is saved with a query ID. We, in turn, use this query to generate a map on kepler.gl by inferring the columns with a location component; this map can now be saved and shared. We called this nifty little tool Grepler meshing Blinkit and kepler.

Grepler also allowed users to upload their custom data along with data queried out of Redash. This data was saved in a persistent volume attached to the Grepler server. Along with the data to be plotted, we also saved the design config changes like stroke color and map type, all of which are stored in a kepler.gl state. We associate this config and the actual data plotted with a unique ID assigned to every map once it’s saved.

All the data and config files go inside a namespace in the volume created for every user logged onto Grepler. The URL of a map follows this format:

grepler-dummy.io/maps/<username>/<unique_id>

E.g. https://grepler-dummy.io/maps/magician/ea377db-20ea-5fca-ac31-3a00c4f6e767

If someone other than a magician accesses this URI, they can view the map. At the same time, our magician can update the map by incrementally making changes. If someone other than the magician tries to make changes, Grepler starts a new save process to fork off the existing map config and saves it as a new map in the non-magician user’s namespace, ensuring isolation.

Here is how a sample map in Grepler looks like:

Sample Grepler Map

What did we build to manage service boundaries?

Grepler served us well for ad-hoc analytics use cases where users just wanted to create, share, and discard maps, but our end goal was always a platform where one could create and manage boundaries along with analytics.

So we set out to build a new tool focusing on making service boundary management better with a vision that both Grepler and this new tool can be easily integrated later into one common platform.

Built again on top of kepler.gl and internally named Sauron (The All-Seeing Boundary Manager), it had the following goals in mind.

• Keep basic visualization features and the toolbar of kepler.gl intact.
• Have RBAC (Role-Based Access Control) over Sauron and also over all the service boundaries. This means that when a user logs into Sauron, even though all the service boundaries are loaded, only the ones the user has edit access to are visible while the rest are hidden.
• Every service boundary is a MultiPolygon; some can be null. Users can add/edit/delete Polygon(s) to those MultiPolygons with Sauron.
  Editing a polygon includes changing vertices by dragging a vertex from one point to another, and editing steps might also involve Adding/removing these vertices.
• Smart ‘editor’ plugins, i.e., auto-completing boundaries by snapping vertices to the nearest known boundary, resolving self-intersections in boundaries, and so on
• Metadata related to the polygons are also stored and edited in Sauron.
• Every change to a polygon goes through an approval process, ensuring there is always a Human in the Loop.

After our learnings from Grepler, the only considerable challenge for us was to find a way to let users edit the polygons while kepler.gl let us draw individual polygons. There was no straightforward way for us to extend it to make all polygons editable.

To do that, we had to identify the part of the app state where drawn polygons are stored and write custom reducers for action: “User clicking on an editable part of the Boundary”. We then allowed users to edit the boundaries and save the changes back to our state internally.

The result looked this way.

Editing Polygons on Sauron

Now that we can edit the blue-bordered polygon with the state saved back internally, we can see the changes to the yellow polygon get reflected in the UI in real-time.

Similarly, we handled metadata editing by managing our internal states and building features around them.

Now Sauron helps folks on the ground manage service boundaries!

The Future

We’ve kept building things on top of KeplerV2. New and cool features are being pushed to the project almost every month.

The prospect of introducing kepler.gl to our GIS stack is much brighter in the future. We’re planning to add features like real-time visualization of the supply chain metrics and enable fine-tuned control over the supply chain.

This article explores only the front-end efforts. Numerous other tricky challenges came up while building services that power these two applications, which deserve a few pages in writing themselves. Saving it for another blog someday soon :)

Huge thanks to Shivashis Padhi for working on this blog and building out our GIS platform into what it is right now. A big shoutout to Deepu Thomas Philip for his guidance and vision that contributed and is paramount to the future of location intelligence here at Blinkit.