We live in a pretty complex world, and sometimes we need a little help to understand where we are. We’re excited to announce Mapzen Search, our new search engine for places that takes our philosophy of open communities creating data and code to its heart. Mapzen Search will launch in the next week, and in the meantime, we wanted to share some background on why the time is right for a new, open search engine for places.

Note: If you’ve been a user of the Pelias web service during our beta, you’ll have to make a few changes as we transition from Pelias beta to 1.0 of Mapzen Search. You may want to skip to this section to understand the changes which we hope are relatively straightforward.

We live in two worlds at once. There’s a human world where we call places by their addresses and names. Then there’s another world for computers in which every place is represented through geographic coordinate systems like latitudes & longitudes or geohashes. Dozens of times every day we cross that boundary between the world of names and the world of coordinates and it’s all facilitated by a process called geocoding. In the coordinate world, it’s easy to do things like find things nearby, measure distances and correlate data, but to get there takes a leap through a boundary – and every time that boundary is crossed there’s a little toll paid. Do it enough and it adds up. But it also comes with all manner of restrictions of how you can see the world.

But what if it didn’t have to be this way? Mapzen Search is our answer to this call. It’s powered completely by open data, available freely to everyone, and importantly open to all to make better. It’s also powered by Mapzen’s open source geocoding project, Pelias. We’re building it in the open, and is open to collaboration from all to make searching our world better and smarter.

Mapzen Search is powered completely by open data, available freely to everyone, and importantly open to all to make better.

We’re still grappling with the implications of these tools, but searching for places has yet to be truly democratized. To search the globe at a scale where you can afford to take things for granted is expensive. It also generally means searching through the lens of a corporation’s owned copy of the world where the data powering it belongs to them, as do the tools that make the data useful.

The Time is Now

We have seen efforts to correct this in the past, with OpenStreetMap’s community driven Nominatim project. It’s a remarkable open geocoder, but is challenging to run at scale without access to some impressive hardware, which makes customizing it extremely difficult (and is also part of the reason Nominatim is unable to offer autocomplete services). The open community genuinely loves Nominatim and, unfortunately, last month the largest provider of a free Nominatim service, Mapquest Open, announced it would start charging high fees for even moderate use of the service.

We believe that the time is right for a new, open experimental geocoder to give everyone a world-class starting point to search the world.

Here’s How

The world is a complex place. Rules about how places are organized are only relevant for certain areas, under certain conditions, and are full of exceptions.

Much of the hard work is in collecting the data so it can represent the world correctly which, thanks to OpenStreetMap, OpenAddresses, and other open data efforts, gives us the greatest accumulation of openly licensed place data the world has ever known. These are projects that give their data to all comers and in the case of OpenStreetMap enable anyone to correct and improve the underlying data, making the tools that use it, and thereby the world just a little bit smarter every time.

So the data collection effort is well underway, and search is a synthesis of this data and how we interpret it. Search requires knowing things like:

• UK postcodes beginning with E represent East London, same for S, N and W.
• Different countries have different address numbering schemes; some ‘zig-zag’ up the street while others go up one side and down the other.
• 0 is a valid street number in some countries.
• US highways with even numbers run East-West and odd routes run North-South.
• Most countries have a clear hierarchy of country/city/state etc. but there are some odd exceptions such as Vatican City.

We believe that an open geocoder can only grow over the long-term if it’s easy to learn as a community and incorporate truly representative local knowledge.

To account for that, you can either hire an army of employees to conduct the research in every far-flung corner of the world, or rely on local knowledge of users around the world to help create open services that are genuinely representative. Mapzen is committed to working with folks in the open mapping community and beyond to make the Pelias engine incrementally better at representing the world the way it really is, through the input of the very people that occupy it.

As we start, our focus is making Mapzen Search & Pelias a project where participating in the design and development is fundamentally open. It’s why we’re building Pelias with Node.js and Elasticsearch.

With Node.js, we’re able to open up Pelias’ development to one of the largest development communities in the world. Javascript has become a lingua franca of programming that has not only shown itself as powerful for building tools, but for building tools that garner powerful communities. Node.js has a huge collection of modules which are shared with other users in the community through the package manager npm. As we develop functionality for Pelias and Mapzen Search, we publish each piece of discrete functionality to npm so that the community can also benefit from it, and contribute features back to us.

And while we orchestrate in Javascript, we power search with Elasticsearch, another remarkable open source project that lets us scale Pelias’ search capabilities from running on just a single desktop to across a data center. It’s a framework that’ll let us continue to build features present in Pelias and Mapzen Search that simply aren’t there in other open geocoders, like autocomplete. It lets us to easily adapt the strategies we use to search for things and what we can offer with those searches.

We think open data, open source, and open strategy win over proprietary solutions at any part of the stack and want to ensure the services we offer are in line with that vision.

We want to give everyone a world-class starting point to search the world. With the revolution in open data being led by communities, and the Open Government movement, anyone who can run Pelias can have a powerful geocoder out of the box based on open data. That’s why we’re building Mapzen Search, our flagship use of Pelias to run entirely on open data - we want to ensure the baseline you get with us is as good as it possibly can be. We think open data, open source, and open strategy win over proprietary solutions at every part of the stack and want to ensure the services we offer are in line with that vision.

You’ve got this data, shouldn’t you be able to use it on any map you want?

At the same time, we want to open the domain of place search and geocoding beyond where it is today, and give others a powerful place to start in customizing a geocoder for specific domains, companies, and purposes we haven’t even imagined today.

We couldn’t be more excited for you to start searching the world with us.

Upcoming Changes

The release of Mapzen Search is a major milestone that we hope will make Pelias easy for developers to start using immediately and to support the improvements we have in store over the next few years. However, it also introduces changes which are incompatible with the beta some developers have been using. We will continue to support calls to the existing API for at least the next two months (through November) to give developers time to transition to Mapzen Search. We realize that any partner API change, no matter the size, is a burden on existing users, and so making this change is something we do not take lightly.

The timing of this transition only immediately affects current users of pelias.mapzen.com. If you are running your own instance of Pelias, you can control timing of the transition by choosing when to roll out the upgrade.

We are formalizing our documentation for this new API. But if you’re an existing Pelias user, we should talk. Because we haven’t previously required registration for Pelias, we can’t get in touch with folks directly.

If you use Pelias today, please let us know by filling out This Form, or sending us an email at search@mapzen.com. We want to get in touch to make this transition as easy as possible for you and your users.

More soon!

Image credits:

• Lionel Pincus and Princess Firyal Map Division, The New York Public Library. “Orbis terrae compendiosa descriptio : quam ex magna universali Gerardi Mercatoris Domino Richardo Gartho, geographie ac ceterarum bonarum artium amatori ac fautori summo, in veteris amicitie ac familiaritatis memoriam Rumoldus Mercator fieri curabat A0. M.D. LXXXVII.” The New York Public Library Digital Collections. 1637. http://digitalcollections.nypl.org/items/510d47e2-0b65-a3d9-e040-e00a18064a99

• Textiles department of Cooper Hewitt, Smithsonian Design Museum. “Jacket, Wearin’ The States, 1990. tyvek, a dupont copyrighted fabric, other synthetics. Gift of Interarts. 1991-14-6.” Cooper Hewitt Collections. 1990. https://collection.cooperhewitt.org/objects/18635643/

• Lionel Pincus and Princess Firyal Map Division, The New York Public Library. “[Plate 77: Map bounded by West 27th Street, East 27th Street, Fourth Avenue, East 22nd Street, West 22nd Street, Sixth Avenue.]” The New York Public Library Digital Collections. 1852 - 1854. http://digitalcollections.nypl.org/items/510d47e4-56ba-a3d9-e040-e00a18064a99

Special thanks to Stephen Hess, Peter Johnson, Nathaniel Vaugn Kelso, Randy Meech, John Oram, Julian Simoni, Kaitlin Thaney, and Diana Shkolnikov for reviewing drafts and generally helping make this piece better.