Home / GIS and Open Map Data for Beginners

OpenStreetMap PBF: The Binary Format the Planet Travels In

Close-up of server racks with active equipment in a data center

Photo: panumas nikhomkhai / Pexels

Download the whole of OpenStreetMap and the file that arrives is not the readable XML you might expect. It is a .osm.pbf file: binary, compact, and unreadable in a text editor, and deliberately so. PBF (Protocol Buffer Format) is the distribution format the project standardized on because the planet outgrew text. This guide explains what is inside a PBF file, how its compression tricks work, how it compares with OSM XML in size and speed, which tools read it, and how to convert between the two formats without losing a single tag. The OpenStreetMap wiki's PBF Format page is the specification this article teaches from.

Interactive tool

Handle the Planet Format: The PBF Checklist

Check off each step as you finish it. Your progress stays on this page.

0 of 7 complete

0% done

What PBF Holds: The Same Map, Packed

A PBF file contains exactly the same data model as OSM XML: nodes (points with coordinates and tags), ways (ordered lists of nodes with tags), and relations (groupings of nodes, ways, and other relations with tags). Nothing about the map changes. What changes is the packing. Per the OSM wiki, a PBF file runs at about half the size of a gzipped planet XML file, and about 30 percent smaller than a bzipped one, while also parsing considerably faster, because a program reads typed binary fields instead of tokenizing text. For a planet-scale file measured in tens of gigabytes, those ratios are the difference between a download that finishes today and one that finishes tomorrow, and between a toolchain that needs a server and one that runs on a laptop.

Inside the File: Fileblocks and Primitive Groups

PBF is organized as a sequence of fileblocks, each independently decodable, which is the property that makes the format practical: a reader can start anywhere, recover from damage, and parallelize. The first block is a header carrying the bounding box, replication information, and the features the file requires and optionally supports. After it come data blocks, each holding a primitive group of roughly 8,000 entities of one kind, nodes together, ways together, relations together. Grouping by kind is itself a compression decision: similar records stored adjacently compress far better than a mixed stream, and most processing pipelines consume the data kind by kind anyway.

The Compression Tricks: String Tables and Delta Encoding

Two ideas do most of the shrinking. First, string tables: tags in OpenStreetMap repeat endlessly (the key highway, the value residential, millions of times), so each block stores its strings once in a table and the entities reference them by index number. Second, delta encoding: instead of storing each node's full ID and coordinates, the file stores the difference from the previous node. IDs in a block run in near sequence, and neighboring nodes sit near each other, so the differences are tiny numbers, and tiny numbers take tiny space. Coordinates are stored as integers in nanodegrees, with a default granularity of 100 nanodegrees, and timestamps with a default granularity of 1,000 milliseconds; offsets stored in the block header place those integers on the real map. None of this is lossy in the ways that matter: decode a PBF and you recover the exact IDs, coordinates, timestamps, and tags that XML would have given you, which the conversion section below relies on.

A server unit glowing blue in a data center room
A server unit glowing blue in a data center room

PBF Against XML: Choosing the Working Format

QuestionOSM XML (.osm)PBF (.osm.pbf)
Can a human read it?Yes, in any text editorNo, binary
Size for the same dataLargest; gzips wellAbout half of gzipped XML
Parse speedSlow, text tokenizingFast, typed fields
Best forSmall extracts, debugging, learning the data modelDownloads, pipelines, anything planet-scale
Tool supportUniversal in OSM toolingUniversal in modern OSM tooling

The working rule: read XML to learn and to debug, use PBF to move and to process. A small extract opened in a text editor teaches the data model better than any documentation, because you can see a node, its tags, and a way referencing it. The moment the file covers a country, switch formats. And note the family resemblance that confuses newcomers: Mapbox Vector Tiles also use the .pbf extension, because they also encode with Protocol Buffers, but a vector tile and an .osm.pbf file are entirely different formats sharing an encoding library, a coincidence our vector tiles guide untangles.

The Toolchain: Osmosis and Its Successors

The wiki names Osmosis as the reference implementation for PBF, the Java tool whose reading and writing defined correct behavior, and Osmosis still works. Around it has grown the modern bench: osmium-tool and the libosmium library for fast filtering, extraction, and conversion on the command line; Osm2pgsql and imposm for loading PBF into databases; GDAL's OSM driver, which is how QGIS opens these files; and readers in every major programming language. The practical point for a beginner is that you never parse PBF by hand. You hand the file to a tool that speaks it, and the tool hands you nodes, ways, and relations. If a workflow asks you to convert a PBF to text to inspect a handful of features, that is legitimate; if it asks you to convert a continent to text to process it, the workflow is wrong, not the format.

Converting Between PBF and XML, Losslessly

Conversion runs both ways with standard tools: osmium and Osmosis both read either format and write the other, and because both formats encode the identical data model, the conversion is lossless: every node, way, relation, tag, timestamp, and version survives. That property makes PBF safe to adopt as your storage format even if your downstream tools are old: keep the PBF as the archive, convert the slice you need to XML when a tool demands it, and never keep the XML as the master copy, since it is the same data wearing five times the disk. One honest caveat: conversion preserves the data, not the file's internal replication metadata in every tool's hands, so pipelines that track minutely updates should convert with the tool their update chain already trusts.

Why the Planet Ships This Way

Step back and the format explains the project's logistics. OpenStreetMap offers the planet file, continental and country extracts through providers such as Geofabrik, and minutely replication diffs, and every one of those distribution channels assumes files that are small to mirror, fast to parse, and safe to resume. PBF's independently decodable blocks, dense packing, and reference implementation are what make a worldwide volunteer mirroring network feasible on donated bandwidth. When you download an extract for your region, in the workflow our OpenStreetMap download guide covers, the file you receive is PBF for these reasons. Understanding the format is understanding how a map made by millions of people physically reaches your desk: not as a web page, but as a dense binary river of nodes, ways, and relations, packed by string tables and deltas into something the internet can afford to carry.

Reading a PBF Once, for the Learning of It

You never parse PBF by hand in production, but decoding one small file by hand, once, teaches the data model better than any diagram. Take the smallest extract you can find, a town rather than a country, and run it through a tool that shows its skeleton: osmium's file information reports the header, the bounding box, the counts of nodes, ways, and relations, and the time the data describes. Then convert just that extract to XML and open it in a text editor. What you see is the entire model with the packing removed: a node with an ID, a version, coordinates, and tags; a way as a list of node references wearing tags of its own; a relation binding members with roles. Every QGIS layer, every tile, every Garmin image built from this data is those three shapes wearing presentation. The exercise also builds the right instinct about scale: the town's XML is readable; the same town in PBF is a fraction of its size; and the country's XML is a file your editor will refuse to open, which is the whole argument of this article experienced at a desk instead of read in one.

Staying Current: Extracts, Diffs, and the Replication Stream

A PBF file is a snapshot, and OpenStreetMap does not stand still for its portrait. The ecosystem keeps snapshots useful two ways. Extract providers regenerate their files on regular cycles (daily is common), so the simple update is a fresh download of the same region, which is the rhythm our Garmin guide builds into trip preparation. For systems that must track the map continuously, the project publishes a replication stream: small change files, issued as often as every minute, that record the edits made since the last one, which tools such as Osmosis and osmium apply to a local PBF or database to roll it forward. The change files use their own change format rather than PBF itself, a reminder that PBF's job in the ecosystem is specifically the bulk state of the map, the thing diffs are applied to. Choose by appetite: a downloaded extract for analysis and publishing, a replicated database for services that must never be a day behind. Either way the snapshot you start from arrives as PBF, because nothing else the project ships can carry a planet at a price the internet will pay.

A Note on the Name: Protocol Buffers Beyond OSM

The PB in PBF stands for Protocol Buffers, Google's general-purpose binary serialization system, and OpenStreetMap is only one of its tenants: the same encoding carries vector tiles, countless internal service messages across the software industry, and data formats that have nothing to do with maps. That generality explains both of the format's quirks a newcomer trips on: the shared .pbf extension with vector tiles (two tenants of one encoding, mistaken for relatives), and the fact that generic Protocol Buffer tools can half-read an .osm.pbf file's structure without understanding a single node. The OSM format is a specific schema written in that general language, maintained by the project, specified on the wiki page this article follows. Knowing the distinction keeps your troubleshooting sane: an encoding question goes to the Protocol Buffers documentation; a question about what a primitive group may contain goes to the OSM wiki, and confusing the two authorities wastes exactly the kind of afternoon this guide exists to save.

Related Guides

If this page answered part of the question, these GIS guides cover the rest of the same ground:

Frequently Asked Questions

What is a PBF file in OpenStreetMap?

It is the project's binary distribution format, .osm.pbf, holding the same nodes, ways, and relations as OSM XML in a compact Protocol Buffer encoding. Per the OSM wiki it is about half the size of a gzipped XML planet file and about 30 percent smaller than a bzipped one, and it parses faster.

Is PBF lossless compared with OSM XML?

Yes. Both formats encode the identical data model, and standard tools such as osmium and Osmosis convert between them preserving every element, tag, timestamp, and version. Keep PBF as the master copy and convert slices to XML only when a tool requires it.

How does PBF get so much smaller?

Mainly two tricks: string tables store each block's repeated tag strings once and reference them by index, and delta encoding stores the difference between successive IDs and coordinates instead of full values. Grouping roughly 8,000 entities of one kind per primitive group makes both tricks work harder.

What opens a .osm.pbf file?

The modern OSM toolchain: osmium-tool for conversion and extraction, Osmosis (the reference implementation), osm2pgsql and imposm for database loading, and GDAL's OSM driver, which is how QGIS reads them. You never parse the binary by hand.

Is a vector tile .pbf the same as an .osm.pbf?

No. Both use Protocol Buffers encoding and share the .pbf extension, but a Mapbox Vector Tile holds styled rendering data for one tile, while an .osm.pbf holds raw OpenStreetMap elements. The shared extension is a coincidence of the encoding library.

Photo credits: panumas nikhomkhai (Pexels); panumas nikhomkhai (Pexels). Photos are served from this site; follow the credit to the library page for the original.