3D Spatial AI Course

Read a LAS file, filter the ground, build a terrain model, and prep your scans for deep learning, all in one guided path through the capture end of the pipeline.

Yes! We are talking about LiDAR. Sorry about that, but I need to plainly share the term: Light Detection and Ranging. Sounds like Sci-Fi (for sure) for any beginner here!

But let's look at history:

In 1969, Apollo 11 left a mirror sitting on the Moon.

Then, that succeeded: Point a laser at it from Earth. Time the round trip. Multiply by the speed of light, halve it, and the Earth-Moon distance drops right out of the arithmetic.

That experiment, Lunar Laser Ranging, is still running today, and it still measures the gap to within a few centimeters.

So here's the thing nobody tells you when you open your first (point cloud) LAS file: you're holding millions of those exact measurements.

Every point is one laser pulse, timed and turned into a distance. Not a picture. A pile of round trips.

And that changes everything.

Because if each point is a physical measurement, then processing it isn't image editing.

It's turning raw arithmetic into geometry a machine can reason about.

So where do you start?

🦚 A quick welcome: Hey, it's Florent Poux, Ph.D. . This is the map for LiDAR and point cloud processing in Python, the part that gets reality onto disk and ready to work with. It's the capture end of a bigger pipeline I think of as capture, understand, deliver: sensing gets the world into the machine, everything after that turns it into meaning and then into something useful. Here you'll learn the core idea at each stage, see what the output actually looks like, and get pointed to the deep tutorial for that step. Every tool is open source, the code is there and every deeper piece is linked so you can go as far as you want. Read it once, then pick your next click. To see the full collection of master guides:

What is LiDAR Data Processing?

Today is the day where you learn to take a raw scan, a binary file full of timed laser returns, and turn it into a clean, structured, training-ready 3D world.

That's the capture phase, the first third of the whole equation. Capture gets reality into the machine. Understand turns that raw data into meaning. Deliver turns meaning into something someone can use.

Understand is the hard part, and I'll be honest about that throughout. But you can't understand data you captured badly. Every choice you make here, how you represent the cloud, how you align sources, how you separate ground, gets inherited by every stage downstream.

Get capture right and the rest has a fighting chance. Get it wrong and you'll be debugging ghosts for weeks.

Zoom into that one marked box and it opens up into a sequence of its own.

The capture end is not a single move but a chain, where each step feeds the next and inherits whatever the previous one got wrong.

The diagram below lays that chain out end to end, from the raw LAS file to a training-ready cloud.

Keep both pictures side by side: one shows where you sit in the whole system; the other shows exactly what you're about to do.

What you'll walk away with:

• A clear mental model of what a point cloud actually is, and why representation decides your speed
• The three-library spine for reading and structuring scans: laspy, PDAL, Open3D
• How to align messy multi-sensor data into one clean coordinate frame
• Cloth Simulation Filtering for ground extraction, with the parameters that make or break it
• A curated reading order through seven deep-dive tutorials, one concept at a time

Why the Pipeline Begins with Representation

Before you touch a single algorithm, answer one question: what shape is this data, really?

A point cloud is a list of XYZ coordinates. Fine. But each point can also carry intensity, a return number, an RGB color, and a classification code. How you hold all of that in memory decides which operations fly and which ones crawl.

That's where you stand at the very start, and it's a decision, not a default.

And there's more than one way to store 3D. Raw points, voxel grids, meshes, depth maps, each trades detail for speed differently. This is what matters here: a voxel grid is easy to convolve over but throws away fine geometry.

A raw cloud keeps every measurement but has no neighborhood structure, so finding a point's neighbors means building a KD-tree first. Pick the wrong representation early and plenty of projects quietly grind to a halt later.

What's still missing at this stage? Neighborhood structure. A raw cloud is just a bag of coordinates with no notion of what's near what, and almost every interesting operation needs neighbors. So the way you push further is to index the cloud early, which is where the KD-tree earns its keep.

🦥 Geeky Note: A KD-tree over N points builds in roughly O(N log N) and answers a nearest-neighbor query in about O(log N). For a 10-million-point urban scan, that's the difference between a radius search finishing in milliseconds versus scanning all 10 million points every single time. Open3D exposes it as geometry.KDTreeFlann.

Aerial LiDAR is a good case in point for where to look next. It's mostly 2.5D, one elevation per ground position, sensed top down, which means some tricks that work for terrain fall apart the moment you switch to a terrestrial scan of a building facade. Knowing which one you hold tells you which algorithms are even valid.

For the mental model that makes everything downstream click, the deep dive is How to represent 3D data, which walks the trade-offs point by point.

🪐 System Thinking Note: This is the capture phase, and it's one part of a whole. Capture, understand, deliver: getting reality into the machine, turning it into meaning, then into something useful. The hard part is understand, going from a messy scan to a faithful digital space a machine can reason over, and you don't get to skip the ugly components. Keep the full pipeline in your head even while you work on this one box. That's the difference between someone who pushes a button and someone who knows what runs underneath it.

In this part: — How to represent 3D data: the four ways to hold a cloud in memory, and why the choice sets your speed ceiling.

So once you know the shape of the data, how do you get it off disk and into Python?

Reading LAS Files and Structuring a Scan

LiDAR usually arrives as a LAS file, or its compressed sibling LAZ. That's where you stand: a binary format governed by the ASPRS LAS specification, which defines point record formats, classification codes, and the scaled-integer coordinate system that keeps files small.

You read it with laspy, which hands you NumPy arrays for X, Y, Z, intensity, and every other dimension the file carries.

Here's what matters about the toolchain. For anything heavier, PDAL gives you a pipeline model where reads, filters, and writes chain together as JSON stages. That's how you process files too big for RAM. And Open3D is where you'll do visualization, downsampling, and neighborhood queries.

Those three, with NumPy underneath, are the spine of the stack. Learn them and you can read almost anything.

Here's the core loop in code. You read the LAS file with laspy, lift the coordinates into an Open3D cloud, thin it with a voxel grid, then index it so neighbor queries return instantly instead of scanning the whole array.

import laspy
import numpy as np
import open3d as o3d

# Read the LAS file; laspy applies scale and offset for real-world XYZ
las = laspy.read("scan.las")
xyz = np.vstack((las.x, las.y, las.z)).transpose()
# Build an Open3D point cloud from the coordinates
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(xyz)
# Voxel-downsample to roughly one point per 0.1 m cell
pcd = pcd.voxel_down_sample(voxel_size=0.1)
# Index the cloud, then query the 8 nearest neighbors of the first point
tree = o3d.geometry.KDTreeFlann(pcd)
k, idx, dist = tree.search_knn_vector_3d(pcd.points[0], 8)

The one number to reason about is voxel_size: it sets the grid cell in meters, so a larger value keeps fewer points and coarser geometry, and Open3D does both the downsampling and the KD-tree over the NumPy arrays laspy handed you.

The hands-on entry point is Discover 3D point cloud processing with Python, from raw file to rendered, filtered cloud.

And when you want the full toolbox laid out so you stop guessing which library does what, Ultimate guide: 3D data science systems and tools maps it as a repeatable process.

What's the trap here? Coordinates. This is the one thing that silently ruins early results, and it's how you push past the beginner wall.

🦚 Florent's Note: When you read a LAS file with laspy, coordinates come back as scaled integers. The real-world value is X * header.scales[0] + header.offsets[0]. Skip that and your points land kilometers from where they should, a bug that looks like corrupted data but is really just an unapplied scale factor.

Where to look next depends on your goal. If you want the toolbox as a system rather than a pile of libraries, the Ultimate guide above is the reference. If you want your hands dirty in ten minutes, start with Discover.

In this part:

• Discover 3D point cloud processing with Python: go from a raw file to a rendered, filtered cloud with laspy and Open3D.
• Ultimate guide: 3D data science systems and tools: the whole toolchain mapped as a repeatable six-step process.

You've got points loaded and structured. But scans from different sensors rarely line up on their own. So what happens when you have more than one source?

Aligning Multiple LiDAR Sources into One Frame

Real projects almost never have one clean scan.

That's where you stand: airborne LiDAR for the roofs, terrestrial scans for the facades, maybe a mobile run down the street, and each lives in its own coordinate frame with its own density and gaps.

This is where coordinate reference systems, datum transforms, and registration all meet. Get the CRS wrong and two perfectly good scans sit meters apart.

What matters is the target state, one consistent world your later stages can trust. Not several clouds that happen to be in the same folder, but a single fused frame where a wall scanned from the ground and a roof scanned from the air actually meet at the eaves.

The tutorial 3D geospatial data integration with Python covers how to bring those sources into one frame so the rest of your pipeline sees a single, consistent cloud.

That's where to look when you have more than one dataset and they refuse to agree.

What's still missing, and easy to underestimate, is how much a small error costs you downstream. Alignment isn't a cosmetic step you can revisit later.

🪐 System Thinking Note: Alignment errors compound. A 30 cm registration offset between airborne and terrestrial scans doesn't just look wrong, it poisons every downstream descriptor: normals point sideways at the seam, ground classification splits into two levels, and your DTM grows a phantom step. Fix registration first, or you'll chase its ghost through every later stage.

So the cloud is loaded, cleaned, and unified. Now comes the step that turns geometry into meaning: which points are ground, and which are everything else?

In this part: 3D geospatial data integration with Python: fuse airborne, terrestrial, and vector sources into one trustworthy coordinate frame.

Ground Filtering and Building a DTM

Separating ground from non-ground is one of the more consequential operations in LiDAR, and it's where you stand once your cloud is unified.

Get it right, and you can build a digital terrain model, measure building heights, map vegetation, model flood risk.

Get it wrong and every derived product inherits the error.

What matters is the method, and there's one that keeps earning its place. A widely used approach is Cloth Simulation Filtering, introduced by Zhang and colleagues in 2016. The idea is almost playful: flip the cloud upside down, drape a virtual cloth over it, and let it settle under gravity.

Where the cloth rests is the ground surface, and nearby points get labeled ground.

It has few knobs; it handles steep terrain better than a naive lowest-point grid, and PDAL ships a filter for it, so you can wire it straight into a pipeline.

🦥 Geeky Note: CSF has three knobs that decide everything: cloth resolution (grid spacing, often 0.5 to 2 m), the rigidness parameter (1 for flat terrain, 3 for steep), and the classification threshold (distance from cloth to a point, commonly around 0.5 m). Tune the threshold too tight and low walls get called ground. Too loose and curbs vanish.

Where's the limitation? CSF gives you ground versus not-ground, and no more. It won't tell you a wall from a tree.

Once ground is separated, though, you interpolate those points into a raster and you've got a DTM. Everything above it, normalized to height-above-ground, becomes far easier to classify, which is exactly how you push this further.

And this loop is exactly the kind of thing you don't want to babysit, which is where to look next: how do you make it repeatable across hundreds of files?

In this part: The Cloth Simulation Filtering paper (Zhang et al., 2016): the ground-filtering method behind the PDAL filter, drape a cloth, read off the terrain.

Automating LiDAR Processing at Scale

Doing this once by hand teaches you the steps. Doing it on 400 tiles by hand teaches you to hate it.

That's where you stand the moment a project stops being a demo. So wrap ingest, filter, classify, and export into a script that chews through a whole folder while you do something else.

One of the first things that script should do to each tile is thin it out. Raw survey tiles carry far more points than the later stages actually need, and pushing every one of them through every step is wasted compute.

Voxel-downsampling early is the cheapest speedup in the whole loop, since it cuts the point count hard while keeping every meaningful surface intact. The sampling below shows what that looks like on one scene at several densities.

What matters is consistency, not just speed. That's the point of How to automate LiDAR point cloud processing with Python.

It shows you how to string the stages together, handle tiling and edge effects, and produce output you can trust, because every tile gets treated identically.

And once you can process a tile reliably, you can push higher: full city models. 3D Python workflows for LiDAR city models takes the automated pipeline toward reconstructing buildings and terrain at urban scale, the kind of output that feeds digital twins and planning tools.

What's the risk when you scale? Silent failure. A pipeline that dies quietly on one tile out of hundreds costs you more than the manual version ever did. So here's how to look at building it.

🌱 Growing Note: If you're building automation for the first time, start with a batch of 5 tiles, not 500. Get the per-tile output pixel-perfect and the logging clear, then scale the loop. A pipeline that fails silently on tile 217 of 500 will cost you more time than it ever saved.

In this part:

• How to automate LiDAR point cloud processing with Python: chain ingest, filter, and export into one script that handles tiling and edges.
• 3D Python workflows for LiDAR city models: push the automated pipeline to reconstruct buildings and terrain at urban scale.

You can now process your own data at scale. So where does machine learning enter, and what does it actually need from you first?

From Classical Descriptors into Deep Learning

Classical processing gets you a long way, and that's where you stand at the end of the capture phase. Hand-designed descriptors: surface normals, curvature, planarity, verticality, height above ground. Feed those to a random forest in scikit-learn and you can classify vegetation, ground, and buildings with respectable accuracy and full interpretability. For a lot of real work, that's genuinely enough.

But when the classes get subtle, cars versus pedestrians, cable versus vegetation, learned descriptors start to win. That's what matters at the boundary of this part: the bridge into deep learning, which lives in the understand phase. Before any of that, though, you need data, and finding good labeled LiDAR is half the battle.

What's still missing before you can train? Ground truth. And it's expensive. Which is why Free LiDAR datasets for self-driving cars is worth your time: it points you to open, labeled autonomous-driving scans you can train on today, so you don't burn a week hand-labeling before you've even started.

How do you push into training cleanly? With one preprocessing habit that saves you from a network that never converges.

🦚 Florent's Note: Before training, normalize each sample to a unit sphere: subtract the centroid, then divide by the max distance from it. Networks that ingest raw meter-scale coordinates converge slowly or not at all, because the loss surface is dominated by absolute position instead of shape. One preprocessing line fixes it.

Where to look next is a whole part of its own, and it's two doors down in the understand phase.

In this part: Free LiDAR datasets for self-driving cars: open, labeled scans you can train on today instead of building ground truth by hand.

So you can capture, structure, and prepare. What does all of this actually let you build?

Why This Matters and What You Can Build

Look back at the result above: the wool factory reconstructed from fused sensors, and the annotated driving scan ready for a network.

That gap, from a raw timing measurement to a labeled, training-ready 3D world, is the whole skill of the capture phase.

And it pays out in concrete things.

A clean DTM feeds flood and slope analysis. Ground-normalized heights feed automatic building and vegetation classification. A repeatable pipeline feeds a digital twin that updates every time a new survey lands.

Each output is one box in that pipeline diagram, built on the box before it.

Here's the part that matters in the age of AI.

Plenty of tools now hand you a button that ingests a scan and spits out a classified cloud. That's fine, until it breaks, until the CRS is wrong, until the ground filter eats a low wall and you have no idea why.

🎩 Fun Anecdote: The same time-of-flight principle behind your desktop scan is what Lunar Laser Ranging has used since 1969 to track the Moon drifting away from Earth at about 3.8 cm per year. Different scale, identical physics: fire light, time the return, read off distance. Your LAS file is that experiment run a few million times.

If you can fix that and know what runs behind the button, you are a small fraction of practitioners. This part is how you become one of them. Models don't replace that understanding; they make it worth more, because someone has to build and judge the buttons everyone else clicks.

⚠️ Warning: Don't optimize the algorithm before you've fixed the representation. If a step is slow, ask whether it's the method or the data structure feeding it. A radius search over a raw cloud with no KD-tree isn't a slow algorithm; it's a missing index.

Pick Your Slowest Stage and Fix It

Here's my question back to you: which stage in your workflow is the slowest, and is it slow because of the algorithm or because of the representation you chose upstream?

That answer usually points straight at what to fix first.

Go open one of the seven spokes above that matches your weakest link, and rebuild just that stage properly.

Continue the Map

This part is one of six under the 3D Spatial AI with Python master guide.

Start there for the full picture of how capture, understand, and deliver connect across the whole pipeline.

Two parts pick up right where capture continues. Once you've captured with laser, the other way to get reality into the machine is from images, which is 3D reconstruction in Python, photogrammetry, AI depth, and Gaussian splatting.

And once your points are clean, the first understand step is grouping them into meaningful parts, which is 3D point cloud segmentation and clustering in Python.

When you're ready to build the whole pipeline hands-on, the free 3D mission is where to start.

Resources

• ASPRS LAS specification, the format spec behind every LAS/LAZ file.
• laspy, PDAL, and Open3D docs, the three-library spine of the stack.
• Cloth Simulation Filtering (Zhang et al., 2016), the ground-filtering method used above.
• Three courses that take this deep: Point Cloud Intelligence for the foundations, Large-Scale Point Cloud Processing for the heavy-data end, and 3D Spatial OS for wiring it into a system.
• The 3D Geodata Academy, where I teach this end-to-end with production code and the parameter intuition you can't get from one article.

Keep Going

If LiDAR and point cloud processing is the kind of thing that pulls you in, the academy has a deep stack of free content to start with, and a structured program for when you're serious. Start free, then go further: begin here.

About the Author

I'm Florent Poux, Ph.D. I research and teach spatial AI, and I wrote 3D Data Science with Python (O'Reilly). I spend my days turning research into systems that ship. My aim is to show you what's behind the buttons you push today, so you can be the one who builds them, not just clicks them.

FAQ

What's the best Python library for LiDAR point cloud processing? There is no single one; you use several together. Read files with laspy, run heavy out-of-core pipelines with PDAL, and handle visualization and neighborhood queries with Open3D, all over NumPy. This three-library spine reads almost any scan.

How do I handle a LiDAR file too large for RAM? Stream and tile it with PDAL pipelines, or downsample early with Open3D voxel grids. Airborne surveys reach tens of millions of points per tile, so out-of-core processing is the norm, not the exception, and voxel-downsampling early can cut point count by 80 percent while keeping every surface.

How do I separate ground points from buildings and trees? Use a ground-filtering algorithm like Cloth Simulation Filtering (Zhang et al., 2016), available as a PDAL filter. It drapes a virtual cloth over the inverted cloud to find the terrain, then labels nearby points as ground so you can interpolate a DTM.

Do I need deep learning to process LiDAR? No. Plenty of tasks are solved well with classical descriptors and a scikit-learn classifier, and that is the right place to start. Deep learning wins on subtle semantic classes, and you can move into it through the free 3D mission once your data is prepared.