Axis-Aligned Rounded Rectangle Collision

If you're the kind of person who likes to make little 2D games from scratch without a “game engine” then this is a handy algorithm to know. It handles not only axis-aligned rounded rectangles, but ordinary rectangles (zero corner radius) and circles (only radius, no extra width or height) as well. So this one algorithm gives you enough for simple games, while giving you more flexibility than just rectangular hit boxes.

And it's relatively simple under the hood, but it introduces some interesting patterns and ideas that are useful when doing more complex collision detection.

So. For the purposes of this algorithm, we want to represent rounded rectangles by:

You may want to store them (or create them) using different parameters, but this is how we want to think about them for this algorithm. So interesting idea number 1: different representations make different problems easier or harder. A lot of interesting math stuff is about shifting to a perspective that makes your particular problem solvable.

Configuration Space Obstacles

Here's a cool viewpoint shift: it's often easier to collide a point against a shape than to collide two shapes against each other. What if we slide one rounded rectangle around the other and trace out the shape that it follows? Then we can check the point against this larger shape, which is often called a Configuration Space Obstacle (CSO for short).

And in this case, it turns out that this larger shape is also a rounded rectangle. So we can simplify the problem to checking whether a point collides with a rounded rectangle.

Note that if you have the same set of “walls” but your moving objects have different sizes and shapes, each object will have a different configuration space. So you can't just store the configuration space.

Unless you treat that as a design constraint. What if all your moving objects have the same collision shape? How different can you make the visual representations? Or how hard can you lean into them goofily all having similar outlines?

Point vs. rounded rectangle

We're thinking of a rounded rectangle as having a center point, an inner width and height (giving a rectangle with sharp corners), and then we apply a radius outside that, giving rounded corners and an outside bounding rectangle.

So to check if a point collides with our shape, first we take the x and y distances from the rectangle center to the point.

We notice that each of these differences could be positive or negative (the point could be above or below, left or right of the rectangle center) but all four quadrants are the same shape. So we take the absolute value of each. This makes the math easier by collapsing all four quadrants down into one.

If we care about which direction the shapes collided in (if you want to bounce them off each other, for instance) then we'll remember the sign of each difference so we can figure out the direction at the end. If all we care about is whether they touch (maybe one shape is a bullet that will be destroyed on impact) then we can skip this.

We also compute the half width and half height, because those are the distances from the center to the edge: we never need the full width and height here.

Are we safely outside the bounding box?

Let's do the easy check first: if the point is outside the bounding box, then there's no collision.

Note that we can check the x and y separately: if the x-coordinate is outside the width of the bounding box, there's no y-coordinate that can make the point hit the rectangle. And similarly for y.

if(dx > r+halfWidth or dy > r+halfHeight) return false

Are we outside the inner rectangle?

Now we know the point is inside the bounding box, so it's probably a hit. The only misses are if it's in one of the corners outside the rounded area. So we calculate how far the point overlaps into the inner rectangle: 
ox = halfWidth - dx
oy = halfHeight - dy

There are several cases here: if the point is inside the inner rectangle on one axis (positive overlap) but not on the other (negative overlap), then we're in one of these four margins. In this case we hit one of the sides, so the “collision normal” (the direction of the collision) points straight sideways or straight up or down.

This point is well inside the inner rectangle on the x-axis, but outside it on the y-axis. So our collision normal would point downwards.

If the overlap is less than zero on both axes then we're on the rounded corner and we have to actually compute the distance to see if the point is inside or outside. In this case ox and oy give the normal: the direction from the (square) corner of the inner rectangle to where the point is.

Inside the inner rectangle

If the point is all the way inside the inner rect, then we choose the normal to be on one of the sides, picking the direction with the least overlap (the smallest distance that we'd have to push the objects apart so they just touch).

So in the following figure our normal would point to the right:

This usually works pretty well, but if two sharp rectangles hit near the corner it can be wrong. This is a notorious source of problems in tile-based 2D platformers. You can try to solve this by using the point's velocity and figuring out exactly where it hits, but sometimes you'll still get the wrong answer because of rounding errors. And what if a bunch of objects clump up and hit each other all at the same time? That quickly gets complicated to resolve.

So I think it's usually best to stick with the simpler code and regard this as a design constraint and get creative about avoiding it: rounded corners on your player character, making sure objects aren't moving “too fast” for the speed of your simulation, merging tiles together into bigger collision boxes (also more efficient!), etc.

Any physics and collision simulation is an approximation and is going to have things that it doesn't do well. Even if you solve everything the hard way so it detects collisions at the exact moment when they happen and “properly” resolves clusters of objects crashing together, then you probably have a simulation that slows way down for complicated collisions rather than one that keeps going but gives wrong results. Which can also be a big problem. So again, I think it's better to treat the weirdness as a creative constraint and find ways to design so that it doesn't matter.

Anyway. If we have a collision, the final step is to return the normal and the overlap. We have the overlap into the inner rectangle, so we add the radius to get the total overlap.

Handling collisions

First you probably want to push the objects apart so they don't overlap. The normal vector (nx, ny) points outward from the rectangle. So if the rectangle represents a fixed object and the point represents a moving object, you can just add overlap times the normal to the point.

If it's the other way around, subtract overlap times the normal from the rectangle's coordinates. Perhaps both of the objects are moving and you want to move each by half, or move each proportionally to the objects' mass. This probably usually doesn't matter too much, since ideally you'd be simulating fast enough (relative to the objects' speed) that they don't interpenetrate very much.

And then you probably want to change the velocity so the objects don't keep pushing together. The basic technique here is to take the dot product of the relative velocity with the normal vector.

You want to get the sign right on this: take the point's velocity minus the rectangle's velocity. This will give you the point's velocity relative to the rectangle.

If we take the dot product of this relative velocity and the normal vector, that will give us the relative speed in the normal direction. Since the normal vector is pointing away from the rectangle, a positive value means the objects are moving apart, while a negative value means they're moving together.

You'll want to put a conditional check here: only bounce your objects if they're moving towards each other (negative speed). Otherwise you can get weird situations where objects get locked together because they're toggling between bouncing away and bouncing back together each frame.

Scale the normal vector by this speed to get the vector component of the velocity in the normal direction. If the rectangle is a fixed object and the point is a floating one, then you want to add twice this to the point's velocity: once to cancel out the closing velocity, and again to mirror it back outwards.

It gets more complicated if you want to bounce both objects: how bouncy are they, what are the relative masses, etc. so I'm not going to cover that here. The resulting equations actually aren't too bad, but deriving them is a fair bit of algebra: Wikipedia's Coefficient of restitution article is pretty good.

Lua code

So here's the whole collision check, in Lua:

local function sign(x)  return x < 0 and -1 or 1  end

-- px/py: point, cx/cy: rectangle center
local function pointVsRoundRect(px,py, cx,cy, w,h, r)
    local halfWidth, halfHeight = 0.5 * w, 0.5 * h
    local dx, dy = px - cx, py - cy
    -- keep the signs of these differences
    local sdx, sdy = sign(dx), sign(dy)
    -- but collapse all four quadrants into one
    dx, dy = math.abs(dx), math.abs(dy)
    -- Are we separated horizontally or vertically?
    if dx > r+halfWidth or dy > r+halfHeight then
        return false
    end
    local nx, ny  -- local variables for collision normal
    -- Distance that point overlaps into the inner rectangle
    local ox, oy = halfWidth - dx, halfHeight - dy
    if ox < 0 and oy < 0 then
        -- We're on the corner of the rectangle,
        -- check the distance to see if
        -- we're inside the rounded corner.
        local dist = ox*ox + oy*oy  -- squared, temporarily
        if dist > r*r then return false end
        dist = math.sqrt(dist)
        local scale = -1 / dist
        nx, ny = ox * sdx * scale, oy * sdy * scale
        return nx,ny, r - dist
    elseif ox < 0 then  -- We're on one of the side margins
        return sdx, 0,  r + ox
    elseif oy < 0 then  -- We're on the top/bottom margin
        return 0, sdy,  r + oy
    else  -- Point is inside the inner rect
        -- Pick the shortest distance to push them apart
        if oy < ox then  return 0, sdy,  r + oy
        else  return sdx, 0,  r + ox
        end
    end
end