It sounds like what you're hunting for isn't the numerical area at all, but a region descriptor, essentially a way to represent "the set of points inside this triangle" in inequalities, similar to how a bounding box can be written:
x_min <= x <= x_max
y_min <= y <= y_max
For an arbitrary triangle though, it won't be simple min/max ranges because of the angled edges. The standard trick is to use half‑space inequalities built from the edges:
1. Take your three vertices A(x1,y1), B(x2,y2), C(x3,y3).
2. For each edge (say AB), compute the line equation through those two points.
- Line AB: (y2 - y1)(x - x1) - (x2 - x1)(y - y1) = 0
3. Decide which side of that line contains the third vertex (C). That tells you whether your inequality should be >=0 or <= 0 for "inside."
4. Do this for all three edges. You'll end up with three inequalities describing exactly the triangular region.
So instead of one scalar "area," you've got something like:
f_AB(x,y) >= 0
f_BC(x,y) >= 0
f_CA(x,y) >= 0
That's basically what axis‑aligned rectangles are doing but generalized, they just happen to reduce neatly into simple less‑than bounds when sides are aligned with axes.
If computation cost is scary:
- You can precompute those coefficients once per triangle and then checking if a point lies inside becomes just 3 multiplies + adds and some comparisons => very cheap.
- If you only need approximations, yes, snapping down to an axis‑aligned bounding box still works, but that's usually coarser than people want.
tl;dr: Represent it as intersection of half spaces (= inequalities). That matches what you're saying about wanting a "less than / greater than" analogue rather than raw numeric area.
Do you need this for testing point‑in‑triangle membership or are you thinking more along collision regions / clipping?