[OCLUG-devel] Re: GNU code with security holes?!! [was: Trouble with strings and files]

[Christopher Smith]

On Fri, 2004-07-09 at 00:53, Christopher Smith wrote:
> Where's the beef? Well, think of the test case in my code:
> 
> (line->size < (SIZE_MAX/2))
> 
> Okay, that's a pretty strong hint. I seem to think the problem is
> happening somewhere around SIZE_MAX/2. Now, technically, there isn't a
> problem if line->size == (SIZE_MAX/2), but since "*= 2" on that would
> work out to SIZE_MAX anyway, I thought why not save ourselves doing a
> multiply? So, what happens once line->size > (SIZE_MAX/2). Well, lets do
> this algebraically.

Okay, so upon *further* review I realized a mistake in my math here.
SIZE_MAX is of course going to be an odd number (unless size_t is
signed, which it never is). So, SIZE_MAX/2 == SIZE_MAX - 1. So, there is
a slight error here. Which helps explain how I came up with

> you'd expect" mod (SIZE_MAX + 1). So it turns out:
> 
> line->size = (SIZE_MAX + 2i) % (SIZE_MAX + 1)
> 
> which is the same as:
> 
> line->size = 2(i - 1)

This is of course wrong. Going from (SIZE_MAX + 2i) % (SIZE_MAX + 1)
should come out to:

2i - 1

The devil is always in the details. If you go back into the algebra,
you'll find that the "SIZE_MAX" I have there is as a result of
(SIZE_MAX/2)*2. Because the division step is done before the multiply,
this doesn't work out to SIZE_MAX, but to SIZE_MAX-1, so while
line->size does end up equally 2(i - 1), that a transformation from:

((SIZE_MAX - 1) + 2i) % (SIZE_MAX + 1)

Anyway, it's a niggly detail, but an important bit to get right.

-- 
Christopher Smith <x at xman.org>