Re: [squid-users] why squid does not support sendfile() ?

Matus UHLAR - fantomas wrote:
>> Weibin Yao wrote:
>>> I'am using squid-2.7. I has checked the configure reference and found
>>> nothing about sendfile(). Why squid does not support sendfile()?
>>> especially the HIT request.
>
> On 26.01.10 21:26, Amos Jeffries wrote:
>> 1) Blocking call. Squid needs to support more than one client request
>> simutaneously.
>
> is it blocking anywhere?

According to the docs sendfile() does not return until the entire file
has been sent. Setting the non-blocking IO flagon the outgoing sockets
wil result in an error code.

Squid with its single thread cannot use this type of call without
terrible performance losses.

>
>> 2) speed. sendfile is limited linearly by disk IO speeds, blocking the
>> entire time.
>
> does it matter for content fetched from disk? I think that sendfile is for
> this cases the most effective option (e.g. from disk direct to network card
> memory). I understand it can be an issue in 3.x where squid wants to
> implement own caching, but wonder if sendfile couldn't help here as you
> indicate.

We could start a new thread for each file send, it might be usable. That
covers the one-client one-file-from-disk sending case...

>
>> 3) HTTP protocol. The current design of Squid stores the headers and
>> data together. They cannot be altered correctly according to protocol
>> requirements during a sendfile() call.
>
> you can read, process and write headers and THEN call sendfile for the rest
> of content. The problem is with chunking which it not supported on client
> connections yet, iirc.

As I said "the current design". Someone with time to do a good re-design
would allow a lot of things to be done better.

Such a re-design has been on the books for a long time to solve the
HTTP/1.1 range-request issues, but none of the current developers has
had both the expertise and time to do it.

>
>> 4) collapsed forwarding. multiple clients may be receiving the same
>> identical object from Squid simultaneously, or even different parts of
>> the same object.
>
> should not be a problem with sendfile, should it?

It increases the disk load for N clients from 1 disk read pass across
the file to N disk read passes. Despite doing it in the kernel, this is
a net gain in lag.

The main loss is that it prevents Squid loading the object back into
memory for Hot-Object re-use and alterations.

>
>> 4) object location. not all HIT objects are from files. some may be in
>> memory, or a range of something partially received by another client.
>
> 5) I think ;-)
> Yes sendfile is only applicable on content fetched from the disk.
>
> Apparently nobody implemented sendfile in squid yet and apparently nobody
> will do it, but I wonder if all those reasons are really that problematic...
>

Well I wouldn't go so far as to say nobody will. I'm just pointing out
the known hurdles as to why nobody has yet. If somebody wants to try it
and see, feel free to do so.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE23
   Current Beta Squid 3.1.0.16