[squid-users] very slow and poor cache hits using ntlm

Hi people: I have a squid using ntlm to authenticate the users. I also
use a external acl but im running out of ideas to make it run faster.

If I disable the ntlm everything works very well, and the cache hits
increses a lot.

I found that the squids asks a lot for username and passwords to the
winbind, and the winbind asks everytime to my pdc. This generates a
lot of traffic between them and a high load on the pdc.

Every hit on any page, squid ask to the winbind for the username and
password. Is this the expected behavior? Is there any way to reduce
(caching maybe?) this? I didn't find a solution in the winbind, to
stop asking the credentials to the pdc.

I have a terminal server enviroment, so where you see 69 clients the
are in fact more than 500 users.

This is my output of squidclient mgr:info

Squid Object Cache: Version 2.6.STABLE5
Start Time: Fri, 05 Feb 2010 07:21:21 GMT
Current Time: Sat, 20 Feb 2010 03:01:08 GMT
Connection information for squid:
         Number of clients accessing cache: 69
         Number of HTTP requests received: 11790881
         Number of ICP messages received: 0
         Number of ICP messages sent: 0
         Number of queued ICP replies: 0
         Number of HTCP messages received: 0
         Number of HTCP messages sent: 0
         Request failure ratio: 0.00
         Average HTTP requests per minute since start: 552.5
         Average ICP messages per minute since start: 0.0
         Select loop called: 154266917 times, 8.300 ms avg
Cache information for squid:
         Request Hit Ratios: 5min: 50.5%, 60min: 18.3%
         Byte Hit Ratios: 5min: 14.1%, 60min: 26.3%
         Request Memory Hit Ratios: 5min: 0.0%, 60min: 10.0%
         Request Disk Hit Ratios: 5min: 19.7%, 60min: 21.9%
         Storage Swap size: 7833612 KB
         Storage Mem size: 409452 KB
         Mean Object Size: 19.26 KB
         Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
         HTTP Requests (All): 0.00919 0.03066
         Cache Misses: 0.35832 0.44492
         Cache Hits: 0.01164 0.01847
         Near Hits: 0.33943 0.37825
         Not-Modified Replies: 0.00286 0.00405
         DNS Lookups: 0.09117 0.10906
         ICP Queries: 0.00000 0.00000
Resource usage for squid:
         UP Time: 1280387.834 seconds
         CPU Time: 5238.387 seconds
         CPU Usage: 0.41%
         CPU Usage, 5 minute avg: 0.07%
         CPU Usage, 60 minute avg: 0.05%
         Process Data Segment Size via sbrk(): 561092 KB
         Maximum Resident Size: 0 KB
         Page faults with physical i/o: 4
Memory usage for squid via mallinfo():
         Total space in arena: 561092 KB
         Ordinary blocks: 555876 KB 13964 blks
         Small blocks: 0 KB 0 blks
         Holding blocks: 1744 KB 4 blks
         Free Small blocks: 0 KB
         Free Ordinary blocks: 5215 KB
         Total in use: 557620 KB 99%
         Total free: 5215 KB 1%
         Total size: 562836 KB
Memory accounted for:
         Total accounted: 511637 KB
         memPoolAlloc calls: 1443436295
         memPoolFree calls: 1441310223
File descriptor usage for squid:
         Maximum number of file descriptors: 1024
         Largest file desc currently in use: 268
         Number of file desc currently in use: 261
         Files queued for open: 0
         Available number of file descriptors: 763
         Reserved number of file descriptors: 100
         Store Disk files open: 2
         IO loop method: epoll
Internal Data Structures:
         407686 StoreEntries
          34175 StoreEntries with MemObjects
          34170 Hot Object Cache Items
         406635 on-disk objects

This is the output of squidclient mgr:ntlmauthenticator

(warning: the avg service time is with NO users, when everyone is
using it the avg service time peeks the 1000 msec. YES 1K msec).

NTLM Authenticator Statistics:
program: /usr/bin/ntlm_auth
number running: 200 of 200
requests sent: 2500498
replies received: 2500498
queue length: 0
avg service time: 19.24 msec

       # FD PID # Requests Flags Time Offset Request
       1 12 17113 168619 0.046 0 (none)
       2 13 17114 62644 0.055 0 (none)
       3 14 17118 31007 0.076 0 (none)
       4 15 17120 15188 0.094 0 (none)
       5 16 17121 5759 0.093 0 (none)
       6 17 17122 2845 0.071 0 (none)
       7 18 17124 1572 0.524 0 (none)
       8 19 17125 891 0.533 0 (none)
       9 21 17130 486 0.584 0 (none)
      10 22 17131 302 0.647 0 (none)
      11 23 17132 194 0.741 0 (none)
      12 24 17135 127 0.818 0 (none)
      13 25 17137 84 0.756 0 (none)
      14 26 17138 56 0.898 0 (none)
      15 27 17143 46 0.954 0 (none)
      16 28 17149 36 1.002 0 (none)
      17 29 17155 24 1.125 0 (none)
      18 30 17161 22 1.094 0 (none)
      19 31 17162 16 1.252 0 (none)
      20 32 17165 10 5.137 0 (none)
      21 33 17167 8 4.807 0 (none)
      22 34 17168 4 1.470 0 (none)
      23 35 17169 4 1.522 0 (none)
      24 36 17170 2 1.185 0 (none)
      25 37 17171 2 0.613 0 (none)
      26 38 17172 2 0.839 0 (none)
      27 39 17173 0 0.000 0 (none)

Any ideas in how to improve this scenario?

This is the squid.conf

hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
http_port 3128
#debug_options ALL,1 33,2
log_fqdn off
cache_store_log none
useragent_log none
cache_log /var/log/squid/cache_log.log
access_log /var/log/squid/access.log
error_directory /usr/share/squid/errors/Spanish
emulate_httpd_log on

offline_mode off
strip_query_terms on
httpd_suppress_version_string on

auth_param ntlm program /usr/bin/ntlm_auth
--helper-protocol=squid-2.5-ntlmssp --domain=MYDOMAIN
auth_param ntlm children 200
auth_param ntlm keep_alive on
authenticate_ttl 60 seconds
authenticate_ip_ttl 2 minutes
authenticate_cache_garbage_interval 10 seconds
auth_param basic program /usr/bin/ntlm_auth
--helper-protocol=squid-2.5-basic --domain=MYDOMAIN
auth_param basic children 10
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours

external_acl_type ldap_group ttl=1200 children=25 %LOGIN
/usr/lib/squid/squid_ldap_group -b "GROUPDN" -f "MYFILTER" -h
LDAPSERVER -v3 -S -P

negative_ttl 5 minutes
positive_dns_ttl 5 hours
negative_dns_ttl 1 minutes
half_closed_clients off
connect_timeout 3 seconds
cache_dir aufs /var/spool/squid 9000 16 256
cache_swap_low 85
cache_swap_high 95
maximum_object_size 81920 KB
maximum_object_size_in_memory 300 KB
cache_mem 400 MB
fqdncache_size 6144
cache_replacement_policy lfuda
memory_replacement_policy lru
pipeline_prefetch off
client_persistent_connections off
server_persistent_connections off
visible_hostname myproxy.mydomain

hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY

acl all src all
acl lan_10_8 src 10.8.0.0/255.255.0.0

acl webservers dst 10.8.50.220/255.255.255.255
10.8.50.221/255.255.255.255 10.8.50.222/255.255.255.255
10.8.50.223/255.255.255.255

acl nomsnurl dstdomain "/etc/squid/nomsn"

acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl SSL_ports port 443 563 1863 6667 4430
acl Safe_ports port 80 # http
acl Safe_ports port 443 563 # https, snews

acl auth proxy_auth REQUIRED
acl noinet external ldap_group noinet
acl linuxadmin external ldap_group linuxadmin
acl nomsn external ldap_group nomsn
acl dummy src 0.0.0.0/0.0.0.0

acl CONNECT method CONNECT
acl PURGE method PURGE

http_access allow PURGE localhost
http_access deny PURGE
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost

http_access allow webservers
http_access deny nomsn nomsnurl
http_access allow linuxadmin dummy
http_access deny noinet dummy
http_access allow auth lan_10_8

http_access deny all
icp_access deny all

acl snmppublic snmp_community public
snmp_port 3401
snmp_access allow snmppublic localhost
snmp_access deny all
snmp_incoming_address 0.0.0.0
snmp_outgoing_address 255.255.255.255
Received on Sat Feb 20 2010 - 03:16:03 MST