Re: [squid-users] very slow and poor cache hits using ntlm

Guido Marino Lorenzutti wrote:
> Hi people: I have a squid using ntlm to authenticate the users. I also
> use a external acl but im running out of ideas to make it run faster.
>
> If I disable the ntlm everything works very well, and the cache hits
> increses a lot.
>
> I found that the squids asks a lot for username and passwords to the
> winbind, and the winbind asks everytime to my pdc. This generates a lot
> of traffic between them and a high load on the pdc.
>
> Every hit on any page, squid ask to the winbind for the username and
> password. Is this the expected behavior? Is there any way to reduce
> (caching maybe?) this? I didn't find a solution in the winbind, to stop
> asking the credentials to the pdc.
>
> I have a terminal server enviroment, so where you see 69 clients the are
> in fact more than 500 users.
>
> This is my output of squidclient mgr:info
>
> Squid Object Cache: Version 2.6.STABLE5
> Start Time: Fri, 05 Feb 2010 07:21:21 GMT
> Current Time: Sat, 20 Feb 2010 03:01:08 GMT
> Connection information for squid:
> Number of clients accessing cache: 69
> Number of HTTP requests received: 11790881
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Number of HTCP messages received: 0
> Number of HTCP messages sent: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 552.5
> Average ICP messages per minute since start: 0.0
> Select loop called: 154266917 times, 8.300 ms avg
> Cache information for squid:
> Request Hit Ratios: 5min: 50.5%, 60min: 18.3%
> Byte Hit Ratios: 5min: 14.1%, 60min: 26.3%
> Request Memory Hit Ratios: 5min: 0.0%, 60min: 10.0%
> Request Disk Hit Ratios: 5min: 19.7%, 60min: 21.9%
> Storage Swap size: 7833612 KB
> Storage Mem size: 409452 KB
> Mean Object Size: 19.26 KB
> Requests given to unlinkd: 0
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 0.00919 0.03066
> Cache Misses: 0.35832 0.44492
> Cache Hits: 0.01164 0.01847
> Near Hits: 0.33943 0.37825
> Not-Modified Replies: 0.00286 0.00405
> DNS Lookups: 0.09117 0.10906
> ICP Queries: 0.00000 0.00000
> Resource usage for squid:
> UP Time: 1280387.834 seconds
> CPU Time: 5238.387 seconds
> CPU Usage: 0.41%
> CPU Usage, 5 minute avg: 0.07%
> CPU Usage, 60 minute avg: 0.05%
> Process Data Segment Size via sbrk(): 561092 KB
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 4
> Memory usage for squid via mallinfo():
> Total space in arena: 561092 KB
> Ordinary blocks: 555876 KB 13964 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 1744 KB 4 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 5215 KB
> Total in use: 557620 KB 99%
> Total free: 5215 KB 1%
> Total size: 562836 KB
> Memory accounted for:
> Total accounted: 511637 KB
> memPoolAlloc calls: 1443436295
> memPoolFree calls: 1441310223
> File descriptor usage for squid:
> Maximum number of file descriptors: 1024
> Largest file desc currently in use: 268
> Number of file desc currently in use: 261
> Files queued for open: 0
> Available number of file descriptors: 763
> Reserved number of file descriptors: 100
> Store Disk files open: 2
> IO loop method: epoll
> Internal Data Structures:
> 407686 StoreEntries
> 34175 StoreEntries with MemObjects
> 34170 Hot Object Cache Items
> 406635 on-disk objects
>
>
> This is the output of squidclient mgr:ntlmauthenticator
>
> (warning: the avg service time is with NO users, when everyone is using
> it the avg service time peeks the 1000 msec. YES 1K msec).
>
> NTLM Authenticator Statistics:
> program: /usr/bin/ntlm_auth
> number running: 200 of 200
> requests sent: 2500498
> replies received: 2500498
> queue length: 0
> avg service time: 19.24 msec
>
> # FD PID # Requests Flags Time Offset Request
> 1 12 17113 168619 0.046 0 (none)
> 2 13 17114 62644 0.055 0 (none)
> 3 14 17118 31007 0.076 0 (none)
> 4 15 17120 15188 0.094 0 (none)
> 5 16 17121 5759 0.093 0 (none)
> 6 17 17122 2845 0.071 0 (none)
> 7 18 17124 1572 0.524 0 (none)
> 8 19 17125 891 0.533 0 (none)
> 9 21 17130 486 0.584 0 (none)
> 10 22 17131 302 0.647 0 (none)
> 11 23 17132 194 0.741 0 (none)
> 12 24 17135 127 0.818 0 (none)
> 13 25 17137 84 0.756 0 (none)
> 14 26 17138 56 0.898 0 (none)
> 15 27 17143 46 0.954 0 (none)
> 16 28 17149 36 1.002 0 (none)
> 17 29 17155 24 1.125 0 (none)
> 18 30 17161 22 1.094 0 (none)
> 19 31 17162 16 1.252 0 (none)
> 20 32 17165 10 5.137 0 (none)
> 21 33 17167 8 4.807 0 (none)
> 22 34 17168 4 1.470 0 (none)
> 23 35 17169 4 1.522 0 (none)
> 24 36 17170 2 1.185 0 (none)
> 25 37 17171 2 0.613 0 (none)
> 26 38 17172 2 0.839 0 (none)
> 27 39 17173 0 0.000 0 (none)
>
>
> Any ideas in how to improve this scenario?
>
> This is the squid.conf
>
> hierarchy_stoplist cgi-bin ?
> acl QUERY urlpath_regex cgi-bin \?
> http_port 3128
> #debug_options ALL,1 33,2
> log_fqdn off
> cache_store_log none
> useragent_log none
> cache_log /var/log/squid/cache_log.log
> access_log /var/log/squid/access.log
> error_directory /usr/share/squid/errors/Spanish
> emulate_httpd_log on
>
> offline_mode off
> strip_query_terms on
> httpd_suppress_version_string on
>
> auth_param ntlm program /usr/bin/ntlm_auth
> --helper-protocol=squid-2.5-ntlmssp --domain=MYDOMAIN
> auth_param ntlm children 200
> auth_param ntlm keep_alive on
> authenticate_ttl 60 seconds
> authenticate_ip_ttl 2 minutes
> authenticate_cache_garbage_interval 10 seconds

Seems a bit extreme to be running the garbage collection 10 seconds. It
happens as needed on top of this.

The defaults are measured in hours and user browsing times are usually
longer than minutes.

> auth_param basic program /usr/bin/ntlm_auth
> --helper-protocol=squid-2.5-basic --domain=MYDOMAIN
> auth_param basic children 10
> auth_param basic realm Squid proxy-caching web server
> auth_param basic credentialsttl 2 hours
>
> external_acl_type ldap_group ttl=1200 children=25 %LOGIN
> /usr/lib/squid/squid_ldap_group -b "GROUPDN" -f "MYFILTER" -h LDAPSERVER
> -v3 -S -P
>
> negative_ttl 5 minutes

This is not really a good idea.
It will extend the period of outage for every service failure and may
hose the network access to a website for 5 minutes following a single
client page error.

> positive_dns_ttl 5 hours
> negative_dns_ttl 1 minutes

Please don't play with DNS TTLs unless you know 100% how they will
affect things.

> half_closed_clients off
> connect_timeout 3 seconds
> cache_dir aufs /var/spool/squid 9000 16 256
> cache_swap_low 85
> cache_swap_high 95
> maximum_object_size 81920 KB
> maximum_object_size_in_memory 300 KB
> cache_mem 400 MB
> fqdncache_size 6144
> cache_replacement_policy lfuda
> memory_replacement_policy lru
> pipeline_prefetch off
> client_persistent_connections off
> server_persistent_connections off

Persistent connections are REQUIRED for NTLM and related
connection-based auth to be used efficiently.

NTLM auth against the proxy requires persistent client connections,
pass-thru to web servers requires both and the connection pinning
feature as well.

> visible_hostname myproxy.mydomain
>
> hierarchy_stoplist cgi-bin ?
> acl QUERY urlpath_regex cgi-bin \?
> no_cache deny QUERY
>
> acl all src all
> acl lan_10_8 src 10.8.0.0/255.255.0.0

acl lan_10_8 src 10.8.0.0/16

>
> acl webservers dst 10.8.50.220/255.255.255.255
> 10.8.50.221/255.255.255.255 10.8.50.222/255.255.255.255
> 10.8.50.223/255.255.255.255

acl webservers dst 10.8.50.220 10.8.50.221 10.8.50.222 10.8.50.223

>
> acl nomsnurl dstdomain "/etc/squid/nomsn"
>
> acl manager proto cache_object
> acl localhost src 127.0.0.1/255.255.255.255

acl localhost src 127.0.0.1

> acl SSL_ports port 443 563 1863 6667 4430
> acl Safe_ports port 80 # http
> acl Safe_ports port 443 563 # https, snews
>
> acl auth proxy_auth REQUIRED
> acl noinet external ldap_group noinet
> acl linuxadmin external ldap_group linuxadmin
> acl nomsn external ldap_group nomsn
> acl dummy src 0.0.0.0/0.0.0.0

acl dummy src all

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE8 or 3.0.STABLE24
   Current Beta Squid 3.1.0.16