Re: [squid-users] Ignoring query string from url

nitesh naik wrote:
> Hi All,
>
> Issues was with Disk I/O. I have used null cache dir and squid
> response is much faster now.
>
> cache_dir null /empty
>
> Thanks everyone for your help.
>
> Regards
> Nitesh

Oh dear, I can't believe I overlooked this.
cache_dir aufs (linux) or diskd (FreeBSD) is likely to solve the disk
speed issues. Particularly if you don't use a blocking IOEngine (leaving
it unset is usually best).

Amos

>
> On Tue, Nov 4, 2008 at 9:40 AM, nitesh naik <niteshnaik_at_gmail.com> wrote:
>> Does these Redirector statistics mean url rewrite helper program is
>> slowing down squid response ? avg service time is 1550 msec.
>>
>> Redirector Statistics:
>> program: /home/zdn/bin/redirect_parallel.pl
>> number running: 2 of 2
>> requests sent: 1069753
>> replies received: 1069752
>> queue length: 0
>> avg service time: 1550 msec
>>
>>
>> # FD PID # Requests Flags Time Offset Request
>> 1 10 18237 12645 B 0.002 38 (none)
>> 2 15 18238 12335 2.144 0 (none)
>>
>> Regards
>> Nitesh
>>
>> On Mon, Nov 3, 2008 at 2:46 PM, nitesh naik <niteshnaik_at_gmail.com> wrote:
>>> Not sure if url rewrite helper is slowing down process because via
>>> cache manager interface it didn't show any connection back log. What
>>> information I should look for in cache manager to find out the cause
>>> of the slow serving of requests ?
>>>
>>> Redirector Statistics:
>>> program: /home/zdn/bin/redirect_parallel.pl
>>> number running: 2 of 2
>>> requests sent: 155697
>>> replies received: 155692
>>> queue length: 0
>>> avg service time: 0 msec
>>>
>>>
>>> # FD PID # Requests Flags Time Offset Request
>>> 1 8 21149 104125
>>> BW 0.033 38 http://s2.xyz.com/1821/78/570/1789/563/i88.js?z=4258
>>> 81.52.249.106/- - GET myip=10.0.0.165 myport=80\n
>>> 2 9 21150 51572 BW 0.039 0 http://s2.xyz.com/1813/2/570/1781/563/i7.js?z=8853
>>> 81.52.249.106/- - GET myip=10.0.0.165 myport=80\n
>>>
>>>
>>> Following are my squid settings.
>>>
>>> acl all src 0.0.0.0/0.0.0.0
>>> acl manager proto cache_object
>>> acl localhost src 127.0.0.1
>>> acl to_localhost dst 127.0.0.0/255.0.0.0
>>> acl localnet src 10.0.0.0/255.0.0.0
>>> acl SSL_ports port 443
>>> acl Safe_ports port 80 21 443 70 210 1025-65535 280 488 591 777
>>> acl CONNECT method CONNECT
>>> http_access Allow manager localhost
>>> http_access Deny manager
>>> http_access Deny !Safe_ports
>>> http_access Deny CONNECT !SSL_ports
>>> http_access Allow all
>>> http_access Allow localnet
>>> http_access Deny all
>>> icp_access Allow localnet
>>> icp_access Deny all
>>> htcp_access Allow localnet
>>> htcp_access Deny all
>>> htcp_clr_access Deny all
>>> ident_lookup_access Deny all
>>> http_port 0.0.0.0:80 defaultsite=s1.xyz.com vhost
>>> cache_peer 10.0.0.175 Parent 80 0 no-query round-robin originserver
>>> cache_peer 10.0.0.177 Parent 80 0 no-query round-robin originserver
>>> cache_peer 10.0.0.179 Parent 80 0 no-query round-robin originserver
>>> cache_peer 10.0.0.181 Parent 80 0 no-query round-robin originserver
>>> dead_peer_timeout 10 seconds
>>> hierarchy_stoplist cgi-bin
>>> hierarchy_stoplist ?
>>> cache_mem 0 bytes
>>> maximum_object_size_in_memory 1048576 bytes
>>> memory_replacement_policy lru
>>> cache_replacement_policy lru
>>> cache_dir ufs /home/zdn/squid/var/cache 6000 16 256 IOEngine=Blocking
>>> store_dir_select_algorithm least-load
>>> max_open_disk_fds 0
>>> minimum_object_size 0 bytes
>>> maximum_object_size 4194304 bytes
>>> cache_swap_low 90
>>> cache_swap_high 95
>>> logformat combined %>a %ui %un [%[tl] "%"rm %"ru HTTP/%">v" %Hs %<st
>>> "%"{Referer}>h" "%"{User-Agent}>h" %Ss:%Sh
>>> access_log /home/zdn/squid/var/logs/access.log squid
>>> cache_log /home/zdn/squid/var/logs/cache.log
>>> cache_store_log /home/zdn/squid/var/logs/store.log
>>> logfile_rotate 10
>>> emulate_httpd_log off
>>> log_ip_on_direct on
>>> mime_table /home/zdn/squid/etc/mime.conf
>>> log_mime_hdrs off
>>> pid_filename /home/zdn/squid/var/logs/squid.pid
>>> debug_options ALL,1
>>> log_fqdn off
>>> client_netmask 255.255.255.255
>>> strip_query_terms off
>>> buffered_logs off
>>> url_rewrite_program /home/zdn/bin/redirect_parallel.pl
>>> url_rewrite_children 2
>>> url_rewrite_concurrency 2000
>>> url_rewrite_host_header off
>>> url_rewrite_bypass off
>>> refresh_pattern ^ftp: 1440 20% 10080
>>>
>>> refresh_pattern ^gopher: 1440 0% 1440
>>>
>>> refresh_pattern (cgi-bin|\?) 0 0% 0
>>>
>>> refresh_pattern . 0 20% 4320
>>>
>>> quick_abort_min 16 KB
>>> quick_abort_max 16 KB
>>> quick_abort_pct 95
>>> read_ahead_gap 16384 bytes
>>> negative_ttl 0 seconds
>>> positive_dns_ttl 21600 seconds
>>> negative_dns_ttl 60 seconds
>>> range_offset_limit 0 bytes
>>> minimum_expiry_time 60 seconds
>>> store_avg_object_size 13 KB
>>> store_objects_per_bucket 20
>>> request_header_max_size 20480 bytes
>>> reply_header_max_size 20480 bytes
>>> request_body_max_size 0 bytes
>>> via off
>>> ie_refresh off
>>> vary_ignore_expire off
>>> request_entities off
>>> relaxed_header_parser on
>>> forward_timeout 240 seconds
>>> connect_timeout 10 seconds
>>> peer_connect_timeout 5 seconds
>>> read_timeout 120 seconds
>>> request_timeout 10 seconds
>>> persistent_request_timeout 120 seconds
>>> client_lifetime 86400 seconds
>>> half_closed_clients off
>>> pconn_timeout 60 seconds
>>> ident_timeout 10 seconds
>>> shutdown_lifetime 30 seconds
>>> cache_mgr webmaster
>>> mail_program mail
>>> cache_effective_user zdn
>>> httpd_suppress_version_string off
>>> umask 23
>>> announce_period 31536000 seconds
>>> announce_host tracker.ircache.net
>>> announce_port 3131
>>> client_persistent_connections off
>>> server_persistent_connections off
>>> persistent_connection_after_error off
>>> detect_broken_pconn off
>>> snmp_port 0
>>> snmp_access Deny all
>>> snmp_incoming_address 0.0.0.0
>>> snmp_outgoing_address 255.255.255.255
>>> icp_port 3130
>>> htcp_port 0
>>> log_icp_queries on
>>> udp_incoming_address 0.0.0.0
>>> udp_outgoing_address 255.255.255.255
>>> icp_hit_stale off
>>> minimum_direct_hops 4
>>> minimum_direct_rtt 400
>>> netdb_low 900
>>> netdb_high 1000
>>> netdb_ping_period 300 seconds
>>> query_icmp off
>>> test_reachability off
>>> icp_query_timeout 4000
>>> maximum_icp_query_timeout 2000
>>> minimum_icp_query_timeout 5
>>> background_ping_rate 10 seconds
>>> mcast_icp_query_timeout 2000
>>> icon_directory /home/zdn/squid/share/icons
>>> global_internal_static on
>>> short_icon_urls on
>>> error_directory /home/zdn/squid/share/errors/templates
>>> err_html_text
>>> email_err_data on
>>> nonhierarchical_direct on
>>> prefer_direct off
>>> incoming_icp_average 6
>>> incoming_http_average 4
>>> incoming_dns_average 4
>>> min_icp_poll_cnt 8
>>> min_dns_poll_cnt 8
>>> min_http_poll_cnt 8
>>> tcp_recv_bufsize 0 bytes
>>> check_hostnames off
>>> allow_underscore on
>>> dns_retransmit_interval 5 seconds
>>> dns_timeout 120 seconds
>>> dns_defnames off
>>> hosts_file /etc/hosts
>>> ignore_unknown_nameservers on
>>> ipcache_size 1024
>>> ipcache_low 90
>>> ipcache_high 95
>>> fqdncache_size 1024
>>> memory_pools on
>>> memory_pools_limit 5242880 bytes
>>> forwarded_for on
>>> cachemgr_passwd XXXXXXXXXX shutdown config reconfigure offline_toggle
>>> client_db on
>>> refresh_all_ims off
>>> reload_into_ims off
>>> maximum_single_addr_tries 1
>>> retry_on_error off
>>> as_whois_server whois.ra.net
>>> offline_mode off
>>> uri_whitespace strip
>>> coredump_dir /home/zdn/squid/var/cache
>>> balance_on_multiple_ip on
>>> pipeline_prefetch off
>>> high_response_time_warning 0
>>> high_page_fault_warning 0
>>> high_memory_warning 0 bytes
>>> sleep_after_fork 0
>>> windows_ipaddrchangemonitor on
>>>
>>> Regards
>>> Nitesh
>>>
>>> On Mon, Nov 3, 2008 at 10:27 AM, nitesh naik <niteshnaik_at_gmail.com> wrote:
>>>> Henrik / Amos,
>>>>
>>>> Tried using these setting and I could see see delay in serving the
>>>> requests even for cached objects.
>>>>
>>>> 1225687535.330 5459 81.52.249.101 TCP_MEM_HIT/200 1475 GET
>>>> http://abc.xyz.com/3613/172/500/248/211/i5.js?z=9059 - NONE/-
>>>> application/x-javascript
>>>> 1225687535.330 5614 81.52.249.100 TCP_MEM_HIT/200 8129 GET
>>>> http://abc.xyz.com/3357/172/211/4/1/i4.js?z=6079 - NONE/-
>>>> application/x-javascript
>>>> 1225687539.661 12327 168.143.241.12 TCP_MISS/200 2064 GET
>>>> http://bc.xyz.com/2333/254/496/158/122/i17.js?z=3473 -
>>>> ROUNDROBIN_PARENT/10.0.0.181 application/x-javascript
>>>>
>>>>
>>>> Following are timeouts that I have set.
>>>>
>>>> connect_timeout 10 seconds
>>>> peer_connect_timeout 5 seconds
>>>> read_timeout 2 minutes
>>>> request_timeout 10 seconds
>>>> icp_query_timeout 4000
>>>>
>>>> and cache peer settings.
>>>>
>>>> cache_peer 10.0.0.175 parent 80 0 no-query originserver round-robin
>>>> cache_peer 10.0.0.177 parent 80 0 no-query originserver round-robin
>>>> cache_peer 10.0.0.179 parent 80 0 no-query originserver round-robin
>>>> cache_peer 10.0.0.181 parent 80 0 no-query originserver round-robin
>>>>
>>>> Regards
>>>> Nitesh
>>>>
>>>>
>>>> On Sun, Nov 2, 2008 at 1:11 AM, Henrik Nordstrom
>>>> <henrik_at_henriknordstrom.net> wrote:
>>>>> On tor, 2008-10-30 at 19:50 +0530, nitesh naik wrote:
>>>>>
>>>>>> url rewrite helper script works fine for few requests ( 100 req/sec )
>>>>>> but slows down response as number of requests increase and it takes
>>>>>> 10+ second to deliver the objects.
>>>>> I'v run setups like this at more than thousand requests/s.
>>>>>
>>>>>> Is there way to optimise it further ?
>>>>>>
>>>>>> url_rewrite_program /home/zdn/bin/redirect_parallel.pl
>>>>>> url_rewrite_children 2000
>>>>>> url_rewrite_concurrency 5
>>>>> Those two should be the other way around.
>>>>>
>>>>> url_rewrite_concurrency 2000
>>>>> url_rewrite_children 2
>>>>>
>>>>> Regards
>>>>> Henrik
>>>>>

-- 
Please be using
   Current Stable Squid 2.7.STABLE5 or 3.0.STABLE10
   Current Beta Squid 3.1.0.1