Re: [squid-users] Ignoring query string from url

Hi All,

Issues was with Disk I/O. I have used null cache dir and squid
response is much faster now.

 cache_dir null /empty

Thanks everyone for your help.

Regards
Nitesh

On Tue, Nov 4, 2008 at 9:40 AM, nitesh naik <niteshnaik_at_gmail.com> wrote:
> Does these Redirector statistics mean url rewrite helper program is
> slowing down squid response ? avg service time is 1550 msec.
>
> Redirector Statistics:
> program: /home/zdn/bin/redirect_parallel.pl
> number running: 2 of 2
> requests sent: 1069753
> replies received: 1069752
> queue length: 0
> avg service time: 1550 msec
>
>
> # FD PID # Requests Flags Time Offset Request
> 1 10 18237 12645 B 0.002 38 (none)
> 2 15 18238 12335 2.144 0 (none)
>
> Regards
> Nitesh
>
> On Mon, Nov 3, 2008 at 2:46 PM, nitesh naik <niteshnaik_at_gmail.com> wrote:
>> Not sure if url rewrite helper is slowing down process because via
>> cache manager interface it didn't show any connection back log. What
>> information I should look for in cache manager to find out the cause
>> of the slow serving of requests ?
>>
>> Redirector Statistics:
>> program: /home/zdn/bin/redirect_parallel.pl
>> number running: 2 of 2
>> requests sent: 155697
>> replies received: 155692
>> queue length: 0
>> avg service time: 0 msec
>>
>>
>> # FD PID # Requests Flags Time Offset Request
>> 1 8 21149 104125
>> BW 0.033 38 http://s2.xyz.com/1821/78/570/1789/563/i88.js?z=4258
>> 81.52.249.106/- - GET myip=10.0.0.165 myport=80\n
>> 2 9 21150 51572 BW 0.039 0 http://s2.xyz.com/1813/2/570/1781/563/i7.js?z=8853
>> 81.52.249.106/- - GET myip=10.0.0.165 myport=80\n
>>
>>
>> Following are my squid settings.
>>
>> acl all src 0.0.0.0/0.0.0.0
>> acl manager proto cache_object
>> acl localhost src 127.0.0.1
>> acl to_localhost dst 127.0.0.0/255.0.0.0
>> acl localnet src 10.0.0.0/255.0.0.0
>> acl SSL_ports port 443
>> acl Safe_ports port 80 21 443 70 210 1025-65535 280 488 591 777
>> acl CONNECT method CONNECT
>> http_access Allow manager localhost
>> http_access Deny manager
>> http_access Deny !Safe_ports
>> http_access Deny CONNECT !SSL_ports
>> http_access Allow all
>> http_access Allow localnet
>> http_access Deny all
>> icp_access Allow localnet
>> icp_access Deny all
>> htcp_access Allow localnet
>> htcp_access Deny all
>> htcp_clr_access Deny all
>> ident_lookup_access Deny all
>> http_port 0.0.0.0:80 defaultsite=s1.xyz.com vhost
>> cache_peer 10.0.0.175 Parent 80 0 no-query round-robin originserver
>> cache_peer 10.0.0.177 Parent 80 0 no-query round-robin originserver
>> cache_peer 10.0.0.179 Parent 80 0 no-query round-robin originserver
>> cache_peer 10.0.0.181 Parent 80 0 no-query round-robin originserver
>> dead_peer_timeout 10 seconds
>> hierarchy_stoplist cgi-bin
>> hierarchy_stoplist ?
>> cache_mem 0 bytes
>> maximum_object_size_in_memory 1048576 bytes
>> memory_replacement_policy lru
>> cache_replacement_policy lru
>> cache_dir ufs /home/zdn/squid/var/cache 6000 16 256 IOEngine=Blocking
>> store_dir_select_algorithm least-load
>> max_open_disk_fds 0
>> minimum_object_size 0 bytes
>> maximum_object_size 4194304 bytes
>> cache_swap_low 90
>> cache_swap_high 95
>> logformat combined %>a %ui %un [%[tl] "%"rm %"ru HTTP/%">v" %Hs %<st
>> "%"{Referer}>h" "%"{User-Agent}>h" %Ss:%Sh
>> access_log /home/zdn/squid/var/logs/access.log squid
>> cache_log /home/zdn/squid/var/logs/cache.log
>> cache_store_log /home/zdn/squid/var/logs/store.log
>> logfile_rotate 10
>> emulate_httpd_log off
>> log_ip_on_direct on
>> mime_table /home/zdn/squid/etc/mime.conf
>> log_mime_hdrs off
>> pid_filename /home/zdn/squid/var/logs/squid.pid
>> debug_options ALL,1
>> log_fqdn off
>> client_netmask 255.255.255.255
>> strip_query_terms off
>> buffered_logs off
>> url_rewrite_program /home/zdn/bin/redirect_parallel.pl
>> url_rewrite_children 2
>> url_rewrite_concurrency 2000
>> url_rewrite_host_header off
>> url_rewrite_bypass off
>> refresh_pattern ^ftp: 1440 20% 10080
>>
>> refresh_pattern ^gopher: 1440 0% 1440
>>
>> refresh_pattern (cgi-bin|\?) 0 0% 0
>>
>> refresh_pattern . 0 20% 4320
>>
>> quick_abort_min 16 KB
>> quick_abort_max 16 KB
>> quick_abort_pct 95
>> read_ahead_gap 16384 bytes
>> negative_ttl 0 seconds
>> positive_dns_ttl 21600 seconds
>> negative_dns_ttl 60 seconds
>> range_offset_limit 0 bytes
>> minimum_expiry_time 60 seconds
>> store_avg_object_size 13 KB
>> store_objects_per_bucket 20
>> request_header_max_size 20480 bytes
>> reply_header_max_size 20480 bytes
>> request_body_max_size 0 bytes
>> via off
>> ie_refresh off
>> vary_ignore_expire off
>> request_entities off
>> relaxed_header_parser on
>> forward_timeout 240 seconds
>> connect_timeout 10 seconds
>> peer_connect_timeout 5 seconds
>> read_timeout 120 seconds
>> request_timeout 10 seconds
>> persistent_request_timeout 120 seconds
>> client_lifetime 86400 seconds
>> half_closed_clients off
>> pconn_timeout 60 seconds
>> ident_timeout 10 seconds
>> shutdown_lifetime 30 seconds
>> cache_mgr webmaster
>> mail_program mail
>> cache_effective_user zdn
>> httpd_suppress_version_string off
>> umask 23
>> announce_period 31536000 seconds
>> announce_host tracker.ircache.net
>> announce_port 3131
>> client_persistent_connections off
>> server_persistent_connections off
>> persistent_connection_after_error off
>> detect_broken_pconn off
>> snmp_port 0
>> snmp_access Deny all
>> snmp_incoming_address 0.0.0.0
>> snmp_outgoing_address 255.255.255.255
>> icp_port 3130
>> htcp_port 0
>> log_icp_queries on
>> udp_incoming_address 0.0.0.0
>> udp_outgoing_address 255.255.255.255
>> icp_hit_stale off
>> minimum_direct_hops 4
>> minimum_direct_rtt 400
>> netdb_low 900
>> netdb_high 1000
>> netdb_ping_period 300 seconds
>> query_icmp off
>> test_reachability off
>> icp_query_timeout 4000
>> maximum_icp_query_timeout 2000
>> minimum_icp_query_timeout 5
>> background_ping_rate 10 seconds
>> mcast_icp_query_timeout 2000
>> icon_directory /home/zdn/squid/share/icons
>> global_internal_static on
>> short_icon_urls on
>> error_directory /home/zdn/squid/share/errors/templates
>> err_html_text
>> email_err_data on
>> nonhierarchical_direct on
>> prefer_direct off
>> incoming_icp_average 6
>> incoming_http_average 4
>> incoming_dns_average 4
>> min_icp_poll_cnt 8
>> min_dns_poll_cnt 8
>> min_http_poll_cnt 8
>> tcp_recv_bufsize 0 bytes
>> check_hostnames off
>> allow_underscore on
>> dns_retransmit_interval 5 seconds
>> dns_timeout 120 seconds
>> dns_defnames off
>> hosts_file /etc/hosts
>> ignore_unknown_nameservers on
>> ipcache_size 1024
>> ipcache_low 90
>> ipcache_high 95
>> fqdncache_size 1024
>> memory_pools on
>> memory_pools_limit 5242880 bytes
>> forwarded_for on
>> cachemgr_passwd XXXXXXXXXX shutdown config reconfigure offline_toggle
>> client_db on
>> refresh_all_ims off
>> reload_into_ims off
>> maximum_single_addr_tries 1
>> retry_on_error off
>> as_whois_server whois.ra.net
>> offline_mode off
>> uri_whitespace strip
>> coredump_dir /home/zdn/squid/var/cache
>> balance_on_multiple_ip on
>> pipeline_prefetch off
>> high_response_time_warning 0
>> high_page_fault_warning 0
>> high_memory_warning 0 bytes
>> sleep_after_fork 0
>> windows_ipaddrchangemonitor on
>>
>> Regards
>> Nitesh
>>
>> On Mon, Nov 3, 2008 at 10:27 AM, nitesh naik <niteshnaik_at_gmail.com> wrote:
>>> Henrik / Amos,
>>>
>>> Tried using these setting and I could see see delay in serving the
>>> requests even for cached objects.
>>>
>>> 1225687535.330 5459 81.52.249.101 TCP_MEM_HIT/200 1475 GET
>>> http://abc.xyz.com/3613/172/500/248/211/i5.js?z=9059 - NONE/-
>>> application/x-javascript
>>> 1225687535.330 5614 81.52.249.100 TCP_MEM_HIT/200 8129 GET
>>> http://abc.xyz.com/3357/172/211/4/1/i4.js?z=6079 - NONE/-
>>> application/x-javascript
>>> 1225687539.661 12327 168.143.241.12 TCP_MISS/200 2064 GET
>>> http://bc.xyz.com/2333/254/496/158/122/i17.js?z=3473 -
>>> ROUNDROBIN_PARENT/10.0.0.181 application/x-javascript
>>>
>>>
>>> Following are timeouts that I have set.
>>>
>>> connect_timeout 10 seconds
>>> peer_connect_timeout 5 seconds
>>> read_timeout 2 minutes
>>> request_timeout 10 seconds
>>> icp_query_timeout 4000
>>>
>>> and cache peer settings.
>>>
>>> cache_peer 10.0.0.175 parent 80 0 no-query originserver round-robin
>>> cache_peer 10.0.0.177 parent 80 0 no-query originserver round-robin
>>> cache_peer 10.0.0.179 parent 80 0 no-query originserver round-robin
>>> cache_peer 10.0.0.181 parent 80 0 no-query originserver round-robin
>>>
>>> Regards
>>> Nitesh
>>>
>>>
>>> On Sun, Nov 2, 2008 at 1:11 AM, Henrik Nordstrom
>>> <henrik_at_henriknordstrom.net> wrote:
>>>> On tor, 2008-10-30 at 19:50 +0530, nitesh naik wrote:
>>>>
>>>>> url rewrite helper script works fine for few requests ( 100 req/sec )
>>>>> but slows down response as number of requests increase and it takes
>>>>> 10+ second to deliver the objects.
>>>>
>>>> I'v run setups like this at more than thousand requests/s.
>>>>
>>>>> Is there way to optimise it further ?
>>>>>
>>>>> url_rewrite_program /home/zdn/bin/redirect_parallel.pl
>>>>> url_rewrite_children 2000
>>>>> url_rewrite_concurrency 5
>>>>
>>>> Those two should be the other way around.
>>>>
>>>> url_rewrite_concurrency 2000
>>>> url_rewrite_children 2
>>>>
>>>> Regards
>>>> Henrik
>>>>
>>>
>>
>
Received on Tue Nov 04 2008 - 07:29:34 MST