Re: [squid-users] gaining the optimal configuration of squid by using a benchmarker

From: Marcos Dutra <macdutra_at_gmail.com>
Date: Tue, 12 Aug 2008 09:34:07 -0300

Hi Dadashi, well I look for webstone and I think this will test only
connection for apache server for example, for this test my suggestion
is apache's ab.
Well about polygraph, for begginers is very dificult really, but is
powerful for squid benchmark.
Look this schema for polygraph.

1 - workload

Bellow workload for simple test. You should modify parameters
max_host_load and max_agent_load. In my tests I use loopback interface
to running in same server.

        Bench benchPolyMix3 = {
                peak_req_rate = undef();
                client_side = {
                        max_host_load = 100/sec;
                        max_agent_load = 0.4/sec;
                        addr_mask = '10.1.0.0';
                };
                server_side = {
                        max_host_load = client_side.max_host_load;
                        max_agent_load = client_side.max_agent_load;
                        addr_mask = '10.1.0.0:80';
                };
        };

        Bench benchWebAxe4 = {
                client_side = {
                        max_host_load = 100/sec;
                        max_agent_load = 0.15/sec;
                        addr_space = ['lo::10.1.0-123.1-250/18'];
                };
                server_side = {
                        max_host_load = client_side.max_host_load;
                        max_agent_load = max_host_load;
                };
        };

        Bench benchPolyMix4 = {
                client_side = {
                        max_host_load = 100/sec;
                        max_agent_load = 0.15/sec;
                        addr_space = ['lo::10.1.0-123.1-250/18'];
                };

                server_side = {
                        max_host_load = client_side.max_host_load;
                        max_agent_load = undef();
                        addr_space = ['lo::10.1.0-123.1-250:80/18'];
                };
        };

        PolyMix3As asPolyMix3;

        PolyMix4As asPolyMix4 = {
                agents_per_addr = 1;
        };

        WebAxe4As asWebAxe4 = {
                agents_per_addr = 2;
        };

        SrvLb4As asNoLb_L4_4 = {
                agents_per_addr = 1;
        };

        SrvLb4As asSrvLb_L4_4 = {
                agents_per_addr = 1;
        };

        SrvLb4As asSrvLb_L7_4 = {
                agents_per_addr = 1;
        };

        Bench TheBench = benchPolyMix4;

        TheBench = {
                client_side = {
                        addr_space = ['lo::10.1.0-123.1-250/18'];
                        hosts = ['127.0.0.1'];
                };
                server_side = {
                        addr_space = ['lo::10.1.0-123.1-250:80/18'];
                        hosts = ['127.0.0.1'];
                };
        };

        TheBench.peak_req_rate = 100/sec;

        rate FillRate = (95%*TheBench.peak_req_rate);

        size ProxyCacheSize = 12GB;

        DnsResolver Resolver;

        Resolver.servers = ['localhost:53'];

        ObjLifeCycle olcStatic = {
                length = const(2year);
                variance = 0%;
                with_lmt = 100%;
                expires = [nmt+const(0sec)];
        };

        ObjLifeCycle olcHTML = {
                length = logn(7day, 1day);
                variance = 33%;
                with_lmt = 100%;
                expires = [nmt+const(0sec)];
        };

        ObjLifeCycle olcImage = {
                length = logn(30day, 7day);
                variance = 50%;
                with_lmt = 100%;
                expires = [nmt+const(0sec)];
        };

        ObjLifeCycle olcDownload = {
                length = logn(0.5year, 30day);
                variance = 33%;
                with_lmt = 100%;
                expires = [nmt+const(0sec)];
        };

        ObjLifeCycle olcOther = {
                length = unif(1day, 1year);
                variance = 50%;
                with_lmt = 100%;
                expires = [nmt+const(0sec)];
        };

        Content cntPolyMix_1 = {
                kind = "polymix-1";
                mime = {

                mime = {
                        type = undef();
                        extensions = [];
                };
                size = exp(13KB);
                obj_life_cycle = olcStatic;
                cachable = 80%;
        };

        Content cntImage = {
                kind = "image";
                mime = {
                        type = undef();
                        extensions = [".gif",".jpeg",".png"];
                };
                obj_life_cycle = olcImage;
                size = exp(4.5KB);
                cachable = 80%;
                checksum = 1%;
        };

        Content cntHTML = {
                kind = "HTML";
                mime = {
                        type = undef();
                        extensions = [".html": 60%,".htm"];
                };
                obj_life_cycle = olcHTML;
                size = exp(8.5KB);
                cachable = 90%;
                checksum = 1%;

                may_contain = [cntImage];
                embedded_obj_cnt = zipf(13);
        };

        Content cntDownload = {
                kind = "download";
                mime = {
                        type = undef();
                        extensions = [".exe": 40%,".zip",".gz"];
                };
                obj_life_cycle = olcDownload;
                size = logn(300KB, 300KB);
                cachable = 95%;
                checksum = 0.01%;
        };

        Content cntOther = {
                kind = "other";
                obj_life_cycle = olcOther;
                size = logn(25KB, 10KB);
                cachable = 72%;
                checksum = 0.1%;
        };

        Server S = {
                kind = "PolyMix-4-srv";
                contents = [cntImage: 65%, cntHTML: 15%, cntDownload:
0.5%, cntOther];
                direct_access = [cntHTML, cntDownload, cntOther];
                xact_think = norm(2.5sec, 1sec);
                pconn_use_lmt = zipf(16);
                idle_pconn_tout = 15sec;
                abort_prob = 0.1%;
                addresses = serverAddrs(asPolyMix4, TheBench);
                http_versions = ["1.0"];
        };

        AddrMap Map = {
                zone = "bench.tst";
                addresses = S.addresses;
                names = ipsToNames(addresses, zone);
        };

        PopModel popModel = {
                pop_distr = popUnif();
                hot_set_frac = 1%;
                hot_set_prob = 10%;
                bhr_discrimination = 90%;
        };

        Resolver.timeout = 5sec;

        float HitIfRepeat = 80%;

        Robot R = {
                kind = "PolyMix-4-rbt";
                recurrence = 55%/HitIfRepeat;
                embed_recur = 100%;
                interests = ["public": 75%,"private"];
                pop_model = popModel;
                req_types = ["basic","Ims200": 5%,"Ims304": 10%,"Reload": 5%];
                req_methods = ["GET","POST": 1.5%,"HEAD": 0.1%];
                abort_prob = 0.1%;
                req_rate = TheBench.client_side.max_agent_load;
                pconn_use_lmt = zipf(64);
                open_conn_lmt = 2;
                dns_resolver = Resolver;
                origins = Map.names;
                addresses = robotAddrs(asPolyMix4, TheBench);
                http_versions = ["1.0"];
        };

        time platDur = 1hour;

        time rampDur = platDur/12;

        int clientHostCount = clientHostCount(TheBench);

        float fillable_ratio = HitIfRepeat;

        rate peak_fill_rate = TheBench.peak_req_rate*0.45*fillable_ratio;

        int wsc = int(peak_fill_rate*platDur);

        working_set_cap(wsc/clientHostCount);

        float smallFactor = 1.0/count(R.addresses);

        Phase phFRamp = {
                name = "framp";
                goal.duration = rampDur;
                recur_factor_beg = 5%/55%;
                special_msg_factor_beg = 0.1;
                populus_factor_beg = smallFactor;
                populus_factor_end = FillRate/TheBench.peak_req_rate;
        };

        Phase phFill = {
                name = "fill";
                goal.fill_size = 2*ProxyCacheSize/clientHostCount;
                wait_wss_freeze = yes;
        };

        Phase phFExit = {
                name = "fexit";
                goal.duration = rampDur;
                recur_factor_end = 1;
                special_msg_factor_end = 1;
                populus_factor_end = smallFactor;
        };

        Phase phInc1 = {
                name = "inc1";
                goal.duration = rampDur;
                populus_factor_end = 1;
        };

        Phase phTop1 = {
                name = "top1";
                goal.duration = platDur;
        };

        Phase phDec1 = {
                name = "dec1";
                goal.duration = rampDur;
                populus_factor_end = 0.1;
        };

        Phase phIdle = {
                name = "idle";
                goal.duration = rampDur;
        };

        Phase phInc2 = {
                name = "inc2";
                goal.duration = rampDur;
                populus_factor_end = 1.0;
        };

        Phase phTop2 = {
                name = "top2";
                goal.duration = platDur;
        };

        Phase phDec2 = {
                name = "dec2";
                goal.duration = rampDur;
                populus_factor_end = 0;
                synchronize = false;
        };
        StatsSample topSample = {
                capacity = 10000;
        };

        StatsSample fillSample = {
                capacity = int(10%*topSample.capacity);
        };

        phFill.stats_samples = [fillSample];

        phTop1.stats_samples = [topSample];

        phTop2.stats_samples = [topSample];

        schedule(phFRamp, phFill, phFExit, phInc1, phTop1, phDec1,
phIdle, phInc2, phTop2, phDec2);

        use(S, R);

        use(Map);

        use(TheBench);

2 - Running dns_config
./dns_config --config workload.pg

3 - Put the configuration in your dns

4 - running polysrv
After running this app. if you running ifconfig you see various ip in
lo interface.

./polysrv --config /usr/local/polygraph/workloads/workload.pg --log
/var/logs/server.log

5 - running polyclt

 ./polyclt --config /usr/local/polygraph/workloads/marcos.pg --proxy
192.168.0.1:3128 --log /var/logs/client.log

6 - running reporter
You may be stop process server and client of polygraph to make a report.
 /usr/local/polygraph/bin/reporter --report_dir /var/www/html/proxy
--label "Proxy ACL" /var/logs/*.log

When you running polygraph you should look mgr info of squid and
analyzing the connections of your squid server.

I hope this informations help you.

Regards,
Marcos

2008/8/12 Stephen Ryan <stephen.ryan_at_corp.aol.com>:
> t.dadashi_at_gmail.com wrote:
>>
>> thanks for your reply,
>> i have a very restricted time,
>> befor i tried to use web polygraph but the
>> commands in the site's document were unknown for
>> the shell,
>> and i really dont know what to do,
>> then i tried webstone :
>> http://www.mindcraft.com/webstone/faq.html
>> (about 11 pages manual against 200 for polygraph!)
>> i think this one is easier but there are some problems
>> in setting it up again!
>> can you please help me on this???
>>
>>
>
> Something as simple as apache_benchmark? You could have some fancy grep and
> awk around it to examine the results, and a shell script around the whole
> thing to run it for you.
>
Received on Tue Aug 12 2008 - 12:34:10 MDT

This archive was generated by hypermail 2.2.0 : Tue Aug 12 2008 - 12:00:03 MDT