Goranka Bjedov–talk about performance testing

This post is my best shot at explaining what I do, why I do it, and why I think it is the right thing to do. Performance testing is a category of testing that seems to evoke strong feelings in people: feelings of fear (Oh, my God, I have no idea what to do because performance testing is so hard!), feelings of inadequacy (We bought this tool that does every aspect of performance testing, we paid so much for it, and we are not getting anything done!), feelings of confusion (So, what the heck am I supposed to be doing again?), and I don’t think this is necessary.

Think of performance testing as another tool in your testing arsenal – something you will do when you need to. It explores several system qualities, that can be simplified to:

• Speed – does the system respond quickly enough
• Capacity – is the infrastructure sized adequately
• Scalability – can the system grow to handle future volumes
• Stability – does the system behave correctly under load

So, I do performance testing of a service when risk analysis indicates that failing in any of the above categories would be more costly to the company than performing the tests. (Which, if your name is Google and you care about your brand, happens with any service you launch.) Note that I am talking about services – I work almost exclusively with servers and spend no time worrying about client-side rendering/processing issues. While those are becoming increasingly more important, and have always been more complex than my work, I consider those to be a part of functionality tests, and they are designed, created and executed by functional testing teams.

Another interesting thing about performance testing is that you will never be able to be 100% “right” or 100% “done. Accept it, deal with it, and move on. Any system in existence today will depend on thousands of different parameters, and if I spent the time analyzing each one of them, understanding the relationships between each two or each three, graphing their impact curves, trying to non-dimensionalize them, I would still be testing my first service two years later. The thought of doing anything less filled me with horror (They cannot seriously expect me to provide meaningful performance results in less than a year, can they?) but I have since learned that I can provide at least 90% of meaningful information to my customers by applying only 10% of my total effort and time. And, 90% is more than enough for vast majority of problems.

So, here is what I really do – I create benchmarks. If I am lucky and have fantastic information about current usage patterns of a particular product (which I usually do), I will make sure this benchmark covers most operations that are top resource hogs (either per single use or cumulative). I’ll run this benchmark with different loads (number of virtual users) against a loosely controlled system (it would be nice to have 100 machines all to myself for every service we have, which I can use once a day or once a week, but that would be expensive and unrealistic) and investigate its behavior. Which transactions are taking the most time? Which transactions seem to get progressively worse with increasing load? Which transactions seem unstable (I cannot explain their behavior)? I call this exploratory performance testing, and I’ll repeat my tests until I am convinced I am observing real system behavior. While I am doing this, I make sure I am not getting biased by investigating the code. If I have questions, I ask programmers, but I know they are biased, and I will avoid getting biased myself!

Once I have my graphs (think, interesting transaction latencies and throughput vs. load here) I meet with the development team and discuss the findings. Usually, there is one or two things they know and have been working on, and a few more they were unaware of. Sometimes, they look over my benchmark and suggest changes (could you make the ratio 80:20, and not 50:50?) After this meeting, we create our final benchmark, I modify the performance testing scripts, and now this benchmark will run as often as possible, but hopefully at least once a night. And, here is the biggest value of this effort: if there is a code change that has impacted performance in an unacceptable way, you will find out about it the next day. Not a week or a month later (How many of us remember what we did in the last month? So, why expect our developers to do so?)

Here is why I think this is the right thing to do: I have seen more bad code developed as a result of premature performance optimizations – before the team even thought they had a problem! Please don’t do that. Develop your service in a clean, maintainable and extensible manner. Let me test it, and keep regression testing it. If we find we have a problem in a particular area, we can then address that problem easily – because our code is not obfuscated with performance optimization that have improved code paths that execute once a month by 5%.

I can usually do this in two – four weeks depending on the complexity of the project. Occasionally, we will find an issue that cannot be explained or understood with performance tests. At that point in time, we look under the hood. This is where performance profiling and performance modeling come in. And, both of those are considerably more complex than performance testing. Both great tools, but should be used only when the easy tool fails.

Tools, tools, tools… So, what do we use? I gave a presentation at Google Test Automation Conference in London on exactly this topic. I use open source tools. I discuss the reasons why in the presentation. In general, even if you have decided to go one of the other two routes (vendor tools or develop your own) check out what is available. You may find out that you will get a lot of information about your service using JMeter and spending some time playing around with it. Sure, you can also spend $500K and get similar information or you can spend two years developing “the next best performance testing tool ever,” but before you are certain free is not good enough, why would you want to?

Final word: monitor your services during performance tests. If you do not have service related monitoring developed and set up to be used during live operations, you do not need performance testing. If the risks of your service failing are not important enough that you would want to know about it *before* it happens, then you should not be wasting time or money on performance testing. I am incredibly lucky in this area – Google infrastructure is developed by a bunch of people who, if they had a meeting where the topic would be “How to make Goranka’s life easy?”, could not have done better. I love them – they make my job trivial. At a minimum, I monitor CPU, memory and I/O usage. I cannot see a case when you would want to do less, but you may want to do a lot more on occasion.

开源的文本比较工具-textdiff

简单分享之：http://www.angusj.com/delphi/textdiff.html

Regular expression

工作中参数化经常用到，随便找了一个做备查，故贴之： http://www.phpe.net/articles/151.shtml

成功安装Loadrunner

这次安装正版的Loadrunner，还有点不太适应：）而且使用的是电子license，需要惠普进行配合生成永久的key，所以步骤要繁琐一些。

这次我采用的是contoller（Full install）+ remote load generator的结构，为了能尝试一下在局域网内，可以互相通信，进行分布式压力测试。

1.增加远程的load generator需要将远程的机器名完整的写进contoller，以便它可以在局域网中寻找。

2.将远程的load generator service 打开，并且保证开启agent的终端服务，这样才可以两端进行通信

3.如果没有防火墙的话，防火墙选项不要选择

今天开始装loadrunner

今天开始装正版的loadrunner，说实话我真的不喜欢这个所谓的市场占有率第一的工具。。。。不知道为什么，虽然试用下来是很强–把许多方面的功能都集成于一体，但安装以及设置的繁琐，使得我渐渐失去些耐心。

而且对电子许可还不是很熟悉，客服的电话经常打不通，上海的客服热线居然是空号！！

不知道是怎么了，难道被收购的企业都会走向没落。。。。。。。

Google performance testing also using open source!

网上关于Google公司进行性能测试的心得的视频是Goranka Bjedov在伦敦一次论坛中录下的，不过那么好的视频实在是声音太小，所以效果大打折扣。。。。。

不过今天在InfoQ上看到了她关于性能测试的帖子，仔细进行拜读，发现很多共鸣（哈哈），很高兴能和这位大师on the same page，lol

现在把他的文章概括如下：

what performance engineer usually feel:

1. fear :when the performance job is coming in

2.inadequacy: when we are not getting anything done

3.confusion:what should i do next

what performance testing should contain:

Speed- does the system respond quickly enough

capacity-is the infrustructure sized adequately

scalability-can the system grow to handle future volumes

stability-does the system behave correctly under load/for a long time

to do performance as a service:

she is more focus on service side and no worry about the client side behavior

she think that should be function QA’s work（this is a big difference between steven sounders and Bjedov or between Yahoo! and Google）

Can not do everything accurately, just 90% important information by applying 10% of total effort and time

There is no fixed answer on your performance, and we can not wait for everything to be clear, or spend long time to find what’s the relationship between each two or each three factors

Create benchmarks:

Got current usage pattern or user modeling of application,ideally

run the benchmark with different loads and investigate its behavior, including:

– Which transactions are taking most of time? (losing index)

– Which transactions seem to get progessively worse with increasing load?(create meeting, list page)

– Which transactions seem unstable (create meeting using same account)

– repeating testing until convinced I am observing real system behavior(consistently up?)

Create new benchmark with dev and run the benchmark every day

so we can find out the problem as soon as possible, ideally the next day

we can easily find a code change that has impacted performance in an unacceptable way

When find an issue that can not be explained or understand with performance tests

here comes the performance profiling and tuning activity

(you should learn to do isolation work, step by step)

Open source tools

Jmeter,Google is using， Me too：）

Monitoring your service during performance testing(Minimun SET)

CPU%

Memory/GC%/Heap graph

I/O utilization

Network utilization, eliminate your network factor as much as you can. It can make things easier.

In addition:

Exploratory performance testing in the future

mike kelly:They open a browser while a test is running and they see what the user experience is for them while the application is under load.

cygwin tool for statistics analysis

cygwin是我在进行性能测试结果分析或者是日志分析的时候用到的。Levi兄帮助我用这个工具写了一些脚本，便于我对大数据量的统计处理。

Loadrunner learning–4

找到了有关Loadrunner的系统流程图，对于理解Loadrunner整个的操作过程还是很有帮助的，故贴之

—–from http://www.wilsonmar.com/1loadrun.htm

Forward—performance testing Goal(simpler edtion)

找到这篇文章，感觉和我目前的做法大体是一致的，所以就不需要我再写了，省去了很多油墨：）

The goal of performance testing is not finding bugs, but to remove the bottlenecks from the application and improve the efficiency.

Before doing a performance testing we basically need to know the following points

1. Expected no of concurrent users or HTTP connections with your application
2. Acceptable response time for your pages

For performance tuning basically we have two approach.

In Approach1(white-box), we can do the following,

Code Analysis, We can search for poor algorithms or looping which is the reason for inefficiency.
Database Analysis, We can use query optimizers and profilers to optimize the database.
Hardware & Network, We can use utilities such as top, iostat to monitor hardware resources and ntop, netstat to monitor the network and Sockets.

In Approach2(black-box), for a Web application, testers will use tools that simulate concurrent users/HTTP connections and measure the response times automatically. If the response time does not meet your expectations tuning has to be done at application/hardware/database level.

In Tuning,

First we need to enhance the application code efficiency, then we can optimize the database.

If still your application doesn’t meet your requirements then the following steps will help you.

1. Using cache mechanisms.
2. Publish highly requested pages statically, so that they don’t hit the database.
3. Scaling Web servers horizontally via load balancing.
4. Scaling database servers horizontally and split them into read/write servers and read-only servers.
5. Scale the servers vertically by adding more hardware resources (CPU,RAM)

Points to remember,

We should take care such that one variable is modified at a time and redo the measurements.

Functionally the application should be well tested and must be in good quality. i.e., the software under test is already stable enough so that performance testing process can proceed smoothly.

playing poker when you do estimation

今天看到scrum项目中一个很有趣的流程，就是玩扑克：）

当项目需要估计所需时间的时候往往会产生分歧或者成员相互间会彼此影响，假若采用扑克牌，将成员之间彼此分割开来，大家在估计的时候互不影响，只是在最后亮出自己的底牌（分数越高表示估计的时间越长），成员底牌相差最大的两个人，也就是分歧最大的两个人需要阐述自己的出牌的原因，这样做既可以了解不同成员对相同项目的理解的深度，也能较为公平的进行项目度量，不至于过于乐观或者悲观。

这种扑克需要特制，但也可以网上免费得到：）很有意思的一种项目管理方式哦