elasticsearch性能测试工具rally深入详解

本文发布时间: 2019-Mar-22
题记elasticsearch性能测试研究了很久,自己想过通过批量导入数据,然后记录时间,统计CPU、内存等变化,计算得出某个性能指标。但显然,数据量起伏不定,非常不准确。研究发现,github上提供了rally作为elasticsearch的性能测试工具,较好的解决了es性能测试问题。1、esrally功能:es的性能测试工具。esrally不支持windows版本,目前只支持Linux和Mac OS。esrally的运行基于以下两点假设:1)所有运行在同一台机器完成。(未来改假设条件可能会去掉)2)你需要向Elasticsearch索引中添加特定的数据集,然后在其上进行基准查询(benchmarking queries).2、esrally安装:前提:验证是否安装了以下内容(注意版本)Python 3.4+ available as python3 on the path (verify with: python3 –version which should print Python 3.4.0 or higher)pip3 available on the path (verify with pip3 –version)JDK 8git如下所示:[root@laoyang git-2.4.0]# pythonPython 3.5.1 (default, Aug 2 2016, 09:53:20)[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linuxType 'help', 'copyright', 'credits' or 'license' for more information.[root@laoyang git-2.4.0]# git --versiongit version 2.4.0[root@laoyang git-2.4.0]# java -versionjava version '1.8.0_91'Java(TM) SE Runtime Environment (build 1.8.0_91-b14)Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)[root@laoyang git-2.4.0]# find / -name 'pip3'/usr/bin/pip3/usr/local/python3/bin/pip3步骤1:安装 Rallypip3 install esrally步骤2:配置 Rally调用: esrally configure 会提示让你输入 java8.home(也就是JAVA_HOME设定的环境变量路径)。仅java8.home使用自己本机的JAVA_HOME环境变量路径,其他采用默认值即可。[root@laoyang .rally]# cat /root/.rally/rally.ini[meta]config.version = 5[system]root.dir = /root/.rally/benchmarkslog.root.dir = logsenv.name = local[source]local.src.dir=/root/.rally/src #注意,此路径并非elaticsearch安装路径。remote.repo.url=https://github.com/elastic/elasticsearch.git[provisioning]local.install.dir = install[runtime]java8.home = /opt/jdk1.8.0_91 #注意,JAVA_HOME 路径为:/opt/jdk1.8.0_91[benchmarks]local.dataset.cache = ${system:root.dir}/data[reporting]datastore.type = elaticsearchdatastore.host = 10.0.1.30datastore.port = 9200datastore.secure = Falsedatastore.user =datastore.password =[tracks]default.url = https://github.com/elastic/rally-tracks步骤3:运行esrally由于elasticsearch的运行必须非root账户。esrally建议用非root账户执行。3、esrally执行常见问题及解决方案问题1:版本用哪个版本?目前用最新版本 Rally 0.3.2(2016-8-7发行)。问题2:注意gradle需要2.13版本,高版本会不兼容。[root@laoyang caches]# ll /usr/bin/gradlelrwxrwxrwx 1 root root 27 Aug 5 17:13 /usr/bin/gradle -> /opt/gradle-2.13/bin/gradle问题3:安装时候提示 documents.json.bz2 无法下载成功。1)手动下载: http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames/documents.json.bz22)移动到 /root/.rally/benchmarks/data/geonames/(对于root用户)对于非root用户,elaticsearch用户,移动路径为:/home/elasticsearch/benchmarks/data/geonames/问题4: esrally单独运行和加参数运行区别?esrally单独运行:非常方便的指令,等价于 :esrally –pipeline=from-sources –version=current. Rally 将要通过Gradle从源码编译elasticsearch。esrally –pipline=from-distribution –distribution-version=2.3.4, Rally将要从 https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.3.4/elasticsearch-2.3.4.tar.gz下载elaticsearch4.3.4版本运行。问题5: Could not resolve all dependencies for configuration ‘:benchmarks:classpath’.37 > Could not download shadow.jar (com.github.jengelman.gradle.plugins:shadow:1.2.3)如何解决?root账户下:cd /home/elasticsearch/.rally/benchmarks/srcgradle :distribution:tar:assemble正确提示信息为:deprecated API.Note: Recompile with -Xlint:deprecation for details.:modules:transport-netty4:processResources UP-TO-DATE:modules:transport-netty4:classes:modules:transport-netty4:jar:modules:transport-netty4:copyPluginPropertiesTemplate:modules:transport-netty4:pluginProperties:modules:transport-netty4:bundlePlugin:distribution:buildModules:distribution:tar:buildTar:distribution:tar:generatePomFileForNebulaPublication:distribution:tar:signArchives SKIPPED:distribution:tar:assembleBUILD SUCCESSFUL4、esrally最终运行结果:Preparing for race (might take a few moments) ... Building from sources .../home/elasticsearch/.rally/benchmarks/races/2016-08-08-07-29-59/local/logs/rally_out.logRacing on track [geonames] and challenge [append-no-conflicts] with car [defaults] Benchmarking indexing at 12183.1 docs/s [100% done] Benchmarking stats (warmup iteration 100/100) [100% done] Benchmarking stats (iteration 100/100) [100% done] Benchmarking search (warmup iteration 100/100) [100% done] Benchmarking search (iteration 100/100) [100% done]------------------------------------------------------ _______ __ _____ / ____(_)___ ____ _/ / / ___/_________ ________ / /_ / / __ / __ `/ / __ / ___/ __ / ___/ _ / __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __//_/ /_/_/ /_/__,_/_/ /____/___/____/_/ ___/------------------------------------------------------| Metric | Value ||----------------------------------------------------------:|----------:|| Min Indexing Throughput [docs/s] | 11949 || Median Indexing Throughput [docs/s] | 12307 || Max Indexing Throughput [docs/s] | 13349 || Indexing time [min] | 164.045 || Merge time [min] | 32.3815 || Refresh time [min] | 8.82333 || Flush time [min] | 1.63852 || Merge throttle time [min] | 1.45482 || Query latency default (90.0 percentile) [ms] | 68.8676 || Query latency default (99.0 percentile) [ms] | 77.6009 || Query latency default (100 percentile) [ms] | 78.8328 || Query latency term (90.0 percentile) [ms] | 4.63227 || Query latency term (99.0 percentile) [ms] | 10.9506 || Query latency term (100 percentile) [ms] | 13.0573 || Query latency phrase (90.0 percentile) [ms] | 6.35018 || Query latency phrase (99.0 percentile) [ms] | 13.1745 || Query latency phrase (100 percentile) [ms] | 14.2399 || Query latency country_agg_uncached (90.0 percentile) [ms] | 361.913 || Query latency country_agg_uncached (99.0 percentile) [ms] | 384.75 || Query latency country_agg_uncached (100 percentile) [ms] | 392.645 || Query latency country_agg_cached (90.0 percentile) [ms] | 7.71451 || Query latency country_agg_cached (99.0 percentile) [ms] | 17.4704 || Query latency country_agg_cached (100 percentile) [ms] | 18.4851 || Query latency scroll (90.0 percentile) [ms] | 50.4234 || Query latency scroll (99.0 percentile) [ms] | 50.9866 || Query latency scroll (100 percentile) [ms] | 51.2115 || Query latency expression (90.0 percentile) [ms] | 482.02 || Query latency expression (99.0 percentile) [ms] | 485.951 || Query latency expression (100 percentile) [ms] | 491.999 || Query latency painless_static (90.0 percentile) [ms] | 890.382 || Query latency painless_static (99.0 percentile) [ms] | 918.354 || Query latency painless_static (100 percentile) [ms] | 927.282 || Query latency painless_dynamic (90.0 percentile) [ms] | 968.908 || Query latency painless_dynamic (99.0 percentile) [ms] | 1019.14 || Query latency painless_dynamic (100 percentile) [ms] | 1021.97 || Median CPU usage (index) [%] | 887.7 || Median CPU usage (stats) [%] | 94.9 || Median CPU usage (search) [%] | 445.05 || Total Young Gen GC [s] | 89.121 || Total Old Gen GC [s] | 12.274 || Index size [GB] | 3.30111 || Totally written [GB] | 20.2123 || Heap used for segments [MB] | 21.6794 || Heap used for doc values [MB] | 0.113129 || Heap used for terms [MB] | 20.0574 || Heap used for norms [MB] | 0.0761719 || Heap used for points [MB] | 0.770404 || Heap used for stored fields [MB] | 0.6623 || Segment count | 96 || Indices Stats(90.0 percentile) [ms] | 11.5116 || Indices Stats(99.0 percentile) [ms] | 14.1724 || Indices Stats(100 percentile) [ms] | 36.8348 || Nodes Stats(90.0 percentile) [ms] | 11.342 || Nodes Stats(99.0 percentile) [ms] | 13.435 || Nodes Stats(100 percentile) [ms] | 16.4768 |Logs for this race are archived in /home/elasticsearch/.rally/benchmarks/races/2016-08-08-07-29-59/local/logs-geonames-append-no-conflicts-defaults.zip5、问题大讨论https://discuss.elastic.co/t/the-below-bugs-appers-when-running-esrally/57063/25源码 作者danielmitterdorfer一一回复, 最终作者回复:I am very happy too and I am glad you had so much persistence. You also uncovered a few usability problems that I need to tackle.帮作者提了一些建议,促使他升级了一个版本0.3.2。参考:https://github.com/elastic/rally小结前面的确费劲周折,花费了接近3整天实践,期间甚至告诉原作者工具太难用,给作者反复沟通,最终问题解决。看到输出结果的那一刻,整个人很兴奋。坚持到底,相信付出的力量!不断思考、反思中前行!2016-08-08 21:00思于家中窗前作者:铭毅天下转载请标明出处,原文地址:http://blog.csdn.net/laoyang360/article/details/52155481如果感觉本文对您有帮助,请点击‘顶’支持一下,您的支持是我坚持写作最大的动力,谢谢!


(以上内容不代表本站观点。)
---------------------------------
本网站以及域名有仲裁协议。
本網站以及域名有仲裁協議。

2024-Mar-04 02:08pm
栏目列表