Skip to content

Instantly share code, notes, and snippets.

@m0veax
Last active November 19, 2015 12:21
Show Gist options
  • Save m0veax/8f6273243eed61f19ff4 to your computer and use it in GitHub Desktop.
Save m0veax/8f6273243eed61f19ff4 to your computer and use it in GitHub Desktop.
Possible Memory Leak in Search::Elasticsearch
#!/usr/bin/perl
use strict;
use warnings;
use utf8;
use FindBin qw($Bin);
use lib "$Bin/../lib";
use Search::Elasticsearch;
use Memory::Usage;
my $mu = Memory::Usage->new;
my $settings = {
host => "localhost",
port => 9200,
cxn => undef,
};
my $es = Search::Elasticsearch->new(
nodes => sprintf( '%s:%s', $settings->{host}, $settings->{port} ),
log_to => 'Stderr',
cxn => $settings->{cxn},
serializer => 'JSON::XS',
);
my $test_record = {
foo => "bar",
bar => "baz"
};
$mu->record('initialized Elastic');
my $data = [];
my $counter = 0;
my $bulk = $es->bulk_helper(
index => "test",
type => "test_items",
max_count => 1000
);
$mu->record('Initialized es_bulk');
for my $i ( 1 .. 100_000 ) {
$bulk->index( { source => $test_record } );
$mu->record('Given to Index');
}
$bulk->flush;
$mu->record('Flushed');
$mu->dump();
Modules used from service/test_elastic_memory.pl:
1. strict 1.08, service/test_elastic_memory.pl line 3 [main]
2. warnings 1.23, service/test_elastic_memory.pl line 4 [main]
3. utf8 1.13_01, service/test_elastic_memory.pl line 5 [main]
4. FindBin 1.51, service/test_elastic_memory.pl line 7 [main]
5. Carp 1.3301, FindBin.pm line 80
6. Exporter 5.71, Carp.pm line 99
15. Exporter::Heavy 5.71, Exporter.pm line 16
7. Cwd 3.48, FindBin.pm line 83
8. vars 1.03, Cwd.pm line 172
9. warnings::register 1.03, vars.pm line 7
10. XSLoader 0.17, Cwd.pm line 248
58. attributes 0.22, XSLoader.pm line 92 [JSON::MaybeXS]
11. File::Basename 2.85, FindBin.pm line 84
12. File::Spec 3.48, FindBin.pm line 85
13. File::Spec::Unix 3.48, File/Spec.pm line 22
14. constant 1.31, File/Spec/Unix.pm line 127
16. lib 0.63, service/test_elastic_memory.pl line 8 [main]
17. Config 5.020001, lib.pm line 6
18. Search::Elasticsearch 2.00, service/test_elastic_memory.pl line 10 [main]
19. Moo 1.006001, Search/Elasticsearch.pm line 3
20. strictures 1.005004, Moo.pm line 3
21. Moo::_Utils, Moo.pm line 4
22. Sub::Name 0.07, Moo/_Utils.pm line 9
23. base 2.22, Sub/Name.pm line 50
24. DynaLoader 1.25, base.pm line 99
115. Log::Any::Adapter::Base 1.032, base.pm line 99
25. Module::Runtime 0.014, Moo/_Utils.pm line 12
50. Package::Stash::XS 0.28, Module/Runtime.pm line 317
52. B::Hooks::EndOfScope::XS 0.13, Module/Runtime.pm line 317
53. Variable::Magic 0.53, Module/Runtime.pm line 317
55. Search::Elasticsearch::Serializer::JSON 2.00, Module/Runtime.pm line 317
56. JSON::MaybeXS 1.002002, Search/Elasticsearch/Serializer/JSON.pm line 4
57. Cpanel::JSON::XS 3.0105, JSON/MaybeXS.pm line 15
64. Class::XSAccessor 1.19, Module/Runtime.pm line 317
65. Class::XSAccessor::Heavy 1.19, Class/XSAccessor.pm line 6
68. Search::Elasticsearch::Role::Serializer::JSON 2.00, Module/Runtime.pm line 317
69. Encode 2.60, Search/Elasticsearch/Role/Serializer/JSON.pm line 8
70. Encode::Alias 2.18, Encode.pm line 47
71. Encode::Config 2.05, Encode.pm line 52
72. Encode::ConfigLocal, Encode.pm line 59 (FAILED)
73. Encode::Encoding 2.07, Encode.pm line 239
74. Search::Elasticsearch::Role::Serializer 2.00, Module/Runtime.pm line 317
75. Search::Elasticsearch::Logger::LogAny 2.00, Module/Runtime.pm line 317
76. Log::Any 1.032, Search/Elasticsearch/Logger/LogAny.pm line 8
77. Log::Any::Manager 1.032, Log/Any.pm line 10
114. Log::Any::Adapter::Stderr 1.032, Log/Any/Manager.pm line 183 (eval 76)
130. Log::Any::Adapter::Null 1.032, Log/Any/Manager.pm line 183 (eval 96)
78. Log::Any::Adapter::Util 1.032, Log/Any.pm line 19
116. Log::Any::Proxy 1.032, Log/Any/Adapter/Util.pm line 173 (eval 77)
79. Log::Any::Adapter 1.032, Search/Elasticsearch/Logger/LogAny.pm line 9
80. Search::Elasticsearch::Role::Logger 2.00, Module/Runtime.pm line 317
81. URI 1.64, Search/Elasticsearch/Role/Logger.pm line 5
82. URI::Escape 3.31, URI.pm line 22
109. URI::http, URI.pm line 132 (eval 73)
110. parent 0.228, URI/http.pm line 6
111. URI::_server, parent.pm line 20
112. URI::_generic, parent.pm line 20
113. URI::_query, parent.pm line 20
83. Search::Elasticsearch::Cxn::Factory 2.00, Module/Runtime.pm line 317
84. Search::Elasticsearch::Cxn::HTTPTiny 2.00, Module/Runtime.pm line 317
85. HTTP::Tiny 0.043, Search/Elasticsearch/Cxn/HTTPTiny.pm line 8
86. Errno 1.2003, HTTP/Tiny.pm line 844 [HTTP::Tiny::Handle]
87. IO::Socket 1.37, HTTP/Tiny.pm line 845 [HTTP::Tiny::Handle]
88. IO::Handle 1.35, IO/Socket.pm line 12
89. Symbol 1.07, IO/Handle.pm line 267
90. SelectSaver 1.02, IO/Handle.pm line 268
91. IO 1.31, IO/Handle.pm line 269
92. Socket 2.013, IO/Socket.pm line 13
93. IO::Socket::INET 1.35, IO/Socket.pm line 22
94. IO::Socket::UNIX 1.26, IO/Socket.pm line 23
95. IO::Socket::IP 0.29, HTTP/Tiny.pm line 852 [HTTP::Tiny::Handle]
96. POSIX 1.38_03, IO/Socket/IP.pm line 33
97. Fcntl 1.11, POSIX.pm line 17
98. Tie::Hash 1.05, POSIX.pm line 422 [POSIX::SigRt]
99. Search::Elasticsearch::Role::Cxn::HTTP 2.00, Module/Runtime.pm line 317
101. Search::Elasticsearch::Role::Cxn 2.00, Module/Runtime.pm line 317
102. Search::Elasticsearch::Role::Is_Sync 2.00, Module/Runtime.pm line 317
103. Search::Elasticsearch::CxnPool::Static 2.00, Module/Runtime.pm line 317
104. Search::Elasticsearch::Role::CxnPool::Static 2.00, Module/Runtime.pm line 317
105. Search::Elasticsearch::Role::CxnPool 2.00, Module/Runtime.pm line 317
106. IO::Select 1.22, Search/Elasticsearch/Role/CxnPool.pm line 6
107. Time::HiRes 1.9726, Search/Elasticsearch/Role/CxnPool.pm line 7
117. Search::Elasticsearch::Transport 2.00, Module/Runtime.pm line 317
118. Search::Elasticsearch::Role::Transport 2.00, Module/Runtime.pm line 317
119. Search::Elasticsearch::Client::2_0::Direct 2.00, Module/Runtime.pm line 317
120. Search::Elasticsearch::Role::API::2_0 2.00, Module/Runtime.pm line 317
121. Search::Elasticsearch::Util::API::QS 2.00, Search/Elasticsearch/Role/API/2_0.pm line 6
122. Search::Elasticsearch::Role::Client::Direct 2.00, Module/Runtime.pm line 317
123. Search::Elasticsearch::Util::API::Path 2.00, Search/Elasticsearch/Role/Client/Direct.pm line 5
124. Any::URI::Escape 0.01, Search/Elasticsearch/Util/API/Path.pm line 5
125. URI::Escape::XS, Any/URI/Escape.pm line 19 (eval 85) (FAILED)
126. Search::Elasticsearch::Role::Client 2.00, Module/Runtime.pm line 317
127. Search::Elasticsearch::Role::Client::Direct::Main 2.00, Module/Runtime.pm line 317
128. Search::Elasticsearch::Bulk 2.00, Module/Runtime.pm line 317
129. Search::Elasticsearch::Role::Bulk 2.00, Module/Runtime.pm line 317
26. Devel::GlobalDestruction 0.12, Moo/_Utils.pm line 14
27. Sub::Exporter::Progressive 0.001011, Devel/GlobalDestruction.pm line 11
28. List::Util 1.38, Sub/Exporter/Progressive.pm line 9
29. Moo::_mro, Moo/_Utils.pm line 16
30. mro 1.16, Moo/_mro.pm line 5
31. Moo::HandleMoose::_TypeMap, Moo/_Utils.pm line 121
32. Scalar::Util 1.38, Moo/HandleMoose/_TypeMap.pm line 10
33. Import::Into 1.002004, Moo.pm line 5
34. Moo::sification, Moo.pm line 10
36. Moo::Object, Moo.pm line 75
59. Method::Generate::Constructor, Moo.pm line 151
60. Sub::Quote 1.006001, Method/Generate/Constructor.pm line 4
61. Sub::Defer 1.006001, Sub/Quote.pm line 7
62. B 1.48, Sub/Quote.pm line 10
108. Method::Generate::BuildAll, Method/Generate/Constructor.pm line 61
63. Method::Generate::Accessor, Moo.pm line 139
66. Moo::Role 1.006001, Moo.pm line 35
67. Role::Tiny 1.003004, Moo/Role.pm line 5
100. Class::Method::Modifiers 2.10, Moo/Role.pm line 64
35. Search::Elasticsearch::Util 2.00, Search/Elasticsearch.pm line 5
37. Search::Elasticsearch::Error 2.00, Search/Elasticsearch/Util.pm line 4
38. overload 1.22, Search/Elasticsearch/Error.pm line 39
39. overloading 0.02, overload.pm line 83
40. Data::Dumper 2.151, Search/Elasticsearch/Error.pm line 41
41. bytes 1.04, Data/Dumper.pm line 718
42. Sub::Exporter 0.987, Search/Elasticsearch/Util.pm line 18
43. Data::OptList 0.109, Sub/Exporter.pm line 11
44. Params::Util 1.07, Data/OptList.pm line 10
45. Sub::Install 0.928, Data/OptList.pm line 11
46. namespace::clean 0.25, Search/Elasticsearch.pm line 6
47. Package::Stash 0.36, namespace/clean.pm line 6
48. Module::Implementation 0.07, Package/Stash.pm line 15
49. Try::Tiny 0.22, Module/Implementation.pm line 13
51. B::Hooks::EndOfScope 0.13, namespace/clean.pm line 11
54. Memory::Usage 0.201, service/test_elastic_memory.pl line 12 [main]
Possible proxies:
26 Module/Runtime.pm line 317, sub Module::Runtime::require_module
3 parent.pm line 20, sub parent::import
2 base.pm line 99, sub base::import
2 Log/Any/Manager.pm line 183, sub Log::Any::Manager::_require_dynamic
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment