Puppet: System Administration Automated

Support

Ticket #469 (new defect)

Opened 2 years ago

Last modified 7 months ago

Rails support reopens the log file and database on every connection

Reported by: luke Assigned to: shadoi
Priority: normal Milestone: elmo
Component: Rails Version: 0.22.0
Severity: normal Keywords:
Cc: Triage Stage: Accepted
Attached Patches: None Complexity: Unknown

Description

I got 'too many open files' because of this, and when I checked the open file list with lsof, there were tens or hundreds of copies of the log and sqlite filehandles.

I'm going to hold this until grover since it's just a server-side error and the Rails support is still explicitly experimental, but it should clearly be solved.

Change History

02/21/07 01:07:34 changed by shadoi

  • owner changed from luke to shadoi.

Possible fix in svn rev 2217

02/21/07 03:48:33 changed by shadoi

After further testing, this apparently doesn't fix it completely. It just slows the accumulation of open sockets/file handles.

I guess we'll need to find the spot (or spots) where all the storeconfigs and export/collection operations finish, and close each connection explicitly. Alternately, we may also be able to wrap all the rails code in something that ensures that all threads using rails use the same connection handle, this is probably the best approach, but it's also the one that requires the most intimate knowledge of puppet internals.

04/05/07 22:41:01 changed by luke

  • specification set to Unnecessary.
  • patch set to None.
  • complexity set to Unknown.
  • milestone changed from grover to elmo.
  • approval set to Unnecessary.
  • compatibility set to Unknown.
  • stage set to Accepted.

03/27/08 11:10:27 changed by wyvern

This seems better, but still leaking on the sqlite (debian package 0.24.1-2, rails 1.1.6-3)

1 3 /dev/null 1 /dev/urandom 1 /etc/group 1 /etc/passwd 2 pipe:[29756375] 1 socket:[29756390] 1 socket:[29756393]

1008 /var/lib/puppet/state/clientconfigs.sqlite3

3 /var/log/puppet/masterhttp.log 1 /var/log/puppet/rails.log

This looks related: http://blog.moertel.com/articles/2006/08/24/database-connection-leak-in-typo-4-0-3-problem-solved

It leaks by 1 FD each node run, with the patch below it stops leaking. I'll leave this running for a little while. I don't know what unexpected side effects this may have.

--- /usr/lib/ruby/1.8/puppet/rails.rb.orig 2008-03-27 10:08:54.000000000 +0000 +++ /usr/lib/ruby/1.8/puppet/rails.rb 2008-03-27 10:07:26.000000000 +0000 @@ -21,7 +21,7 @@

ActiveRecord::Base.logger.level = Logger::DEBUG

end

- ActiveRecord::Base.allow_concurrency = true + ActiveRecord::Base.allow_concurrency = false

ActiveRecord::Base.verify_active_connections!

begin

03/27/08 11:14:07 changed by wyvern

let's try that again, remembering the formatting this time (can't see an edit unfortunately)

This seems better, but still leaking on the sqlite (debian package 0.24.1-2, rails 1.1.6-3)

  3 /dev/null
  1 /dev/urandom
  1 /etc/group
  1 /etc/passwd
  2 pipe:[29756375]
  1 socket:[29756390]
  1 socket:[29756393]
  1008 /var/lib/puppet/state/clientconfigs.sqlite3
  3 /var/log/puppet/masterhttp.log
  1 /var/log/puppet/rails.log

This looks related: http://blog.moertel.com/articles/2006/08/24/database-connection-leak-in-typo-4-0-3-problem-solved

It leaks by 1 FD each node run, with the patch below it stops leaking. I'll leave this running for a little while. I don't know what unexpected side effects this may have.

--- /usr/lib/ruby/1.8/puppet/rails.rb.orig	2008-03-27 10:08:54.000000000 +0000
+++ /usr/lib/ruby/1.8/puppet/rails.rb	2008-03-27 10:07:26.000000000 +0000
@@ -21,7 +21,7 @@
             ActiveRecord::Base.logger.level = Logger::DEBUG
         end
 
-        ActiveRecord::Base.allow_concurrency = true
+        ActiveRecord::Base.allow_concurrency = false
         ActiveRecord::Base.verify_active_connections!
 
         begin

04/24/08 06:51:04 changed by luke

  • component changed from library to Rails.