Disclaimer: the real substance of this article is a verbatim copy of “How This Package Maintains Persistent Connections” from the README of the alm.solrindex package (see e.g. https://pypi.python.org/pypi/alm.solrindex). I take no credit for any of it. A big thanks to authors of alm.solrindex for documenting their approach.
Why re-publish here? Well, the original documentation does not necessarily come up top when one googles the topic, and I wanted to make sure I can later get back to it even if I forget where I read about it. Perhaps this will help some others find about the mechanism a bit easier as well. After all, the need to make external connections from Plone will likely increase in our more and more networked world. And who knows some improvements or suggestions will come up that will help anyone struggling with the topic (myself included).
So, when one needs to make external network connections from Plone that need to be persistent (kept open), how do you maintain them in Plone, with all the issues related to threading etc. ?
The alm.solrindex approach, re-documented here:
This package uses a new method of maintaining an external database connection from a ZODB object. Previous approaches included storing_v_ (volatile) attributes, keeping connections in a thread local variable, and reusing the multi-database support inside ZODB, but those approaches each have significant drawbacks.
The new method is to add dictionary called foreign_connections to the ZODB Connection object (the _p_jar attribute of any persisted object). Each key in the dictionary is the OID of the object that needs to maintain a persistent connection. Each value is an implementation-dependent database connection or connection wrapper. If it is possible to write to the external database, the database connection or connection wrapper should implement the IDataManager interface so that it can be included in transaction commit or abort.
When a SolrIndex needs a connection to Solr, it first looks in the foreign_connections dictionary to see if a connection has already been made. If no connection has been made, the SolrIndex makes the connection immediately. Each ZODB connection has its own foreign_connections attribute, so database connections are not shared by concurrent threads, making this a thread safe solution.
This solution is better than _v_ attributes because connections will not be dropped due to ordinary object deactivation. This solution is better than thread local variables because it allows the object database to hold any number of external connections and it does not break when you pass control between threads. This solution is better than using multi-database support because participants in a multi-database are required to fulfill a complex contract that is irrelevant to databases other than ZODB.
Other packages that maintain an external database connection should try out this scheme to see if it improves reliability or readability. Other packages should use the same ZODB Connection attribute name, foreign_connections, which should not cause any clashes, since OIDs can not be shared.
An implementation note: when ZODB objects are first created, they are not stored in any database, so there is no simple way for the object to get a foreign_connections dictionary. During that time, one way to hold a database connection is to temporarily fall back to the volatile attribute solution. That is what SolrIndex does (see the _v_temp_cm attribute).