[4suite] [amara] Possible race condition of amara/Ft within mod_python subinterpreter context?

[Jeremy Kloth]

On Tuesday 19 December 2006 5:47 am, Remco Wendt wrote:
> Hello all,
>
> First of all (to all who have helped in the creation of Ft & amara),
> thank you for the beautiful amara library, which I use for all my
> dealings with XML in Python. 

Thank you, much appreciated.

> Of course it would be nice of me to just 
> email with a plain thank you. But alas, that is not the case.
>
> I develop web applications, and for that I use python (mod_python) and
> django. In the project I'm currently involved in, we use amara to deal
> with xml that is used within our web application. In our development
> server there is no problem whatsoever with amara (and Ft for that
> matter). But once we deploy the application in our production
> environment, importing amara can sometimes lead to a TypeError which
> originates from Ft's xmlparser.c (see django /mod_python trace down
> below):

From the traceback, it shows that the object `source` is indeed an 
xml.sax.xmlreader.InputSource, however, one InputSource is not the same as 
another if a module has been reloaded (using reload() or a 
PyImport_ReloadModule() in C).

$ python
Python 2.5 (r25:51908, Oct  6 2006, 15:22:41)
[GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from xml.sax import xmlreader, InputSource
>>> issubclass(InputSource, xmlreader.InputSource)
True
>>> reload(xmlreader)
<module 'xml.sax.xmlreader' 
from '/usr/lib/python2.5/site-packages/_xmlplus/sax/xmlreader.pyc'>
>>> issubclass(InputSource, xmlreader.InputSource)
False

This is relevant because Ft/Xml/src/domlette/xmlparser.c loads the 
xml.sax.xmlreader.InputSource class when it is first imported and uses that 
for its "isinstance" check in the prepareInputSource() function.  If memory 
serves, I believe that django and/or mod_python do module reloading at some 
point.

> Testing our web application shows that the error mainly occurs when
> multiple requests are send to parts of the application that use amara.
> Is it possible that some kind of race condition occurs within/with
> xmlparser.c in the context of mod_python? I read the following on the
> mod_python homepage about mod_python's use of subinterpreters:
>
> "Note that if any third party module is being used which has a C code
> component that uses the simplified API for access to the Global
> Interpreter Lock (GIL) for Python extension modules, then the
> interpreter name must be forcibly set to be "main_interpreter". This
> is necessary as such a module will only work correctly if run within
> the context of the first Python interpreter created by the process. If
> not forced to run under the "main_interpreter", a range of Python
> errors can arise, each typically referring to code being run in
> restricted mode."
>  -- http://www.modpython.org/live/current/doc-html/pyapi-interps.html
>
> Could this have something to do with the error that occurs?

Nope, red herring.

> The software that is used on both machines:
> - Apache 2.0.54-prefork
> - mod_python 3.2.10
> - 4Suite XML 1.0.1
> - Amara 1.2a2
> - python 2.5.0-final
>
> Now I'm not really sure what information is useful and what I can
> precisely do to trace the origins of this error. So any insights are
> more then welcome.

If you could provide insight into when if ever either django or mod_python do 
a module reload and if they do, if they fixup a C module's dictionary 
(although Python stores a prestine copy of it just for this purpose, so it 
would be difficult for them to do so).

Basically, reload() is evil if a module has imports. It may be possible to 
work around it if enough is known about how the reload is performed. I would 
look into it myself, but ATM I'm involved in other tasks and I don't know 
either package at all to make a quick look.

If you could do the legwork of finding when/where/how of reload(), I'll 
attempt to make sure that Ft can support that usage.

-- 
Jeremy Kloth
http://4suite.org/