ADF BC Tuning IX: Application Modules, Part 3

This is the last in a series of posts about tuning business components. The complete list:

The last post was about understanding application module pooling, so I’m going to assume you understand it now. And if you understand application module pooling, you can understand the pain points from a performance perspective, and you can gain some idea of how to balance out differing concerns. Here are the pain points, in decreasing order of importance.

Pain Point 1: Passivation and Activation

The passivation/recycling/activation process is how application module pooling accomplishes its goal: Keeping the number of caches in memory at any given time down to a reasonable size. That said, you want to minimize how often this process is needed. Writing an application module’s state to the database takes time–a lot of time, compared to in-memory operations, and reading it back in takes a lot of time too.

You can control how often this happens by changing the referenced pool size mentioned above (I’ll talk about how you change all these things in a minute). You want to keep it low enough that you won’t pose risks of running out of memory, but within those guidelines, you want it as high as possible–because when the referenced pool size is reached, the passivation-activation cycle starts happening.

The default referenced pool size is 10. For most applications–all but the very most data-intensive–this is way too small. You should have enough server memory to handle way more that 10 application module instances per application, and you don’t want to see a huge slowdown after the 10th simultaneous user. I’d recommend starting with 20.

Then, you need to do a little load testing. Here’s what I recommend: Test your application both at your expected average hit rate and at your expected peak hit rate, and pay attention to where there are performance slowdowns that are worse than what you’d expect from a linear scale. If the slowdowns are because your swap space keeps getting hit, lower the referenced pool size. If the slowdowns are because you have more than linearly increasing database reads/writes, raise the referenced pool size. You’re looking for two things: acceptable performance at peak hit rate and the best performance at average hit rate that’s compatible with that. If your usage statistics change, repeat the tests, because you might need to adjust the referenced pool size again.

Pain Point 2: Peak Time of Doom

But what if, at peak hit rate, you’re swapping memory to an unacceptable degree no matter *how* low you set the recycle threshold? In that case, your maximum pool size may need to be adjusted.

Reading the descriptions above, you might think that the total number of application module instances will never grow beyond the referenced pool size, but the fact is, it can. This can happen when the hit rate is high enough that not only will multiple sessions be going at the same time, but lots of multiple requests are going at the same time. Suppose your referenced pool size is 50, and there are 50 application module instances currently checked out of the pool, and another request comes in. There are no instances, not even referenced instances, in the pool to recycle, so that 51st application module is going to get created.

If this is really happening a lot–which doesn’t apply to lots of applications but does apply to some high-usage ones–you could end up with far more application module instances in your pool than the referenced pool size. You can solve this problem by setting another property: The maximum pool size. If your maximum pool size is 150, and you have 150 application module instances checked out, and another request comes in, that request is just going to have to wait until one of those other instances become available. You want to be a little careful about doing this, and set this number as high as you can, because it makes some requests wait on others. But better that than slowing down everyone’s response time with lots of disk swaps.

The default maximum pool size is 4096, which is very likely too high, but only high-usage applications will ever notice, so only bother changing this if you’re really getting unacceptable performance at peak hit rate and can’t make it go away by changing the referenced pool size.

Pain Point 3: Ramp-Ups and Ramp-Downs

This is only an issue if you have an extended “peak time” with lots of ups and downs in actual usage. If that’s the case, here’s what you should see:

  1. As peak-time begins, you’ll have a slowdown both as additional application module instances have to be created and as recycling starts. Memory consumption will increase, although hopefully it will stay within a manageable level.
  2. As peak-time plateaus (possibly a very jagged plateau, with lots of little ups and downs), you should gain some of your performance back, since you’ll stop the instantiation of new application module instances.
  3. When peak-time passes, memory consumption should decrease.

Why? Well, the idea is, once you create all the extra application module instances required by peak time, they should hang around throughout the full duration of the peak, even though the exact usage level may rise and dip moment-to-moment. When peak time is gone, you don’t want all those extra application module instances hanging around, so they should “time out” and disappear in a reasonably timely fashion.

You may find that you’re not seeing either #2 or #3 happen: Either performance will continue to be as bad throughout peak time as it was at the very beginning of the peak, or you’ll be very slow to get back the memory lost in step 1 after peak time has truly passed. This means that there’s something wrong with the way your application is handling timeouts.

Timeouts are controlled by four application module pool settings:

  • The Idle Instance Timeout, which controls how long an application module can be considered inactive before it is marked as eligible for timing out. You want this to be big enough that it smooths over the little dips in your peak time, but small enough that it allows rapid elimination of application module instances when the peak time has passed. The default, 600 seconds (10 minutes) is a good guess for many applications, but depending on your usage patterns, you might want a higher or lower number.
  • The Pool Polling Interval, which indicates how often the pool is checked for timed-out instances. In a worst-case situation, an application module instance will have to be (idle instance timeout) + (pool polling interval) seconds old before it will really be considered for timeout. You don’t want to set this too low, though, because polling the pool can be expensive. Generally, setting this figure to be equal to the idle instance timeout seems to give pretty good results.
  • The Minimum Available Pool Size. When it’s time to poll the pool, all the timed-out instances will disappear, down to the minimum available pool size. This allows you to keep a certain “baseline” number of instances in the pool. The number should be similar to the number of active sessions your application has under normal (i.e., non-peak) load. Any lower and you’ll be needlessly re-instantiating application module instances; any higher and you’ll consume more memory than you need.
  • The Maximum Available Pool Size. Suppose it’s time to poll the pool, and all the timed-out instances are removed. If the maximum available pool size is exceeded, non-timed-out instances will start going away too, oldest first. This may be useful if you have an occasional gigantic but momentary surge of users, but personally I think the default of 25 is probably to small; you usually want a more gradual ramp-down of extra instances. If you don’t anticipate such odd surges, set this to be equal to your maximum pool size.

Pain Point 4: Application Startup

This is generally a pretty minor concern, but it’s worth mentioning. Suppose you’ve just deployed or redeployed your application, or you’ve just bounced your server. And suppose that your average number of simultaneous sessions (and, hopefully, your minimum pool size) is 20. By default, each of your first 20 users is going to need to wait for an application module instance to be instantiated. Probably pretty negligible, unless you have a huge data model and aren’t using lazy loading, but still perhaps an issue.

Instead, you can set your pool’s Initial Pool Size to 20. If you do this, instead of the first 20 users having to wait for one application module instantiation each, your very first user is going to have to wait for 20 application module instantiations. Very painful for one person, but faster for the next 19.

This sounds like a lousy trade-off, right? Well, not so fast. Unless you’re providing 24/7 availability, you can keep your application off your public portal (or whatever your entry site is) for a few minutes after you deploy it. That allows you to make sure that *you* are the first user, and therefore that you are the one who suffers all the slowness of initial application module instantiation. This may well be worth it.

How Do I Do All This?

All these settings are part of an application module configuration, a bit of metadata an ADF data control uses when it instantiates its application module. Probably the easiest way to edit an application module’s configuration is to right-click it, select Configurations from the context menu, select a particular configuration (if you don’t know which to select, you probably want to select the one that ends with “Local”), and click Edit. That opens the Edit Business Components Configuration dialog. For the purposes of pool tuning, you want to select the Pooling and Scalability tab:

The Pooling and Scalability Tab
The Pooling and Scalability Tab

You should recognize all the labels on the left-hand side from earlier in this post. You probably don’t want to use the right-hand side of this tab. Although connection pooling sounds good, if you don’t use it, each application module instance will maintain its own connection, and since you’re pooling application modules, you’ll effectively be pooling connections. The only time you’d want to use it is if you want to pool connections among multiple pools of application modules (that is, across application module definitions, not just instances), and even then, you’re better off setting connection pooling properties in your Java EE container.

Well, that’s it for business components tuning, at least for now! Hope you found this series useful.

3 thoughts on “ADF BC Tuning IX: Application Modules, Part 3”

  1. Avrom,

    Can’t thank you enough for organizing and sharing all the information related to Performance Tuning ADF applications.
    Great Work !!

    Thanks,
    Husain

  2. I agree – a very useful set of posts… I’m looking forward to reading your book too!

    Just one comment: if you’re deploying your ADF app in a High Availability configuration (i.e. “Failover Transaction State Upon Managed Release” is ticked) the state is written back to the database (default) on every page request. Strictly this is not a full passivation though, i.e. you won’t get a subsequent recycle/activation unless it would need to do that anyway for the number of concurrently active users.

    Simon

  3. That’s right. There’s of course one other case you’ll get an activation here: If the process holding the AM instance goes down, and the instance needs to be reconstructed in another process.

    This makes a good point in general: Don’t assume that because something has a cool-sounding name (e.g., “High Availability”) that you always want it. Failing over is great if you can actually get your servers back up quickly and if the application module instance is managing a critical transaction (such as the finalization of a purchase), but it’s a cost you should always consider for non-critical transactions, and it’s an entirely pointless cost if you’re servers won’t be back up before the user gives up and closes their browser.

Leave a Reply

Your email address will not be published. Required fields are marked *