Support action-level concurrency in Java runtime.#130
Uh oh!
There was an error while loading. Please reload this page.
Conversation
rabbah left a comment
rabbah left a comment There was a problem hiding this comment.
Thank you for the contribution. Have you thought about how we can unit test this change?
Scusemua commented Sep 6, 2022
Not extensively. We could probably create a unit test in which we have a single container serve multiple requests simultaneously and watch for this by checking the value of a variable that gets incremented for each concurrent request or something like that. |
bdoyle0182 commented Sep 6, 2022
Hey Ben I replied on your slack thread and didn't realize you had an open pr for this will take a look |
| if (Boolean.parseBoolean(System.getenv("__OW_ALLOW_CONCURRENT"))){ | ||
| ThreadPoolExecutor executor = new ThreadPoolExecutor( | ||
| 10, // Core size. |
There was a problem hiding this comment.
I'm not sure how we should optimally determine this number since every function will have a different concurrency setting. Should it err on the side of maximizing low concurrency, i.e. most functions probably won't increase their setting past 1.
| if (Boolean.parseBoolean(System.getenv("__OW_ALLOW_CONCURRENT"))){ | ||
| ThreadPoolExecutor executor = new ThreadPoolExecutor( | ||
| 10, // Core size. | ||
| 25, // Max size. |
There was a problem hiding this comment.
I'm not sure if we want to set the max pool size as this becomes a hard bound on the language when the openwhisk setting per function is defined elsewhere. So the user would have to know explicitly that for java they can't increase their concurrency past 25 without getting performance degradation even if they aren't cpu bounded.
There was a problem hiding this comment.
Yeah. This was basically an arbitrary setting on my part as I was moreso interested in getting something working. We could just as well set the max pool size to something else (or a configurable limit).
There was a problem hiding this comment.
It would be a much larger change for the system to pass in the configured action limit as a part of the payload on the init request (and not every language would need the value so the system would need special logic per language for this). I think leaving it unbounded for now is probably the easiest way to get the initial working capability and can be optimized later if needed
On top of @rabbah 's point, the action concurrency feature in openwhisk is still fairly new and experimental. It adds an additional level of capacity planning required on top of the user correctly allocating an appropriate amount of memory to the function to know how many executions an instance can support. However this will be really hard for the user to test in practice as there's no guarantee the function execution will go to the same container or really any way for the user to verify that they ran things concurrently within the same container without writing special code to recognize it in their function. Only a few that I know of are leveraging this system feature, I wonder if anyone has attempted to solve this or thought about how a user can deterministically stress test their function. Not a question for this pr, but we can move this to one of our generic forums like slack or the mailing list |
3 participants
This change would enable support for action-level concurrency in the Java OpenWhisk runtime. The design is roughly modeled after the action-level concurrency support of the NodeJS environments. The Java
Proxychecks the__OW_ALLOW_CONCURRENTenvironment variable to determine whether support should be enabled or not.This does not address the action-level concurrency limit, and I imagine the values passed to the
ThreadPoolExecutorwill need to be changed for this to be used within OpenWhisk. I am not sure if the action-level concurrency configured for the action should affect the underlyingThreadPoolExecutorobject or not, though it seems like it should.At the very least, this PR serves as a starting point for evaluating whether the Java runtime can support action-level concurrency in this manner. For what it's worth, I have been using this modified runtime in personal projects for several weeks.