More than 50 million people use GitHub to discover, fork, and contribute to over 100 million projects. The first step is to create a ThreadPoolBulkheadConfig: This creates a ThreadPoolBulkheadConfig with default values for coreThreadPoolSize (number of processors available - 1) , maxThreadPoolSize (maximum number of processors available) , keepAliveDuration (20ms) and queueCapacity (100). io.github.resilience4j » resilience4j-bulkhead Apache Resilience4j is a lightweight, easy-to-use fault tolerance library designed for Java8 and functional programming Last Release on Oct 9, 2020 For a given remote service the Bulkhead must be a singleton. A Bulkhead represent an entity limiting the amount of parallel operations. Then you will see that service-call-1 till service-call-5 (5 concurrent threads) processing finished together around 17:01: Then you will see that service-call-6 till service-call-10 (5 concurrent threads) processing finished together around 17:01: Then you will see that service-call-16 till service-call-20 ended up with. At first you will see “Starting service call” for all 20 threads. He enjoys both sharing with and learning from others. In this article we will look at very simple basic example of Resilience4j bulkhead feature & look at runtime behavior of bulkhead. We do this to avoid explicitly passing the data as a parameter between method chains, especially when the value is not directly related to the core business logic we are implementing. While what happened to our service was an extreme example, we can see how a slow upstream dependency can impact an unrelated area of the calling service. We will talk about its modules and functionality later, but first, let's briefly discuss why you should even bother with fault tolerance. Our service talks to a remote service encapsulated by the class FlightSearchService. With a clean and minimalist approach to design, he is passionate about code - the aesthetics of it and creating maintainable and flexible solutions. If a thread is not able to acquire a permit in the 2s maxWaitDuration we specified, a BulkheadFullException is thrown: Apart from the first line, the other lines in the stack trace are not adding much value. Then you will see that service-call-11 till service-call-15 (5 concurrent threads) processing finished together around 17:01. In a previous post we talked about the implementation of the Circuit Breaker pattern.As a reminder, the Circuit Breaker is a pattern that prevents cascading the failure of a single micro-service in the whole architecture, ensuring the system is resilient. Subscribe to my mailing list to get notified about new content and get my eBook "Get Your Hands Dirty on Clean Architecture" for just $5! If the queueCapacity has been reached, then the remote call is rejected with a BulkheadFullException. We will find out what problem it solves, when and how to use it, and also look at a few examples. GitHub is where people build software. Even as we began investigating the issue, there was a second alert - another server had stopped responding to health checks and had also been taken out of the pool. Assume that we are building a website for an airline to allow its customers to search for and book flights. To prevent this, the actual call to the remote service should be in a core, internal layer and other areas should use the bulkhead decorator exposed by the internal layer. The term bulkhead itself comes from its usage in ships where the bottom portion of the ship is divided into sections separated from each other. coreThreadPoolSize , maxThreadPoolSize , keepAliveDuration and queueCapacity are the main configurations associated with the ThreadPoolBulkhead. Conclusion We may want to retry after some delay if there is a BulkheadFullException, for example. If you are a video person here is the video tutorial. This is because the ThreadPoolBulkHead does not execute the code synchronously on the current thread. Additionally, it shows how to enforce this by codifying the intent in ArchUnit tests. Overview: In this tutorial, I would like to demo Bulkhead Pattern, one of the Microservice Design Patterns for designing highly resilient Microservices using a library called resilience4j along with Spring Boot. I have a web application running in a Tomcat server that performs calls to an external server. We next set a request tracking id before calling the bulkhead-decorated flight search operation: The sample output shows that this value was not available in the bulkhead-managed thread: To solve this problem, ThreadPoolBulkhead provides a ContextPropagator. Build: LAST BUILD BRANCH: v0.14.0. BulkheadConfig encapsulates the maxConcurrentCalls, maxWaitDuration, writableStackTraceEnabled, and fairCallHandlingEnabled configurations. Resilience4j provides higher-order functions (decorators) to enhance any functional interface, lambda expression or method reference with a Circuit Breaker, Rate Limiter, Retry or Bulkhead. Within minutes, all 2000 threads across the cluster had blocked indefinitely - there were no free threads to even respond to health checks from the load balancer. Do not burden service with calls more than it can consume at a time. The idea behind bulkheads is to set a limit on the number of concurrent calls we make to a remote service. A new library always has one advantage over a previous library - it can learn from the mis… It internally uses a java.util.concurrent.ArrayBlockingQueue and a java.util.concurrent.ThreadPoolExecutor to control the number of concurrent calls. Resilience4j provides two types of bulkheads - SemaphoreBulkhead and ThreadPoolBulkhead. It is kind of replacement of Hystrix because Hystrix is not in active development, instead in maintenance mode. It is lightweight compared to Hystrix as it has the Vavr library as its only dependency. When using the thread pool-based bulkhead, ThreadPoolBulkheadRegistry, ThreadPoolBulkheadConfig, and ThreadPoolBulkhead are the main abstractions we work with. This module provides configurable Filters to provide CircuitBreaking, RateLimiting, Retrying and Bulkheading, by integrating with the awesome Resilience4J library.. Photo Credits Create mock external service which takes 2 seconds to finish its processing. If there is a breach, and water starts flowing in, only that section gets filled with water. How can we ensure that a new developer understands this intent in the future? (adsbygoogle = window.adsbygoogle || []).push({}); Here is a mock service which takes 2 seconds to finish processing. Usually when reading logs, just knowing that a BulkheadFullException has occurred is enough. In this article, we learned how we can use Resilience4j’s Bulkhead module to set a limit on the concurrent calls that we make to a remote service. First, we create BulkheadConfig, BulkheadRegistry, and Bulkhead as usual. resilience4j content on DEV. Similar to the other Resilience4j modules we have seen, the Bulkhead also provides additional methods like decorateCheckedSupplier(), decorateCompletionStage(), decorateRunnable(), decorateConsumer() etc. We provide it the code we want to execute as a functional construct - a lambda expression that makes a remote call or a Supplier of some value which is retrieved from a remote service, etc. Each Bulkhead object is associated with a BulkheadConfig. ThreadPoolBulkhead also has awritableStackTraceEnabled configuration to control the amount of information in the stack trace of a BulkheadFullException. The prefix resilience4j.bulkhead is missing. As we found out later, there was some issue with the Redis cluster at the same time and it had stopped accepting new connections. If the BulkheadFullException occurs multiple times, these stack trace lines would repeat in our log files. so we can provide our code in other constructs than a Supplier. Resilience4jis a fault tolerance library designed for Java 8 and functional programming. Resilience4j provides higher-order functions (decorators) to enhance any functional interface, lambda expression or method reference with a Circuit Breaker, Rate Limiter, Retry or Bulkhead. The ThreadPoolBulkhead uses a thread from a thread pool to execute our code. Resilience4j | Retry basics & runtime behavior | Simple example for beginners, Resilience4j | Rate Limiter basics & runtime behavior | Simple example for beginners, Resilience4j | Circuit breaker basics & runtime behavior/state changes | Simple example for…, Resilience4j Complete Tutorial | Basics with runtime behavior | Simple examples for beginners, Resilience4j | Expose Circuit Breaker Metrics to JMX using Micrometer | Simple example, Java 13 Feature | Text blocks using three double quotes “”” | Examples with multiline HTML, SQL, Paragraph with indentation. Yes, it's normal that all threads are waiting until maxWaitDuration is elapsed before the exception is thrown. Spring Boot ve Resilience4j /2 25 Jun 2020 spring • boot • resilience4j • gemi • bölmesi • bulkhead • semaphore • semafor • test • türkçe. We were using the Jedis library to connect to Redis and the default behavior of that library was to block the calling thread indefinitely until a connection was established. We have context value set in ThreadLocal. ContextPropagator is an abstraction for retrieving, copying and cleaning up values across thread boundaries. When using the semaphore-based bulkhead, BulkheadRegistry, BulkheadConfig, and Bulkhead are the main abstractions we work with. In this series so far, we have learned about Resilience4j and its Retry, RateLimiter, and TimeLimiter modules. Finally, let’s call the decorated operation a few times to understand how the bulkhead works: The timestamps and thread names in the output show that while the first two requests executed immediately, the third request was queued and later executed by one of the threads that freed up: If there are no free threads and no capacity in the queue, a BulkheadFullException is thrown: We can use the writableStackTraceEnabled configuration to reduce the amount of information that is generated in the stack trace: Sometimes we store data in a ThreadLocal variable and read it in a different area of the code. Any thread which attempts to call the remote service over this limit can either get a BulkheadFullException immediately or wait for some time for a permit to be released by another thread. When using the thread pool-based bulkhead, ThreadPoolBulkheadRegistry, … We can reduce the amount of information that is generated in the stack trace by setting the writableStackTraceEnabled configuration to false: Now, when a BulkheadFullException occurs, only a single line is present in the stack trace: Similar to the other Resilience4j modules we have seen, the Bulkhead also provides additional methods like decorateCheckedSupplier(), decorateCompletionStage(), decorateRunnable(), decorateConsumer() etc. Required fields are marked *, * Create bulk head of 5 max concurrent calls with 2 seconds wait time for. Resilience4j has been inspired by Netflix Hystrix but is designed for Java 8 and functional programming. When there are multiple threads waiting for permits, the fairCallHandlingEnabled configuration determines if the waiting threads acquire permits in a first-in, first-out order. Let’s say we want to limit the number of concurrent calls to 2 and that we are willing to wait 2s for a thread to acquire a permit: Let’s now express our code to run a flight search as a Supplier and decorate it using the bulkhead: Finally, let’s call the decorated operation a few times to understand how the bulkhead works. It’s more effective to combine a bulkhead with one or more of the other Resilience4j modules like retry and rate limiter. This article is accompanied by a working code example on GitHub. But when a problem occurred in this one area, it ended up impacting the entire service. For example, we might want to log the current user ID or a transaction ID or some request tracking ID to every log statement to make it easier to search logs. Resilience4J has been selected as a replacement for Netflix’s Hystrix, which has been moved to maintenance mode. The SAP Cloud SDK now builds upon the Resilience4j library in order to provide resilience for your cloud applications. Copy link Member RobWin commented Aug 28, 2019. on parent, main thread before calling flight search", "Getting request tracking id from thread: ", "Clearing request tracking id on thread: ", organizing the package structure to make such intents clear, Gotchas and Good Practices When Implementing Bulkhead, the maximum number of available permissions (, the core and maximum sizes of the thread pool (. // Wrap service call in bulkhead & call service. I am doing a lot of tests using Resilience4j and monitoring the behaviour of threads. Netflix Hystrix, by contrast, has a dependency on Archaius which has several other external library dependencies such as Guava and Apache Commons. In previous versions 2.x we used the Hystrix library, which has been in maintenance mode for some time now.. Resilience4j comes with many modules to protect your application from failures. Here is the maven dependency for resilience4j-bulkhead required for this example. When you work with distributed systems, always remember this number one rule – anything could happen. resilience4j-bulkhead works similar to the other Resilience4j modules. This can be useful because without it, our logs could get filled with a lot of similar information when the exception occurs multiple times. In a few minutes, every server had stopped responding to health probes and our service was completely down. It means they won't review issue, merge pull requests and release new versions. Then, we create a MeterRegistry and bind the ThreadPoolBulkheadRegistry to it: In a real application, we would export the data to a monitoring system periodically and analyze it on a dashboard. You are using a semaphore-based Bulkhead. Here we will call service client in 20 parallel threads which might mimic 20 parallel users or 20 parallel executions. Bulkhead Concept. First we define a RequestTrackingIdHolder class, a wrapper class around a ThreadLocal: The static methods make it easy to set and get the value stored on the ThreadLocal. What is Resilience4j? Then, we create a MeterRegistry and bind the BulkheadRegistry to it: After running the bulkhead-decorated operation a few times, we display the captured metrics: First, we create ThreadPoolBulkheadConfig, ThreadPoolBulkheadRegistry, and ThreadPoolBulkhead as usual. Our service was hosted on Tomcat and it had a default request handling thread pool size of 200 threads. I am using spring-boot 2.2.5, resilience4j 1.4.0. The remaining request handling threads could have continued serving other requests. Home » io.github.resilience4j » resilience4j-bulkhead Resilience4j. BulkheadFullException (Showing top 13 results out of 315) Common ways to obtain BulkheadFullException The service itself supported several features and not all of them required accessing the Redis cache. A constructive and inclusive social network. A Bulkhead instance is thread-safe can be used to decorate multiple requests. Resilience4j is a lightweight fault tolerance library inspired by Netflix Hystrix, but designed for functional programming. All calls to a given remote service should go through the same Bulkhead instance. Bulkhead is configured to allow only 5 concurrent calls at a time. - and the bulkhead decorates it with the code to control the number of concurrent calls. When using the ThreadPoolBulkhead, since our code is not executed on the current thread, the data we had stored on ThreadLocal variables will not be available in the other thread. Resilience4j is a lightweight, easy-to-use fault tolerance library designed for Java8 and functional programming License: Apache 2.0: Tags: github io: Used By: 25 artifacts: Central (23) Version Also a service client which decorates calls to external service using bulkhead. ThreadPoolBulkheadConfig encapsulates the coreThreadPoolSize , maxThreadPoolSize , keepAliveDuration and queueCapacity configurations. Subscribe to my Mailing List and get my book Get Your Hands Dirty on Clean Architecture for just $5! " Here is the maven dependency for resilience4j-bulkhead required for this example. resilience4j.bulkhead: instances: get: maxConcurrentCall: 10 maxWaitDuration: 10ms. We treat calls to different remote services as different, isolated pools and set a limit on how many calls can be made concurrently. Need For Resiliency: Microservices are distributed in nature. For example, Resilience4j also provides other modules like RateLimiter, Bulkhead, Retry in addition to the CircuitBreaker and TimeLimiter modules used in this article. In case already 5 concurrent calls are in process, then it is configured to wait for maximum of 5 seconds. Hi team, We are trying to migrate from Hystrix to Resiliency4j. Saajan is an architect with deep experience building systems in several business domains. This prevents the entire ship from sinking. ThreadPoolBulkhead internally uses these configurations to construct a ThreadPoolExecutor. Do not burden service with calls more than it … This is because we do not exhaust the resources of product-service. Rated 4.8 stars on Amazon Best Java code snippets using io.github.resilience4j.bulkhead. Let’s see how to use the various features available in the resilience4j-bulkhead module. This is exactly the problem that bulkhead addresses - it prevents a problem in one area of the service from affecting the entire service. Let’s look at an example to understand this problem. We can think of this value as the number of permits that the semaphore is initialized with. // Make 20 calls using service client mimicking 20 parallel users. Check out Tom’s article which shows one way of solving such problems by organizing the package structure to make such intents clear. In this tutorial, we'll talk about the Resilience4j library. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. If we don’t enforce this, some areas of our codebase may make a direct call to the remote service, bypassing the Bulkhead. We can use CompletableFuture to simulate concurrent flight search requests from users: The timestamps and thread names in the output show that out of the 4 concurrent requests, the first two requests went through immediately: The third and the fourth requests were able to acquire permits only 1s later, after the previous requests completed. Repo Added 28 Mar 2017 07:41PM UTC Total Files 112 # Builds 678 Last Badge. Liked this article? Please refer to the description in the previous article for a quick intro into how Resilience4j works in general. We learned why this is important and also saw some practical examples on how to configure it. so we can provide our code in other constructs than a Supplier.. ThreadPoolBulkhead. Resilience4j is a lightweight fault tolerance library inspired by Netflix Hystrix, but designed for functional programming. The library helps with implementing resilient systems by managing fault tolerance for remote communications. If call load is greater than service’s consumption at that time, then keep calls waiting for reasonable time until existing concurrent calls can finish & they can get turn. Now, we will be creating a LibraryConfig class to define a bean for RestTemplate to make a call to Book Management service. If we had had a limit of, say, 20 concurrent requests to Redis set on each of the server instances, only those threads would have been affected when the Redis connectivity issue occurred. Let’s look at the configurations associated with the semaphore bulkhead and what they mean. Resilience4j Tutorial | Basics with runtime behavior | Simple examples for beginners, Your email address will not be published. Mimic 20 parallel users/executions by calling service client in 20 threads. Use of resilience4j bulkhead Showing 1-1 of 1 messages. The first step is to create a BulkheadConfig: This creates a BulkheadConfig with default values formaxConcurrentCalls(25), maxWaitDuration(0s), writableStackTraceEnabled(true), and fairCallHandlingEnabled(true). Please show some love and subscribe to my channel Subscribe Hacker Heap. resilience4j / resilience4j. Resilience4j is a lightweight fault tolerance library inspired by Netflix Hystrix, but designed for functional programming. It is lightweight, modular, and really fast. However, Spring Cloud Circuit Breaker is an abstraction over only the circuit breaker part. Each ThreadPoolBulkhead object is associated with a ThreadPoolBulkheadConfig. Finally, the writableStackTraceEnabled configuration lets us reduce the amount of information in the stack trace when a BulkheadFullException occurs. Resilience4j is a lightweight, easy-to-use fault tolerance library designed for Java 8 and functional programming, it is a lightweight as it doesn't depend on any other external libraries. So every request which went through a code path that connected to Redis ended up blocking the thread indefinitely. ThreadPoolBulkheadRegistry is a factory for creating and managing ThreadPoolBulkhead objects. You can play around with a complete application illustrating these ideas using the code on GitHub. The above libraries have more capabilities than we've explored here. Circuit Breaking ¶ If no thread is free to execute an incoming task, the task is enqueued for executing later when a thread becomes available. The SemaphoreBulkhead internally uses java.util.concurrent.Semaphore to control the number of concurrent calls and executes our code on the current thread. Its All Binary – Coding Posts, Examples, Projects & More, Simplest example of resilience4j bulkhead, Create your own screen sharing web application using Java and JavaScript (WebRTC), Create your own video conference web application using Java & JavaScript, Java Server two-way communication with browser | Simple Websocket example, Detailed Comparison of SQL (MySQL) vs. NoSQL (MongoDB) vs. Graph Query (Neo4j) | Data-structure, Queries, Data types, Functions, CSS in Action (Tutorial) | Watch, understand & learn CSS live in your own browser, Getting started with artificial intelligence in java, json parent multiple child serialize json. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. Use of resilience4j bulkhead: El R1co: 5/24/20 3:36 PM: Hi, I have a vertx application with RxJava 2 and I would like to use a Bulkhead on one of my services, to limit the number of concurrent calls specifically for this one. Let’s say we want to limit the number of concurrent calls to 2: Unlike the SemaphoreBulkhead.decorateSupplier() which returned a Supplier>, the ThreadPoolBulkhead.decorateSupplier() returns a Supplier>. We will use the same example as the previous articles in this series. Resilience4j is a lightweight, easy-to-use fault tolerance library inspired by Netflix Hystrix, but designed for Java 8 and functional programming. We will observe how bulkhead behaves for all threads. 90% master: 100% DEFAULT BRANCH: master. 8. The internalThreadPoolExecutor executes incoming tasks using one of the available, free threads. This is determined by the maxWaitDuration value. We can listen for these events and log them, for example: The bulkhead.available metric is the same as maxConcurrentCalls that we configure on the BulkheadConfig. The most important ones are timeouts, bulkheads, and circuit breakers. Your email address will not be published. About¶. Resilience4j is a lightweight fault tolerance library designed for Java 8 and functional programming. In this article we will look at very simple basic example of Resilience4j bulkhead feature & look at runtime behavior of bulkhead. Resilience4j provides higher-order functions (decorators) to enhance any functional interface, lambda expression or method reference with a Circuit Breaker, Rate Limiter, Retry or Bulkhead. Resilience4j provides higher-order functions (decorators) to enhance any functional interface, lambda expression or method reference with a Circuit Breaker, Rate Limiter, Retry or Bulkhead. We were using Redis for caching some data for a couple of features supported by the application. A few years back we had a production issue where one of the servers stopped responding to health checks and the load balancer took the server out of the pool. Products requests average response time is 106 milliseconds compared to 3.6 seconds without bulkhead implementation. In this article, we will explore the Bulkhead module. Feature overview of the http4k-resilience4j module. It defines an interface with methods to get a value from the current thread (retrieve()), copy it to the new executing thread (copy()) and finally cleaning up on the executing thread (clear()). The following examples show how to use io.github.resilience4j.bulkhead.Bulkhead.These examples are extracted from open source projects. maxConcurrentCalls determines the maximum number of concurrent calls we can make to the remote service. Bulkhead Configurations: Maximum 5 concurrent calls at any given time. Open source and radically transparent. Mikroservislerin hata dayanıklılığını (fault tolerance) tesis etmenin birinci yolu olan sigortalardan (circuit breaker) serimizin ilk yazısında bahsetmiştik. The following examples show how to use io.github.resilience4j.bulkhead.BulkheadConfig.These examples are extracted from open source projects. BulkheadRegistry is a factory for creating and managing Bulkhead objects. Implementing Bulkhead with Resilience4j Next up in the Resilience4j series - this article explains the Bulkhead module and how to use it to build resilient applications. Otherwise just timeout the calls & go for alternate recovery path. Illustrating these ideas using the thread indefinitely people use GitHub to discover, fork, and breakers! It means they wo n't review issue, merge pull requests and release versions! Overloading of service & also give a graceful way to provide CircuitBreaking RateLimiting. Users or 20 parallel executions copy link Member RobWin commented Aug 28, 2019 few examples maximum 5 concurrent we! Because Hystrix is not in active development, instead in maintenance mode messages... Problem it solves, when and how to enforce this by codifying the in. As its only dependency client in 20 threads some delay if there is lightweight! Netflix ’ s more effective to combine a bulkhead represent an entity limiting the amount of information in stack. Configurations associated with the code synchronously on the number of permits that the bulkhead... Commented Aug 28, 2019 not burden service with calls more than 50 million people use GitHub discover! Go through the same example as the number of concurrent calls at a.! Ones are timeouts, bulkheads, and bulkhead as usual maintenance mode Amazon and 4.6 stars on Amazon and stars. Bulkhead objects control the number of concurrent calls monitoring the behaviour of threads executes! The task is enqueued for executing later when a problem in one area, 's... And it had a DEFAULT request handling threads could have continued serving other requests bean for RestTemplate to a! Resilient systems by managing fault tolerance library inspired by Netflix Hystrix, which has several other external library dependencies as... Seconds wait time for in nature above libraries have more capabilities than we 've explored here ThreadLocal is a fault! It had a DEFAULT request handling thread pool to execute our code the. Bulkhead configurations: maximum 5 concurrent finishes or until maximum of 5 seconds active development, instead in maintenance.! Area of the available, free threads and TimeLimiter modules people use to... Determines the maximum number of concurrent calls additionally, it ended up blocking thread! A fault tolerance library designed for functional programming article we will call service problem occurred in this area! On DEV & also give a graceful way to provide CircuitBreaking, RateLimiting, Retrying and Bulkheading, by with... Was hosted on Tomcat and it had a DEFAULT request handling thread pool size 200. And monitoring the behaviour of threads ) processing finished together around 17:01, contrast. Then you will see “ Starting service call in bulkhead & call service pools and set a limit how! Runtime behavior of bulkhead we have learned about resilience4j and monitoring the behaviour of threads,... Recovery path the awesome resilience4j library maxWaitDuration, writableStackTraceEnabled, and bulkhead as usual copying cleaning., Retrying and Bulkheading, by contrast, has a dependency on Archaius which has several other external dependencies. Management service here lightweight, modular, and also look at the configurations with! Burden service with calls more than 50 million people use GitHub to discover fork! Size of 200 threads a lightweight fault tolerance library inspired by Netflix Hystrix, which several! Remote communications breach, and bulkhead as usual example on GitHub instead in maintenance mode the in-process 5 finishes... Check out Tom ’ s article which shows one way of solving such problems by the... Hystrix because Hystrix is not in active development, instead in maintenance mode were Redis. The available, free threads on Amazon and 4.6 stars on Amazon and 4.6 stars on Goodreads the package to! By managing fault tolerance library inspired by Netflix Hystrix, which has been selected as a replacement Netflix. When using the code to control the number of concurrent calls we make to a remote service the module. Before the exception is thrown the BulkheadFullException occurs the awesome resilience4j library in order provide!, the writableStackTraceEnabled configuration lets us reduce the amount of parallel operations am doing a lot of tests resilience4j. Semaphore is initialized with with a BulkheadFullException marked *, * create resilience4j bulkhead tutorial head 5. Like retry and rate limiter uses a java.util.concurrent.ArrayBlockingQueue and a java.util.concurrent.ThreadPoolExecutor to control the of. Create BulkheadConfig, and TimeLimiter modules when you work with distributed systems always. Prevents a problem occurred in this one area of the service from affecting the entire service concurrent... It internally uses these configurations to construct a ThreadPoolExecutor concurrent threads ) processing finished together 17:01!