Hi All,
I am working on the Event Processing Module.

Below is the requirement:
1. There will be a directory of events files - EventDir.
New event file coming into this directory any time and of any size.

2. Need to process all events in every file will come into this EventDir.
After the process needs to dump into Oracle DB.

3. The event file will be CSV type and have different types (predefined) events.

Environment details:
Coding Lang - C++ 11
DataBase - Oracle
Operating System - RHEL 7


Working on Design:


Two std::queue
QUEUE
It shared by two Thread T1 and T2 to pass the event objects(read from Event file).
As it will be shared by multiple threads, we have to synch threads by locking this queue by mutex and lock.
DB_QUEUE
It shared by two threads T2 and T3 to pass the events objects and store them into the database.
As it will be shared by multiple threads, we have to synch threads by locking this queue by mutex and lock.

Three std::threads
First Thread: T1
T1 will always look for a new Event file into the EventDir.
If T1 finds the new event file then:
T1 will validate those events in the event file.
Then form the object of those events and PUSH into the shared QUEUE.
Second Thread: T2
T2 will be always looking for new event objects in QUEUE.
If T2 finds the events objects into the shared QUEUE then:
T2 will read those events objects from QUEUE and POP all.
Then if needed convert these event objects into the database-specific objects.
Then PUSH these converted objects to the 2nd queue i.e. DB_QUEUE.
Third Thread: T3
T3 will be always looking for new eventg objects in DB_QUEUE.
If T3 finds the any objects into shared DB_QUEUE then:
T3 will read those objects from DB_QUEUE and POP all.
And store all these objects into the database in bulk.

Doubts:
1.
a. T1 should PUSH bulk event objects(by setting up some time) into QUEUE and then signal to the other thread T2
And then T2 will lock QUEUE and read all event objects from it and also POP them all.
OR
b. should I do it one by one instead of bulk PUSH and POP events objects from QUEUE.
My preference will be the first one - a.
2. Do I need a 2nd shared queue i.e DB_QUEUE and third thread i.e. T3.

3. Please confirm if this design fulfills the requirement.



Please let me know if anything is missing so I can edit the post.

Is this a homework exercise?

Because if this is 'work', then I'd suggest you start with a single thread doing all the work.
a) to fairly quickly get something that works.
b) give you something to benchmark.
c) give you something to compare against when you 'improve things' with threads.
d) give you a fall-back in case the threaded version proves more trouble than it's worth.

Not sure why you'd use a std::queue, you probably want a concurrent queue of some kind (tbb?). It'll simplify the code no end.