hi ho!
i have a question about the general handling or rather the best practice when dealing with large, very volatile object stores.
i am interested in persisting many large object stores, such as 3d architectural plans, in kafka. for example, if there are thousands of architectural plans, each with millions of objects that are constantly being changed online, which i think would be perfect for an event store.
but how would i present them in kafka?
- one topic per architectural plan?
- one topic with all plans, but each plan in a separate partition?
- or other ways?
since every object movement is saved for the undo/redo functionality, the number of messages would be many times greater than the actual objects. which would make loading a plan incredibly long. but with a compacted topic, the history would be missing. would you use another topic for this and then move each event into two toppics?
i’m not so much interested in a direct answer here, but rather where i can read up on such questions? because i’m afraid this will only be the beginning of these kinds of questions.
there are so many papers on the confluent pages that i can read, but almost too many
can someone give me a tip where i can read such application design with kafka questions, because i think my question above is rather a standard question.
thank you very much for your help.