12 more things I learned from Designing Data Intensive Applications by Martin Kleppmann

This is good to keep in mind even in the early stages of a project that might get very large. The author points out the advantages of robust replication instead of statement based application for basis. When you duplicated the rows, you're simply copying the data over. If you were writing, a statement and the statement uses a function based on time or date functions , a stored procedure, or any other method that could be dependent on the machine, you'll get different results.

The write ahead logs mean any changes such as insert, update, or delete get written to a log before they go to any database. Then they are executed/tried on the leaders and later tried on the followers (can be milliseconds later).

In the case of something like Twitter, where you have users contributing values through a website, you would use the leader to reflect the values back to the user they just change them. And you would use the followers to show those changes to everybody else. This does a few things: it reduces the reload on the leaders database significantly, and it gives immediate feedback to the person that enter the data. It then allows the follower databases to pick up later in time, which could be seconds, minutes, or even longer in the case of system failures, but the databases should eventually agree.

This is a way to make sure that a user will always see consistent data. The most straightforward way to achieve this is to make sure that every user session is using the same database or data center. The example given was if somebody connects to one data center and sees a post that exists there and then refresh the page and connect to a different data center it doesn't have the same post yet. They will see data inconsistencies. So unless there is a failure, you want to lock a user to the same data system.

Make sure that things are always seen in the same order, like a tweet or timeline. If you are using a shared a database where records stored in order might not be on the same physical machine these problems can happen. You will need to sort your results so that they are always given in the same chronological order.

Also group document editing has the same replication issues with each browser acting as a data center

This is kind of the opposite relationship of a traditional database. In a traditional database the data is stored, and the queries are ephemeral in stream analytics the queries are stored, and they are analyzing a stream of data. With something matches the query it is then picked off and stored depending on the application.

These are ways to check and verify data integrity. For example, constantly running check sums against databases to be sure that the data has not been corrupted. This could be by either software or hardware errors, but this example is focus. Mostly on hardware is like this failures.

ACID and other technologies are based on assumptions that the database will always store what you tell it to. Events are easier to keep track of than individual insert statements and they can be more resilient to failure.

He calls this "algorithmic jail" where people are hurt by a biased system. There is bias built into society and any algorithmic way to make predictions amplifies these issues. This is similar to what ANT is doing and using AI to guarantee loans for people. It will be biased against certain geographic areas, etc.

"We use customer data to improve our products " to " we use customer surveillance to improve our products "

We need to monitor and build our data stewardship with a mindset that we are entering a new era. We are going to be storing all data forever from here forward and will need ways to scale it.