Generative AI isn’t just for chatbots. Artificial intelligence and machine learning are changing how data is stored, structured, and consulted in traditional databases. The news affects all levels of storage.
One of the biggest transformations that charismatic generative AI brings is buried under its software. Hidden in plain sight, AI algorithms are changing database technology on a global scale. They are revamping systems built to track data across the planet in endless regular tables and replacing them with newer artificial intelligence capabilities that are complex, adaptable, and seemingly intuitive.
The news affects all levels of data storage. The basic data structures are currently being revised. Database developers are changing how we store information to work better with AI models. The role of the database administrator, once static and mechanical, is evolving to be more communicative. Data loggers leave the scene, and mind-reading magicians enter.
Here are ways database technology changes, adapts to this new era, and improves as AI becomes increasingly ubiquitous.
Vectors and Embeds
AI developers like to store information as long vectors of numbers. In the past, databases stored these values as rows, with each number in a separate column. Now some databases support pure vectors, so there is no need to split the information into rows and columns. Instead, databases store them together. Some vectors used for storage have hundreds or even thousands of numbers.
Such vectors are usually combined with embedding, a scheme for turning complex data into a single list of numbers. Embed design is still an art often based on knowledge of the underlying domain. When embeddings are well-designed, databases can offer fast access and complex queries.
Some companies like Pinecone, Vespa, Milvus, Margo, and Weaviate are creating new databases specializing in vector storage. Others, like PostgreSQL, are adding vectors to their current tools.
Query models
Adding vectors to databases is more than convenient. The new query functions can do more than find exact matches. They can locate the “nearest” values, which helps implement systems like recommendation engines or anomaly detection. Embedding data in the vector space simplifies complicated problems involving matching and association to mere geometric distance.
Vector databases such as Pinecone, Vespa, Milvus, Margo, and Weaviate offer vector queries. Some unexpected tools like Lucene or Solr also offer similarity matching, which can give similar results with large blocks of unstructured text.
Recommendations
The new vector-based query systems have a halo of magic and mystery much greater than those of the past. Old queries looked for matches; new AI-powered databases read the user’s mind. They use similarity searches to find data elements that are “close” and often a good match for what users want. The underlying math may be as simple as finding the distance in n-dimensional space. Still, that’s enough to generate the unexpected. These algorithms have long run separately as full applications. Still, they are slowly being incorporated into the database, where they can support better and more complex queries.
Oracle is just one example of a database targeting this market. Oracle has long offered several functions for fuzzy matches and similarities. It now directly offers custom tools for industries such as online retail.
Indexing paradigms
In the past, databases created simple indexes that supported faster searches for particular columns. Database administrators could build elaborate queries with joins and filter clauses that executed faster with the correct indexes. Vector databases are now designed to create indices that effectively span all the values in a vector. We’re just beginning to discover all the apps for finding vectors that are “close” to each other.
But that’s just the beginning. When the AI is trained on the database, it absorbs all its information. Now we can send queries to the AI in plain language, and it will search in a complex and adaptive way.
Data classification
Including artificial intelligence in database technology is about more than just adding a new structure to the latter. Sometimes it involves adding a new structure within the data itself. Some data arrives in an unordered set of bits. There may be images without annotations or large BLOBs (Binary Large Objects) of text written by someone long ago.
Artificial intelligence algorithms are beginning to clean up chaos, filter out noise, and impose order on messy data sets. They fill in the tables automatically. They can classify the emotional tone of a block of text or guess the attitude of a face in a photograph. Tiny details can be extracted from images, and algorithms can also learn to detect patterns. They sort through the data, extract important details, and create a regular, clearly delineated tabular view of the information.
Amazon Web Services offers several data classification services that connect AI tools like SageMaker with databases like Aurora.
Best Performance
Good databases handle many of the details of data storage. In the past, programmers still had to spend time analyzing various parameters and schemas used by the database to make them work efficiently. The database administrator role was precisely established to manage these tasks.
Many high-level meta-tasks are becoming automated, often using machine learning algorithms to understand query patterns and data structures. They can observe the traffic on a server and develop a plan to accommodate the demands. They can adapt in real-time and learn to predict what users will need.
Oracle offers one of the best examples. Previously, companies paid large salaries to administrators who looked after their databases. Now, Oracle calls its databases autonomous because they come with sophisticated AI algorithms that adjust performance on the fly.
Cleaner data
Running a good database requires keeping the software running and ensuring that the data is as clean and error-free as possible. AIs simplify this workload by looking for anomalies, flagging them, and suggesting fixes. They can find sites where a customer’s name is misspelled and then find the correct spelling by looking through the rest of the data. They can also learn incoming data formats and ingest the data to produce a single, unified corpus where all names, dates, and other details are represented consistently.
Microsoft’s SQL Server is an example of a database tightly integrated with data quality services to clean up any data with problems such as missing fields or duplicate dates.
Fraud detection
One application of machine learning is to create more secure data storage. Some people use machine learning algorithms to look for anomalies in their data sources because they can indicate fraud. Is anyone going to the ATM late at night for the first time? Has the person ever used a credit card on this continent? AI algorithms can detect dangerous rows and turn a database into a fraud detection system.
Google web services, for example, offer several options for integrating fraud detection into your data storage pipeline.
Greater security
Some organizations apply these algorithms internally. AIs are not just trying to optimize the database for usage patterns but also looking for unusual cases that might indicate someone is breaking in illegally. Not only sometimes emote, but the user also requests full copies of entire tables. A good AI can smell something fishy.
IBM Guardium Security is an example of a tool integrated with data storage layers to control access and detect anomalies.
Fusion of the database and generative AI
In the past, AIs were kept out of the database. When it was time to train the model, the data was pulled from the database, reformatted, and then fed to the AI. New systems train the model directly from existing data. This can save time and energy for larger jobs, where just moving the data can take days or weeks. It also simplifies the lives of DevOps teams by making training an AI model as simple as issuing a command.
There is even talk of replacing the database entirely. Instead of sending the query to a relational database, they will send it directly to an AI that magically answers the queries in any format. Google offers Bard, and Microsoft is pushing ChatGPT. Both are serious contenders to replace the search engine. There’s no reason they can’t replace the traditional database, either.
The approach has its drawbacks. In some cases, the AIs hallucinate and give completely wrong answers. In other cases, they may change their output format on a whim.
But artificial intelligence can deliver satisfying results when the domain is narrow enough and the training set is deep and comprehensive. And it does so without the hassle of defining tabular structures and forcing the user to write queries that find data within them. Data storage and search with generative AI can be more flexible for users and creators.