100 new options. 20 new integrations. 2 large product launches. A primary of its type buyer convention. At Atlan, 2023 was the yr of constructing the longer term, with information groups which might be envisioning it. And we couldn’t be extra excited to recap the yr, so let’s get began.
The First AI Copilot for Information Groups
Generative AI modified the expertise panorama in 2023 and each information crew began excited about the methods during which they might use AI to grow to be extra productive.
Armed with learnings from after we had been a knowledge crew, we seized the chance to make the lives of people of information higher with Generative AI. In April 2023, we known as for a company-wide AI hackathon, sourcing concepts for the way forward for information catalogs and information governance.
Two months later, in June 2023, we launched Atlan AI. By partnering with Microsoft and utilizing Azure OpenAI Service, Atlan was the primary to deliver AI superpowers to information catalogs, main a wave of AI innovation for information governance.
Make your crew love documentation with Atlan AI
Within the outdated world, everybody uncared for documentation, blocking companies from trusting information and making sooner, higher selections. Atlan AI brings an finish to the ache by auto-generating descriptions for a variety of information property in Atlan. All information producers must do is overview, edit, and publish.
After 6 months of testing with prospects beneath a closed preview, we’ve seen Atlan AI’s description strategies shine with an acceptance charge of 60%, giving people of information extra time to work on business-critical initiatives.
Clarify lineage transformations with Atlan AI
Lineage simplifies transformations by taking complexity away, displaying relationships between information property. However when a knowledge analyst needs to dive deeper into how and why a knowledge asset was reworked, the method of parsing by means of advanced SQL queries can take hours. However not anymore.
Atlan AI helps information analysts and engineers perceive advanced lineage transformations by explaining what’s occurring in pure language.
It’s not an exaggeration to say that information catalogs and information governance are altering ceaselessly, and for the higher, due to AI.
The Way forward for Information Governance
In 2023, we hosted a primary of its type group convention on fashionable information governance — Re:Govern. Almost a thousand people of information got here collectively to listen to visionary information leaders, from corporations like Nasdaq, Fox, Autodesk, Elastic, and HelloFresh, discuss their methods and playbooks for contemporary information governance.
If there was just one key takeaway from Re:Govern, it will be this: the way forward for information governance will look very totally different from its previous.
Amy Raygada (Swiss Market Group) and Mark Kidwell (Autodesk), shared their visionary approaches to constructing a knowledge mesh. Takashi Ueki (Elastic) make clear constructing automating belief by means of information contracts. And Mihir Modi (FOX) defined his imaginative and prescient for information merchandise and AI in information governance.
Information groups are forsaking the outdated world of handbook, siloed, one-size-fits-all approaches for a brand new world the place automation & AI, collaboration, and suppleness are key to success.
That’s why, in 2023, we began innovating in direction of a future the place information mesh involves life, AI is embedded in our workflows, and handbook, conventional governance is automated. With that, Let’s recap 2023’s information governance updates:
Deliver your information mesh to life with Atlan Mesh
Atlan Mesh is the primary ever native information mesh expertise in a knowledge catalog. It’s an expertise that caters to information shoppers and improves their understanding of the information property. Right here’s how:
- Information merchandise as first-class residents: Treating information as a product requires a local resolution, not workarounds. Information merchandise can now be created & curated natively in Atlan, inside a brand-new dwelling for simple, context-rich discovery for enterprise customers.
- Devoted areas for federated domains: With Atlan Mesh, every area will get its crew’s personal workspace and touchdown web page to deal with curated information merchandise and documentation for information shoppers.
- Enterprise lineage: Historically, lineage has been a technical device for information producers to grasp affect and discover root causes. Atlan Mesh introduces enterprise lineage for information shoppers, who wish to perceive the provenance, not the technical structure, behind how information merchandise are created to information utilization selections.
- Information contracts: To help the creation of information merchandise and proactively bridge the hole between information producers and shoppers, we launched a brand new imaginative and prescient for information contracts in Atlan.
We’ve already began rolling out Atlan Mesh to the primary few prospects, who shall be part of the Atlan Mesh Advisory Council, and we’re excited to construct the way forward for federated, versatile information governance collectively.
Handle compliance in a single dwelling with Tag Administration
As the fashionable information stack continues to evolve, information groups want to make sure the correct individuals have the correct entry to the correct information. This entails figuring out delicate information and defending it with the correct entry controls, whereas serving trusted information to information shoppers.
To unravel this problem, we launched Tag Administration — a brand new manner to your information crew to handle compliance and safety — and have become one of many first Snowflake information governance companions to allow bi-directional tag sync between Snowflake and Atlan.
Tag Administration allows you to:
- Create tags natively or import tags from instruments like Snowflake into Atlan
- Classify information property with tags at scale utilizing Atlan’s Playbooks
- Sync tag updates in Atlan again to information sources like Snowflake
With Atlan’s Tag Administration for Snowflake, our crew can have one central dwelling to handle tags. Bi-directional tag sync will empower our information producers to tag property the place they work and allow our platform crew to handle tags and permissions seamlessly.”
Roi Levoso Fernandez, Information Engineering Supervisor, Taxfix
Perceive affect and optimize prices with Reputation & Utilization metrics
As a knowledge chief, you’re at all times seeking to get probably the most out of your information, whereas controlling prices. However you want visibility into who’s truly utilizing what information during which instruments.
In 2023, we launched Reputation & Utilization for 4 connectors — Snowflake, Databricks, Energy BI, and Redshift — giving information groups the power to:
- Uncover probably the most or least used property by sorting by recognition
- Perceive recognition in lineage with recognition indicators & pop-ups
- See who’s utilizing information with Prime Customers and Latest Customers
- Optimize the information property with Standard, Gradual, and Costly queries
With the launch of Reputation and Utilization for Snowflake, Mistertemp, a frontrunner in recruitment and non permanent work based mostly in France, deprecated 50% of unused Snowflake tables and over 60% of their Looker property:
All the things downstream modified. We had been in a position to see each current connection in Fivetran. We may see what was truly used. We stored these, and for all the things else, we might disconnect.”
David Milosevic, Head of Information & Analytics, Mistertemp
The Period of Energetic Metadata
Energetic metadata has at all times been core to Atlan’s platform, and in 2023, we noticed it turning into the middle of information estates throughout industries and companies too.
This yr, we had 20 new information leaders, from companies like Docker, Purple, and Datacamp, be part of the Energetic Metadata Pioneers membership — a visionary group that’s pushing the boundaries of metadata ahead by making energetic metadata a precedence. And to shut out the yr, in November 2023, G2 launched its first Energetic Metadata Grid Report, pushed by opinions from actual customers, with Atlan being the one chief within the class.
With that, let’s recap 2023’s energetic metadata developments:
Don’t go breaking my coronary heart dashboards with Metadata CI/CD
Influence evaluation is a tiring, time consuming, and disheartening course of for information engineers. However with out it, one small change may break 1000’s of dashboards. What if affect evaluation may very well be proactive and preventative, as an alternative of reactive and handbook? Say howdy to Metadata CI/CD.
With integrations for GitHub and GitLab, Metadata CI/CD routinely surfaces impacted property proper within the information producer workflow. This implies information engineers don’t must manually test affect and enterprise customers can belief their dashboards, which break much less usually.
After its launch, Metadata CI/CD helped a knowledge crew on Atlan understand that the request for a column identify change may affect greater than 1,000 business-critical dashboards.
Atlan has been an awesome assist. We now not must depend on these paperwork, and we’re in a position to do affect assessments on the click on of a button.”
Nestor Jarquin, World Information & Analytics Lead, Aliaxis
Bringing metadata to everybody’s favourite device: spreadsheets
There’s one information device that has stood the check of time: Excel.
In 2023, we launched and upgraded our integrations with Microsoft Excel and Google Sheets to allow new use circumstances:
- Speed up documentation by enriching metadata at scale
By importing information property from Atlan into Excel or Sheets, now you can doc descriptions, certificates, house owners, tags, and bulletins to your column property utilizing spreadsheet flexibility and sync the metadata updates to Atlan with a single click on.
- Construct belief and hold finish customers knowledgeable with affect evaluation
Now you can analyze affect sooner by importing affect evaluation right into a spreadsheet and add bulletins to maintain finish customers knowledgeable.
Deliver enterprise and information collectively in Microsoft Groups
To create true company-wide adoption, you could meet your customers the place they work. That’s why Atlan now integrates with Microsoft Groups. Now you can speed up your information and enterprise initiatives with higher, cross-functional collaboration round information.
Atlan’s integration with Microsoft Groups allows you to:
- Share information property & ask information questions in a Microsoft Groups channels with out leaving Atlan
- Hyperlink vital, context-rich Microsoft Groups threads to Atlan property
- Get notifications & alerts in chosen Microsoft Groups channels
Push the boundaries of metadata with Webhooks and new Python and Java SDKs
Making ready your information property and crew for mission-critical information initiatives, like AI fashions, wants a platform method to metadata. From event-driven metadata use circumstances like alerting to derived metadata use circumstances like Information as a Product scoring, information groups are making the way forward for metadata come to life with this yr’s extensibility enhancements.
In 2023, we launched the Java and Python SDKs, enabling information groups to construct customized energetic metadata use circumstances like:
- Governance reporting: Measure the success your governance initiatives by automating metadata enrichment stories.
- Customized connections: Join Atlan to your enterprise homegrown methods to allow end-to-end discovery and lineage.
- Derived metadata: Create customized metadata comparable to a “Metadata Completeness Rating” or “Information as a Product Rating” by analyzing metadata enrichment.
- Metadata migration: Robotically migrate all of your current metadata out of your legacy information catalog to Atlan.
Together with Python and Java SDKs, we additionally launched help for Webhooks in 2023 — opening up the world of event-driven metadata use circumstances.
Webhooks mean you can monitor occasions taking place in Atlan, obtain notifications to a URL of your selection, and take motion instantly. For instance, you may create a webhook to ship notifications to your e mail handle or collaboration app, like Slack or Microsoft Groups, when a time period is up to date or an asset is tagged.
The probabilities are really infinite and we’re excited to see the way forward for energetic metadata, constructed not by Atlan, however by information groups around the globe.
A Collaboration to Ship Trusted Information
With each device within the fashionable information stack turning into more and more siloed, the people of information have gotten siloed as nicely.
Information shoppers, who reside in BI instruments, don’t have visibility into the upstream pipeline world of information producers. So when issues go unsuitable, they’re usually the final to know. And information producers don’t know the way their code modifications are breaking downstream dashboards.
This yr, we launched native, out-of-the-box connectors for 18 new instruments, protecting areas like Information High quality, Information Observability, Information Orchestration, and Enterprise Intelligence, to deliver the world of information producers and shoppers nearer collectively. Let’s recap 2023’s key integrations.
All-New Partnerships with Information High quality Instruments
Almost 75% of the time, when issues go unsuitable, enterprise stakeholders are the primary to establish information points. It doesn’t must be this manner. Information high quality is a basic sign into information belief, however it wants to fulfill enterprise customers the place they work, together with the correct metadata context.
That’s why, in 2023, we launched two new out-of-the-box integrations with leaders within the Information High quality and Observability market: Monte Carlo and Soda.
Monte Carlo x Atlan
With Monte Carlo and Atlan, companies can achieve an up-to-date understanding of their information well being, construct belief in information, and help revolutionary new methods to method distributed information infrastructure. The native Monte Carlo integration provides you the power to:
- Uncover Monte Carlo incidents and screens in Atlan with Monte Carlo-specific filters.
- Democratize Monte Carlo’s information high quality indicators wherever enterprise customers work with Atlan’s Chrome extension.
- Speed up root trigger and affect evaluation by means of Monte Carlo incidents being surfaced in Atlan’s column-level lineage.
With Monte Carlo and Atlan, we are able to catch information incidents early on, and supply everybody with clear visibility into the present standing of information accuracy. This has been vital for the chief crew to trust we are able to ship on our promise of dependable, reliable information.“
Michael Weiss, Senior Director of Product Administration (NAM, Information Entry and Analytics), NASDAQ
Soda x Atlan
Atlan and Soda’s native integration supplies information groups with an intuitive and complete platform to seek out, belief, and use the correct information. The native Soda integration provides you the power to:
- Uncover Soda information high quality metrics and ends in Atlan.
- Inform information customers of information points earlier than they make selections by highlighting Soda’s test ends in Atlan’s Chrome extension.
- Speed up root trigger and affect evaluation by means of Soda checks being surfaced in Atlan’s column-level lineage.
BI Connectors for Enterprise Adoption
To drive adoption of your information catalog platform and information governance initiatives, you could meet enterprise customers the place they work: in BI instruments. In 2023, we launched native, out-of-the-box connectors for six new BI instruments to assist information groups drive company-wide adoption:
- Sigma
- Qlik Sense
- Amazon Quicksight
- MicroStrategy
- Thoughtspot
- Sisense
These BI connectors allow companies to:
- Construct a verified, single supply of reality for BI property by enabling discovery and documentation of BI property in Atlan.
- Allow proactive affect evaluation of downstream dashboards and use circumstances by connecting BI property to upstream warehouse, ELT, and supply property with cross-system lineage.
- Construct belief in information by surfacing belief indicators from information high quality, observability, and orchestration instruments.
I’ve had at the least two conversations the place questions on downstream affect would have taken allocation of plenty of assets. Then truly getting the work achieved would have taken at the least 4 to 6 weeks, however I managed to take a seat alongside one other architect and remedy that inside half-hour, saying ‘When you’re altering the column identify or including an additional column, that is what it’s going to interrupt or affect.”
Karthik Ramani, World Head of Information Structure, Dr. Martens
Bringing ELT & Orchestration Context to the Enterprise
Operational metadata can deliver highly effective context from pipeline processes for each enterprise customers and information groups. In 2023, we invested in constructing native connectors to instruments that would assist reply this query: when a pipeline fails, how do you inform a enterprise person and assist a knowledge engineer discover the basis trigger sooner? That’s why we constructed our first event-driven integration with Airflow and OpenLineage.
Airflow x OpenLineage x Atlan
In 2023, Airflow, OpenLineage, and Atlan partnered to construct an ecosystem of belief by making real-time pipeline observability a actuality. Right here’s how:
- Get a complete overview of your Airflow pipelines in Atlan by cataloging and documenting Airflow property, comparable to DAGs and duties.
- Observe and monitor your pipeline in a single dwelling with real-time operational metadata like job run particulars and statuses.
Perceive affect of your Airflow pipelines by visualizing how Airflow DAGs and duties hook up with your information property with cross-system lineage.
Trying out Atlan? Strive our product tour.
Enthusiastic about information governance? Chat with our crew about your governance initiatives and the way Atlan may also help.
